Records Processed10.2Mlast run
Throughput847K/speak ingestion
Data Quality99.1%pass rate
Avg Run Time4m 11slast 7 days
Pipeline Graph
FungalFlow_Daily · GBIF → Synapse → Power BI
ExtractDONE
GBIF Source10.2M · 8sValidateDONE
Schema & Quality10.2M · 12sTransformDONE
Apache Spark10.1M · 45sAggregateDONE
Spark Aggregation10.1M · 23sLoadDONE
Azure Synapse10.1M · 18sReportDONE
Power BI Refresh10.1M · 4sDead LetterIDLE
98 recordsRetry QueueIDLE
412 recordsEvent Log
Pipeline execution events
05:00:00RUN-2843: connection timeout to Azure Synapse (retried 3×)
07:00:13RUN-2846 completed successfully — 9.8M records
08:01:55RUN-2847 triggered by daily scheduler
08:01:58Extract: 10.2M records received from GBIF
08:02:13Validate: 98 records failed quality check — dead-letter
08:02:58Transform: enrichment complete — 10.1M records
08:03:22Aggregate: 412 records in retry queue
08:03:45Aggregate: Spark job completed in 23s
08:04:10Load: Synapse commit successful — 10.1M rows (18s)
08:04:12RUN-2847 completed successfully — 10.2M records processed
Run History
FungalFlow_Daily · last 5 executions
| Run ID | Started | Records | Duration | Status |
|---|---|---|---|---|
| RUN-2847 | 08:00 · Today | 10.2M | 4m 12s | OK |
| RUN-2846 | 07:00 · Today | 9.8M | 3m 58s | OK |
| RUN-2845 | 06:00 · Today | 10.5M | 5m 33s | WARN |
| RUN-2844 | 05:00 · Today | 8.9M | 4m 01s | OK |
| RUN-2843 | 04:00 · Today | — | — | FAIL |