End-to-end latency monitoring answers the question users actually care about: is the data in this report current right now? This requires tracking the full chain from source to report, not just monitoring individual components in isolation.
The chain that determines data freshness
A typical enterprise data chain has multiple hops that each add latency:
- Source system updates its data (e.g., at 23:00 when the day's transactions close)
- An ADF pipeline starts at midnight and runs for 45 minutes
- A dbt Cloud job transforms the data, completing at 02:00
- The Power BI dataset refreshes at 05:00 and takes 30 minutes
- Users open reports starting at 07:00
The total end-to-end latency is 8 hours from when the source data was created to when it's available in the report. Each hop contributes to this total. Monitoring each hop individually tells you whether each step completed — but not whether the total latency met the business SLA.
What to measure
End-to-end latency monitoring tracks three timestamps for each dataset:
- Data watermark: The maximum value of a date field in the loaded dataset — this tells you how recent the data actually is, regardless of when the refresh ran
- Pipeline completion time: When the upstream pipeline (ADF, dbt, Databricks) finished writing to the source table
- Refresh completion time: When the Power BI refresh finished
The gap between the data watermark and the current time is the total data age. The gap between pipeline completion and refresh start is the scheduling slack (buffer time between pipeline completion and the next Power BI refresh window). If the pipeline takes longer than expected and eats into this buffer, the refresh may miss its window.
SLA monitoring in practice
For most organizations, SLA monitoring doesn't require real-time measurement — a daily check is sufficient. The check confirms: "did this dataset contain data fresher than X hours when users started working today?"
For datasets backing real-time operational dashboards, more frequent checks are warranted: "is the max event timestamp less than 2 hours old?"
Where end-to-end latency monitoring differs from individual step monitoring
A team monitoring ADF and Power BI separately might see both components as healthy — all pipelines succeeded, all refreshes completed. But if the ADF pipeline takes 30 minutes longer than usual, the Power BI refresh might start before the pipeline finishes — loading the previous run's data. Both components look healthy. The end-to-end latency measurement reveals the problem: the data is 24 hours old instead of 1 hour old.