Data Freshness Monitoring: What It Is and Why Stale Data Is Worse Than Missing Data

A Power BI refresh 'succeeded' is one of the least informative status messages in the modern data stack. It tells you the import-mode dataset called the data source, received a response, and completed its load without an API error. It says nothing about the age of the data it loaded.

Here is the scenario that plays out more often than most data teams want to admit. An external source system — a CRM, an ERP, a partner API — stops writing new records at 14:00 on Friday due to a backend issue. Your nightly ADF pipeline runs at 02:00 Saturday and copies the available data faithfully. The Power BI dataset refreshes at 06:00 Saturday and loads it without error. By Monday morning, the 'current' data in the dashboard is from Friday afternoon, a 65-hour gap for a dataset the business expects to be 24-hour fresh.

Every tool in the chain reported success. Nobody got an alert. The only way to catch this is to check the freshness of the data itself — not the freshness of the pipeline run.

Data freshness monitoring is a core signal in any mature data observability platform. It is distinct from refresh monitoring and distinct from data quality rules, because it watches what the data says about when it was written — not whether the process of moving it succeeded.

The primary mechanism for data freshness monitoring is a watermark check. The principle is simple: read the maximum value of the most recent timestamp column in the table and compare it against the current time.

For a daily batch table that refreshes at 06:00, a watermark check running at 07:00 should see a maximum timestamp from the current day. If it shows yesterday, the source stopped writing before the pipeline ran. If it shows three hours ago, the data is current. The check itself is one query:

SELECT MAX(updated_at) AS watermark FROM fact_orders

The output is compared against a configured staleness threshold — typically one to two refresh cycles. Anything beyond that threshold triggers an alert.

Watermark checks are cheap to run, have near-zero false positive rates on tables with known refresh schedules, and catch the entire category of 'pipeline ran but source went stale' failures that no other check surfaces. They belong on every fact table and every table with a clear refresh schedule in your warehouse.

For Snowflake datasets loaded via ADF or Databricks, MetricSign runs this check automatically per connector — tracking the watermark alongside volume and schema signals so the same incident feed shows freshness failures next to pipeline failures. Teams monitoring Snowflake with Power BI get this for the full source-to-dashboard chain.

Watermark checks on full-refresh tables are straightforward. Incremental loads are harder, because partial staleness can accumulate without the table looking stale at the aggregate level.

Consider a sales fact table that loads incrementally by day. Each night, yesterday's transactions are appended. If the source system has a 48-hour processing delay for certain transaction types — refunds, late invoices, corrections — those records appear in the table two days after the event date. The table is technically 'fresh' in that the most recent watermark is current. But a report that aggregates the last 7 days is understating revenue by whatever amount landed in the delay window.

This type of staleness is harder to catch with a watermark check alone. What it requires is a combination of: - Per-partition freshness: checking whether the data for each date partition was fully loaded before moving to the next - Expected row counts per period: verifying that Tuesday's transactions show expected volume by Wednesday morning - End-to-end latency tracking: measuring the time from source write to BI availability, not just pipeline duration

For teams running Databricks pipelines that feed Power BI, the incremental staleness problem often surfaces at the connector layer — a Delta table shows recent data but a specific time window is sparse. See Databricks and Power BI monitoring for how this failure mode shows up in practice.

A freshness SLA is a commitment: the data in this table will be at most X hours old when users access it. Without an explicit SLA, freshness monitoring has no threshold to compare against — you cannot alert on 'too stale' if nobody has decided what 'stale' means for a given table.

The process for setting SLAs is shorter than most teams expect:

1. Identify who uses the table and when. If the finance team runs Monday morning reports from a dataset, that dataset needs to be fresh before Monday 08:00. The SLA follows the consumption pattern, not the pipeline schedule.

1. Work backward from the consumption time. If the report runs at 08:00 and the dataset refreshes at 06:00, the pipeline needs to complete before 06:00, which means the source data needs to be ready before the pipeline runs. The SLA for the source is not 'refresh at 02:00 successfully' — it is 'data written by 01:30'.

1. Set the threshold conservatively. A 24-hour staleness threshold on a daily table with a 06:00 refresh is appropriate. A 4-hour threshold on a near-real-time dataset means any upstream batch delay will trigger an alert — intentional if the SLA is genuinely tight, noisy if it is not.

Once SLAs are defined, the watermark check becomes a comparison against a specific threshold rather than an open-ended anomaly score. For Power BI teams, MetricSign's freshness monitoring surfaces the staleness per dataset relative to its schedule — so the alert says '18 hours stale, SLA 8 hours' rather than 'freshness anomaly detected'.

Most enterprise data stacks have at least three hops between the source and the BI layer: source → staging → warehouse → semantic model. Freshness at the destination depends on freshness at every hop.

A common failure mode: the source writes data on time, but the staging-to-warehouse transformation job (a dbt model or a Databricks notebook) runs two hours late because of a cluster configuration change. The semantic model refreshes on schedule and picks up whatever is in the warehouse at that point — which does not yet include today's source data.

A freshness check on the warehouse table catches this. A freshness check on the semantic model also catches it. But neither tells you where in the chain the staleness was introduced. That requires end-to-end latency tracking — measuring the timestamp at each hop and comparing against the expected schedule.

For the typical modern data stack (source → ADF or Databricks → warehouse → Power BI), end-to-end freshness monitoring means: - Source system watermark (when did the source last write?) - Pipeline completion timestamp (when did the load finish?) - Warehouse table watermark (what is the newest record in the target table?) - BI dataset refresh completion (when did Power BI finish loading?)

The gap between source watermark and BI refresh completion is the actual end-to-end latency your users experience. This is the number that should be compared against the SLA — not 'did the pipeline run on time' but 'how old is the data your users see right now'.