Azure Data Factory Monitoring: What Native Tools Miss
ADF Studio tells you a pipeline run failed. It doesn't tell you which downstream Power BI dataset is now stale, whether the same pipeline was supposed to run at 04:00 but never started, or what the actual fix is for the error code it returned.
Azure Data Factory's native monitoring is event-driven: it records what happened, surfaces raw status codes, and sends one email per failure. Three categories of problems remain invisible: missing runs (pipelines that never start), cross-stack impact (which Power BI datasets are now serving stale data), and resolution guidance (what the error code actually means and how to fix it).
MetricSign vs ADF Native
What ADF native monitoring covers — and where it stops
Azure Data Factory provides three monitoring mechanisms. The Monitor hub in ADF Studio shows pipeline runs, activity runs, trigger runs, and integration runtime status in near-real-time. Azure Monitor integration forwards diagnostic logs and metrics to Log Analytics, enabling custom KQL alerts and dashboards. Email alerts via Azure Monitor can notify a distribution list when a pipeline fails.
Four gaps persist regardless of which mechanism you use.
First, ADF monitoring is event-driven. A pipeline that is scheduled to run at 04:00 but never triggers produces no log entry and no alert. The first signal that something is wrong comes the next morning when a report contains data from the day before.
Second, error messages surface as raw codes. An error like UserErrorOdbcInvalidQueryTimeout or MappingColumnNameNotFoundInSource appears in the activity run output without context; finding the resolution requires a separate documentation search or Azure support query.
Third, there is no connection between ADF failures and downstream Power BI datasets. When a pipeline fails at 03:30 and the Power BI refresh fails at 04:00, they appear as two separate unrelated events — in two separate monitoring consoles.
Fourth, ADF retains run data for 45 days. Pipeline patterns that span months — seasonal failures, gradual performance degradation, intermittent source connectivity issues — are invisible to native tooling.
The missing run problem
Most ADF monitoring discussion focuses on failures. The harder problem is the run that never starts.
Common causes: a trigger is disabled after a change window and not re-enabled, a dependency chain stalls and a downstream pipeline waits indefinitely, a Managed VNet integration runtime takes longer than expected to warm up.
In all these cases, ADF's event-driven monitoring produces silence — which looks identical to a healthy night where all pipelines ran successfully. Teams discover the problem when a report shows stale data, not when the missed run occurred.
MetricSign learns the expected schedule for each pipeline. When a run does not appear within the learned window, an incident opens — before anyone opens a report and before any downstream Power BI refresh is attempted.
When MetricSign replaces scattered monitoring
A common ADF architecture: an ADF pipeline extracts from Snowflake, transforms via a dbt model, loads to an Azure SQL staging table, and the Power BI dataset refreshes from that table.
With native tooling, each layer has its own monitoring console: Snowflake query history, dbt Cloud job runs, ADF Monitor hub, Power BI refresh history. An incident that starts in Snowflake at 02:00 and surfaces in Power BI at 05:00 requires manual correlation across four dashboards.
MetricSign monitors all four layers simultaneously. The Snowflake query timeout at 02:00, the dbt model failure at 02:15, the ADF pipeline failure at 02:30, and the Power BI dataset failure at 05:00 are grouped into one incident — with the Snowflake timeout identified as the root cause.
The Fix Tab translates each error code in the chain into plain English and provides resolution steps with direct links to the relevant settings pages.
Pricing: €299/month per organisation. 45-day trial, no credit card.
Frequently asked questions
Does Azure Data Factory have built-in monitoring?
Yes. ADF Studio includes a Monitor hub that shows pipeline run status, activity run details, trigger history, and integration runtime health in near-real-time. Azure Monitor integration enables custom alerts via email when pipelines fail. However, native monitoring does not detect missing runs, does not correlate failures with downstream Power BI impact, and retains run data for only 45 days.
How do I get alerts when an ADF pipeline fails?
Native option: configure a Failure alert rule in Azure Monitor for your data factory resource — this sends an email when a pipeline run fails. Limitation: one email per failure, no incident tracking, no Telegram or webhook delivery without Logic Apps. MetricSign provides email, Telegram, Slack, Teams, and webhook alerts with incident lifecycle tracking (open / acknowledge / resolve) and no additional Azure services required.
How do I detect when an ADF pipeline doesn't run at all?
Azure Data Factory does not alert on missing runs — it only logs events that occur. If a scheduled pipeline never starts, no alert fires. MetricSign learns the expected schedule for each pipeline and opens an incident when a run does not appear within the learned window.
How long does Azure Data Factory store pipeline run history?
45 days. Run data older than 45 days is permanently deleted from ADF. If you need longer retention for audit, trend analysis, or SLA reporting, you need to either export to Azure Monitor Log Analytics (which incurs separate storage costs) or use a third-party tool like MetricSign, which retains full run history.
What is the difference between ADF monitoring and MetricSign?
ADF native monitoring is scoped to ADF. It shows what happened inside ADF pipelines but has no visibility into upstream Snowflake or Databricks failures that caused the ADF run to fail, no connection to downstream Power BI datasets that are now serving stale data, and no fix guidance for error codes. MetricSign monitors the full pipeline stack — ADF, Databricks, dbt, Snowflake, Fabric, and Power BI — and correlates cross-tool failures into one incident with root cause identification and guided resolution.
Does MetricSign work with Azure Data Factory?
Yes. MetricSign connects to ADF via the Azure Data Factory REST API. It monitors pipeline run status, activity run failures, trigger health, and integration runtime availability. Incidents are correlated with upstream Databricks, dbt, and Snowflake runs and with downstream Power BI dataset refreshes. Setup takes approximately 15 minutes and requires a service principal with Contributor access to the ADF resource.