ADF pipeline failure monitoring: where native alerts stop working

The source team renamed customer_id to cust_id in the Salesforce export. The ADF Copy activity reads from a stored procedure that still references the old column. The activity returns Succeeded — zero rows copied, no exception thrown, because the column mapping was set to 'auto' and the schema drift policy silently dropped the unmapped field. The pipeline reports Succeeded. The downstream Power BI dataset refreshes against the empty staging table. At 9am, the CFO opens a dashboard showing yesterday's revenue at $0.

ADF surfaces monitoring in four places: the Azure Monitor metrics blade, the pipeline run history in the studio, alert rules wired through Action Groups, and Log Analytics with diagnostic logs. Most teams configure two of them — pipeline run history (it's on by default) and a single Action Group that emails the team distribution list when a pipeline fails. That covers the case where the pipeline throws. It does not cover the case above.

The supported path for pipeline-level failure alerts is five steps. In your Data Factory, open Diagnostic Settings and route the PipelineRuns and ActivityRuns categories to a Log Analytics workspace. Wait for logs to flow (2-5 minutes after the next run). In Azure Monitor, create a scheduled query rule against that workspace. Attach an Action Group with email, Teams webhook, or an ITSM connector. The base KQL for pipeline failures is:

``kusto ADFPipelineRun | where TimeGenerated > ago(15m) | where Status == "Failed" | project TimeGenerated, PipelineName, RunId, Status, FailureType, Message=ErrorMessage ``

End-to-end latency: 3-10 minutes from failure to email. The rule fires once per evaluation window. Coverage is pipeline-level Status only — anything that bubbles up as a pipeline failure will be caught. What it misses: activity-level failures inside a succeeded pipeline, schema drift that produces wrong output, downstream impact on Power BI or Databricks, and the question of whether the ten alerts you just got at 3am are one incident or ten.

A pipeline's Status is computed from the orchestration outcome, not the data outcome. A Copy activity with enableSkipIncompatibleRow: true will skip rows and report Succeeded. A Lookup activity with a malformed query that returns an empty result set is a successful execution. A ForEach over an empty array completes in milliseconds with Status Succeeded.

The schema-mismatch case is the one that bites hardest. ADF's tabular translator with mapComplexValuesToString and auto-mapping will tolerate a renamed column by dropping it from the projection. The Copy activity reports rowsCopied: 487293 and Status Succeeded — the row count is real, the columns are wrong. Pipeline-level monitoring will never see this. The only signal is the row count itself, parsed from the activity Output JSON, compared against an expected baseline.

Activity-level alerting requires querying ADFActivityRun directly and parsing the Output column. Here's a working query that catches both explicit failures and zero-row Copy activities:

``kusto ADFActivityRun | where TimeGenerated > ago(15m) | where ActivityType in ("Copy", "Lookup", "ExecuteDataFlow") | extend Output = parse_json(Output) | extend rowsCopied = toint(Output.rowsCopied), rowsRead = toint(Output.rowsRead), errors = toint(Output.errors) | where Status == "Failed" or (ActivityType == "Copy" and Status == "Succeeded" and (rowsCopied == 0 or errors > 0)) | project TimeGenerated, PipelineName, ActivityName, ActivityType, Status, rowsRead, rowsCopied, errors, ErrorMessage=tostring(Error.message) ``

This works. It also requires you to know which activities have a meaningful zero-row baseline (a delta load may legitimately copy zero rows on a quiet day), which expected row count to compare against, and which pipelines need this treatment. Budget 15-30 minutes per pipeline to write, test, and tune. Multiply by your portfolio. The cost is not the KQL — it's the maintenance when activity names change, when new pipelines ship without alerts, and when the same failure triggers five rules across a fan-out.

Three problems show up once your portfolio crosses roughly five pipelines or your stack mixes ADF with Databricks, dbt, or Power BI.

Cross-pipeline correlation. The ADF Copy activity failure at 03:14 is the same incident as the dbt staging_orders model failure at 03:22 and the Power BI Sales Executive dataset refresh_delayed signal at 06:00. Native tooling treats these as three alerts in three inboxes. They are one incident, with a clear causal chain.

Deduplication. A pipeline that fans out to 12 datasets via ForEach generates 12 activity failures on the same root cause. Action Groups will email you 12 times, or once per rule, depending on how you wrote the threshold — neither is the right answer. The right answer groups by root cause.

Root cause surfacing. ADF's ErrorMessage column gives you Operation on target Copy data1 failed: ErrorCode=2200, ... followed by 2KB of stack trace. The actionable detail — column 'customer_id' not found in source — is in there, but you have to read for it. Dedicated tooling parses these, surfaces the column name, and links it to the schema change that introduced it.

This is where MetricSign fits: it reads ADF activity logs through the same diagnostic stream you already configured, groups failures into incidents across ADF, Databricks, dbt, and Power BI, and surfaces the root cause hint instead of the stack trace.

For one to four pipelines with no downstream BI dependencies: configure Diagnostic Settings, write the pipeline-level alert rule above, attach an Action Group, and stop. The native path is sufficient. Spend the time on data quality tests inside the pipelines instead.

For portfolios with mixed orchestration (ADF triggering Databricks, dbt running on a schedule, Power BI refreshing on top), or for pipelines under an SLA where stakeholders will notice within an hour: the per-pipeline KQL approach hits a maintenance wall around five to ten pipelines. The math flips — engineering hours spent on alert plumbing exceed the cost of a tool that does correlation, dedup, and root cause parsing out of the box. Decide based on portfolio size and stack diversity, not on whether the native path 'works'. It works. It just doesn't scale to the failure mode that wakes you up.