Airflow DAG Failed: Causes, Detection, and Alerting for Data Engineers

When an Airflow DAG fails, the UI marks the task red and the DAG run shows as failed. But that's where the visibility ends. Airflow tells you what failed inside its own boundary — it does not tell you what broke outside it.

There are three failure modes that look similar in the UI but have very different implications:

• Failed task: one task in the DAG returned a non-zero exit code or raised an exception. Other tasks that don't depend on it may still run.
• Failed DAG run: the DAG run as a whole is marked failed, usually because a required task failed. Downstream tasks with trigger_rule=ALL_SUCCESS (the default) will be skipped.
• Upstream dependency failure: a task that produces data for another system failed, but Airflow considers the downstream task its responsibility only up to the boundary of the DAG. What happens in Snowflake, dbt, or Power BI after that is invisible to Airflow.

This boundary problem is the core issue. A DAG that extracts data from a source and loads it to Snowflake may fail on the load step. Airflow logs the failure. But the Snowflake table is now stale, the dbt model that reads it will produce incorrect results on its next run, and the Power BI report will serve yesterday's numbers to stakeholders who have no idea anything went wrong.

Debugging an Airflow failure therefore requires two parallel tracks: fixing the immediate task failure, and understanding what downstream systems were affected.

Most Airflow DAG failures fall into one of six categories:

PythonOperator exceptions — the most frequent cause. The Python callable raises an unhandled exception. The task log will contain the full Python traceback. AttributeError, KeyError, and requests.exceptions.ConnectionError are the most common culprits in data engineering pipelines.

BashOperator non-zero exit — a shell command returned exit code != 0. Airflow treats any non-zero exit as a failure. Check the task log for the stderr output of the command.

Upstream task dependency failure — with the default trigger_rule=ALL_SUCCESS, a task will not run if any of its direct upstream tasks failed or were skipped. A single task failure can cascade to mark multiple downstream tasks as upstream_failed.

Connection errors — an Airflow Hook cannot reach the target system. The database is offline, an API key expired, or the network path changed. Check the connection config in the Airflow UI under Admin > Connections.

Resource exhaustion — the Airflow scheduler is backlogged and cannot queue new task instances, or a worker ran out of memory (OOM) and was killed. Symptoms include tasks stuck in queued state for longer than expected, or tasks that fail without a Python traceback.

XCom deserialization errors — tasks that pass data between each other via XCom can fail if the serialized value is too large, or if the receiving task expects a different type than what was pushed. The default XCom backend stores values in the Airflow metadata database with a size limit.

Trigger rule misconfiguration — using trigger_rule=ALL_DONE or ONE_FAILED produces behaviour that surprises engineers who expect the default. A task with trigger_rule=ALL_DONE runs even if its upstream tasks failed, which can produce misleading success signals for downstream notification tasks.

Start with the task logs, not the scheduler logs. Click into the failed task in the Airflow UI, open the logs for the specific task instance, and look for the error at the bottom of the log output. The traceback will tell you whether this is an application error (exception in user code) or an infrastructure error (connection refused, OOM kill).

For infrastructure-level failures — scheduler backlogs, worker crashes — you need the scheduler logs, which are not exposed in the UI. On a managed Airflow deployment (MWAA, Cloud Composer, Astronomer), use the platform's log viewer. On a self-hosted deployment:

# Scheduler logs
journalctl -u airflow-scheduler -n 200 --no-pager

# Or if running in Docker
docker logs airflow-scheduler --tail 200

To test a specific task without triggering the full DAG:

airflow tasks test my_dag my_task_id 2026-06-23

This runs the task in isolation using the specified execution date. It bypasses dependencies and does not record a DAG run — useful for testing whether the task itself works without running the full pipeline.

For backfill and re-run scenarios:

# Re-run a specific DAG run
airflow dags backfill my_dag --start-date 2026-06-23 --end-date 2026-06-23

# Clear a failed task instance to reset it to scheduled
airflow tasks clear my_dag -t my_task_id --start-date 2026-06-23

For connection errors, test the connection directly:

airflow connections test my_connection_id

If the connection test fails, fix the connection config first before debugging the task code — there is nothing wrong with the task.

Airflow has built-in email alerting on failure (email_on_failure=True in the default_args), but it requires SMTP configuration and sends emails only to the email field on the DAG — which is typically a personal address that may be outdated.

For reliable alerting, use on_failure_callback. This is a Python callable that Airflow invokes when a task fails, receiving a context dict with the DAG run, task instance, and exception information.

from airflow import DAG
from airflow.operators.python import PythonOperator
import requests

def post_to_slack(context):
    dag_id = context["dag"].dag_id
    task_id = context["task_instance"].task_id
    execution_date = context["execution_date"]
    exception = context.get("exception", "Unknown error")

    message = (
        f":red_circle: *DAG failed* — `{dag_id}`\n"
        f"Task: `{task_id}`\n"
        f"Execution date: `{execution_date}`\n"
        f"Error: `{str(exception)[:200]}`"
    )

    requests.post(
        "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK",
        json={"text": message},
        timeout=5,
    )

with DAG(
    "my_pipeline",
    default_args={
        "on_failure_callback": post_to_slack,
    },
    schedule_interval="0 6 * * *",
    catchup=False,
) as dag:
    ...

The same pattern works for Teams (use the Teams incoming webhook URL) and PagerDuty (use the Events API v2 endpoint).

For DAG-level failure (as opposed to task-level), set on_failure_callback on the DAG object itself:

dag = DAG(
    "my_pipeline",
    on_failure_callback=post_to_slack,
    ...
)

Note that Airflow calls this callback in the scheduler process, so keep it lightweight. Network calls with a short timeout are fine; heavy computation is not.

The most dangerous aspect of an Airflow DAG failure is what happens outside Airflow's visibility window.

A common pattern: Airflow orchestrates an ELT pipeline. A DAG extracts data from a source system, loads it to Snowflake, and triggers a dbt Cloud job via the dbt Cloud API. If the load task fails at 2:00am, the failure is logged in Airflow. But:

• The dbt trigger task does not run (upstream_failed), so the dbt job is never submitted
• The Snowflake table is not updated
• The Power BI dataset scheduled to refresh at 4:00am runs against stale data and shows yesterday's numbers
• No one knows until a stakeholder opens the dashboard at 8:00am and notices the date in a data label

The failure happened at 2:00am in Airflow. The symptom appeared at 8:00am in Power BI. The four-hour gap is invisible to every alerting system you have — unless you are monitoring at the BI layer.

This is not a hypothetical edge case. It is the standard failure mode for any data stack that uses Airflow as the orchestrator for pipelines that feed BI tools. The gap between the orchestration failure and the BI symptom is filled with stale reports and unanswered questions.

When Airflow is part of a broader data stack, alerting on the DAG alone is not enough. You need visibility across the orchestration layer (Airflow), the transformation layer (dbt), the storage layer (Snowflake or Databricks), and the consumption layer (Power BI, Tableau).

MetricSign monitors all of these layers from a single connection. When an Airflow DAG fails, MetricSign creates a dag_failed incident and immediately shows which downstream systems are affected — which dbt jobs depend on the DAG output, which Snowflake tables will be stale, and which Power BI reports or Tableau workbooks are connected to those tables.

Instead of discovering the problem from a stakeholder complaint, you get a single alert at 2:00am that says: DAG load_sales_data failed. Downstream: dbt_sales_model will not run. Affected reports: Sales Dashboard, Weekly Revenue Report.

MetricSign connects to Airflow via the Airflow REST API (no agents, no code changes, 15-minute setup). Alerts route to Slack, Teams, email, or webhook. The connection is read-only — MetricSign does not trigger or modify DAG runs.