Three generations running simultaneously — three monitoring consoles with no shared view
If your organization started building data pipelines more than five years ago, there's a reasonable chance you're running three generations of ETL tooling at the same time: SSIS packages from the SQL Server era, ADF pipelines from the first Azure migration, and Fabric Data Pipelines from the current modernization initiative.
Maintaining oversight across all three is the same challenge that data observability platforms are built to solve — normalizing a fragmented monitoring landscape into one coherent view.
Each generation works. Each has its use cases. And each has a completely separate monitoring interface. SSIS jobs surface in SQL Server Agent history or in SSISDB execution reports. ADF pipelines are visible in Azure Monitor and the ADF Studio monitoring pane. Fabric Pipelines appear in the Fabric workspace monitoring hub. None of these surfaces provides any visibility into the others.
The practical consequence is that when something breaks, the on-call engineer has to check three consoles to establish what ran, what didn't, and what depends on what. For a pipeline failure that spans a generation boundary — an SSIS extract that feeds an ADF transformation that feeds a Fabric semantic model — the investigation requires hopping between three monitoring tools with no shared context.
This isn't a tooling immaturity problem. It's an architectural consequence of incremental migration. The solution isn't consolidating all pipelines onto one platform before you can monitor them. It's abstracting the monitoring layer.
SSIS: the packages your cloud monitoring cannot see
Most organizations underestimate how much SSIS is still running in production. The packages work, they've run for years without issues, and migrating them to ADF requires effort that doesn't make it onto the sprint board when higher-priority work is competing. The result: business-critical SSIS packages running on SQL Server Agent with no integration into any cloud monitoring system.
The failure mode is invisible by design. When an SSIS package fails at 02:30, SQL Server Agent records it in msdb.dbo.sysjobhistory. Azure Monitor doesn't see it. Power BI's refresh monitoring doesn't see it. The ADF pipeline that was supposed to pick up the SSIS output runs on schedule, finds no new data, copies nothing (successfully), and marks itself complete. Power BI refreshes the downstream dataset and serves yesterday's data without an error.
Fixing this doesn't require migrating the SSIS packages. It requires one addition: a step at the end of each package that writes execution status, row count, and timestamp to a SQL Server table accessible from Azure. A simple INSERT into a shared logging table after each successful package creates cross-environment visibility that no native tool provides. The cloud monitoring layer reads from that table alongside ADF run logs and Fabric pipeline status.
ADF: managing two very different workloads with one console
ADF in a three-generation environment is simultaneously doing two things: replacing SSIS for on-premises-connected workloads (requiring the data gateway), and feeding data to Fabric and Power BI for the forward-looking work. These two categories of pipeline have different failure patterns, different SLAs, and different maintenance needs — but they're managed from the same console.
Pipelines in maintenance mode awaiting eventual migration to Fabric are a distinct monitoring category. They're running, but nobody is actively developing them. Schema changes in their upstream sources go unnoticed longer. Dependencies built into them years ago may not be documented. A failure in a maintenance-mode ADF pipeline can be harder to diagnose than a failure in an actively-developed one, because the engineer who built it may no longer be on the team.
The practical response is to explicitly tag pipelines by status: actively developed, stable production, maintenance mode awaiting migration. Maintenance-mode pipelines warrant more aggressive volume monitoring and more detailed logging, not less — because when they fail, the on-call engineer will need that context to diagnose a pipeline they didn't build.
Fabric Pipelines: new capabilities, new failure modes
Fabric Data Pipelines are Microsoft's current-generation orchestration layer. As of 2026, they're well-established for lakehouse workloads. The failure modes are familiar in structure but specific in some details that matter for monitoring.
The most significant difference is the Direct Lake semantic model. Unlike import-mode Power BI datasets, which load data into an in-memory cache during a scheduled refresh, Direct Lake models read directly from OneLake at query time. There is no separate refresh step that loads a buffer. When a Fabric pipeline fails to update the underlying lakehouse data, users see wrong data immediately when they open the report — without any refresh failure to alert on.
This changes the monitoring model. For import-mode datasets, a pipeline failure gives you a window: the dataset still holds the last successfully loaded data until it tries to refresh again. For Direct Lake models, that buffer doesn't exist. Monitoring the Fabric pipeline that writes to the lakehouse becomes as urgent as monitoring the semantic model itself — because the pipeline failure is the refresh failure, collapsed into one event.
Fabric's native monitoring hub provides per-pipeline run status, duration, and error details. The gap is the same as SSIS and ADF: no cross-generation view without abstraction.
A normalized pipeline run record makes cross-generation monitoring practical
You don't need to consolidate every pipeline onto one platform before monitoring makes sense. You need a monitoring view that works regardless of which generation each pipeline is on — and stays stable as pipelines migrate between generations.
A normalized pipeline run record captures four fields regardless of the tool: name (a consistent identifier for the logical pipeline, not the tool-specific job name), status (succeeded, failed, warning), row count (rows written to the destination), and destination (the table or file the pipeline produced). Captured consistently across SSIS, ADF, and Fabric, a single query against that table gives you status across your entire stack — without caring which tool ran each job.
Building it requires modest per-tool instrumentation: a logging INSERT at the end of each SSIS package, a custom activity in each ADF pipeline that writes run metadata, a Fabric notebook or pipeline activity for Fabric jobs. The monitoring surface then survives migration: when a pipeline moves from ADF to Fabric, the normalized log keeps being written with the same fields. Nothing in the monitoring configuration changes.