Monitoring During Cloud Migration: What Teams Miss

Cloud migration projects rarely have a clean cutover date. The reality for most organizations moving from on-premises SQL Server and SSIS to Azure SQL and ADF is a period — often months, sometimes years — where both environments run simultaneously. Some datasets have been migrated; others haven't. Some business units are on Power BI; others are still on SSRS.

This middle state is when data monitoring is hardest and most important. You're not monitoring one environment; you're monitoring two environments with different failure modes, different tooling, and different expertise on your team. An SSIS package that used to fail cleanly might now fail because the Azure gateway it uses for connectivity is misconfigured. An ADF pipeline that looks healthy in Azure Monitor might be writing to the wrong database because a connection string was set up incorrectly.

Most monitoring tools are built for one world or the other. Azure Monitor covers ADF and Azure SQL. SQL Server Agent logs cover on-premises SSIS. Power BI Service covers refresh status. None of these tools gives you the cross-environment picture — the one that shows you the complete state of your data stack, on-premises and cloud, in a single view.

The on-premises data gateway is the bridge between your legacy on-premises systems and the Azure cloud. During migration, it's also your biggest single point of failure.

The gateway runs as a Windows service on a dedicated server (or, ideally, a cluster). It handles every data refresh that touches an on-premises SQL Server, Oracle database, SSAS cube, or file system. When the gateway goes down, every dataset that depends on it stops refreshing. There is no fallback.

Gateway monitoring needs to cover several things that Power BI Service's built-in monitoring misses:

Gateway service health: Is the Windows service running? Gateway services crash, and Power BI cloud can take up to 10 minutes to detect a gateway going offline (based on the gateway's heartbeat polling interval). By the time the portal shows "offline", the scheduled refreshes that ran in that window have already silently failed.

Cluster capacity: How many concurrent refreshes is the gateway handling? If you've added 20 new datasets during migration and the gateway is handling 40 concurrent refreshes, performance degrades and refreshes start queuing — appearing slow before they start failing.

Gateway version: Microsoft regularly updates the gateway software. Running an outdated version causes authentication failures and connection errors that are hard to diagnose. Monitoring the installed version against the latest release is a low-effort, high-value check.

Connectivity per data source: Regularly test connectivity from the gateway to each registered data source. A firewall rule change or SQL Server credential expiry can silently break connectivity before any refresh attempts to run.

Credential management is one of the most common sources of migration-related incidents. You have credentials in multiple places: SSIS package connection managers, SQL Server Agent job steps, Power BI data source credentials, ADF linked services, and Key Vault secrets. When a password changes or a service account is rotated, missing one location causes a failure.

During migration, the credential surface area is at its largest. Some pipelines use on-premises service accounts. New ADF pipelines use managed identities or Azure AD service principals. Power BI datasets may have a mix of embedded credentials, gateway-mapped credentials, and OAuth tokens.

Specific risks to watch during migration:

Expired gateway credentials: Power BI datasets that use gateway connections need credentials stored in the gateway's connection configuration. These are separate from the database password itself. If the database password changes but the gateway credential isn't updated, every refresh through that connection fails — often with a generic "Credentials are invalid" error that doesn't specify which credential.

Cross-environment connection string drift: An ADF pipeline migrated from SSIS might inherit a connection string that points at the on-premises database, while ADF's linked service is configured to point at Azure SQL. One environment gets updated; the other doesn't. The pipeline runs but writes to the wrong destination.

Service account lockout: Accounts used by SSIS or ADF to access SQL Server can get locked out due to password policies or unusual login patterns from new IP ranges. Monitoring authentication failures at the gateway layer catches this early — before multiple refreshes fail.

Before migration, your data loading schedule was probably maintained in one place: SQL Server Agent jobs. The schedule was straightforward to audit — one system, one list of jobs.

During migration, schedules spread across multiple systems: SQL Server Agent (legacy jobs still running), ADF triggers (new pipelines), Power BI scheduled refresh (dataset refresh windows), and possibly Azure Logic Apps or Event Grid for event-driven flows.

The risk is conflicts and ordering violations. Two jobs running simultaneously that write to the same table. A Power BI refresh triggering before the ADF pipeline has finished loading the staging table — because someone changed the pipeline schedule but forgot to update the Power BI refresh schedule. An SSIS package that still runs at midnight loading from the same source database at the same time the new ADF pipeline is running a full reload.

Schedule monitoring during migration means maintaining a single view of all scheduled jobs across all tools, checking for conflicts on shared resources, and validating that refresh schedules are downstream of the pipelines that feed them. Even a simple spreadsheet with all scheduled jobs, their times, and their dependencies is more useful than assuming each tool's schedule is independent.

Migration monitoring needs to track progress, not just failures. At each stage, define explicitly what healthy means.

Pre-migration baseline: On-premises pipelines running on schedule, Power BI refreshes completing within SLA. Row counts, durations, and error rates captured as a baseline. You need this data to verify parity during and after migration.

Parallel running: Both on-premises and cloud pipelines running. Data from both environments compared to validate parity — do the row counts match? Are the values consistent? This is the most complex monitoring phase because you're validating two systems simultaneously.

Post-migration, pre-decommission: Cloud pipelines running reliably. On-premises jobs disabled (not just deprecated — actually disabled, so a restart event doesn't reactivate them). Gateway traffic decreasing as datasets migrate to cloud-native direct connections.

Steady state: Gateway used only for genuinely on-premises sources (databases that will never move to cloud). Clear documentation of which datasets still require gateway access and why. Monitoring returning to a single-environment view.

The monitoring system should reflect this progression. During parallel running, alerting thresholds and notification rules are different from steady state. Temporary failures during migration cutover are expected; new failure modes that weren't present before are not.