What Databricks native monitoring covers
Databricks ships with a substantial set of monitoring tools built into the platform. For teams new to Databricks monitoring, understanding what each tool covers — and what boundary it stops at — is the starting point.
Job Run History
The Jobs UI in the Databricks workspace shows the run history for every configured job. Each run entry shows status (Succeeded, Failed, Timed Out, Cancelled), start time, duration, and the triggering mechanism (scheduled, manual, API). For failed runs, clicking into the run detail shows which task in the job failed and the error message from that task.
Job Run History is the first place to look when investigating a Databricks failure. Its limitation is scope: it shows runs within a single workspace. Teams with multiple workspaces need to check each workspace separately, or build aggregated monitoring using the Jobs REST API.
Cluster Event Log
Every Databricks cluster records a structured event log covering cluster lifecycle events: creation, resize, restart, termination, node additions and removals. The cluster event log is useful for diagnosing failures caused by cluster infrastructure problems — autoscaling events that took too long, node evictions on spot instance clusters, and driver or worker failures that cause a running job to terminate.
The cluster event log is not a performance monitoring tool. It records events but does not track metrics like CPU utilization, memory pressure, or network throughput over time. For Spark-level performance metrics, Ganglia is the relevant tool.
Ganglia Metrics
Databricks exposes a Ganglia metrics interface on every running cluster that shows real-time CPU, memory, network I/O, and disk usage per node. Ganglia is useful for diagnosing performance issues during an active job: if a Spark stage is running slowly, Ganglia shows whether the workers are CPU-bound, memory-constrained, or network-throttled.
Ganglia's limitation is retention: metrics are only available while the cluster is running. After a job completes and the cluster terminates (or auto-terminates), the Ganglia data is gone. Post-mortem analysis of a completed job's resource utilization requires exporting metrics to an external system during the run.
Spark UI
The Spark UI is accessible from within the Databricks cluster view for active or recently completed jobs. It shows the Directed Acyclic Graph (DAG) of stages and tasks, stage durations, task skew metrics, shuffle data volumes, and executor logs. The Spark UI is the primary tool for diagnosing performance bottlenecks inside a Spark job: identifying skewed partitions, slow stages, and excessive garbage collection.
Like Ganglia, the Spark UI is ephemeral — it is only available while the cluster is running. Databricks stores a truncated version of Spark UI data in the cluster driver logs for a limited retention period, but full Spark history requires exporting event logs to ADLS or S3 before the cluster terminates.
Databricks Alert Policies
Databricks supports configuring alerts on SQL queries (for SQL warehouse usage). For job monitoring, Databricks provides email notifications on job failure — configurable per job with recipient email addresses. This is the built-in alerting mechanism for job failures, and for many small teams it is sufficient as a starting point.
The limitation is granularity: job failure notifications tell you the job failed, the job name, and a link to the run. They do not include task-level error detail in the notification, they do not aggregate across multiple workspaces, and they have no duration-based alerting for slow runs.
What Databricks monitoring misses for production BI stacks
The native Databricks monitoring tools are well-suited for the use cases they were designed for: debugging individual job failures within a workspace and monitoring cluster resource utilization during active runs. They were not designed for monitoring the health of a production data pipeline that feeds downstream BI tools.
Slow runs with no failure signal
A Databricks job that runs 3x longer than its normal duration is a production incident in most data stacks — it means data lands late, Power BI refreshes load stale data, and reports miss their SLA window. Databricks native monitoring has no concept of a historical duration baseline. A job that takes 45 minutes when it normally takes 15 minutes reports as Succeeded. No alert fires. The only way to detect this within Databricks is to set a job timeout — but timeout-based detection triggers after the problem has already exceeded the acceptable window, not as the run approaches it.
Detecting slow runs requires tracking duration history and computing a baseline per job, then alerting when a current run deviates significantly from that baseline. This is a common capability in dedicated monitoring tools but not available natively in Databricks.
Cross-workspace aggregation
Enterprise Databricks deployments commonly span multiple workspaces: one workspace per environment (dev, staging, prod), or one workspace per business unit. The Databricks UI is workspace-scoped. A data engineering team responsible for jobs across three production workspaces has no single view of all job runs — they need to open each workspace separately. The Jobs REST API allows programmatic aggregation across workspaces, but building and maintaining a cross-workspace dashboard is a non-trivial operational investment.
Schedule drift and missing runs
A Databricks job that is scheduled to run at 05:00 UTC every morning and doesn't trigger because of a job scheduler failure, a disabled trigger, or a Databricks platform issue simply does not appear in the job run history for that day. There is no 'expected run did not start' event. Without external monitoring that tracks expected run times and alerts on the absence of a run, a silent schedule failure is invisible until a downstream pipeline detects the missing data — or until a user reports stale reports.
The downstream impact on Power BI
This is the largest gap in Databricks-only monitoring. When a Databricks job fails or runs late, the immediate consequence within Databricks is a failed or slow run. The downstream consequence — a Power BI semantic model that refreshes against incomplete Delta table data, or a Power BI report that shows numbers from the previous run — is entirely outside Databricks' visibility.
From Databricks' perspective, the job ended. From Power BI's perspective, the refresh might have succeeded. From the user's perspective, the data is wrong. None of the three systems produced an alert that connected the job failure to the report impact.
How a Databricks job failure reaches Power BI without anyone noticing
The mechanism by which a Databricks job failure produces stale data in Power BI is worth examining in detail, because it explains why standard Databricks alerting is insufficient for data teams that also run Power BI.
The standard architecture
In the most common architecture: a Databricks job runs daily transformations and writes output to a Delta table. A Power BI semantic model uses the Databricks connector (or reads from a storage layer like ADLS or Unity Catalog) and has a scheduled refresh that runs after the Databricks job is expected to complete. The scheduling dependency is informal — the Power BI refresh is configured to start 30 minutes after the Databricks job's normal completion time.
What happens when the Databricks job fails
The Databricks job fails at task 3 of 5 and is marked as Failed in the job run history. The Delta table that the job writes to contains data from the previous successful run — the failed run did not complete the final write operations. The Power BI semantic model refresh runs at its scheduled time, connects to the Delta table (which exists and is readable), and loads the data it finds. The refresh completes with a Succeeded status.
The Power BI report now shows data from yesterday's Databricks run. There is no error in Power BI. The Databricks failure alert went to a distribution list that the BI team is not on. Nobody connects the two events.
The partial write scenario
A more subtle failure mode: the Databricks job writes to multiple Delta tables in sequence. It completes writing tables A and B, then fails writing table C. Power BI refreshes a semantic model that reads from A and B (which are updated) and another semantic model that reads from C (which is stale). The first model has fresh data; the second has yesterday's data. A report that joins data from both models shows a mismatch that looks like a business anomaly rather than a pipeline failure.
Why standard alerting does not connect these
Databricks sends a job failure alert. Power BI sends no alert because the refresh succeeded. The data engineer who receives the Databricks alert investigates the Databricks job and opens a ticket. The BI engineer who gets a user complaint about mismatched data investigates the Power BI model and opens a separate ticket. The connection is made when someone correlates the timestamps manually — typically 2–4 hours after the incident began.
This cross-stack correlation problem is the core reason why Databricks-only monitoring is insufficient for data teams with Power BI in their stack.
What cross-stack Databricks monitoring looks like in practice
Closing the gap between Databricks job monitoring and downstream Power BI impact requires monitoring that spans both layers and connects events across tool boundaries.
Duration baselines per job
For each Databricks job, compute a normal duration range using historical run data. A job that ran between 12 and 18 minutes over the trailing 30 runs has an expected range. A job that runs 35 minutes in that context is a slow run worth alerting on — even if it eventually succeeds. This baseline-based detection catches the failure mode that timeout-based alerting misses: the job that takes too long but does complete.
MetricSign implements this using the MAD (median absolute deviation) algorithm over the trailing run history. MAD is more robust than standard deviation for job durations because it is not skewed by occasional very long outlier runs that shift the mean significantly.
Job-to-dataset lineage
Knowing which Databricks jobs feed which Power BI semantic models is the prerequisite for cross-stack alerting. This linkage is typically established through two mechanisms: workspace naming conventions (a Databricks job named etl_sales_daily writes to a Delta table that a Power BI semantic model named Sales Model reads), or explicit configuration in a monitoring tool where the relationship is declared.
With this lineage in place, a Databricks job failure can immediately produce an alert that includes both the failed job and the downstream semantic models that depend on it — without requiring manual cross-referencing.
Cross-stack incident view
When a Databricks job fails and a downstream Power BI semantic model is affected, a cross-stack incident view shows both events with the relationship between them. Instead of a Databricks alert and a separate Power BI alert (or no Power BI alert at all), the on-call engineer sees a single incident: Databricks job X failed at 05:43 UTC; Power BI model Y, which depends on job X, refreshed at 06:00 UTC and is now serving data from the previous run.
This incident view replaces the manual correlation step and cuts mean time to resolution by eliminating the investigation time spent connecting two separate alert streams.
MetricSign's Databricks connector
MetricSign connects to the Databricks Jobs REST API and monitors all job runs across configured workspaces. It creates two incident types for Databricks: job_failed (the run returned a failed state) and job_slow (the run exceeded its historical duration baseline). When a Databricks job is linked to a downstream Power BI semantic model, both the Databricks incident and the Power BI stale-data impact appear in a single correlated incident.
For teams running Databricks alongside ADF, dbt, or Fabric, MetricSign provides a unified incident feed where a Databricks job failure is visible in the same view as the ADF pipeline failure that may have caused it, and the Power BI refresh impact it produces downstream.
Choosing the right level of Databricks monitoring for your team
The appropriate monitoring setup depends on the complexity of your data stack and the downstream consequences of Databricks job failures.
Starting point: native job failure alerts + Power BI refresh alerts
For small teams with a simple architecture — a handful of Databricks jobs feeding a few Power BI datasets — the native Databricks job failure email notification combined with Power BI refresh failure alerts is a reasonable starting point. The gap is the cross-stack correlation, but a small enough team can perform this correlation manually when both alerts fire within the same window.
The failure mode for this approach is the slow-run scenario: a Databricks job that runs late but succeeds produces no alert in either system. If the Power BI refresh SLA is tight enough that a 30-minute delay causes user-visible impact, this gap will surface as a user complaint before it surfaces as an alert.
Intermediate: Databricks REST API polling with duration tracking
Teams with a data engineering discipline who are comfortable building and maintaining tooling can implement cross-workspace job polling using the Databricks REST API and track duration per job over time. This closes the slow-run gap and enables cross-workspace aggregation. The remaining gap is the downstream Power BI linkage — connecting a slow Databricks run to the impact on specific reports requires additional integration with the Power BI REST API.
Full cross-stack monitoring: dedicated monitoring tool
For teams where Databricks job failures have visible business impact — stale reports used in operational decisions, SLA commitments to downstream consumers, or complex dependencies across multiple tools — a dedicated cross-stack monitoring tool closes all the gaps simultaneously: slow-run detection, cross-workspace aggregation, job-to-dataset lineage, and unified alerting across the full stack.
MetricSign is free to start, does not require pipeline instrumentation, and connects to both Databricks and Power BI through their respective REST APIs. The lineage between a Databricks job and a downstream Power BI semantic model is configured in MetricSign's interface and takes effect immediately for all subsequent monitoring.