Spot instances get reclaimed mid-shuffle and the error says nothing useful
Every cost-conscious Databricks deployment uses spot instances for worker nodes. AWS can reclaim them with two minutes notice. Azure gives thirty seconds. When a spot node disappears mid-task, Spark reattempts the task on surviving executors. That works — until the reclaimed node held shuffle data that other tasks depend on.
The error you see is FetchFailedException: Failed to connect to host. Not "your spot instance was reclaimed." Not "try on-demand nodes." A network fetch error that looks like a transient glitch. Teams retry the job, hit the same reclamation window, and retry again.
The fix depends on scale. For jobs under 2 hours, setting spark.speculation to true and using spot instances with fallback to on-demand (first_on_demand: 1 in cluster policy) keeps the driver stable while tolerating worker churn. For long-running jobs — anything doing large shuffles or writing hundreds of Delta partitions — on-demand workers cost less than three retries of a 4-hour job at spot pricing.
Databricks added spot fall-back policies in 2024, but the default cluster configuration still uses spot-only for workers. If you created your job template before that update, you're running the old default. Check your cluster policy JSON for aws_attributes.availability or azure_attributes.availability — if it says SPOT or SPOT_WITH_FALLBACK_AZURE, you're exposed. Change it to SPOT_WITH_FALLBACK on AWS or ON_DEMAND_AZURE for critical overnight jobs.
The pattern is predictable: spot reclamation rates spike during business hours in US-East and EU-West regions when general compute demand rises. Your 2am EST job that ran fine for months starts failing when a cloud provider's other customers scale up their morning workloads in overlapping time zones.
Autoscaling adds workers too late and your job hits the timeout
Databricks autoscaling looks at pending task counts and adds workers when the backlog exceeds capacity. The problem: provisioning a new node, installing libraries, and registering with the Spark driver takes 3-7 minutes on AWS and 5-12 minutes on Azure. For bursty workloads — jobs that suddenly need 10x the executors for a large join — the scale-up arrives after the stage has already been running at reduced parallelism for minutes.
In development you ran the job on an interactive cluster that was already warm with 8 nodes. In production, the job cluster starts with min_workers: 2 and max_workers: 20. The first stages run fine. Then the wide transformation hits, Spark schedules 200 tasks across 2 workers, and the autoscaler begins requesting 18 more nodes. Half of those requests fail because the instance pool is cold or the cloud region is capacity-constrained. Your job limps through with 6 workers instead of 20, takes three times longer than expected, and hits the timeout_seconds you set based on the development run.
The error is Run exceeded timeout of 7200 seconds. Nothing points to autoscaling as the root cause.
Two mitigations actually work. First, use instance pools with idle instances set to min_idle_instances: 4 — this keeps warm nodes ready and cuts provisioning time to under 90 seconds. Second, set autoscale.min_workers closer to your actual steady-state need rather than the minimum. The cost of 6 idle nodes for 10 minutes before the job starts is trivial compared to a failed 2-hour job that has to fully restart.
The Stack Overflow 2024 survey shows only 1.9% of professional developers use Databricks SQL, which means most Databricks production workloads are Spark jobs managed by small specialized teams. When the one engineer who tuned the cluster config leaves, these settings drift.
Delta concurrent write conflicts escalate when you retry the failed job
Delta Lake's ACID transactions use optimistic concurrency control. Two jobs writing to the same table simultaneously will both succeed — unless their operations conflict. A ConcurrentAppendException means two commits tried to add files to the same partition. A ConcurrentDeleteReadException means one job read files that another job deleted (via MERGE or DELETE).
In development, you run one job at a time. In production, the nightly ingestion job overlaps with the late-running transformation from yesterday. Or the retry of the failed 2am job collides with the 6am job that reads the same table.
Here's where it gets worse. The default retry behavior in Databricks Workflows retries the entire job from the beginning. If the job failed at task 847 of 900 due to a transient spot reclamation, the retry re-executes all 900 tasks — including the writes that already committed to Delta. Those writes now conflict with the partially-committed data from the first attempt.
For ConcurrentAppendException, the fix is partition isolation: ensure concurrent writers target different partitions by using replaceWhere with non-overlapping predicates. For ConcurrentDeleteReadException during MERGE operations, enable spark.databricks.delta.merge.repartitionBeforeWrite.enabled to reduce the file-level conflict surface.
But the real fix is architectural. Don't retry write-heavy jobs blindly. Use idempotent_token on Databricks job runs (available since late 2025) to make retries safe, or design your pipeline with a staging table pattern where each run writes to a unique path and a final atomic RENAME swaps the result into place. The retry then overwrites the staging path harmlessly.
The commit log in _delta_log tells you exactly which transaction conflicted and when — but only if you look before the next successful run compacts the log.
Unity Catalog permissions propagate on a delay that tests never reveal
Unity Catalog replaced workspace-level access controls with a centralized governance layer. Permission grants propagate through a distributed cache. In practice, a GRANT SELECT ON TABLE catalog.schema.table TO group takes 30-120 seconds to become effective across all clusters in the workspace.
Your CI/CD pipeline grants permissions in step 3 and runs the integration test in step 4. In the test environment with one cluster, the propagation completes in under 5 seconds. In production with 40 concurrent clusters, it takes 90 seconds. The test passes. The production job — which starts a new cluster after the deployment — hits AnalysisException: User does not have permission to SELECT on table.
This fails intermittently, which is the worst kind of failure. The same job succeeds on retry because by then the permission has propagated. Teams label it flaky and add a retry. The retry masks the problem until a job has two permission-dependent steps and the retry only re-executes from the failed step, leaving the first step's stale permission cache intact.
The mitigation is a permissions warm-up query. After granting permissions in your deployment script, execute a SELECT 1 FROM catalog.schema.table LIMIT 1 using the service principal that will run the production job. Wait for it to succeed with a 3-retry loop and a 30-second backoff. This forces the permission into the cache of at least one cluster, and subsequent clusters inherit from the metastore cache rather than waiting for full propagation.
Also check system.access.audit logs — Unity Catalog logs every permission check. If you see DENIED entries followed by ALLOWED entries for the same principal and table within minutes, you have a propagation timing issue, not a misconfiguration.
The monitoring gap between job status and actual data freshness
Databricks Workflows reports job status: succeeded, failed, timed out, or cancelled. What it doesn't report is whether the job that succeeded actually produced the data your downstream consumers expect.
A job can succeed while writing zero rows because the source table was empty. It can succeed while silently dropping 40% of records due to a schema evolution that converted a non-nullable column to nullable. It can succeed 90 minutes late because of the autoscaling delays described above, and every dashboard that depends on that data shows stale numbers through the morning standup.
The Databricks job run API (/api/2.1/jobs/runs/get) returns result_state: SUCCESS for all of these cases. The run duration is available, but there's no built-in alerting for "this job usually takes 45 minutes and today it took 130 minutes." You'd need to build that yourself by querying the runs list API, computing a rolling average, and setting a threshold.
MetricSign connects to your Databricks workspace and tracks job execution patterns — not just pass/fail, but duration anomalies and delayed completions. When a job that normally finishes by 5:15am is still running at 6:30am, MetricSign fires a refresh_delayed signal before the job times out or your stakeholders notice stale dashboards. That early warning is the difference between proactively notifying consumers and reactively explaining why the 8am report used yesterday's numbers.
The operational reality for most Databricks teams: the job succeeded, the data is wrong, and nobody knows until someone opens a dashboard and notices the numbers haven't changed. That gap between job status and data freshness is where production trust breaks down.