Microsoft Fabric Error:
Pipeline Error 3202
What does this error mean?
A Fabric Data Factory pipeline that triggers Databricks jobs hit the Databricks workspace rate limit of 1,000 job creations per hour. This occurs when multiple pipelines collectively create too many Databricks runs in a rolling 60-minute window.
Common causes
- 1Multiple pipelines share a single Databricks workspace and together create more than 1,000 jobs per hour
- 2A pipeline is triggering a very high number of Databricks runs — e.g., a ForEach loop with many iterations each spawning a Databricks job
- 3Scheduled pipelines are all set to run simultaneously, causing a burst of simultaneous Databricks job submissions
How to fix it
- 1Audit all pipelines in the workspace that target the same Databricks workspace and sum their hourly job creation rates.
- 2Migrate some pipelines to a different Databricks workspace to distribute the load below the 1,000/hour limit.
Beyond the docs
Common practitioner solutions not covered in the official documentation.
- 1Refactor ForEach loops to batch multiple items into a single Databricks job instead of spawning one job per item
- 2Stagger pipeline trigger times to spread job submissions over the hour
- 3Consider using the Databricks Jobs API's multi-task job feature to consolidate multiple related operations into one job