What causes STREAM_FAILED?

An exception was thrown inside a foreachBatch or foreach sink function and was not caught. The upstream Kafka, Kinesis, or Delta source became unavailable mid-run. A schema mismatch or data type error occurred in a stateful operation. The driver ran out of memory while maintaining streaming state. An incompatible checkpoint was found after a job definition change

How do I fix STREAM_FAILED?

Inspect the full stack trace in the cluster event log or streaming query exception to identify the inner cause.. If the failure is in a foreachBatch function, wrap its body in a try/except and handle errors explicitly.. Check source connectivity — verify the Kafka broker, Kinesis stream, or Delta table is accessible.. If the query definition changed since the last checkpoint, delete the checkpoint directory and restart.. Review streaming query metrics in the Spark UI Structured Streaming tab to identify the failing batch.

High severitystreaming

Power BI Refresh Error:
STREAM_FAILED

What does this error mean?

A Structured Streaming query terminated with an unrecoverable failure. The streaming query cannot continue processing microbatches until the root cause is resolved and the query is restarted.

Common causes

1An exception was thrown inside a foreachBatch or foreach sink function and was not caught
2The upstream Kafka, Kinesis, or Delta source became unavailable mid-run
3A schema mismatch or data type error occurred in a stateful operation
4The driver ran out of memory while maintaining streaming state
5An incompatible checkpoint was found after a job definition change

How to fix it

1Inspect the full stack trace in the cluster event log or streaming query exception to identify the inner cause.
2If the failure is in a foreachBatch function, wrap its body in a try/except and handle errors explicitly.
3Check source connectivity — verify the Kafka broker, Kinesis stream, or Delta table is accessible.
4If the query definition changed since the last checkpoint, delete the checkpoint directory and restart.
5Review streaming query metrics in the Spark UI Structured Streaming tab to identify the failing batch.

Frequently asked questions

Does STREAM_FAILED mean the checkpoint is corrupt?

Not necessarily. STREAM_FAILED indicates the query stopped; checkpoint corruption is one possible cause. Check the inner exception first before deleting the checkpoint directory.

Can I restart a STREAM_FAILED query automatically?

Yes — configure the streaming job with restart-on-failure using Databricks Jobs. For idempotent sinks this is safe; for non-idempotent sinks verify deduplication before enabling auto-restart.