What causes DF-File-InvalidSparkFolder?

The sink path is configured to write into a Spark-managed directory (e.g., a Structured Streaming checkpoint folder or a Delta table folder that Spark manages). The target folder contains Spark metadata files (_spark_metadata, _delta_log) and the sink is configured in a write mode that conflicts with those. The folder path is shared between a Spark streaming job and an ADF batch sink, creating a write collision

How do I fix DF-File-InvalidSparkFolder?

Check the sink path in the data flow and verify it does not point to a folder that already has Spark metadata (_spark_metadata, _delta_log) unless the sink is configured for Delta.. Change the sink output path to a dedicated, clean folder that is not managed by any other Spark job.. If writing Delta format, ensure the sink type is set to 'Delta' — writing Parquet or CSV to a Delta table folder will trigger this error.. Check whether another ADF or Databricks job is writing to the same folder concurrently — folder conflicts cause this error.. Enable Debug mode and run a data preview to verify the sink path resolves to an accessible, non-conflicting location.

Medium severitydata source

Power BI Refresh Error:
DF-File-InvalidSparkFolder

What does this error mean?

The ADF Mapping Data Flow sink is configured to write to a folder path that overlaps with or conflicts with an existing Spark-managed folder structure. Spark uses internal folder conventions (like _spark_metadata) that ADF cannot write into directly.

Common causes

1The sink path is configured to write into a Spark-managed directory (e.g., a Structured Streaming checkpoint folder or a Delta table folder that Spark manages)
2The target folder contains Spark metadata files (_spark_metadata, _delta_log) and the sink is configured in a write mode that conflicts with those
3The folder path is shared between a Spark streaming job and an ADF batch sink, creating a write collision

How to fix it

1Check the sink path in the data flow and verify it does not point to a folder that already has Spark metadata (_spark_metadata, _delta_log) unless the sink is configured for Delta.
2Change the sink output path to a dedicated, clean folder that is not managed by any other Spark job.
3If writing Delta format, ensure the sink type is set to 'Delta' — writing Parquet or CSV to a Delta table folder will trigger this error.
4Check whether another ADF or Databricks job is writing to the same folder concurrently — folder conflicts cause this error.
5Enable Debug mode and run a data preview to verify the sink path resolves to an accessible, non-conflicting location.

Frequently asked questions

Can I write to a Delta table folder from ADF?

Yes — but only if the ADF sink is configured with Delta format. Setting it to Parquet, CSV, or JSON will conflict with the Delta metadata in the folder and trigger this error.

What is a _spark_metadata folder and why does it conflict?

Spark Structured Streaming creates a _spark_metadata folder for checkpoint state. ADF cannot write into this folder hierarchy — it is reserved for Spark's internal bookkeeping.

How do I safely share a storage account between ADF and Databricks?

Use separate, non-overlapping folder paths. Define a clear convention: e.g., /adf-output/ for ADF writes and /databricks-output/ for Spark writes — never point both at the same path.

Official documentation: https://learn.microsoft.com/en-us/azure/data-factory/data-flow-troubleshoot-guide