What causes MISSING_REQUIRED_FIELD?

An upstream service stopped sending a field that was previously always present in the JSON payload. An Avro or Protobuf schema used for deserialization declares a field as required but the producer omitted it. A Delta table was read with a user-supplied schema that marks a column as NOT NULL and the file lacks that column. A from_json call uses a schema with a required field but the JSON string does not include that key. A schema registry version mismatch between producer and consumer

How do I fix MISSING_REQUIRED_FIELD?

Switch the field to optional in the schema, or provide a default value using COALESCE after parsing.. Use from_json with PERMISSIVE mode to allow missing fields to become NULL.. Check the upstream producer for a schema change — validate the new payload against the expected schema before ingestion.. For Avro or Protobuf, update the consumer schema to set a default value for the required field.. Add an Auto Loader schema hint: .option('cloudFiles.schemaHints', 'field_name INT') to handle missing columns.

Medium severitydata quality

Power BI Refresh Error:
MISSING_REQUIRED_FIELD

What does this error mean?

A required field was absent from a record during parsing or schema validation. This error appears when reading structured data (JSON, Avro, Protobuf) and a field declared as required in the schema is not present in the incoming record.

Common causes

1An upstream service stopped sending a field that was previously always present in the JSON payload
2An Avro or Protobuf schema used for deserialization declares a field as required but the producer omitted it
3A Delta table was read with a user-supplied schema that marks a column as NOT NULL and the file lacks that column
4A from_json call uses a schema with a required field but the JSON string does not include that key
5A schema registry version mismatch between producer and consumer

How to fix it

1Switch the field to optional in the schema, or provide a default value using COALESCE after parsing.
2Use from_json with PERMISSIVE mode to allow missing fields to become NULL.
3Check the upstream producer for a schema change — validate the new payload against the expected schema before ingestion.
4For Avro or Protobuf, update the consumer schema to set a default value for the required field.
5Add an Auto Loader schema hint: .option('cloudFiles.schemaHints', 'field_name INT') to handle missing columns.

Frequently asked questions

How do I distinguish a truly missing field from a null-valued field?

In JSON, a missing key and a key with a null value are different. from_json will produce NULL for a missing key and NULL for an explicit null value — use a sentinel default value or schema validation to distinguish them.

Can schema evolution in Auto Loader handle new missing fields automatically?

Auto Loader schema evolution adds new columns when they appear, but it does not handle a previously present column going missing. That requires explicit schema handling with cloudFiles.schemaHints or a rescue column.