ADF upsert logic inserts a row that already exists instead of updating it. dbt incremental models insert duplicate keys when the unique_key configuration is wrong. Race condition — two concurrent transactions insert the same key. Source data itself contains duplicates that are not deduplicated before load

Step 1: Identify the duplicate value from the error message, e.g., `Duplicate entry '123' for key 'PRIMARY'`.. Step 2: Use INSERT ... ON DUPLICATE KEY UPDATE to handle upserts: `INSERT INTO your_table (id, col) VALUES (1, 'val') ON DUPLICATE KEY UPDATE col='val';`. Step 3: In dbt, set a unique_key in the incremental config: `{{ config(materialized='incremental', unique_key='id') }}`.. Step 4: Deduplicate source data before loading: `INSERT INTO target SELECT DISTINCT * FROM source;`

High severitydata integrity

Power BI Refresh Error:
1062

What does this error mean?

An INSERT or UPDATE tried to create a duplicate value in a column with a UNIQUE or PRIMARY KEY constraint.

Common causes

1ADF upsert logic inserts a row that already exists instead of updating it
2dbt incremental models insert duplicate keys when the unique_key configuration is wrong
3Race condition — two concurrent transactions insert the same key
4Source data itself contains duplicates that are not deduplicated before load

How to fix it

1Step 1: Identify the duplicate value from the error message, e.g., `Duplicate entry '123' for key 'PRIMARY'`.
2Step 2: Use INSERT ... ON DUPLICATE KEY UPDATE to handle upserts: `INSERT INTO your_table (id, col) VALUES (1, 'val') ON DUPLICATE KEY UPDATE col='val';`
3Step 3: In dbt, set a unique_key in the incremental config: `{{ config(materialized='incremental', unique_key='id') }}`.
4Step 4: Deduplicate source data before loading: `INSERT INTO target SELECT DISTINCT * FROM source;`

Frequently asked questions

How do I make ADF inserts idempotent in MySQL?

Use a Stored Procedure activity with `INSERT ... ON DUPLICATE KEY UPDATE` logic, or configure ADF's upsert write behavior in the MySQL sink dataset settings.

How does dbt prevent duplicate key errors in MySQL?

Set `unique_key` in your incremental model config. dbt will use REPLACE or INSERT ... ON DUPLICATE KEY UPDATE depending on the adapter to handle duplicates automatically.

Can MetricSign alert me when duplicate key errors break a pipeline?

Yes — MetricSign captures ADF pipeline failures and surfaces the MySQL error code, letting you see exactly which pipeline run hit the constraint violation.

Official documentation: https://dev.mysql.com/doc/mysql-errors/8.0/en/server-error-reference.html