Dead letter queue

The dead letter queue (DLQ) holds records that failed processing after retries: validation errors, schema mismatches, API rejections, or transform exceptions. You triage DLQ items to fix data, adjust pipelines, and reprocess without rerunning entire historical loads when the platform supports targeted replay.

​

DLQ management

Open Observability → Dead letter queue (or the DLQ tab on a pipeline/sync detail page). Filters include:

• Pipeline or sync name
• Error code or message substring
• Time range
• Destination or stage

Select an entry to inspect the payload snapshot, stack summary, and links to the run that produced it.

​

Stats

The DLQ summary charts depth over time, top error types, and affected partitions. Sudden spikes after a deploy often point to mapping or parser changes; slow growth may indicate upstream source quality decay.

​

Bulk operations

Perform bulk actions on selected entries:

• Reprocess after you fix SQL, mappings, or credentials
• Discard when records are invalid by business rule and should never retry
• Assign to an owner for investigation when workflow integrations are enabled

​

Export

Export DLQ entries to CSV or JSON for spreadsheet analysis or ticketing systems. Redact regulated fields before attaching exports to external tickets.

​

Reprocessing

Reprocessing sends the stored payload back through the configured path:

1. Confirm the root cause is fixed.
2. Select entries or filter by error type.
3. Choose Reprocess and monitor the child run IDs.
4. Verify DLQ depth decreases and downstream counts match expectations.

​

DLQ routing from lake write contract failures

When a data contract is assigned to a Managed Lakehouse destination node in block mode, records that fail validation are automatically routed to the DLQ instead of being written to the lakehouse.

​

How it works

1. The Managed Lakehouse destination validates each batch against the assigned contract
2. Valid records are committed to Iceberg + Delta as normal
3. Invalid records are written to the DLQ with error code CONTRACT_VIOLATION

​

DLQ entry contents

Each contract-failed DLQ entry includes:

​

Reprocessing contract failures

After fixing the upstream data or updating the contract:

1. Filter DLQ entries by error code CONTRACT_VIOLATION
2. Review sample payloads to confirm the fix
3. Select entries and click Reprocess
4. The records re-enter the pipeline and are validated against the current contract version

​

Related topics