Overview
Work Stream 5 adds two capabilities to the Managed Lakehouse destination:- Data Contract Enforcement — validate every record against a data contract before writing to Iceberg/Delta
- Partition Advisor — analyze column profiles and recommend optimal partition strategies
Data Contract Enforcement
When a data contract is assigned to a Managed Lakehouse node, every batch of records is validated before being written to cloud storage.Enforcement Modes
How It Works
Configuration
In the pipeline canvas, open the Managed Lakehouse node settings:- Expand Advanced Settings
- Under Data Contract Enforcement, select a contract
- Choose enforcement mode: Warn or Block
API Configuration
Contracts are configured as part of the managed lakehouse node settings:Viewing Violations
Contract violations are recorded in the Data Contracts page under the contract’s violation history. Each violation includes:- The pipeline run ID
- Affected column and violation type
- Expected vs actual values
- Sample data
CONTRACT_VIOLATION.
Pipeline Run Summary
After a run completes, the Managed Lakehouse node’s status note shows violation counts:Partition Advisor
The Partition Advisor analyzes your column profiles and recommends partition strategies based on data distribution patterns.How It Works
The advisor examines each column and scores partition strategies:Using the Advisor
- Open a Managed Lakehouse node’s settings
- Expand Advanced Settings → Partition Strategy
- Click Analyze Columns
- Review suggestions with confidence scores
- Select one or more and click Apply