Overview
Streaming mode enables continuous data ingestion by committing data at regular intervals instead of waiting for the entire pipeline run to complete.Configuration
Enable streaming mode in the Managed Lakehouse destination node:
Whichever threshold is reached first triggers the commit.
Backpressure
When storage upload latency increases, the system automatically reduces batch sizes to prevent memory buildup:
The engine queries the destination’s
GetBackpressureProfile() to adjust ArrowTuning accordingly.
Metrics
Monitor streaming throughput in the pipeline run summary:Tier Limits
Best Practices
Start with 60s intervals
Begin with the default 60-second interval and adjust based on observed latency.
Monitor backpressure
If you see batch size reductions in logs, your storage may need scaling.
Use with compaction
Streaming creates many small files. Enable compaction via table maintenance.
Arrow-native sources
For maximum throughput, use Arrow-native sources (Kafka, Parquet) with streaming mode.