Managed Lakehouse

The Managed Lakehouse destination writes Parquet data files once and commits metadata to both Apache Iceberg and Delta Lake in a single atomic operation. Every query engine in your stack — whether it speaks Iceberg or Delta — can read the same underlying data without conversion or duplication.

Managed Lakehouse is available on the Professional plan and above. Upgrade →

Architecture

Key design decisions:

Single write, dual commit — Parquet files are uploaded once. Iceberg and Delta metadata are committed separately, eliminating data duplication.
Iceberg-primary — Iceberg is the transactional source of truth. If the Iceberg commit succeeds but Delta fails, the pipeline retries Delta once and logs a warning without failing the run.
Catalog-backed — Iceberg tables are registered in a catalog (AWS Glue Data Catalog or REST Catalog) for schema governance, time travel, and partition pruning.

Supported cloud providers

Amazon S3
Google Cloud Storage
Azure Blob Storage

Field	Description
Credential	AWS credential with `s3:PutObject`, `s3:GetObject`, `s3:DeleteObject`, `s3:ListBucket` on the target bucket
Bucket	S3 bucket name (e.g., `my-data-lake`)
Prefix	Path prefix for data files (e.g., `lakehouse/`)
Region	AWS region (e.g., `us-west-2`)

For Iceberg via AWS Glue, the credential also needs:

glue:GetDatabase, glue:GetDatabases
glue:GetTable, glue:GetTables, glue:CreateTable, glue:UpdateTable

Field	Description
Credential	GCP service account with `storage.objects.create`, `storage.objects.get`, `storage.objects.delete`
Bucket	GCS bucket name
Prefix	Optional object prefix

Field	Description
Credential	Azure credential (storage key, connection string, or service principal) with Blob Data Contributor role
Container	Blob container name
Prefix	Optional blob prefix
Storage Account	Azure storage account name

Iceberg catalog configuration

AWS Glue Catalog
REST Catalog

Field	Description
Catalog Type	`glue`
Region	AWS region of the Glue Data Catalog
Namespace	Glue database name (e.g., `analytics`)
Warehouse	S3 path for Iceberg data files (e.g., `s3://my-bucket/lakehouse/`)

AWS credentials are shared with the S3 storage credential. Glue permissions must include table create/update access.

Field	Description
Catalog Type	`rest`
Catalog URL	REST catalog endpoint (e.g., `https://iceberg-catalog.example.com`)
Token	Bearer token or API key for authentication
Namespace	Catalog namespace or database
Warehouse	Warehouse identifier or storage path

REST catalogs are compatible with Tabular, Nessie, Polaris, and other Iceberg REST catalog implementations.

Write modes

Append
Overwrite
Merge (Upsert)

Adds new Parquet files and commits a new snapshot to both Iceberg and Delta. Existing data is preserved.Best for: event streams, logs, incremental loads, and any workload where historical data should not be modified.

Table formats

You can enable one or both formats:

Format	Description	When to use
Apache Iceberg	Catalog-backed, ACID-transactional table format with time travel, schema evolution, and hidden partitioning	Primary format for analytics engines (Spark, Trino, Athena, Flink)
Delta Lake	File-based transaction log with column statistics	When you also need Databricks/DuckDB compatibility or dual-engine access

By default, both formats are enabled. If you only need one, uncheck the other in the node configuration.

Advanced settings

Partition strategy

Partitioning organizes data files by column values for faster queries. Supported partition transforms:

Transform	Syntax	Example
Identity	`column_name`	`region`
Year	`year(column)`	`year(created_at)`
Month	`month(column)`	`month(created_at)`
Day	`day(column)`	`day(created_at)`
Hour	`hour(column)`	`hour(event_time)`
Bucket	`bucket(n, column)`	`bucket(16, user_id)`
Truncate	`truncate(w, column)`	`truncate(4, zip_code)`

Schema evolution

When enabled (default), the destination automatically adapts to upstream schema changes:

First batch — schema inference

Column types are inferred from the data and registered in both the Iceberg catalog and Delta log.

New columns

If a new column appears in a later batch, it is added to the schema. Existing columns retain their original types.

Iceberg schema IDs

Iceberg tracks column identity by field ID, enabling safe renames and reordering without breaking downstream consumers.

Maintenance settings

Setting	Default	Description
Snapshot Retention	7 days	How long to keep expired Iceberg snapshots and Delta versions before cleanup
Compaction Target	128 MB	Target file size for compaction operations — smaller values produce more files but faster writes

Maintenance can be triggered manually via the API or scheduled automatically.

Reading your tables

SELECT * FROM glue_catalog.analytics.events;

SELECT * FROM iceberg.analytics.events;

INSTALL delta;
LOAD delta;
SELECT * FROM delta_scan('s3://my-bucket/lakehouse/analytics/events');

from pyiceberg.catalog import load_catalog
catalog = load_catalog("glue", **{"type": "glue", "glue.region": "us-west-2"})
table = catalog.load_table("analytics.events")
df = table.scan().to_pandas()

API reference

The Managed Lakehouse API provides endpoints for table management, commit history, and maintenance operations.

List registered tables

GET /api/managed-lakehouse/tables

Register a new table

POST /api/managed-lakehouse/tables
Content-Type: application/json

{
  "connectionId": "uuid",
  "tableName": "events",
  "storageProvider": "s3",
  "storagePath": "s3://my-bucket/lakehouse/events",
  "icebergEnabled": true,
  "deltaEnabled": true,
  "icebergCatalogType": "glue",
  "icebergNamespace": "analytics"
}

View commit history

GET /api/managed-lakehouse/tables/{tableId}/commits?limit=50

Trigger maintenance

POST /api/managed-lakehouse/tables/{tableId}/maintenance
Content-Type: application/json

{
  "operation": "full_maintenance"
}

Available operations: snapshot_expiry, orphan_cleanup, compaction, metadata_cleanup, delta_checkpoint, full_maintenance.

Troubleshooting

Symptom	Likely cause	Fix
”Iceberg catalog not found”	Glue database doesn’t exist or wrong region	Create the Glue database first; verify the region matches your bucket
”io scheme not registered”	Missing S3/GCS I/O driver	This is an internal error — contact support
”PermanentRedirect” on S3	Region mismatch between bucket and Glue	Ensure the bucket region and Glue catalog region are the same
Delta commit failed, Iceberg succeeded	Transient storage error	The destination retries Delta once automatically. Check S3 permissions.
”Access denied” on Glue	IAM credential lacks Glue permissions	Add `glue:CreateTable`, `glue:UpdateTable`, `glue:GetTable` to the IAM policy
Table visible in Iceberg but not Delta	Only Iceberg format enabled	Check the format selection — enable Delta in the node configuration
Slow writes	Many small batches	Increase pipeline batch size to 10,000+ rows

Comparison with other destinations

Feature	Managed Lakehouse	Delta Lake	Iceberg	Fabric / OneLake
Formats	Iceberg + Delta	Delta only	Iceberg only	Delta only
Cloud support	S3, GCS, Azure	S3, GCS, Azure	S3, GCS, Azure	OneLake only
Catalog	Glue, REST	None (file-based)	Glue, REST	Fabric
Write modes	Append, Overwrite, Merge	Append, Overwrite	Append, Overwrite, Merge	Append, Upsert
Time travel	Yes (Iceberg snapshots)	Yes (Delta versions)	Yes	Yes
Schema evolution	Yes	Yes	Yes	Yes
Dual-engine access	Yes	Delta engines only	Iceberg engines only	Fabric only
Tier	Professional+	All tiers	Professional+	All tiers

Delta Lake destination

Standalone Delta Lake destination for simpler single-format workflows.

Destination nodes

All destination node types including Write, Cloud Destination, and Iceberg.

Cloud storage

Configure S3, GCS, and Azure Blob connections used by the lakehouse.

Data contracts

Enforce schema and quality rules before data lands in your lakehouse.

Getting started

Platform

Connections

Pipelines

Nodes

Reverse ETL

AI Copilot

Streaming & CDC

Orchestration

Data governance

Observability

Notebooks

Troubleshooting

Connectors

Integrations

Managed Lakehouse

Architecture

Supported cloud providers

Iceberg catalog configuration

Write modes

Table formats

Advanced settings

Partition strategy

Schema evolution

Maintenance settings

Reading your tables

API reference

List registered tables

Register a new table

View commit history

Trigger maintenance

Troubleshooting

Comparison with other destinations

Delta Lake destination

Destination nodes

Cloud storage

Data contracts

​Architecture

​Supported cloud providers

​Iceberg catalog configuration

​Write modes

​Table formats

​Advanced settings

​Partition strategy

​Schema evolution

​Maintenance settings

​Reading your tables

​API reference

​List registered tables

​Register a new table

​View commit history

​Trigger maintenance

​Troubleshooting

​Comparison with other destinations

​Related topics

Delta Lake destination

Destination nodes

Cloud storage

Data contracts

Architecture

Supported cloud providers

Iceberg catalog configuration

Write modes

Table formats

Advanced settings

Partition strategy

Schema evolution

Maintenance settings

Reading your tables

API reference

List registered tables

Register a new table

View commit history

Trigger maintenance

Troubleshooting

Comparison with other destinations

Related topics