DuckIceLake
Keep DuckLake. Speak Iceberg to every client.
An open-source Iceberg v3 REST Catalog proxy on top of DuckLake.
Kevin Keller · kevinkeller.org
Why it matters
Interoperability without migration
DuckLake is a compelling format: Postgres as the catalog, Parquet on S3 as data, DuckDB as the engine. But many tools speak Iceberg.
Connect PyIceberg, Trino, Spark or DuckDB's iceberg extension without leaving DuckLake.The idea
Translate DuckLake state into Iceberg manifests on demand
Two paths, Iceberg REST and DuckLake direct, see exactly the same data.
The bridge
One snapshot identity, always in sync
- The materialiser turns DuckLake's Postgres catalog into
vN.metadata.json, manifest lists and manifests on S3 DuckLake HEAD == Iceberg current-snapshot-id- A row written through one path appears in the other automatically
Eager materialisation
~1s
for the Iceberg metadata chain to appear after a direct DuckLake write
A Postgres NOTIFY trigger wakes one elected listener. The lazy LoadTable path remains the correctness floor.
Full commit surface
Every Iceberg action becomes DuckLake SQL
| Iceberg action | Translation |
|---|---|
add-snapshot | ducklake_add_data_files() |
add-schema | ALTER TABLE ADD/DROP COLUMN |
add-partition-spec | ALTER TABLE … SET PARTITIONED BY |
upgrade-format-version | Manifests re-emit in matching Avro schema |
Iceberg v3 today
A small shim unlocks v3 writes from PyIceberg
- One
install()call before anyRestCatalogoperation - Puffin deletion vectors, one blob per affected data file
- Row lineage and new types:
variant,geometry,geography - v2 tables keep Parquet position deletes, so v2 readers still work
Governed access
No shared root keys handed to query engines
- Credential vending: a real
AssumeRolecall, scoped to one table's data and metadata paths - Short-lived: credentials come with an expiry
- OAuth2: HMAC-signed JWTs on every
/v1/*route - RBAC: per-namespace scopes such as
ns:analytics:rw
Operations
Built to be observed
- Cache-hit
LoadTablemeasured at ~349 req/s at concurrency 32 - Prometheus
/metrics, plus/healthzand/readyz - One Pixi environment runs Postgres, MinIO, the proxy and clients
- Local and production environments are identical
What's left
The spec is effectively complete. The gaps are elsewhere.
- DuckLake-blocked: true divergent branches, per-table
set-location, real KMS encryption - Upstream: Spark v3 writes, DuckDB iceberg-ext v3 features
- Ops: HA backends, TLS, tracing, Spark and Trino integration tests
The takeaway
Standard Iceberg tooling, no lock-in, DuckLake underneath.
Open source: github.com/KellerKev/duckicelake