DuckIceLake

Keep DuckLake. Speak Iceberg to every client.

An open-source Iceberg v3 REST Catalog proxy on top of DuckLake.

Kevin Keller · kevinkeller.org

Why it matters

Interoperability without migration

DuckLake is a compelling format: Postgres as the catalog, Parquet on S3 as data, DuckDB as the engine. But many tools speak Iceberg.

Connect PyIceberg, Trino, Spark or DuckDB's iceberg extension without leaving DuckLake.

The idea

Translate DuckLake state into Iceberg manifests on demand

Two paths, Iceberg REST and DuckLake direct, see exactly the same data.

The bridge

One snapshot identity, always in sync

  • The materialiser turns DuckLake's Postgres catalog into vN.metadata.json, manifest lists and manifests on S3
  • DuckLake HEAD == Iceberg current-snapshot-id
  • A row written through one path appears in the other automatically

Eager materialisation

~1s

for the Iceberg metadata chain to appear after a direct DuckLake write

A Postgres NOTIFY trigger wakes one elected listener. The lazy LoadTable path remains the correctness floor.

Full commit surface

Every Iceberg action becomes DuckLake SQL

Iceberg actionTranslation
add-snapshotducklake_add_data_files()
add-schemaALTER TABLE ADD/DROP COLUMN
add-partition-specALTER TABLE … SET PARTITIONED BY
upgrade-format-versionManifests re-emit in matching Avro schema

Iceberg v3 today

A small shim unlocks v3 writes from PyIceberg

  • One install() call before any RestCatalog operation
  • Puffin deletion vectors, one blob per affected data file
  • Row lineage and new types: variant, geometry, geography
  • v2 tables keep Parquet position deletes, so v2 readers still work

Governed access

No shared root keys handed to query engines

  • Credential vending: a real AssumeRole call, scoped to one table's data and metadata paths
  • Short-lived: credentials come with an expiry
  • OAuth2: HMAC-signed JWTs on every /v1/* route
  • RBAC: per-namespace scopes such as ns:analytics:rw

Operations

Built to be observed

  • Cache-hit LoadTable measured at ~349 req/s at concurrency 32
  • Prometheus /metrics, plus /healthz and /readyz
  • One Pixi environment runs Postgres, MinIO, the proxy and clients
  • Local and production environments are identical

What's left

The spec is effectively complete. The gaps are elsewhere.

  • DuckLake-blocked: true divergent branches, per-table set-location, real KMS encryption
  • Upstream: Spark v3 writes, DuckDB iceberg-ext v3 features
  • Ops: HA backends, TLS, tracing, Spark and Trino integration tests

The takeaway

Standard Iceberg tooling, no lock-in, DuckLake underneath.

Read the full article

Open source: github.com/KellerKev/duckicelake