DuckLake with Pixi
A governed data lake for under 10 euros a month
Row-level access control, S3 storage and SQL analytics, from one pixi.toml.
Kevin Keller ยท kevinkeller.org
Why it matters
A data lake doesn't need heavyweight infrastructure
- No Hive Metastore, no Iceberg REST catalog
- Access control without a commercial platform
- The same catalog format locally and in production
Inspired by Floyd Berndsen's ducklake-hetzner and ducklake-guard.
The stack
Four parts, each doing one job
- DuckDB: the query engine, SQL directly on Parquet in S3
- PostgreSQL: the catalog, plus row-level security policies
- S3 storage: MinIO locally, Hetzner Object Storage in production
- Pixi: installs and runs all of it
Why DuckLake
The catalog is just a SQL database
Iceberg, Delta Lake and Hudi keep metadata in files and need a catalog service. DuckLake stores tables, snapshots and file references in PostgreSQL.
- Concurrent access via standard connections
- Access control via standard SQL grants
- Any PostgreSQL client can inspect it
Why Pixi
One file defines everything
- Dependencies, tasks and environments in a single
pixi.toml pixi.lockpins every dependency to an exact version and hash- DuckDB, PostgreSQL and MinIO come from conda-forge, nothing touches the system
- No Makefile, no Docker Compose, no requirements.txt
Guard
Two layers: hide it in the catalog, block it in storage
Access control that holds even if someone bypasses the catalog.
How Guard works
Catalog and storage both enforce access
- PostgreSQL RLS: a reader without access to
customerdoesn't even see it in the catalog - S3 bucket policies: reader credentials are scoped to specific key prefixes
- One command each:
guard-acl-grant,guard-acl-revoke,guard-acl-show
Production cost
~9 EUR/mo
for the full governed stack on Hetzner
| Component | Service | Cost |
|---|---|---|
| PostgreSQL + DuckDB | CX22 VPS (2 vCPU, 4 GB) | ~5.50 EUR |
| Object Storage | Hetzner S3, 1 TB included | ~3.50 EUR |
Local to production
Develop on MinIO, switch with a config change
pixi run local-upstarts PostgreSQL and MinIO- Load the TPC-H sample (~86K rows, 8 tables) to test access control
pixi run prod-upcreates the Hetzner bucket- The DuckLake catalog format is identical in both
When to use it
Simplicity and cost over scale
- A lightweight lake without Iceberg or Delta Lake infrastructure
- Row-level access control on a budget
- Analytics teams, side projects, local/production parity
Not a replacement for Snowflake or Databricks at large production scale.
The takeaway
Just SQL, Parquet and one Pixi file.
Code: github.com/KellerKev/ducklake-pixi