DuckLake with Pixi

A governed data lake for under 10 euros a month

Row-level access control, S3 storage and SQL analytics, from one pixi.toml.

Kevin Keller ยท kevinkeller.org

Why it matters

A data lake doesn't need heavyweight infrastructure

  • No Hive Metastore, no Iceberg REST catalog
  • Access control without a commercial platform
  • The same catalog format locally and in production

Inspired by Floyd Berndsen's ducklake-hetzner and ducklake-guard.

The stack

Four parts, each doing one job

  • DuckDB: the query engine, SQL directly on Parquet in S3
  • PostgreSQL: the catalog, plus row-level security policies
  • S3 storage: MinIO locally, Hetzner Object Storage in production
  • Pixi: installs and runs all of it

Why DuckLake

The catalog is just a SQL database

Iceberg, Delta Lake and Hudi keep metadata in files and need a catalog service. DuckLake stores tables, snapshots and file references in PostgreSQL.

  • Concurrent access via standard connections
  • Access control via standard SQL grants
  • Any PostgreSQL client can inspect it

Why Pixi

One file defines everything

  • Dependencies, tasks and environments in a single pixi.toml
  • pixi.lock pins every dependency to an exact version and hash
  • DuckDB, PostgreSQL and MinIO come from conda-forge, nothing touches the system
  • No Makefile, no Docker Compose, no requirements.txt

Guard

Two layers: hide it in the catalog, block it in storage

Access control that holds even if someone bypasses the catalog.

How Guard works

Catalog and storage both enforce access

  1. PostgreSQL RLS: a reader without access to customer doesn't even see it in the catalog
  2. S3 bucket policies: reader credentials are scoped to specific key prefixes
  3. One command each: guard-acl-grant, guard-acl-revoke, guard-acl-show

Production cost

~9 EUR/mo

for the full governed stack on Hetzner

ComponentServiceCost
PostgreSQL + DuckDBCX22 VPS (2 vCPU, 4 GB)~5.50 EUR
Object StorageHetzner S3, 1 TB included~3.50 EUR

Local to production

Develop on MinIO, switch with a config change

  • pixi run local-up starts PostgreSQL and MinIO
  • Load the TPC-H sample (~86K rows, 8 tables) to test access control
  • pixi run prod-up creates the Hetzner bucket
  • The DuckLake catalog format is identical in both

When to use it

Simplicity and cost over scale

  • A lightweight lake without Iceberg or Delta Lake infrastructure
  • Row-level access control on a budget
  • Analytics teams, side projects, local/production parity

Not a replacement for Snowflake or Databricks at large production scale.

The takeaway

Just SQL, Parquet and one Pixi file.

Read the full article

Code: github.com/KellerKev/ducklake-pixi