Hybrid cloud, part 1

Query on-premise data from Snowflake, with every ingress port closed

An SSH tunnel from Snowflake Container Services to your DMZ.

Kevin Keller ยท kevinkeller.org

Why it matters

Use the cloud today, even for what can't move

  • Reach resources that aren't migrated yet, during your move to the cloud
  • Some APIs, AI models and data will never leave for compliance reasons
  • Access from Snowflake can be cut at any time

The architecture

The DMZ is the bridge

  1. The private network makes only an outbound connection to the DMZ
  2. The DMZ opens one SSH port
  3. An SPCS container establishes the SSH tunnel at start-up

Ingress ports on the internal firewall stay closed.

Why SSH

VPNs need a privilege containers don't get

OpenVPN and WireGuard require NET_ADMIN for a container, which Snowflake Container Services doesn't grant today. SSH works without it.

The demo

Two engines, one Iceberg lake behind a firewall

  • DuckDB and Spark in one container
  • Iceberg REST Catalog and MinIO on private IPs
  • Jupyter and VS Code in the container
  • Streamlit app and a REST API callable as a UDF

The control

Cut the tunnel, lose the access

Spark and DuckDB run in the container, not a warehouse, so no result set cache holds on-premise data in Snowflake.

Three ways to query

Each with its own trade-off

PathNote
Jupyter notebookJoin on-premise and Snowflake dataframes freely
Snowflake UDFResults are cached; scalar only, up to 16MB
Streamlit appEdit live in the container's VS Code

Setup

From a Linux VM to a running tunnel

  1. Run the Iceberg lake with Docker Compose and load sample data
  2. Close every port except 22
  3. Push the image to a Snowflake image repository
  4. Store the SSH user, DMZ IP and key in a Snowflake secret
  5. Create the service and open its endpoints

Before production

What the proof of concept doesn't do yet

  • Replace SSH private keys with short-lived SSH certificates
  • Broker credentials, for example Device Code OAuth for notebooks
  • Fetch secrets from a secret manager instead of code
  • Consider a catalog with role and policy controls, such as Apache Polaris

Beyond Iceberg

Any resource behind the firewall

On-premise databases, APIs and AI models work the same way. Part 2 reverses the direction, exposing services in Snowflake Container Services to apps outside Snowflake.

The takeaway

Reach data that will never move, and keep the switch in your hands.

Read the full article

A proof of concept, not a full production setup