Data integration
Apache NiFi inside Snowflake, with the fixes it needs
A patched NiFi 2.6.0 for Snowpark Container Services, ready to deploy.
Kevin Keller · kevinkeller.org
Why it matters
Keep the integration workhorse, drop the separate infrastructure
- NiFi handles CDC, file ingestion, API polling and protocol translation in a visual designer.
- On SPCS it runs inside Snowflake's compute and security boundary.
- Native Snowflake OAuth: no stored passwords, no key files.
- Auto-suspend, auto-resume and compute pool scaling.
The catch
NiFi was not designed to run behind SPCS ingress
Several compatibility issues stop the flow designer working at all.
This project patches all of them so teams can focus on building flows.
The patches
Three blockers, three fixes
| Problem | Fix |
|---|---|
X-Forwarded-Port: 80 breaks every UI link | A patched servlet JAR in the image |
| Some components still misread ingress headers | An API proxy strips the headers |
| Default CSP blocks dynamic ingress hostnames | A permissive connect-src * |
Authentication
OAuth tokens, not passwords
- SPCS mounts a session token in every container, refreshed every few minutes.
- A refresh daemon reads it and builds a complete JDBC URL.
- NiFi's connection pool reads the URL from a file.
No username, no password, nothing stored in the image or config.
The idea
Let Snowflake rotate the credential. Let a daemon keep NiFi current.
The JDBC URL follows the token automatically.
Operations
Three endpoints, all behind role-based ingress
- NiFi UI (8080): the full flow designer
- Token debug (8081): inspect tokens, decode JWTs, copy the JDBC URL
- Web terminal (7681): a shell for debugging
- Access: only users with the right Snowflake role
Sample flow
CDC from PostgreSQL to Snowflake out of the box
QueryDatabaseTableRecordpolls PostgreSQL for rows added since the last run.PutDatabaseRecordinserts them into Snowflake over JDBC with the SPCS token.
Swap PostgreSQL for MySQL, Oracle or SQL Server and the same approach works.
Getting started
Build locally, deploy when ready
- Run the SQL setup scripts for the Snowflake infrastructure.
- Build the patched image and push it to your image repository.
- Deploy the service and open the
nifi-uiendpoint. - Change the default admin credentials in the service spec.
test-local.shgives the same image with local PostgreSQL.
The takeaway
NiFi works on SPCS, with patches. This project ships them.
Kevin Keller · kevinkeller.org