Data integration

Apache NiFi inside Snowflake, with the fixes it needs

A patched NiFi 2.6.0 for Snowpark Container Services, ready to deploy.

Kevin Keller · kevinkeller.org

Why it matters

Keep the integration workhorse, drop the separate infrastructure

  • NiFi handles CDC, file ingestion, API polling and protocol translation in a visual designer.
  • On SPCS it runs inside Snowflake's compute and security boundary.
  • Native Snowflake OAuth: no stored passwords, no key files.
  • Auto-suspend, auto-resume and compute pool scaling.

The catch

NiFi was not designed to run behind SPCS ingress

Several compatibility issues stop the flow designer working at all.

This project patches all of them so teams can focus on building flows.

The patches

Three blockers, three fixes

ProblemFix
X-Forwarded-Port: 80 breaks every UI linkA patched servlet JAR in the image
Some components still misread ingress headersAn API proxy strips the headers
Default CSP blocks dynamic ingress hostnamesA permissive connect-src *

Authentication

OAuth tokens, not passwords

  1. SPCS mounts a session token in every container, refreshed every few minutes.
  2. A refresh daemon reads it and builds a complete JDBC URL.
  3. NiFi's connection pool reads the URL from a file.

No username, no password, nothing stored in the image or config.

The idea

Let Snowflake rotate the credential. Let a daemon keep NiFi current.

The JDBC URL follows the token automatically.

Operations

Three endpoints, all behind role-based ingress

  • NiFi UI (8080): the full flow designer
  • Token debug (8081): inspect tokens, decode JWTs, copy the JDBC URL
  • Web terminal (7681): a shell for debugging
  • Access: only users with the right Snowflake role

Sample flow

CDC from PostgreSQL to Snowflake out of the box

  1. QueryDatabaseTableRecord polls PostgreSQL for rows added since the last run.
  2. PutDatabaseRecord inserts them into Snowflake over JDBC with the SPCS token.

Swap PostgreSQL for MySQL, Oracle or SQL Server and the same approach works.

Getting started

Build locally, deploy when ready

  • Run the SQL setup scripts for the Snowflake infrastructure.
  • Build the patched image and push it to your image repository.
  • Deploy the service and open the nifi-ui endpoint.
  • Change the default admin credentials in the service spec.
  • test-local.sh gives the same image with local PostgreSQL.

The takeaway

NiFi works on SPCS, with patches. This project ships them.

Read the full article

Kevin Keller · kevinkeller.org