Observability data lake

Logs, traces and metrics, always ready to query

Ingesting OpenTelemetry data directly into Snowflake to build an open SIEM and APM data lake.

Kevin Keller · kevinkeller.org

Why it matters

Long-term telemetry usually sits in storage nobody can query

  • SIEM and APM data often moves to cold storage and must be rehydrated first.
  • Storage costs and scalability limit how much you keep.
  • Snowflake with Iceberg tables gives an open lake any Iceberg engine can query.

The idea

Plug the OpenTelemetry ecosystem straight into Snowflake

A receiver that accepts logs, metrics and traces over OTLP, running inside Snowflake Container Services.

The data flow

From application to open tables in four steps

  1. An application sends telemetry to an OpenTelemetry Collector.
  2. The collector forwards it over HTTPS to the receiver in Snowflake.
  3. The receiver writes raw logs, metrics and traces tables.
  4. Streams and a task transform them into Elastic Common Schema Iceberg tables.

Why a standard schema

Elastic Common Schema makes many sources one dataset

  • Logs, metrics and traces from different sources share one schema.
  • It makes the data easier to work with and to scale.
  • Iceberg tables keep it open to any compatible query engine.

What you can do with it

One place for SIEM and APM analysis

  • Incidents: a 10-minute log table for time-sensitive alerts
  • Automated response: notifications and triggered actions
  • Threat hunting and bottleneck analysis in notebooks
  • Dashboards: Streamlit in Snowflake, PromQL (experimental) and Splunk SPL samples
  • Enrichment: HR, identity provider and Marketplace data

The sweet spot

Augment your SIEM rather than replace it

Keep your current SIEM and APM for recent data, and send everything else to Snowflake instead of long-term storage.

All data stays ready to query, with no rehydration.

Getting started

Run it locally first, then move the receiver into Snowflake

StepWhat runs where
1. LocalSample Flask app, collector and receiver in Docker Compose
2. SnowflakeReceiver in Container Services, app and collector outside

Authentication to the Snowflake endpoint uses an OAuth token from a key pair.

What's next

More of the pipeline moves into Snowflake

  • Run all containers in Snowflake Container Services.
  • Notebooks for data crunching and threat hunting.
  • Batching on the collector for more efficient delivery.
  • Ingest Vector and Fluentd data the same way.

The takeaway

Centralise SIEM and APM data in Snowflake, open, standardised and queryable.

Read the full article

Kevin Keller · kevinkeller.org