Observability data lake
Logs, traces and metrics, always ready to query
Ingesting OpenTelemetry data directly into Snowflake to build an open SIEM and APM data lake.
Kevin Keller · kevinkeller.org
Why it matters
Long-term telemetry usually sits in storage nobody can query
- SIEM and APM data often moves to cold storage and must be rehydrated first.
- Storage costs and scalability limit how much you keep.
- Snowflake with Iceberg tables gives an open lake any Iceberg engine can query.
The idea
Plug the OpenTelemetry ecosystem straight into Snowflake
A receiver that accepts logs, metrics and traces over OTLP, running inside Snowflake Container Services.
The data flow
From application to open tables in four steps
- An application sends telemetry to an OpenTelemetry Collector.
- The collector forwards it over HTTPS to the receiver in Snowflake.
- The receiver writes raw logs, metrics and traces tables.
- Streams and a task transform them into Elastic Common Schema Iceberg tables.
Why a standard schema
Elastic Common Schema makes many sources one dataset
- Logs, metrics and traces from different sources share one schema.
- It makes the data easier to work with and to scale.
- Iceberg tables keep it open to any compatible query engine.
What you can do with it
One place for SIEM and APM analysis
- Incidents: a 10-minute log table for time-sensitive alerts
- Automated response: notifications and triggered actions
- Threat hunting and bottleneck analysis in notebooks
- Dashboards: Streamlit in Snowflake, PromQL (experimental) and Splunk SPL samples
- Enrichment: HR, identity provider and Marketplace data
The sweet spot
Augment your SIEM rather than replace it
Keep your current SIEM and APM for recent data, and send everything else to Snowflake instead of long-term storage.
All data stays ready to query, with no rehydration.
Getting started
Run it locally first, then move the receiver into Snowflake
| Step | What runs where |
|---|---|
| 1. Local | Sample Flask app, collector and receiver in Docker Compose |
| 2. Snowflake | Receiver in Container Services, app and collector outside |
Authentication to the Snowflake endpoint uses an OAuth token from a key pair.
What's next
More of the pipeline moves into Snowflake
- Run all containers in Snowflake Container Services.
- Notebooks for data crunching and threat hunting.
- Batching on the collector for more efficient delivery.
- Ingest Vector and Fluentd data the same way.
The takeaway
Centralise SIEM and APM data in Snowflake, open, standardised and queryable.
Kevin Keller · kevinkeller.org