Security and observability data
Most logs are evidence, not alarms. Stop paying alarm prices for evidence.
Keep everything in a lakehouse you own, and query it where it lives.
Kevin Keller · kevinkeller.org
The problem
The bill grows with the business, whether anyone reads the data or not
- SIEM and observability tools are often priced by the data they take in
- More services, more cloud accounts, more AI agents: more logs
- The usual fix, cutting sources or retention, quietly deletes the evidence
Three forces
Everything pushes the same way
- Volume grows: pricing by volume turns growth into cost
- Most data is rarely read: DNS, flows, audit trails, debug logs
- Retention goes up: 30 to 90 days in the SIEM, one to seven years asked by compliance
- The result: budget versus evidence, a false choice
The idea
One record, two speeds.
Separate the system of record from the systems of reaction.
Two tiers
Pay for the tier each kind of data needs
| Real-time tools (SIEM, APM) | Open lakehouse | |
|---|---|---|
| What goes in | The time-critical slice | Everything |
| For | Alerts within minutes | Investigations, compliance, AI |
| Retention | Days to weeks | As long as policy requires |
| Query speed | Milliseconds | Seconds |
The building blocks
Four parts, none tied to one vendor
- A routing layer that decides where each event goes.
- Open storage you own, such as Apache Iceberg on object storage.
- Federated queries from the tools analysts already use.
- Detections on the lake where seconds are enough.
Deciding what goes where
Does this event need someone to react within minutes?
- Yes: endpoint alerts, suspicious logins, privilege escalation, on-call errors
- No: web access logs, DNS, network flows, cloud audit trails, debug logs
- Start conservative and reduce as confidence grows
What it's worth
70 to 80%
lower monthly cost in one worked example
100 GB of security logs a day: roughly $15,000 to $20,000 a month with everything in the SIEM, roughly $3,500 to $6,500 with the critical slice there and the rest in the lake.
One scenario's estimates, not a quote
Beyond the licence line
What leaders get
- Retention you can afford: five years became practical
- Better investigations: months of history, not 30 days
- One copy, many uses: security, operations, finance, AI
- Negotiating power: your history isn't locked in a tool
A phased plan
Nothing is taken away before the replacement works
- Measure ingest by source and by how it's used.
- Build the lake and routing, alongside today's setup.
- Connect federated search for analysts.
- Move high-volume, low-urgency sources one at a time.
- Renegotiate the licence for the real-time slice.
When it doesn't pay off
Be honest about the fit
- Strongest above about 50 GB a day, not justified below about 10
- Detections that need every source in one place: audit them first
- Two platforms to run, and someone must own the routing rules
- The lake holds sensitive data and needs the same controls
The takeaway
Make ingest volume an architecture decision, not a licence surprise.
Read the full article · The executive brief
Kevin Keller · Figures are estimates for one scenario · Not an official Snowflake solution