Most logs are evidence, not alarms. Stop paying alarm prices for evidence.
Executive summary#
The business problem. Security tools (SIEM) and observability tools (APM, log management) are often priced by the volume of data they take in. Log volumes grow with the business, so the bill grows with it, whether or not anyone looks at the data. The usual response is to drop sources or cut retention, which saves money and quietly throws away the evidence an investigation or an auditor will ask for later.
What changes. Treat logs, traces and metrics as data, not as a by-product of a tool:
- Keep everything in an open lakehouse you own: low-cost object storage, open table formats such as Apache Iceberg, and SQL on top.
- Send only the time-critical slice to the real-time tools. The SIEM keeps detecting attacks and the APM keeps paging on-call engineers, on the events that need second-by-second attention.
- Query history where it lives. Analysts and engineers reach the lakehouse from the tools they already use, with federated queries, instead of copying data back.
What it’s worth. In a worked example from my hybrid security data lake build, 100 GB of security logs a day cost roughly $15,000 to $20,000 a month with everything in the SIEM, and roughly $3,500 to $6,500 with only the critical 20 to 30% there and everything in the lake: a 70 to 80% reduction. Five years of retention became affordable instead of impractical. These are one scenario’s estimates, not a quote, but the shape of the saving is the point.
The decision for leadership. Make the volume of data a tool ingests an architecture decision, not a licence surprise. Decide what each kind of data is for, and pay for the tier it actually needs.
Why the bill keeps growing#
Three forces push in the same direction.
- Volume grows with the business. More services, more cloud accounts, more users, more AI agents: every one of them writes logs. Pricing by volume turns growth into cost automatically.
- Most data is rarely read. Web access logs, DNS lookups, cloud audit trails, network flows and debug-level application logs are invaluable during an incident or an audit, and almost never needed in the next minute.
- Retention requirements go up, not down. In the security build, typical SIEM-only retention was 30 to 90 days, while compliance frameworks such as GDPR, DORA, PCI DSS and HIPAA often ask for one to seven years.
When everything lands in the most expensive tier, the organisation ends up choosing between the budget and the evidence. That is a false choice created by the architecture, not a law of nature.
The pattern: one record, two speeds#
The idea is simple: separate the system of record from the systems of reaction.
| Real-time tools (SIEM, APM) | Open lakehouse | |
|---|---|---|
| What goes in | The time-critical slice | Everything, including a copy of the critical slice |
| What it’s for | Alerting and paging within seconds or minutes | Investigations, threat hunting, root-cause analysis, compliance, reporting, AI |
| Retention | Short: days to weeks | Long: as long as policy requires, with older data moved to cheaper storage |
| Query speed | Milliseconds | Seconds, which is fine for an investigation, not for live alerting |
| Format | The tool’s own | Open (for example Apache Iceberg), readable by many engines |
Four building blocks make it work, and none of them is tied to one vendor:
- A routing layer. One pipeline collects everything and decides where each event goes. Open options exist for every part: the OpenTelemetry Collector for logs, traces and metrics, Vector, Fluent Bit or Kafka for log streams.
- Open storage you own. Object storage with an open table format. Your data stays in your account and your format, and more than one query engine can read it. That keeps your options open at the next renewal.
- Federated queries. The SIEM or APM queries the lakehouse when an analyst needs history, and pushes the heavy lifting (filtering, aggregation) down to the lakehouse instead of pulling raw data back. Other teams query the same data directly with SQL, notebooks or dashboards.
- Detections that run on the lake, where seconds are enough. Scheduled or continuously refreshed queries can catch slower patterns, such as brute force over hours or unusual data access over days, and raise an alert in the real-time tool when they find something.
The same design covers both worlds. In my OpenTelemetry build, logs, traces and metrics flow through the OpenTelemetry Collector into Iceberg tables, while the SIEM and APM keep the most recent data. In the security build, the SIEM keeps the critical slice and searches the rest by federated query.
Deciding what goes where#
One question decides the routing: does this event need someone to react within minutes?
- Real-time tier (and the lake): endpoint alerts, failed and suspicious logins, blocked firewall traffic, privilege escalation, data loss alerts, and the service-level errors and latency that page on-call engineers. In the security build this was around 20 to 30% of volume.
- Lake only: web access logs, DNS, network flows, cloud audit trails, debug and info-level application logs, full traces, and the long tail of metrics. Everything here is still searchable, from the same tools, in seconds.
Start conservative: send more to the real-time tier at first and reduce as confidence grows. Routing is a living rule set, not a one-off project.
What it’s worth, beyond the licence line#
- Lower running cost. Pay real-time prices only for real-time data. The 100 GB a day example above is the clearest number, and the saving grows with volume.
- Retention you can afford. In the same example, five years of security logs (about 182 TB) came to roughly $4,200 to $7,300 a month in lake storage. Keeping them in the SIEM was described as not practical for most organisations.
- Better investigations. “Have we seen this attacker before?” and “when did this error really start?” need months of history, not the last 30 days.
- One copy for many uses. The same data serves security, operations, finance (cost per service), product (usage) and AI use cases, instead of each team paying to collect it again.
- Negotiating power. When your data lives in an open format you control, the real-time tool is a choice you renew on merit, not a place your history is locked in.
A phased plan#
- Measure. Break down ingest by source, and by how each source is actually used: live alerting, investigation, compliance or nothing at all. The answer is usually a surprise.
- Build the lake and the routing layer. Send everything to the lake first, alongside today’s setup, so nothing changes for the teams yet.
- Connect federated search. Make sure analysts and engineers can query the lake from the tools they already use before anything is taken away from them.
- Move sources one at a time. Start with high-volume, low-urgency sources such as DNS, network flows or debug logs. Check which detections and dashboards depend on each source before you move it.
- Set retention by policy. Retention per data type from legal and regulatory needs, with automatic tiering to cheaper storage and deletion at the end.
- Renegotiate. With volumes down and history safe in the lake, size the next licence for the real-time slice only.
When it doesn’t pay off#
Be honest about the fit. The security build sees the case as strongest above about 50 GB a day and not justified below about 10 GB, where running two systems costs more attention than it saves. Other watch-outs from that work:
- Detections that need every source in one place can break if data is split. Audit them before moving a source.
- Two platforms to run. Someone has to own the routing rules, the lake and its access controls.
- Federated queries take seconds, not milliseconds. Fine for investigations and dashboards, wrong for live correlation.
- The lake holds sensitive data. Logs contain user names, IP addresses and sometimes personal data, so the lake needs the same access control, encryption and audit as any other sensitive store.
Questions to ask your team#
- How many GB a day do we send to our SIEM and observability tools, and what share of it ever triggers an alert or gets looked at?
- How long do we keep logs today, and how long are we required to?
- If we needed a year of history for an investigation tomorrow, where would it come from?
- Is our log and trace history in a format we own, or only inside a vendor’s tool?
- Which detections and dashboards depend on which sources?
- What would the numbers look like with our own volumes and contracts?
Go deeper#
- Stop Paying Real-Time Prices for Archive Security Data: the two-page executive brief.
- A Hybrid Security Data Lake with Federated Queries: the full cost model, routing tables, retention tiers and federated query examples, built with Splunk and Snowflake.
- OpenTelemetry Logs, Traces and Metrics into an Iceberg Security Data Lake: the observability side, from the OpenTelemetry Collector into open Iceberg tables.
- DuckLake on Hetzner with Pixi: what a small governed data lake can look like with open-source tools on European infrastructure.
The figures come from the linked articles and are estimates for one scenario at the prices used there; redo them with your own volumes, platforms and contracts. Snowflake-related articles explore the art of the possible and are not official Snowflake solutions or endorsed by Snowflake.
