↓ Skip to main content

Stop Paying Real-Time Prices for Archive Security Data

C-level corner · Executive brief

Real-time security tools are often priced by the volume of data they take in, so organisations pay premium rates for logs that are only ever read during an investigation or an audit. Send only the data that needs instant alerting to the real-time tool and keep everything in a cheaper, searchable data lake. In the technical article's worked example, that cuts the monthly bill by 70 to 80% and makes years of retention affordable.

The business problem
#

Log volumes grow with the business. The real-time detection tool (SIEM) is where your security team spots attacks as they happen, and its value is real. But much of what flows into it does not need second-by-second attention: web access logs, DNS lookups, cloud audit trails, network traffic records. These are invaluable when you investigate an incident or answer an auditor, and they are rarely needed in the next minute.

When everything goes into the most expensive tier, two things happen. The bill becomes one of the largest lines in the security budget. And retention gets cut to fit the budget: the article puts typical retention in a SIEM-only setup at 30 to 90 days, while compliance frameworks often ask for one to seven years. Teams end up choosing between cost and the evidence they may later need.

What changes
#

Tier security data by what it is used for, not by where it happens to land today. One question decides the routing: does this event need detection and alerting within a minute?

  • Real-time tier. Endpoint alerts, failed logins, blocked firewall traffic, privilege escalation, data loss alerts. In the article this is around 20 to 30% of volume.
  • Data lake. Everything, including a copy of the critical events. It becomes the complete record for investigations, threat hunting and compliance reporting.
  • Search across both. Analysts query the lake from the tool they already use, without moving raw data back. Those searches take seconds rather than milliseconds, which is fine for an investigation and not for live alerting.
  • Older data gets cheaper. Storage can move aging logs to lower-cost tiers automatically and delete them when the retention period ends.

What it’s worth
#

  • Lower running cost. In the article’s example of 100 GB of logs a day, sending everything to the real-time tool comes to roughly $15,000 to $20,000 a month. Routing only the critical 20 to 30% there, with all data in the lake, comes to roughly $3,500 to $6,500. These are one scenario’s estimates, not a quote.
  • Compliance you can afford. The same example puts five years of logs (about 182 TB) at roughly $4,200 to $7,300 a month in lake storage, where keeping them in the real-time tool is described as not practical for most organisations.
  • Faster investigations. Analysts can search months of history, not only the real-time tool’s retention window, when they need to know whether an attacker has been seen before.
  • No loss of live detection. The real-time tool keeps doing what it is best at, on the events that need it. The lake adds to it rather than replacing it.

Decisions for leadership
#

  1. Ask for a breakdown of security log volume by source and by how each source is actually used: live alerting, investigation or compliance.
  2. Set retention periods per data type from your regulatory and legal needs, then fund them, instead of letting the licence decide.
  3. Fund the routing work as a project with an owner. Deciding what is critical needs ongoing tuning, and the article advises starting conservative and reducing over time.
  4. Check the fit before committing. The article sees the case as strongest above about 50 GB a day and not justified below about 10 GB, and warns that detections relying on all log sources in one place can break if data is split.

Questions to ask your team
#

  • How many GB a day do we send to the real-time tool, and what share of it ever triggers an alert?
  • How long do we keep security logs today, and how long are we required to?
  • Which of our detection rules depend on sources we might move out of the real-time tier?
  • Do we have the people to run two systems and maintain the routing rules?
  • What would our own numbers look like with our volumes and contracts?

Technical background, for your architects

Figures come from the linked articles and their sources; check them against your own situation. Snowflake-related articles explore the art of the possible and are not official Snowflake solutions or endorsed by Snowflake.

Share this article