Better together

Keep Splunk's detection. Keep every log for years.

A hybrid security data lake with Splunk, Snowflake and federated search.

Kevin Keller · kevinkeller.org

Why it matters

Not every log needs sub-second alerting

Log volume grows with the organisation. Web access, DNS, cloud audit and netflow are vital for investigations and compliance, but they don't all belong in the real-time detection tier.

The architecture

Everything to Snowflake. The critical subset to Splunk.

A routing layer such as Vector, Cribl, Kafka or Fluent Bit sends each event to one or both.

The routing question

One question decides where each log goes

Does this event need sub-minute detection and alerting?
  • Splunk, 20-30%: EDR alerts, authentication failures, firewall denies, privilege escalation, DLP
  • Snowflake, 100%: web, DNS, netflow, cloud audit, email, proxy, and the critical logs too
  • Snowflake is always the complete record

The savings

70-80%

lower monthly spend at 100 GB/day

Splunk-onlyHybrid
Total monthly cost~$15,000-20,000~$3,500-6,500
Retention30-90 days30 days Splunk + years in Snowflake

The compliance multiplier

Five years of logs becomes affordable

RetentionSplunk-onlySnowflake storage
1 year (36 TB)Significantly higher licence~$830-1,440/mo
2 years (73 TB)Often cost-prohibitive~$1,680-2,920/mo
5 years (182 TB)Not practical for most~$4,200-7,300/mo

Storage lifecycle policies can move older data to COOL and COLD tiers.

Federated search

Analysts query Snowflake without leaving Splunk

  • Splunk DB Connect runs dbxquery against Snowflake
  • Aggregation pushdown: the SQL runs on Snowflake, only results return
  • Hunt for SQL injection or XSS across months of logs
  • Join today's EDR alerts with an IP's history in Snowflake

Detection in Snowflake

Dynamic Tables add a near-real-time layer

  • Refresh on a target lag, typically 1 to 5 minutes
  • Chain into multi-stage pipelines, such as brute force detection
  • Declarative: define the SQL and the lag, Snowflake does the rest
  • Separate SECURITY_WH and SPLUNK_WH to track costs

Fit

Worth it at volume, not for everyone

Good fitNot a good fit
50+ GB/dayUnder 10 GB/day
Multi-year retention needsHeavy reliance on ES notable models across all sources
Historical threat huntingReal-time-only use cases
Snowflake already in useNo capacity for the routing layer

Trade-offs

Know these before you split

  • Federated queries take seconds, not milliseconds
  • Two platforms to monitor, secure and run
  • "Critical" needs ongoing tuning: start by sending more to Splunk
  • Audit which ES searches depend on a source before rerouting it

The takeaway

Not a replacement story: each platform for what it does best.

Read the full article

Kevin Keller · kevinkeller.org