Security Data Lake Lab
Your mission
Last night something happened at a 50-person company. Its logs from the last 21 days are below, and an attack is hidden in them.
- Route. The defaults follow the article: the time-critical slice goes to the SIEM, and everything goes to the lake.
- Open the SIEM. Which alerts fired in real time? Look at the dashboard.
- Hunt in the lake. The endpoint alert on WS-042 is real. How long has the attacker been on that machine? Ask the SIEM first, then the lake.
- Break it. Set Sign-ins to "Lake only", or DNS to "Drop", and see which detections go blind and which hunts lose their evidence.
- Check the cost of your routing at your own daily volume.
Starting DuckDB in your browser...
About 7 MB, loaded once. Nothing leaves your machine.
One question decides the routing: does this event need a reaction within minutes?
For each source, choose where it goes. "Slice to SIEM" sends only the events that match the slice rule to the SIEM. The lake always gets the full stream, unless you drop the source.
See the routing rules as SQL
Alerts
Real-time rules run on the SIEM's copy of the data. Detections on the lake run every hour and raise their alerts here too. "Search the lake" runs a federated hunt on the alert's IP, host or user.
Dashboard
Failed sign-ins per hour, last 48 hours
Events in the SIEM per day, by source
Real-time detection rules
Query the lake directly
The tables are lake.auth, lake.edr, lake.cloud_audit, lake.netflow, lake.dns and lake.proxy. The SIEM's copies use the same names under siem., plus siem.alerts. A query can join both: that is the federated query. It's DuckDB SQL, and times are in UTC.
Detections on the lake (scheduled, every hour)
These catch slower patterns, where minutes or hours are fast enough, without paying SIEM prices for the data.
What your routing costs per month
How this is calculated: unit prices come from the 100 GB a day worked example in the hybrid security data lake article. That's about $150 to $200 a month for each GB a day of SIEM licence, about $5 to $9 a month for each GB a day of lake compute and storage, and $23 to $40 per TB a month for each extra month of lake storage beyond the first. The SIEM prices assume its usual 30 to 90 days of retention. Volume shares come from the simulated logs. These are estimates for one scenario, not a quote.
A simplified simulation for learning. The company, the people, the logs and the attack are made up, and the detection rules are teaching examples, not production rules. In a real setup the routing layer would be something like the OpenTelemetry Collector, Vector or Kafka, and the lake would be open tables such as Apache Iceberg queried from the SIEM. The article explains the architecture.
Articles that use this playground
- Your Log Bill Is a Design Choice: Observability and Security Data in a Lakehouse, with Federated QueriesSIEM and observability tools are often priced by the data they take in, so companies pay real-time prices for logs that are only read during an investigation or an audit, …
- A Hybrid Security Data Lake with Federated Queries, Built with Splunk and SnowflakeSIEM licences charge by volume, so logs get dropped. The pattern: route all logs to the data platform as the system of record, send the time-sensitive subset to the SIEM …