↓ Skip to main content

Security Data Lake Lab

The article behind this: Your Log Bill Is a Design Choice: Observability and Security Data in a Lakehouse, with Federated Queries All playgrounds · Articles · Subscribe

Your mission

Last night something happened at a 50-person company. Its logs from the last 21 days are below, and an attack is hidden in them.

  1. Route. The defaults follow the article: the time-critical slice goes to the SIEM, and everything goes to the lake.
  2. Open the SIEM. Which alerts fired in real time? Look at the dashboard.
  3. Hunt in the lake. The endpoint alert on WS-042 is real. How long has the attacker been on that machine? Ask the SIEM first, then the lake.
  4. Break it. Set Sign-ins to "Lake only", or DNS to "Drop", and see which detections go blind and which hunts lose their evidence.
  5. Check the cost of your routing at your own daily volume.

Starting DuckDB in your browser...

About 7 MB, loaded once. Nothing leaves your machine.

A simplified simulation for learning. The company, the people, the logs and the attack are made up, and the detection rules are teaching examples, not production rules. In a real setup the routing layer would be something like the OpenTelemetry Collector, Vector or Kafka, and the lake would be open tables such as Apache Iceberg queried from the SIEM. The article explains the architecture.

Articles that use this playground

Share this article