Local LLMs

The memory wall is the KV cache, not the weights

TurboQuant compresses the cache so long contexts and 128B models fit on consumer hardware.

Kevin Keller · kevinkeller.org

Why it matters

Context length, not model size, runs you out of memory

  • Weight quantization (4-bit, 8-bit) is mature and well understood.
  • The KV cache grows linearly with every token of context.
  • At 32K tokens a 26B model's cache alone can take several gigabytes.
  • At 128K, or with a 128B model, you run out of memory or truncate.

The usual workaround

Squeezing the weights harder hurts quality

Aggressive 3-bit or 2-bit model quantization frees memory but costs quality. TurboQuant leaves the weights at the precision you prefer and compresses the cache instead, where the structure allows it without perceptible loss.

The result

3.8–5.1x

less KV cache memory, with negligible quality loss.

Measured on a Mac Studio M4 Max with the turbo4 mode.

Benchmarks

Large models become viable locally

ModelKV compression
Gemma 4 26B5.1x
Mistral Medium 3.5 128B3.8x
Llama 3.xSupported
Qwen 3.5Supported

Models that had to truncate to 8K tokens now run at 32K and beyond.

Two modes

Use 4-bit by default

  • turbo4: 4-bit, reliable across every tested architecture
  • Recommended default for keys and values
  • turbo3: 3-bit, higher compression
  • Caveat: corrupted output on Mistral Medium 3.5, safe on Llama, Qwen and Gemma

The idea

Compress the cache, not the model.

Keep the weights you trust and win back the memory that long contexts eat.

Getting started

One install, no compilation

  1. Add llama-cpp-turboquant from the prefix.dev channel with Pixi or conda.
  2. Start llama-server with --cache-type-k turbo4 and --cache-type-v turbo4.
  3. Start at a context of 32,768 tokens.
  4. Point any OpenAI-compatible agent at http://localhost:8080/v1.

Why it suits agents

Coding agents need the headroom

  • A typical coding agent session can use 15 to 20K tokens before answering.
  • Without TurboQuant, that leaves little room for responses on consumer hardware.
  • Builds cover Apple Silicon with Metal, Linux CPU, CUDA 12+ and ARM.

Why conda, not Docker

Native execution wins for local inference

  • Same dependency reproducibility as Docker, with no container boundary.
  • Metal runs directly against the GPU on Apple Silicon.
  • The trade-off: you need a compatible OS and GPU drivers.

The takeaway

Compress the KV cache and serious context runs locally.

Read the full article

Kevin Keller · kevinkeller.org