Local LLMs
The memory wall is the KV cache, not the weights
TurboQuant compresses the cache so long contexts and 128B models fit on consumer hardware.
Kevin Keller · kevinkeller.org
Why it matters
Context length, not model size, runs you out of memory
- Weight quantization (4-bit, 8-bit) is mature and well understood.
- The KV cache grows linearly with every token of context.
- At 32K tokens a 26B model's cache alone can take several gigabytes.
- At 128K, or with a 128B model, you run out of memory or truncate.
The usual workaround
Squeezing the weights harder hurts quality
Aggressive 3-bit or 2-bit model quantization frees memory but costs quality. TurboQuant leaves the weights at the precision you prefer and compresses the cache instead, where the structure allows it without perceptible loss.
The result
3.8–5.1x
less KV cache memory, with negligible quality loss.
Measured on a Mac Studio M4 Max with the turbo4 mode.
Benchmarks
Large models become viable locally
| Model | KV compression |
|---|---|
| Gemma 4 26B | 5.1x |
| Mistral Medium 3.5 128B | 3.8x |
| Llama 3.x | Supported |
| Qwen 3.5 | Supported |
Models that had to truncate to 8K tokens now run at 32K and beyond.
Two modes
Use 4-bit by default
- turbo4: 4-bit, reliable across every tested architecture
- Recommended default for keys and values
- turbo3: 3-bit, higher compression
- Caveat: corrupted output on Mistral Medium 3.5, safe on Llama, Qwen and Gemma
The idea
Compress the cache, not the model.
Keep the weights you trust and win back the memory that long contexts eat.
Getting started
One install, no compilation
- Add
llama-cpp-turboquantfrom the prefix.dev channel with Pixi or conda. - Start
llama-serverwith--cache-type-k turbo4and--cache-type-v turbo4. - Start at a context of 32,768 tokens.
- Point any OpenAI-compatible agent at
http://localhost:8080/v1.
Why it suits agents
Coding agents need the headroom
- A typical coding agent session can use 15 to 20K tokens before answering.
- Without TurboQuant, that leaves little room for responses on consumer hardware.
- Builds cover Apple Silicon with Metal, Linux CPU, CUDA 12+ and ARM.
Why conda, not Docker
Native execution wins for local inference
- Same dependency reproducibility as Docker, with no container boundary.
- Metal runs directly against the GPU on Apple Silicon.
- The trade-off: you need a compatible OS and GPU drivers.
The takeaway
Compress the KV cache and serious context runs locally.
Kevin Keller · kevinkeller.org