Agent memory

Agents that remember store embeddings. TurboQuant makes them 6–8x smaller.

Vector compression with under 2% recall loss and no training.

Kevin Keller · kevinkeller.org

Why it matters

Vector storage gets expensive fast

  • Every memory, document chunk and saved tool result becomes a float32 vector.
  • 10,000 memories at 1,536 dimensions: 60 MB of vectors.
  • 100K memories across tenants: 600 MB, before metadata and indexes.

The result

3–4 bits

per element instead of 32, with under 2% recall loss.

Google's TurboQuant, published at ICLR 2026. No training required.

How it works

Two stages, no codebook

  1. PolarQuant: rotate each vector with a random orthogonal matrix, so every coordinate follows a known distribution, then quantise each coordinate.
  2. QJL: a 1-bit residual correction that preserves inner product accuracy for similarity search.

The rotation comes from a seed shared by writer and reader.

What changes for agents

Four reasons it fits agent memory

  • Remember more: the same RAM holds 6 to 8x more memories
  • No cold start: works on the first vector like the millionth
  • Cheaper persistence: saved memory files 8x smaller than float32
  • Quality kept: queries stay at full precision

No cold start

Data-oblivious compression suits incremental memory

You don't retrain a codebook every time a new memory arrives.

Product Quantization needs a representative sample first. TurboQuant fits the incremental add() pattern of agent memory.

The idea

Compress the stored vectors, keep the query at full precision.

Asymmetric search finds almost always the same memories at 4 bits as at 32.

In practice

One parameter in zettelkasten-memory

  • A TurboQuantCompressor(bits=4) sits between embedding and storage.
  • Usage of add and search stays identical.
  • The MCP server enables it with a --compress flag.
  • Six embedding providers, including Ollama and Snowflake Cortex.

Already spreading

TurboQuant is showing up in agent memory projects

  • Prism MCP advertises it as a headline feature.
  • LangChain has a community integration, langchain-turboquant.
  • The turboquant PyPI package has a first stable release.

The takeaway

If your agent stores embeddings, compression is the easiest win available.

Read the full article

Kevin Keller · kevinkeller.org