Research division announced TurboQuant, a vector quantization algorithm that combines 2 other methods - QJL and PolarQuant, which solves the problem of increasing KV-cache when working with a long context.
TurboQuant will be presented at ICLR 2026, PolarQuant - at AISTATS 2026.
➡️ Why it matters?
The KV-cache stores intermediate representations of tokens so that the model does not recalculate them at each step of generation. With the growth of the context, it becomes a memory bottleneck.
Conventional vector quantization compresses this data, but it introduces overhead: for each block, it is necessary to store quantization constants in full accuracy, and this is plus 1–2 bits per element, which partially devalues the compression itself.
➜ TurboQuant is a two-stage pipeline.
First, PolarQuant: a random rotation aligns the geometry of vectors, after which they are translated from Cartesian coordinates to polar (radius and angle). The distribution of angles turns out to be predictable and concentrated, so normalization and storage of additional constants are no longer necessary.
At the second stage, QJL is connected, a method based on the Johnson-Lindenstrauss transformation, which encodes the residual error of the first stage with just one sign bit and, through a built-in estimation function, combines a high-precision query with low-precision compressed data, correctly calculating the attention score.
None of the methods requires training or retraining and works in the mode of "without preliminary analysis of the dataset".
The algorithms were tested on benchmarks for long context: LongBench, Needle In A Haystack, ZeroSCROLLS, RULER and L-Eval with the Gemma and Mistral models.
When quantizing the KV-cache to 3 bits, TurboQuant showed zero degradation in accuracy on all tasks: searching for a "needle in a haystack", QA, code generation, summarization.
The volume of the KV-cache at the same time decreased by 6 times. On H100, the four-bit TurboQuant accelerated the calculation of attention logits up to 8 times compared to 32-bit keys.
Area of application is not limited to KV-cache. In experiments with high-dimensional vector search, TurboQuant consistently surpassed PQ and RaBitQ methods in recall, despite fact that they used large code books and tuning to a specific dataset.
Article, Arxiv • #AI #ML #LLM #TurboQuant #Google
••••••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore