Google Research introduces TurboQuant for AI model compression

What TurboQuant does

TurboQuant is a set of advanced quantization algorithms that enable massive compression for large language models and vector search engines. It specifically addresses bottlenecks in the key-value cache - a high-speed storage system that stores frequently used information under simple labels for instant retrieval.

How it works

TurboQuant achieves high reduction in model size with zero accuracy loss through two key steps:

• High-quality compression (PolarQuant method): Starts by randomly rotating data vectors to simplify geometry, then applies a standard quantizer to each part of the vector individually. This stage uses most of the compression power to capture the main concept and strength of the original vector.
• Eliminating hidden errors: Uses a small residual amount of compression power (just 1 bit) to apply the QJL algorithm to the tiny amount of error left over from the first stage. QJL acts as a mathematical error-checker that eliminates bias, leading to more accurate attention scores.

Key components

QJL (Quantized Johnson-Lindenstrauss): Uses the Johnson-Lindenstrauss Transform to shrink high-dimensional data while preserving distances between data points. It reduces each resulting vector number to a single sign bit (+1 or -1) with zero memory overhead. Uses a special estimator that balances high-precision queries with low-precision data to accurately calculate attention scores.

PolarQuant: Addresses memory overhead by converting vectors into polar coordinates using a Cartesian coordinate system. Instead of standard coordinates (X, Y, Z), it uses a format comparable to "Go 5 blocks total at a 37-degree angle" rather than "Go 3 blocks East, 4 blocks North."

Technical context

Traditional vector quantization typically introduces memory overhead of 1-2 extra bits per number due to storing quantization constants for every small data block. TurboQuant optimally addresses this challenge. The techniques showed promise in testing for reducing key-value bottlenecks without sacrificing AI model performance.

TurboQuant will be presented at ICLR 2026, while PolarQuant will be presented at AISTATS 2026.