Subquadratic Debuts 12M Token Context Window for AI Models

Subquadratic has announced a 12-million-token context window, claiming a breakthrough in subquadratic attention mechanisms. This compares to typical 128K-1M token windows in current models. The technique allows models to handle vastly larger contexts without quadratic scaling of compute or memory.

Key Details

• Context window: 12 million tokens (12x larger than GPT-4's 128K tokens)
• Based on subquadratic attention, likely using linear or near-linear complexity in sequence length
• Enables processing entire large codebases, long documents, or multi-hour video transcripts in a single forward pass
• Potential applications: code review of entire repos, long-document analysis, multi-turn dialog with full history
• Compatible with existing transformer-based LLMs via drop-in attention replacement

The approach reduces O(n²) attention to near-O(n) using techniques like state-space models or low-rank factorizations. No specific benchmark numbers are provided in the source, but the claim is that this makes 12M-token windows practical on a single GPU.

Who It's For

AI engineers working on code analysis, document processing, or any task requiring long-context understanding without expensive chunking or retrieval.