Analyzing AI Coding Tools: Dissecting 3,177 API Calls
The recent analysis conducted on four AI coding tools—Claude Code Opus 4.6, Claude Code Sonnet 4.5, Codex GPT-5.3, and Gemini 2.5 Pro—highlights substantial differences in managing API call context windows. Using the Context Lens tracer, the study intercepted 3,177 API calls to evaluate the tools’ efficiency and strategy in handling the context window when tasked with bug fixes in an Express.js environment.
Each coding tool tackled a specific bug—an incorrectly reordered null check in res.send(). Opus, Sonnet, Codex, and Gemini were tasked with identifying and fixing the bug, followed by running the test suite to verify the fix. They all succeeded, albeit with varying approaches and resources.
Claude Code Opus 4.6 consistently used around 23K to 27K tokens, mainly consisting of tool definitions (69% of the context). This indicates a reliance on re-sending these definitions due to the architecture, causing significant caching overhead. Codex (GPT-5.3) presented a wider range from 29.3K to 47.2K tokens, mostly tool results (72%), providing more variability depending on test command specificity. Sonnet, with similar variance, mixed definitions and results more evenly.
Gemini stands out due to its disproportionate use of tokens, peaking at 350.5K, utilizing almost exclusively tool results (96%), exploiting its large 1M context window. Despite a lower cost per token, Gemini’s inconsistent and expansive usage pattern without convergence across runs indicates a unique, albeit less efficient strategy.
These findings illustrate considerable disparities in how AI coding tools manage context windows, impacting both performance and cost efficiency. Developers should weigh token usage strategies when choosing the appropriate tool for their needs, particularly for tasks involving iterative changes or extensive project histories.
📖 Read the full source: HN LLM Tools
👀 See Also
Building and Testing an MCP Server in Claude Desktop: Architecture and Lessons
A developer shares their experience building and testing an MCP server within Claude Desktop, detailing their architecture setup and practical lessons learned about tool schemas, debugging, and limitations.
Local RAG Tool Built with Nemotron Nano 9B v2 and vLLM Tool Calling
A developer built a local-first RAG research tool that runs entirely on a single GPU using Nemotron Nano 9B v2 Japanese on vLLM with custom parser plugins for tool calling. The system features a two-step extract-execute flow with bilingual keyword extraction and parallel FTS5/DuckDuckGo search.
Efficient Workflow Using Claude Code: Planning Before Execution
Boris Tane leverages Claude Code with a structured planning-first approach, focusing on detailed research and planning to maintain control over architecture decisions.
Total Recall: Local Knowledge Graph for Claude Code Conversation History
Total Recall is an open-source system that ingests Claude Code's JSONL conversation transcripts into a SQLite database with full-text search and vector embeddings, making conversation history searchable across sessions. It retrieves actual conversation excerpts with DAG-aware context and includes a ChatGPT importer.