Benchmark shows context engine reduces AI coding agent costs by 3x on SWE-bench
A developer benchmarked four AI coding agents on SWE-bench Verified using the same Claude Opus 4.5 model, with context management as the only variable. The results show significant cost differences for similar performance levels.
Benchmark setup
The test used a 100-task stratified subset of SWE-bench Verified with all 12 repositories represented proportionally. All agents ran Claude Opus 4.5 with the same $3/task budget and 250-turn limit. The only difference was the context layer in front of the model.
Results
- Context engine + Claude Code: 73.0% Pass@1, $0.67/task
- Live-SWE-Agent: 72.0% Pass@1, $0.86/task
- OpenHands: 70.0% Pass@1, $1.77/task
- Sonar Foundation: 70.0% Pass@1, $1.98/task
The most expensive setup costs 3x more per task for a lower resolution rate. Eight tasks were solved only by the setup with the context layer - bugs that the model couldn't fix without seeing the right code.
Limitations
On matplotlib (rendering-heavy, visual output code), the context engine scored 43% while Sonar Foundation hit 86%. Graph-based context is less effective when relevant code doesn't follow dependency chains.
How the context layer works
Instead of letting Claude read entire files, it pre-indexes the codebase into a dependency graph using tree-sitter + SQLite (30 languages supported) and returns a ranked context capsule: full source for functions that matter, skeletonized signatures for everything connected to them. The agent starts every task already knowing what's relevant.
It includes session memory that persists across sessions via MCP. When code changes, previous observations get flagged as stale automatically, so the agent doesn't re-explore the same things.
The system is 100% local with no cloud, no account, and no code leaving your machine. It works with Claude Code and 11 other agents via MCP.
Open source availability
The benchmark harness, all evaluation logs, per-instance results, and comparison scripts are available on GitHub at github.com/Vexp-ai/vexp-swe-bench. The tool itself is available at vexp.dev with a free tier, VS Code extension, or CLI. Full benchmark results with charts are at vexp.dev/benchmark.
📖 Read the full source: r/ClaudeAI
👀 See Also
Vektori's Memory Architecture: Principles from Claude's Leaked System
Vektori implements a three-layer hierarchical sentence graph for AI memory, inspired by leaked principles from Claude's architecture. The system uses strict quality filters, skeptical retrieval with a 0.3 minimum score, and maintains correction history across sessions.
ClawCut: A Python Proxy That Makes Small Local LLMs Usable with OpenClaw
ClawCut is a Python Flask proxy that solves common problems when connecting 7B/14B local models to OpenClaw, including context poisoning, infinite loops, and failed cron job outputs. It implements dynamic amnesia during tool calls and auto-delivery for scheduled tasks.
Local PII Redaction Skill for OpenClaw Uses GLiNER Model
A new OpenClaw skill intercepts outgoing responses and runs them through the local nvidia/gliner-PII model to detect and redact sensitive information like API keys and PII, replacing them with labels like [API_KEY] and adding removal notices.
Pi Coding Agent with Qwen 35B Q2: Using Filesystem as External Memory and Enforcing Context Guards
A Reddit user built a stack around Pi coding agent with Qwen 35B Q2_K_XL quant that enforces guards — rejects edits over 100 lines, caps thinking blocks at 2000 chars, and monitors context at 65%/80% — treating the filesystem as the model's memory, not the context window.