Benchmark Results: Claude Agent Swarm with Memory System Shows 30-43% Token Cost Savings
Memory System Benchmark for Claude Agent Swarms
A developer has been building a memory system called Stompy for nine months, evolving from file-based to SQLite to PostgreSQL. The goal was to minimize token usage when running Claude agent swarms. They conducted a benchmark comparing performance with and without the memory system.
Test Setup
The benchmark used a 40-point coding task requiring a full booking feature with backend, frontend, and tests. A 6-agent swarm was tested with three different Claude models as lead: Sonnet 4.6, Opus 4.6, and Haiku 4.5. All tests used the same codebase, same teammates, and same scoring system. Teammate agents always ran Opus regardless of the lead model.
Benchmark Results
- Sonnet 4.6 + memory: 40/40, $3.98, 6.5min, 2 turns
- Sonnet 4.6 no memory: 40/40, $7.04, 9.6min, 4 turns
- Opus 4.6 + memory: 40/40, $4.34, 9.6min, 29 turns
- Opus 4.6 no memory: 40/40, $7.65, 10.0min, 70 turns
- Haiku 4.5 + memory: 39/40, $4.95, 7.5min, 2 turns
- Haiku 4.5 no memory: 0/40, $3.97, 5.8min, 3 turns
Key Findings
Opus and Sonnet with memory saved about 43% on cost compared to running without memory. The developer notes that these models are smart enough to complete the task without memory, but they burn tokens on codebase exploration that the memory system eliminates.
The Haiku result was unexpected: it scored 0/40 without memory but 39/40 with memory. The developer observed that Haiku couldn't coordinate the Opus teammate agents without understanding the project structure, but became a competent lead with memory access.
Sonnet with memory was the best overall configuration, beating memoryless Opus on every metric at roughly half the cost. The takeaway is that making project knowledge available to the model matters more than using expensive models.
Technical Details
The memory system is called Stompy and is MCP/API/CLI-based, working with Claude Code. The benchmark setup is available on GitHub for others to use or improve. The developer notes this is n=1 per condition so far, with more runs planned.
📖 Read the full source: r/ClaudeAI
👀 See Also
oMLX introduces SSD KV caching for Apple Silicon, reducing OpenClaw response times from 30-90 seconds to 5 seconds
oMLX is a new backend that persists KV cache blocks to SSD in safetensors format, preventing cache invalidation when context shifts. This reduces OpenClaw response times from 30-90 seconds down to 5 seconds on subsequent turns.
Efficient Token Management with Open-Source MCP Servers: Pare
Pare MCP servers reduce token waste and enhance efficiency when AI coding agents use developer tools by providing structured output.
Building a Geological Clock with Claude Code: Single HTML + Three.js
A product designer built eona.earth, a geological clock that maps Earth's 4.5 billion years onto 12 hours, using Claude Code, Three.js, and custom WebGL shaders — all as a single HTML file with no build step.
AutoProber: AI-driven flying probe automation for hardware hacking
AutoProber is a hardware hacker's flying probe automation stack that enables AI agents to discover targets, map microscope images, perform safety-monitored CNC motion, review probes, and conduct controlled pin probing. It includes Python control code, a web dashboard, CAD files, and operates with GRBL CNC controllers, USB microscopes, and oscilloscope safety monitoring.