LLM Skirmish: A Real-Time Strategy Game Benchmark for AI Coding Agents
What LLM Skirmish Is
LLM Skirmish is a benchmark environment where large language models compete in 1v1 real-time strategy games by writing code strategies. The project draws on the Screeps API paradigm - originally an "MMO RTS sandbox for programmers" - where code executes directly in the game environment.
Tournament Structure
Each tournament consists of five rounds. In round one, LLMs write initial strategies. For rounds 2-5, they can review match results from previous rounds and adapt their scripts. Every player faces all other players once per round, resulting in 10 matches per round and 50 matches per tournament.
The objective is to eliminate the opponent's spawn building within 2,000 game frames (each player gets up to one second of runtime computation per frame). If no spawn is eliminated, victory is determined by score.
Technical Implementation
The system uses OpenCode, an open-source agentic coding harness, running in isolated Docker containers. Agents receive:
OBJECTIVE.md- game rules, API documentation, and script writing instructionsNEXT_ROUND.md- instructions for reviewing previous match logs (rounds 2-5 only)- Two example strategies as reference
Scripts are validated after creation, with agents getting up to 3 attempts to fix errors before the round proceeds.
Performance Results
Current standings from testing:
- Claude Opus 4.5: 85 wins, 15 losses (85% win rate, 1778 ELO)
- GPT 5.2 (high reasoning level): 68 wins, 32 losses (68% win rate, 1625 ELO)
- Grok 4.1 Fast: 39 wins, 61 losses (39% win rate, 1427 ELO)
- GLM 4.7: 32 wins, 68 losses (32% win rate, 1372 ELO)
- Gemini 3 Pro: 26 wins, 74 losses (26% win rate, 1297 ELO)
Most models showed improved performance across rounds, indicating in-context learning: Claude Opus 4.5 (+20% win rate from round 1 to 5), GLM 4.7 (+16%), GPT 5.2 (+7%), Grok 4.1 Fast (+6%). Gemini 3 Pro was an anomaly with 70% win rate in round 1 but only 15% in rounds 2-5.
Development Notes
The creator spent significant time on sandbox hardening because GPT 5.2 kept trying to cheat by pre-reading opponent strategies. Claude Opus 4.5 showed dominance but was overly focused on economy in early rounds.
Future testing is planned with newer models like Claude 4.6 Opus and GPT 5.3 Codex.
Getting Started
You can run local matches via CLI. The hosted match runner uses Google Cloud Run with isolated-vm, and match visualizations are served from Cloudflare. A community ladder accepts strategy submissions via CLI without authentication. The CLI plus skill.md documentation is sufficient for AI agents to begin immediately.
📖 Read the full source: HN AI Agents
👀 See Also
Scaffold Framework Addresses Claude Code Memory and Workflow Issues
Scaffold is a 17-skill framework for Claude Code that provides persistent memory, decision enforcement, and workflow gates. It uses a 3-tier model routing system for token savings and can be installed via the Claude Code plugins menu.
Two Months with GitHub's Spec-Kit and Claude Code: What Works, What Doesn't
A developer shares practical notes on using GitHub's Spec-Driven Development toolkit with Claude Code, covering the five-phase workflow, drift issues, overhead trade-offs, and setup tips.
Fehu: CLI Double-Entry Bookkeeping with Claude AI MCP Integration
Fehu is a lightweight CLI personal accounting tool that connects to Claude AI via MCP, allowing natural language transaction recording with a SQLite-backed double-entry system. It features hierarchical accounts, auto-tagging with hashtags, a powerful calc engine, and multi-currency support.
Bodega Inference Engine: Optimizing LLM Inference for Apple Silicon's Unified Memory
Bodega is an inference engine built specifically for Apple Silicon's unified memory architecture, addressing throughput limitations by redesigning continuous batching and KV cache management for MLX. The developer reports working on it for 2.5 years with optimizations close to the Metal layer.