Benchmark Results for Small Local and OpenRouter Models on Agentic Text-to-SQL Task
A developer has published benchmark results for small local and OpenRouter models on an agentic text-to-SQL task. The benchmark takes English queries like "Show order lines, revenue, units sold, revenue per unit (total revenue ÷ total units sold), average list price per product in the subcategory, gross profit, and margin percentage for each product subcategory" and converts them to SQL that is tested against database tables.
Benchmark Details
The agent can see query results and modify SQL to fix issues, with a limit on debugging rounds. The benchmark is deliberately short with 25 questions and runs in much less than 5 minutes for most models, making it practical for testing different configurations. It's designed to be tough enough to separate the best models from others.
Key Findings
- The best open models identified were kimi-k2.5, Qwen 3.5 397B-A17B, and Qwen 3.5 27B
- NVIDIA Nemotron-Cascade-2-30B-A3B outscores Qwen 3.5-35B-A3B and matches Codex 5.3
- Mimo v2 Flash was described as "a gem of a model"
Self-Hosted Option
The benchmark now includes the ability to run it yourself against your own server using the WASM version of Llama.cpp. The developer is seeking feedback on what to change for version 2 and wants to see scores others get with different configurations.
📖 Read the full source: r/LocalLLaMA
👀 See Also
MemRosetta adds persistent memory to Claude Code with one command setup
MemRosetta v0.2.4 provides Claude Code with cross-session memory via a single npm install command. The tool includes a MCP server with 6 memory tools, automatic session capture, and local SQLite storage that can be shared with Cursor.
Bodega Inference Engine: Optimizing LLM Inference for Apple Silicon's Unified Memory
Bodega is an inference engine built specifically for Apple Silicon's unified memory architecture, addressing throughput limitations by redesigning continuous batching and KV cache management for MLX. The developer reports working on it for 2.5 years with optimizations close to the Metal layer.
Keyoku Plugin Replaces OpenClaw's Static Heartbeat with Memory-Driven Autonomy
Keyoku is a free OpenClaw plugin that changes the agent's heartbeat from reading a static HEARTBEAT.md file to scanning the agent's actual memory store for stalled work, dropped commitments, conflicting information, and quiet relationships. It uses a local Go engine with SQLite + HNSW and offers three autonomy levels: observe, suggest, and act.
Superglue CLI: Let AI Agents Execute API Calls Without Pre-Built Tools
Superglue CLI provides a skill that teaches AI coding agents how to use its commands, handle authentication, build tools, and debug failures. Instead of creating pre-built tools for every API integration, agents can read API specs at runtime and plan multi-step calls.