Benchmark Results for Small Local and OpenRouter Models on Agentic Text-to-SQL Task
A developer has published benchmark results for small local and OpenRouter models on an agentic text-to-SQL task. The benchmark takes English queries like "Show order lines, revenue, units sold, revenue per unit (total revenue ÷ total units sold), average list price per product in the subcategory, gross profit, and margin percentage for each product subcategory" and converts them to SQL that is tested against database tables.
Benchmark Details
The agent can see query results and modify SQL to fix issues, with a limit on debugging rounds. The benchmark is deliberately short with 25 questions and runs in much less than 5 minutes for most models, making it practical for testing different configurations. It's designed to be tough enough to separate the best models from others.
Key Findings
- The best open models identified were kimi-k2.5, Qwen 3.5 397B-A17B, and Qwen 3.5 27B
- NVIDIA Nemotron-Cascade-2-30B-A3B outscores Qwen 3.5-35B-A3B and matches Codex 5.3
- Mimo v2 Flash was described as "a gem of a model"
Self-Hosted Option
The benchmark now includes the ability to run it yourself against your own server using the WASM version of Llama.cpp. The developer is seeking feedback on what to change for version 2 and wants to see scores others get with different configurations.
📖 Read the full source: r/LocalLLaMA
👀 See Also
Developer shares solution for Claude AI ignoring rules beyond 50-count threshold
A developer reports Claude Code started silently dropping rules once their shared rule set exceeded approximately 50 items, particularly during frontend-heavy tasks. They built a hook that scans prompts and loads only 2-3 relevant rules based on keyword matching.
mistral.rs Adds Support for Gemma 4 12B: Multimodal, Agentic, and MTP
mistral.rs now supports Gemma 4 12B with multimodal, agentic, and MTP integration. One-step install and run with web search, code execution, and built-in UI.
Scalpel v2.0: Codebase Scanner and AI Agent Orchestrator
Scalpel v2.0 is an open-source tool that scans codebases across 12 dimensions and assembles custom AI agent teams. It includes a pure bash scanner that runs without AI tokens and works with Claude Code, Codex, Gemini, Cursor, Windsurf, Aider, and OpenCode.
Tendril: A self-extending agent that builds and registers tools on the fly
Tendril is an agentic sandbox that autonomously discovers, builds, and registers tools. It starts with just three bootstrap tools and dynamically grows its capability registry without asking the user.