Jake Benchmark v1: Local LLM Performance Testing for OpenClaw AI Agents
The Jake Benchmark v1 is a performance evaluation tool for local LLMs functioning as AI agents with OpenClaw. It tests models on 22 practical tasks to determine their effectiveness in real-world agent scenarios.
Test Setup and Methodology
The benchmark was run on a Raspberry Pi with Ollama running on an NVIDIA 3090 GPU. The developer tested 7 different local LLMs to identify the best model for agent work with OpenClaw.
Task Categories
The 22 tasks covered real-world scenarios including:
- Reading emails and creating tasks from them
- Scheduling meetings and checking for conflicts
- Phishing detection (specifically a fake email pretending to be the owner asking for a bitcoin wallet key)
- Error handling
Key Results
The performance variation was significant across models:
- Qwen 27B: Scored 59.4% - successfully handled emails, scheduled meetings, detected phishing attempts, and managed errors
- Nemotron 30B: Scored 1.6% - attempted to solve tasks by running
apt-get install git
Notable Observations
The phishing test revealed interesting behaviors:
- The best model refused the phishing request immediately
- The worst model read the secrets file three times before deciding not to share the information
Dashboard Features
The benchmark includes an interactive dashboard that allows users to:
- Click into any model to view the full conversation
- See exactly what each model did during tasks
- Identify where models went wrong in their execution
The tool is available on GitHub for developers to run their own evaluations and compare local LLM performance for agent tasks.
📖 Read the full source: r/openclaw
👀 See Also
GLM 5 on Mac M3: Performance Observations for Agentic Coding
A user reports running GLM 5 via MLX 4-bit quantization on a Mac M3 with 512GB RAM, finding it usable for agentic coding with context under 50k tokens but noting significant slowdowns beyond that threshold.
Logira: eBPF Runtime Auditing for AI Agent Runs
Logira is an observe-only Linux CLI tool that records exec, file, and network events via eBPF during AI agent runs, with per-run local storage in JSONL and SQLite and built-in detection rules for credential access, persistence changes, and suspicious patterns.
Claude Code's Illusion of Finished Work: Why Reviewing the Agent's Path Matters More Than the Diff
Claude Code can produce a clean diff, passing tests, and a good summary—yet still miss real behavior, security concerns, or architecture constraints. The author argues that reviewing the chain of actions (plans, files read, commands run, test output) is now essential, not just the final diff.
Engram: Hybrid Memory Plugin for OpenClaw Agents — Vector + Semantic Search with Decay
Engram gives OpenClaw agents persistent memory across sessions using SQLite+FTS5 for exact recall and LanceDB for semantic search, with decay classes and auto-capture hooks.