Running OmniCoder-9B locally with llama.cpp configuration details
Hardware and Model Setup
The setup uses mid-range hardware: AMD Ryzen 9 5900X CPU (12 threads used for inference), 62GB DDR4 RAM, NVIDIA RTX 3080 with 10GB VRAM, NVMe SSD, and Ubuntu 22.04 on a remote server.
The model is OmniCoder-9B, based on Qwen3.5-9B, fine-tuned on 425k+ coding agent trajectories by Tesslate. It uses Q6_K quantization (6.85GB file size) with 128K token context window, sourced from HuggingFace.
llama.cpp Configuration
The model runs via llama.cpp server with these specific flags:
llama-server \ --model /home/openclaw/models/omnicoder-9b/omnicoder-9b-q6_k.gguf \ --host 0.0.0.0 --port 8080 \ --ctx-size 131072 \ --n-gpu-layers 99 \ --cache-type-k q8_0 \ --cache-type-v q4_0 \ --threads 12 \ --batch-size 128 \ --flash-attn on \ --temp 0.4 \ --top-k 20 \ --top-p 0.95 \ --jinja \ --reasoning-budget 0
Key parameters explained:
--ctx-size 131072: 128K context window for large codebases--n-gpu-layers 99: Offload all layers to GPU--cache-type-k q8_0 --cache-type-v q4_0: Compressed KV cache to fit 128K context in 10GB VRAM--threads 12: Match physical cores (not hyperthreads)--flash-attn on: Faster attention computation--reasoning-budget 0: Disables chain-of-thought output in the reasoning_content field, making the model output code directly
Performance and Testing
Performance metrics: prompt evaluation at ~300 tokens/s, generation at ~80-90 tokens/s, VRAM usage ~8.5GB/10GB, latency 1-5 seconds for typical coding tasks.
The testing was conducted by Agent Zero, an autonomous agent framework using GLM-5 as its main brain. Agent Zero discovered the --reasoning-budget 0 flag, SSH'd into the remote server, updated the systemd service, created benchmark scripts from scratch, ran multiple benchmarks (HumanEval base, HumanEval Pro, MBPP, MultiPL-E), and iterated on prompt engineering.
Benchmark Results
Benchmark results compared to official claims:
- HumanEval base: Official 92.7%, Run 1: 100%, Run 2: 95%, Run 3: 95%, Average: 96.7%
- HumanEval Pro: Official 70.1%, Run 1: 70%, Average: 70%
The average HumanEval base score of 96.7% exceeds the official 92.7%, while HumanEval Pro matches exactly at 70%.
📖 Read the full source: r/LocalLLaMA
👀 See Also
Optimizing Qwen3.5-9B on RTX 3070 Mobile with ik_llama.cpp: Config Tweaks and Benchmarks
A developer shares optimization findings for running Qwen3.5-9B Q4_K_M on an RTX 3070 Mobile 8GB GPU using ik_llama.cpp, achieving ~50 tokens/second generation speed and significant prompt evaluation improvements through configuration adjustments.
OpenClaw 2026.3.7 breaks Kimi tool calls, downgrade to 2026.3.2 fixes regression
OpenClaw version 2026.3.7 has a regression where the Kimi API provider outputs raw <function_calls> XML instead of executing tools. The solution is to downgrade to version 2026.3.2 and restore a compatible config file.
Three-layer memory architecture for persistent OpenClaw agent context
A developer built a 3-layer memory system on top of OpenClaw's infrastructure to prevent agents from starting each session without context. The architecture includes L1 workspace files injected every turn, L2 semantic memory search, and L3 reference documents opened on demand.
Four aarch64-specific failure modes when running vLLM on Blackwell GB10 with CUDA 13.0
A developer encountered four specific failure modes when setting up vLLM v0.7.1 with DeepSeek-R1-32B on a Blackwell GB10 system running aarch64 architecture with CUDA 13.0, including ABI mismatches and missing dependencies.