Qwen 3.6 27B with MTP on V100 32GB: 54 t/s via llama.cpp Branch
A user on r/LocalLLaMA reports impressive results running Qwen 3.6 27B with Multi-Token Prediction (MTP) on a V100 32GB SXM module using a PCIe adapter. The setup uses am17an's MTP branch of llama.cpp and the corresponding MTP GGUF quant. Key specs: Q8_0 KV cache with 200k cache limit, running as a VS Code Copilot backend via llama-server.
Performance Numbers
- Without MTP: 29-30 tokens/second
- With MTP: 54-55 tokens/second (at 150W power limit)
- After 50k tokens context: drops to 40-45 t/s
Branch: am17an's MTP fork. Build and run were straightforward — 'pulled and built in one shot' with llama-server running without issues. The setup handles tool calls and sub-agents well, and delivered 'very insightful code reviews and refactors' despite the VRAM limitation (32GB).
This is particularly relevant for developers running LLMs on older datacenter hardware like V100s. MTP effectively doubles throughput for this model, demonstrating practical gains for coding assistant workloads.
📖 Read the full source: r/LocalLLaMA
👀 See Also
obsidian-mcp: Graph-Aware MCP Server for Claude with 25 Tools Targeting Large Vaults
obsidian-mcp is an MCP server exposing 25 tools (including get_note, traverse_graph, query_dataview, move_note, create_notes) that gives Claude graph-aware access to your Obsidian vault — avoiding context window death on 5k-note vaults. MIT, works with Claude Desktop, Claude Code, Cursor, Cline, Continue, Zed.
Developer shares 10+ MCP servers for AI agent settlement, reputation, and micropayments
A developer built BlindOracle on Claude Code with 100+ agents and created 10+ MCP servers for settlement, reputation, and micropayments. The architecture includes private commit-reveal forecasts, on-chain scoring, per-request micropayments, and verifiable agent attestation.
Nyx: Autonomous Testing Harness for AI Agents
Nyx is a blackbox testing harness that probes AI agents for failure modes like logic bugs, reasoning failures, and security vulnerabilities through multi-turn adaptive conversations. It tests in under 10 minutes what manual audits take hours to surface.
Microsoft BitNet: 1-bit LLM inference framework for CPU and GPU
Microsoft released BitNet, an inference framework for 1-bit LLMs that achieves 1.37x to 6.17x speedups on CPUs and reduces energy consumption by 55.4% to 82.2%. It can run a 100B parameter model on a single CPU at 5-7 tokens per second.