TestThread: Open Source Testing Framework for AI Agents
What TestThread Does
TestThread is an open source testing framework designed specifically for AI agents, similar to how pytest works for traditional code. It addresses the problem of agents breaking silently in production with wrong outputs, hallucinations, or failed tool calls that only become apparent when downstream systems crash.
Key Features
- 4 match types including semantic matching where AI judges meaning rather than just text
- AI diagnosis on failures that explains why tests failed and suggests fixes
- Regression detection that flags when pass rates drop
- PII detection that automatically fails tests if agents leak sensitive data
- Trajectory assertions that test agent steps in addition to final outputs
- CI/CD GitHub Action that runs tests on every push
- Scheduled runs at hourly, daily, or weekly intervals
- Cost estimation per run
Installation and Setup
Install via package managers:
pip install testthreadnpm install testthreadThe framework includes a live API, dashboard, and Python/JavaScript SDKs. It's part of the Thread Suite alongside Iron-Thread, which validates outputs while TestThread tests behavior.
How It Works
You define what your agent should do, run it against your live endpoint, and receive pass/fail results with AI-powered explanations of failures. This approach helps catch issues before they impact production systems.
📖 Read the full source: r/LocalLLaMA
👀 See Also
Benchmark shows context engine reduces AI coding agent costs by 3x on SWE-bench
A benchmark of 4 coding agents using Claude Opus 4.5 on SWE-bench Verified shows a context engine achieved 73% pass rate at $0.67/task, while other agents cost up to $1.98/task for similar or lower performance.
GGUF Model Merging Script and Workflow for Qwen3.5-35B Variants
A Reddit user shared a Python script for merging GGUF model files with minimal loss, specifically combining HauhauCS's Qwen3.5-35B-A3B-Uncensored model with samuelcardillo's Claude-4.6-Opus-Reasoning-Distilled version. The script runs on Google Colab Free Tier and includes quantization support via llama-quantize.
Mnemos: an MCP server for persistent Claude Code memory
Mnemos is an open-source MCP server that gives Claude Code persistent memory across sessions, recording corrections as structured patterns and pushing ranked context at startup. Single 15 MB Go binary, no Docker or vector DB needed.
Open-source persistent memory system for Claude Code solves context loss between sessions
A developer built a file-based memory system for Claude Code that automatically captures project context without plugins or API keys. It uses conversation transcripts, an inbox file, and nightly cron jobs to maintain persistent memory across sessions.