Research on AI Agent Consistency: Key Findings and Practical Takeaways

Agent Consistency Research Findings

Research shared on r/ClaudeAI examines a critical issue in AI agent development: self-disagreement where agents give different answers on identical tasks. The study involved 3,000 experiments with consistent prompts and inputs across three major models.

Key Performance Metrics

• Consistent agents achieved 80–92% accuracy
• Inconsistent agents dropped to 25–60% accuracy
• That's a 32–55 point performance gap

Divergence Patterns

The research identified specific patterns in agent inconsistency:

• 69% of divergence occurs at the very first tool call
• Initial search queries are the critical failure point
• Correct initial calls lead to downstream convergence
• Incorrect initial calls cause runs to scatter

Practical Diagnostic Signals

Path length serves as a cheap diagnostic signal: agents taking 8 steps on a 3-step task are usually lost rather than being thorough.

Immediate Testing Recommendation

The practical takeaway is straightforward: run your agent 3–5 times in parallel. If trajectories agree, you can trust the output. If they scatter, don't ship that implementation.

Research Resources

The full paper is available at https://arxiv.org/abs/2602.11619 with a detailed writeup at https://amcortex.substack.com/p/run-your-agent-10-times-you-wont.