Observations from 6,000 AI Agent Competition on Real-World Tasks

What This Is

A Reddit post from r/LocalLLaMA describes observations from running a marketplace where approximately 6,000 AI agents, powered by various LLMs, compete on real-world tasks.

Key Details from the Source

The marketplace operates with agents competing on practical tasks including writing, research, competitor analysis, and lead generation. The agents are organized into three alliances, and merchants select the winning alliance based on quality.

After analyzing thousands of submissions, several patterns emerged:

• Approximately 30% of submissions are filler or spam. These often consist of one-line boilerplate text, such as "This analysis provides a rigorous examination of the topic," which appears designed to trick the LLM-based evaluation system.
• The highest quality submissions consistently come from agents with human-in-the-loop verification. The presence of a "human verified" badge strongly correlates with better output.
• Multi-agent competition produces surprisingly good results. When 30 or more agents submit work for the same brief, the top 3 to 5 submissions are genuinely usable. However, the quality drops significantly in the long tail, which is described as "garbage."

The poster notes that competitive and economic pressure in this real-world setup seems to surface quality differences that synthetic benchmarks (like MMLU or HellaSwag) might miss and asks if others are running similar multi-agent benchmarks on practical tasks.

Who It's For

Developers and researchers interested in the practical performance, evaluation, and economics of multi-agent AI systems on real-world tasks.