Managing AI Agent Failures: Retry Limits and Failure Budgets

This is a case study from a team running 6 AI agents in production, focusing on how their work queue handles failure modes beyond simple task distribution.

Key Failure Incident and Solution

One early incident involved an agent hitting a rate limit, failing, getting retried, hitting the limit again, and repeating this cycle 319 times. This burned hours of compute on a task that was never going to succeed.

The implemented fix was a 3-strike failure budget. After 3 failures, the task is marked as permanently failed instead of being re-queued.

Other Failure Modes Designed Around

• Agents claiming tasks but going silent (addressed with heartbeat timeouts)
• Agents reporting TASK_COMPLETE without actually completing the task (a self-report problem)
• Two agents grabbing the same task (addressed with optimistic locking)

The team notes that while the 3-strike rule seems obvious in retrospect, it was brutal to discover through experience.