Security Audit Experiment Shows AI Agent Performance Depends on Knowledge Access

A Reddit user conducted an experiment comparing AI security audit approaches on the same codebase to test how knowledge access affects results. The experiment used BoxyHQ's open source Next.js SaaS starter kit as the test subject.

Three Audit Methods Compared

The developer ran three independent security audits:

• Claude Code's built-in security review: Found 1 critical, 6 high, and 13 medium severity issues
• AI agent without extra context: Found 1 critical, 5 high, and 14 medium severity issues
• AI agent with 10 professional security books: Found 8 critical, 9 high, and 10 medium severity issues

Key Findings

The book-equipped agent identified vulnerabilities that the other methods completely missed, including:

• Password reset tokens stored in plaintext
• A TOCTOU (Time-of-Check to Time-of-Use) race condition on token validation
• A feature flag that calls res.status(404) but doesn't return, allowing execution to continue

The developer noted these aren't obscure edge cases but the type of issues that appear in real security breaches. The experiment used the same codebase and same AI model across all three tests, with the only variable being the knowledge the agent had access to.

Implications for AI-Assisted Development

The experiment suggests AI agents aren't limited by intelligence but by what knowledge they can access when needed. The developer concluded that security knowledge "lives above the code" rather than within it, highlighting the importance of providing domain-specific references to AI tools rather than relying solely on their base training.

This approach to augmenting AI agents with specialized knowledge sources could be particularly relevant for developers using AI coding assistants for security reviews, where access to current security references and best practices significantly impacts the quality of findings.