Security Benchmark: 10 LLMs Tested Against 211 Adversarial Probes
A security researcher conducted a systematic test of 10 different LLMs against 211 adversarial security probes to evaluate how they handle attacks in real-world scenarios.
Test Methodology
The researcher used a standardized setup with temperature 0 and identical API calls for every model. The test included 82 extraction probes (attempting to steal system prompts) and 109 injection probes (attempting to hijack model behavior). A honeypot system prompt loaded with fake PII, SSH keys, and API credentials was used as bait.
Key Findings
- Extraction resistance is mostly solved: Most models are decent at blocking "repeat your system prompt" type attacks. The average across all models is around 85%.
- Injection resistance is not solved: Average is 46.2%, meaning more than half of injection attacks succeed across the board.
- Universal failures: Every single model failed on delimiter attacks, distractor injection, and style injection. 0% resistance on those categories across all 10 models.
- Dead attack patterns: Every model resisted payload splitting and typo evasion at 100%.
Model-Specific Results
- Claude Opus: Scored 72.7% on injection resistance, the best of any model tested. Still means over 1 in 4 injection attacks work.
- GPT-5.4: Has perfect extraction and boundary scores but only 50% injection resistance.
- GPT-5.3 Codex: The model behind Codex CLI that runs code on your machine scored 34.5% on injection. 2 out of 3 injection attempts succeed.
- DeepSeek V3.2: Scored 17.4% on injection, basically no resistance.
- Qwen 3.5 API vs local: Almost identical extraction (81.6% vs 81.7%) but the local version is worse on injection (46.9% vs 29.8%) and much worse on boundary integrity (59.8% vs 44.6%). Running locally doesn't make it less capable at blocking extraction but does make it more vulnerable to injection.
Why Injection Matters
Extraction means someone steals your system prompt - bad, but recoverable. Injection means someone hijacks what your agent does. If your agent has tool access, file system access, or can make API calls, a successful injection can lead to data exfiltration, file deletion, or worse. Right now the best model in the world only blocks 73% of injection attempts.
Full methodology and results are public at agentseal.org/benchmark. The test prompt is also published so anyone can reproduce the results.
📖 Read the full source: r/LocalLLaMA
👀 See Also
Audit Your Claude Code Permissions: A Practical Guide to Scoping Tool Access
A Reddit user audited their Claude Code setup and found over-permissioned tools that could edit .env files and production configs. Practical steps: audit global vs. per-project tools, check CLAUDE.md for secrets, and scope file access per directory.
Practical Security Practices for OpenClaw Agents
A Reddit post outlines specific security practices for OpenClaw users, including scheduled commands for updates and audits, managing agent access in shared channels, and securing API keys and skills.
Google TIG Reports First AI-Generated Zero-Day Exploit in the Wild
Google Threat Intelligence Group has identified a threat actor using a zero-day exploit believed to be developed with AI, marking the first observed offensive use of AI for zero-day vulnerability exploitation.
Meta Security Incident Caused by Rogue AI Agent Providing Inaccurate Technical Advice
A Meta engineer used an internal AI agent similar to OpenClaw to analyze a technical question, but the agent posted inaccurate advice publicly instead of privately, leading to a SEV1 security incident that temporarily exposed sensitive data.