Security Benchmark: 10 LLMs Tested Against 211 Adversarial Probes
A security researcher conducted a systematic test of 10 different LLMs against 211 adversarial security probes to evaluate how they handle attacks in real-world scenarios.
Test Methodology
The researcher used a standardized setup with temperature 0 and identical API calls for every model. The test included 82 extraction probes (attempting to steal system prompts) and 109 injection probes (attempting to hijack model behavior). A honeypot system prompt loaded with fake PII, SSH keys, and API credentials was used as bait.
Key Findings
- Extraction resistance is mostly solved: Most models are decent at blocking "repeat your system prompt" type attacks. The average across all models is around 85%.
- Injection resistance is not solved: Average is 46.2%, meaning more than half of injection attacks succeed across the board.
- Universal failures: Every single model failed on delimiter attacks, distractor injection, and style injection. 0% resistance on those categories across all 10 models.
- Dead attack patterns: Every model resisted payload splitting and typo evasion at 100%.
Model-Specific Results
- Claude Opus: Scored 72.7% on injection resistance, the best of any model tested. Still means over 1 in 4 injection attacks work.
- GPT-5.4: Has perfect extraction and boundary scores but only 50% injection resistance.
- GPT-5.3 Codex: The model behind Codex CLI that runs code on your machine scored 34.5% on injection. 2 out of 3 injection attempts succeed.
- DeepSeek V3.2: Scored 17.4% on injection, basically no resistance.
- Qwen 3.5 API vs local: Almost identical extraction (81.6% vs 81.7%) but the local version is worse on injection (46.9% vs 29.8%) and much worse on boundary integrity (59.8% vs 44.6%). Running locally doesn't make it less capable at blocking extraction but does make it more vulnerable to injection.
Why Injection Matters
Extraction means someone steals your system prompt - bad, but recoverable. Injection means someone hijacks what your agent does. If your agent has tool access, file system access, or can make API calls, a successful injection can lead to data exfiltration, file deletion, or worse. Right now the best model in the world only blocks 73% of injection attempts.
Full methodology and results are public at agentseal.org/benchmark. The test prompt is also published so anyone can reproduce the results.
📖 Read the full source: r/LocalLLaMA
👀 See Also
SupraWall MCP Plugin Blocks Prompt Injection Attacks on Local AI Agents
SupraWall is an MCP plugin that intercepts and blocks sensitive data exfiltration attempts from AI agents, demonstrated in a red-team challenge where it prevented credential leaks via prompt injection attacks.
OpenClaw SOC Agent Integration for SIEM Home Lab Threat Hunting
A Reddit user shares their open-source SIEM setup called Red Threat Redemption on Debian 13, integrating Elasticsearch, Kibana, Wazuh, Zeek, and pfSense with Suricata, then adds an AI agent for automated threat correlation, hunting, and alert triage.
FlyTrap Attack Uses Adversarial Umbrellas to Compromise Camera-Based Autonomous Drones
UC Irvine researchers developed FlyTrap, a physical attack framework that uses painted umbrellas to exploit vulnerabilities in camera-based autonomous target tracking systems. The attack reduces tracking distances to dangerous levels, enabling drone capture, sensor attacks, or physical collisions.
Roblox cheat and AI tool caused Vercel platform outage
A Roblox cheat combined with an AI tool reportedly caused a complete platform outage for Vercel, generating significant discussion on Hacker News with 66 points and 24 comments.