PolyRange: Contamination-Resistant Offensive-AI Benchmark with LLM-Generated Targets
PolyRange v1.0 is an MIT-licensed, contamination-resistant offensive-AI benchmark for web security agents. Instead of static targets that leak into training corpora, each PolyRange deploy is freshly generated by the researcher's choice of LLM — satisfying the 'newly constructed tasks' criterion that OpenAI, Anthropic, and UK AISI have publicly called for.
What PolyRange addresses
The author, CEO of Aether AI, notes that existing cyber-AI benchmarks fall into two lanes that don't measure what labs need: CTF-style benchmarks (DVWA, NYU CTF Bench, CyberGym, AutoPenBench) use static targets that contaminate future models, and bug-bounty-style benchmarks (XBOW) have undefined defensive infrastructure. PolyRange bridges this gap with production-shape conditions including active defenders.
Technical specifications
- 84 WSTG-derived classes spanning all 12 OWASP testing-guide categories
- Two defense tiers approximating active-defender conditions
- Real backends: Postgres dialects, real PHP for LFI, real shell for command injection, real Jinja2 for SSTI
- Agent-submits-flag oracle convention for scoring
- Single-command eval CLI
- Self-hostable on Fly.io or any Docker host
Because targets are regenerated per run via LLM (researcher's choice of generator model), there is no static artifact for future models to ingest — addressing Anthropic's concern that 'this report will, itself, likely contribute to the problem.'
The benchmark uses a two-bucket entropy framing separating exploit-recall axes from cosmetic/realism axes, which the author believes is over-conflated in adjacent benchmark literature.
Funding for a full empirical paper (with publishable-N results) depends on partnership funding, but the framework is available now.
📖 Read the full source: r/LocalLLaMA
👀 See Also
5 Malicious OpenClaw Skills That Passed ClawScan + VirusTotal: Unit 42 Analysis
Unit 42 found 5 malicious OpenClaw skills that bypassed ClawScan and VirusTotal. Techniques included runtime referral swapping, SOL pooling for pump-and-dump, and 22MB README padding to hide an AMOS dropper.
Cybercriminals Are Pushing Back Against AI-Generated Slop on Underground Forums
New research shows low-level hackers and scammers are complaining about AI-generated posts on cybercrime forums, viewing them as low-quality noise that undermines community trust and social interaction.
BlindKey: Blind Credential Injection for AI Agents
BlindKey is a security tool that prevents AI agents from accessing plaintext API credentials by using encrypted vault tokens and a local proxy. Agents reference tokens like bk://stripe, and the proxy injects the real credential at request time.
Skill Analyzer Now Available on ClawHub with One-Command Install
The OpenClaw Skill Analyzer security scanner is now available on ClawHub with a single command install. The tool scans skill folders for malicious patterns like prompt injection and credential theft, and includes Docker sandbox support for safe execution.