Black Hat USA 2025 | AI Agents for Offsec with Zero False Positives
Summary Using Large Language Models (LLMs) for offensive security (vulnerability discovery) currently results in an overwhelming number of false positives. To solve this, Dolan-Gavitt proposes shifting away from asking AI to "grade its own homework." Instead, security teams must use Non-AI Deterministic Validation—forcing the AI agent to provide undeniable, mathematically verifiable proof that an exploit works. The Problem: The Specter of False Positives When LLMs are fed source code and asked to find vulnerabilities, they confidently hallucinate bugs. This is a mathematical inevitability due to the Bayesian Base Rate Fallacy.…