“Hack To The Future: Owning AI-Powered Tools With Old School Vulns” by Nils Amiet and Nathan Hamiel at Black Hat USA 2025:
Core Thesis
The integration of generative AI into developer productivity tools (like AI code reviewers and data analytics assistants) is creating massive new attack surfaces. While the underlying Large Language Models (LLMs) are not being “hacked,” the applications wrapping them are poorly designed, overly permissive, and riddled with classic, “old-school” vulnerabilities like Remote Code Execution (RCE), Prompt Injection, and Insecure Direct Object Reference (IDOR).
Because these AI tools are deeply integrated into development environments and databases, the impact of these classic vulnerabilities is now catastrophic.
Key Case Studies & Findings
The researchers analyzed over a dozen AI-powered developer tools and found vulnerabilities in all of them. They highlighted a few major examples:
- CodeRabbit (The 1 Million Repo Compromise): The most popular AI code review app on GitHub. The researchers found an RCE vulnerability by exploiting how CodeRabbit integrated with RuboCop (a Ruby static analyzer). By submitting a malicious pull request containing a crafted .rubocop.yml file, they executed arbitrary code. This allowed them to extract CodeRabbit’s GitHub App Private Key, which effectively granted them write access to over 1 million public and private GitHub repositories that had the app installed.
- Qodo Merge (formerly PR-Agent): An AI code review tool. In the open-source version, researchers found ways to escalate privileges (e.g., in GitLab) and overwrite critical configuration settings via prompt injection. In the SaaS version (Qodo Merge Pro), they manipulated the configuration to dump environment variables, which leaked a highly privileged, “Administrator Access” AWS secret key.
- Dot (AI Data Analytics): A tool that allows users to “talk” to their databases. It contained an unauthenticated API endpoint that allowed RCE. Exploiting this leaked the environment variables, which included the master credentials to the PostgreSQL database containing all customer database credentials.
- Sourcery: An AI code review tool that suffered from an IDOR (Insecure Direct Object Reference) API vulnerability. This allowed the researchers to access internal Sentry error logs of other customers, which contained leaked GitHub access tokens and private repository data.
Why is this happening?
The speakers identified several dangerous trends causing these vulnerabilities:
- Blind Execution of Input: AI apps are taking user/attacker data, mixing it with AI prompts, and blindly executing the output (e.g., executing AI-generated Python or SQL code without sandboxing).
- Excessive Permissions: To make tools “plug and play,” developers are granting AI apps sweeping permissions (like GitHub Repo Read/Write or AWS Admin access) without understanding the risks.
- Security is Ignored for Speed: Developers are chasing the promise of “10x productivity” and adopting tools without putting them through standard Application Security (AppSec) reviews.
- Generative AI is a New Execution Environment: AI introduces vast, undocumented protocols into applications. Developers do not know exactly what code will execute at runtime because the AI acts as a “black box.”
Recommendations & Takeaways
The speakers stressed that AI Security is largely just traditional Application and Product Security. Organizations must stop treating AI as magic and apply standard security principles:
- Never Trust the Output: If an attacker can get data into your generative AI tool (via a prompt, a pull request, or a config file), you cannot trust the code or data the AI outputs. Design your systems expecting the AI to output malicious commands.
- Apply Least Privilege: Scrutinize the permissions you grant to third-party AI GitHub apps and database tools. Never grant them admin-level access.
- Refrain, Restrict, Trap: Refrain from using AI for safety-critical tasks, restrict its access and permissions, and put “traps” (guardrails) around its inputs and outputs.
- Treat AI like an “Intoxicated Robot”: Do not treat AI agents as highly capable, super-intelligent systems. Treat them as “lazy, intoxicated robots.” This mindset forces developers to build better guardrails, validation checks, and sanitization around the AI’s actions.