LEETLABS
§ 00 / FIELD NOTE
BY Said Betmurzaev · Founder, Leetlabs

AI audit tools: auditing code, performance, and how you use AI

On 7 April 2026, Anthropic published a technical assessment of Claude Mythos Preview. Against a set of Firefox 147 vulnerabilities, the model produced 181 working shell exploits, where its predecessor managed 2. Across roughly 7,000 OSS-Fuzz entry points it reached complete control-flow hijack ten times. It surfaced a 27-year-old denial-of-service flaw in OpenBSD’s TCP SACK implementation, a 16-year-old integer overflow in FFmpeg’s H.264 codec, and CVE-2026-4747, a 17-year-old remote code execution bug in the FreeBSD NFS server granting unauthenticated root. Given 100 recent Linux kernel CVEs, it filtered 40 plausible candidates and built working privilege-escalation exploits for more than half, at under $2,000 each.

None of those bugs were hiding. They sat in the most-reviewed code on earth, through decades of human audit and continuous fuzzing. That is the state of AI auditing tools now, and it changes what a technical audit is worth and what it has to cover.

The evidence, with receipts

The downstream effect showed up in public data within weeks. Anthropic’s Project Glasswing partners, including Microsoft, Google, Apple and AWS, reported over 10,000 high- or critical-severity vulnerabilities found ahead of the model’s release. Epoch AI, tracking disclosures from 21 major vendors to filter out noise, found that June 2026 saw around 1,500 high- and critical-severity CVEs published, more than 3.5× the previous monthly record. They are careful to call the link correlative rather than proven, and that caveat is fair. The direction is not in doubt.

The offensive-testing side moved in parallel. XBOW’s autonomous agent reached #1 on HackerOne’s global leaderboard in April 2026 with more than 1,060 fully automated submissions against real production targets, a year after first topping the US board. Its recent disclosures include critical remote code execution flaws in Microsoft’s Bing image search rated 9.8 on CVSS, two of them carrying CVE-2026-32194 and CVE-2026-32191. Google has been running the same play defensively for longer: Big Sleep found the first AI-discovered zero-day in production software, AI-generated fuzz targets exposed 26 further vulnerabilities in open source, and DeepMind’s CodeMender upstreamed 72 security fixes in its first six months by writing the patches, not just the findings.

Attackers got the same capability on the same schedule. Google’s Threat Intelligence Group reported on 12 May 2026 that it had identified a threat actor deploying an AI-developed zero-day in the wild, a 2FA bypass in a widely used open-source system administration tool, alongside malware families making live model calls for obfuscation and a March 2026 supply-chain compromise that hit the LiteLLM and BerriAI repositories to harvest AI API secrets out of build environments. The asymmetry that used to protect slow, boring, unfashionable code is gone. If your codebase has a 15-year-old parsing bug, someone’s agent will reach it, and the only question is whose.

What a technical audit covers now

Three things, and they are increasingly the same review looked at from different angles.

The codebase and its security. Where the exploitable bugs are, what your dependency tree actually pulls in, how secrets are handled, and what an attacker reaches first. This is the part where the tooling above changed the economics: work that was priced as weeks of specialist time is now a matter of running the right agents against the right slices and verifying what comes back. The catch is that verification is now the expensive half, which is the subject of the next two sections.

Performance. Slow is a defect with a revenue number attached, and it is the easiest thing to measure honestly and the most commonly measured dishonestly. Real audits use measured runs, not vibes: page-level metrics under mobile emulation, backend traces on the paths that actually carry load, and database queries ranked by aggregate cost rather than by worst single case. We publish measured Lighthouse numbers on our own work for the same reason we insist on them in an audit. A number you can reproduce is worth more than a number that flatters you.

How you use AI. This one is newest and, for most teams, the largest unpriced risk. It covers what AI is doing inside your product and inside your engineering process: what data reaches a model and whether you are allowed to send it there, whether an agent has permissions broader than any human user, whether anyone can reconstruct why a model produced a given answer, and how much of your shipped code was generated by a tool nobody reviewed closely. That last question is not rhetorical. The data below explains why.

AI writes the vulnerabilities too

The same tools accelerating discovery are also the largest new source of the bugs being discovered, and this is the part vendors skip when they sell you a scanner.

Veracode’s Spring 2026 GenAI Code Security update, published 24 March 2026, ran 80 coding tasks across four languages against a longitudinal set of more than 150 models. Syntax correctness is above 95%. The security pass rate is 55%. Roughly 45% of generated code carried a known vulnerability, and the distribution is worse than the average suggests: Java passed 29% of the time, cross-site-scripting defences 15%, log injection 13%. Reasoning-focused models improved to around 70%, which is a real gain and still nowhere near shippable.

The enterprise picture has the same shape. Analysis summarised in the Cloud Security Alliance’s April 2026 research note found AI-assisted developers committing at three to four times the rate of their peers while introducing security findings at ten times the rate. Syntax errors fell 76%; privilege-escalation paths rose 322% and architectural design flaws 153%. That is the signature of a tool that has eliminated the errors a compiler catches and multiplied the ones only a reviewer with context catches. Add that around 20% of generated samples reference packages that do not exist, which attackers now register deliberately, and you have a class of risk that did not exist three years ago.

So both things are true at once: AI is the best code auditor available, and it is the fastest-growing producer of the flaws that need auditing. Anyone selling you only the first half is selling you the tool, not the outcome.

Running the tool is not the audit

The curl project is the cleanest case study in mistaking volume for value. Daniel Stenberg ended curl’s paid bug bounty at the end of January 2026 after the confirmed-vulnerability rate fell below 5%, meaning more than nineteen in twenty reports were burning a small volunteer security team’s time. The instructive part is what happened next: curl returned to HackerOne about a month later without cash rewards, and by April 2026 was receiving roughly twice the 2025 volume of reports with 15–16% confirmed valid. The models had genuinely improved. The bottleneck simply moved from generation to verification, which is where it remains.

That is the operational lesson for anyone buying an audit. An agent that emits 400 findings has not reduced your risk; it has handed you a triage problem with a plausible-sounding cover page. The value sits entirely in what happens after generation: reproducing each finding, discarding the ones that do not survive contact with your actual configuration, ranking what is left by exploitability in your deployment rather than by CVSS in the abstract, and producing a fix a human can review. Anthropic’s own Claude Code Security product, launched 22 February 2026, is built around exactly that constraint: severity ratings, confidence scores, automated rechecks to kill false positives, and no automatic code modification. The tool proposes. A person decides.

How we run it

We audit with our own tooling, because the off-the-shelf options solve the easy half of the problem.

Our audits run inside Vibehub, the multi-agent orchestration platform we built and use every day for our own engineering. Three things about it matter here. One agent with one context window cannot hold a real codebase, so Vibehub runs many isolated sessions in parallel, each owning a slice — authentication, data access, dependencies and supply chain, infrastructure and secrets, performance, and the AI and agent layer if you have one — each reporting into a single reviewed record. Agents reach real tools through the Model Context Protocol rather than guessing: version control history, dependency graphs, profilers, a real browser driven against your running staging environment. A finding that cannot be reproduced against the running system does not make the report. And anything hard to reverse pauses for human approval by design, so nothing touches your code without a person confirming it.

On top of that sit our own audit harnesses: the prompt architecture, the slice definitions, the verification pass that independently re-tests every candidate finding before it can be written up, and the scoring model that ranks by exploitability and cost in your deployment rather than by generic severity. That layer is ours, and it is what turns raw model output into something you can hand an engineer on Monday. It is also why we can tell you plainly when a finding is noise, since we are not paid by the finding.

The last part of the review is the one AI is worst at and the 2026 data makes urgent: how much of this codebase was generated, and does anyone left on the team understand it? Given a 322% rise in privilege-escalation paths in AI-assisted commits, “who reviewed this and could they explain it” is now a first-order question. It is the same instinct behind our technical due diligence work. The risk lives in what nobody can reconstruct.

Get a free audit

If you have shipped meaningfully with AI assistance in the last year, you have accumulated security and performance debt at a rate the industry data now quantifies, and you probably have not looked. We would rather show you than argue about it.

We will run a free initial audit of your codebase. You get a written report: verified findings ranked by real exploitability, measured performance numbers rather than estimates, a clear read on how much of the code is AI-generated and what that implies, and the specific things we would fix first. Reproduced, not guessed, and short enough to actually read. If nothing serious turns up, we will tell you that too, which is a useful thing to know for free.

Tell us what you want reviewed and we will scope it in a day. If you would rather start further upstream, with where AI belongs in your business at all, that is our AI readiness assessment.

Sources