Maintainers don’t need more AI bug reports. They need fixes.
- Source
- github.blog
- Date

Context
GitHub and Microsoft ran a pilot with six open source projects to test whether AI could hand maintainers security fixes rather than more findings. The problem they set out to address is that AI has made finding possible vulnerabilities cheap, while fixing them still takes human judgment and falls to each project's maintainers, many of whom are already receiving AI-generated reports that are duplicates or not real vulnerabilities at all. The pilot's two rules were to send only findings that matter and never to send a finding nobody had verified.
The work ran in stages that each narrowed the pile. Codename MDASH, Microsoft Security's agentic scanning system, scanned each project with Microsoft's MAI Cyber model, flagged thousands of possible issues and suggested fixes. A triage from the GitHub Security Lab, built on Copilot, then removed duplicates, checked validity and kept the high-impact bugs. Seven researchers from Microsoft's FORGE Lab checked every remaining finding by hand. What reached maintainers was 99 reports, sent through GitHub's private vulnerability reporting so the bugs were not made public before a fix, each with a proof of concept showing the bug and a suggested patch. As of November 15, maintainers had fixed 55, and 14 of those fixes were already in a release.
The projects were AutoGPT, the Caddy web server, the Express web framework for Node.js, FastMCP, ImageMagick and the Java networking framework Netty. Maintainers describe the saving as time spent understanding a report. Caddy's team fixed all 22 of its reports in a week or less. ImageMagick, run by two people, also received 22, and one fix took 15 minutes where, a maintainer said, an advisory normally takes 30 minutes to an hour to read, understand and fix. A Netty maintainer put the shift plainly: finding vulnerabilities is no longer the bottleneck, fixing them is.
The pilot does not show that this can run at scale. GitHub says it worked because people did careful work at every step and that this work does not scale yet: the hand review took about two days per project. Six projects is a small sample, and GitHub itself calls the results early lessons. What it does establish is a standard GitHub now asks anyone sending AI-found vulnerabilities to meet: check each finding against the project's threat model, report it privately, include a fix and let maintainers set the pace. GitHub plans to offer checked AI scans to every alumni project of its Secure Open Source Fund, the program the six projects came from, and says it is working with Microsoft to scale the researcher review.
- Of thousands of possible issues flagged across six projects, only 99 private reports reached maintainers after AI triage and hand review; as of November 15, 55 had been fixed and 14 of those fixes had shipped in a release.
- The human verification step is what GitHub admits does not scale yet: seven Microsoft FORGE Lab researchers spent about two days per project checking findings by hand before anything was sent.
- GitHub's proposed standard for AI-generated vulnerability reports is to check each finding against what the project already knows plus its threat model and security policy, report privately, include a fix, and let maintainers set the pace.
- The pilot surfaced remaining gaps: maintainers need to run CI/CD tests on private forks to verify fixes, and need access to cyber-capable models to verify a proof of concept before and after a fix.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Checking sign-in…
Loading comments…

