Quiet Zones: seeded-bug tests that show what an AI vulnerability scanner never looked at

No product was found that plants known bugs into a buyer's own repositories to run a neutral multi-vendor bake-off, or that keeps rotating seeded canaries in CI to detect silent regressions after a vendor update. The method is widely recommended (Cycode and Endor Labs tell buyers to do it themselves) and has been hand-built once in an open-source project (dormouse), while public benchmarks like OWASP Benchmark, OpenSSF and RealVuln use public code and the head-to-head numbers buyers see are vendor-authored. A new entrant can win on neutrality, on running inside the bank's environment to satisfy code-custody concerns, on reporting which files and paths an AI agent never examined (nobody surfaces this), and on audit-ready evidence banks can show regulators and model-risk teams. The main risks are that realistic seeded bugs, especially authorization and business-logic flaws, are technically hard to generate (research shows naive injection is easy to game or unrepresentative), that banks may resist any third-party handling of source code, and that scanner vendors or AppSec consultancies could add seeded-evaluation features, though vendor ownership would undercut the neutrality that is the core value.

• diffplug/dormouse PR #1064 (seeded canaries for CI security audit)

  Open-source, project-specific CI job that plants known vulnerabilities in a throwaway repo copy, runs an AI security audit, and reports per-domain recall plus findings on untouched files. Essentially the coverage canary idea built by hand for one repo; not a product.

• OWASP Benchmark

  Free, runnable Java/Python test suite with known vulnerabilities used to score scanner accuracy. Fixed synthetic code, not the buyer's own repositories.

• sast-bench

  Open-source benchmark that scores SAST tools on annotated vulnerable regions in agentic codebases. Public corpus, not seeded into customer code.

• RealVuln benchmark

  2026 research benchmark comparing rule-based, general LLM and security-specialized scanners on real code with hand-labeled findings and false-positive traps. Academic, public code only.

• DryRun Security SAST Accuracy Report

  Vendor-run comparison that seeded 26 vulnerabilities into intentionally vulnerable apps and scored tools. Shows the method but is self-interested and not run on customer code.

• Corgea best SAST tools comparison

  Vendor-authored July 2026 recall/precision comparison of SAST tools, ranking itself first. Example of the non-neutral benchmarks buyers currently rely on.

• Cycode AI SAST guide

  Vendor guidance telling buyers to run evaluations on their own repos seeded with known vulnerabilities. Validates the need but leaves the work to the buyer.

• AISLE AI SAST proof of value

  Vendor-run proof of value on the customer's codebase comparing against existing SAST. Single-vendor and without seeded ground truth.

• Automated Customized Bug-Benchmark Generation (research)

  Academic work on injecting bugs into programs to measure static analyzer recall, finding LAVA-style injection yields near-zero static recall. Shows the injection technique exists in research and is hard to make realistic.

• Beyond Single Bugs: LLM multi-vulnerability injection benchmark

  Research pipeline that uses an LLM to inject plausible CWEs into real files to test detectors. Close to the core technique but academic code, not a buyer-facing product.

Checked 2026-10-22