A cross-customer KV cache leak on Workers AI makes shared inference an audit item

For the testing suite, the opening is real but narrow. Methods exist in public research (the Stanford timing audit, PromptPeek), and firms like Knostic and RedFox publish analysis and test methods, but I found no company running continuous canary and timing tests against hosted inference endpoints and issuing per-provider, per-model attestations that banks and healthcare buyers can attach to procurement or third-party-risk reviews. An entrant could win by focusing on cases one-off audits skip: very long contexts near cache-capacity limits (where the Workers AI leak appeared), servers that split processing into stages, and batching under load. It would also need to package results as compliance-ready evidence and partner with providers who want a third-party badge after Cloudflare's incident. Two caveats: outside testing can show a leak but cannot prove isolation, and long-context probes across many models are costly. The isolated cache tier is weak as a standalone business. vLLM's cache_salt makes tenant-partitioned prefix caching a free configuration option, OpenAI already limits caches to one organization, dedicated deployments are a standard upsell, and Tinfoil and Phala sell stronger hardware-attested isolation to the same regulated buyers. Also, splitting prefix-cache reuse would not have prevented the Cloudflare bug, which came from reading leftover cache pages rather than intentional cache sharing. That tier is best treated as a feature providers will add themselves.

• Cloudflare post-mortem: GLM-5.3 Workers AI KV cache exposure

  The triggering incident. 155 responses leaked other customers' content. The cause was a 1M-token context window with a 681K-token cache pool on the first-stage GPUs: rejected prompts still had cache built for them, and the second-stage node read leftover pages. A researcher found it, not a monitoring service.

• LMSYS / SGLang security update on request isolation

  The open-source fix, shipped in SGLang v0.5.20 to v0.5.22. Shows these bugs live in the shared serving engines that many hosts run, so the same class of bug can recur at other providers.

• Auditing Prompt Caching in Language Model APIs (Stanford, ICML 2025)

  Open, statistically rigorous method for detecting cross-user cache sharing from outside. Found global sharing at seven providers, including OpenAI. Effectively a free version of the test suite's core, but a one-off study, not a continuous service.

• PromptPeek / I Know What You Asked (NDSS 2025)

  Academic attack recovering other users' prompts through KV cache sharing in vLLM and SGLang. Research only, no product.

• Knostic

  AI security firm that published analysis of the Workers AI cross-tenant leak. A likely entrant into provider leak testing, but it currently publishes incident analysis rather than continuous testing with attestations for buyers.

• RedFox Security

  Pen-testing firm that publishes the canary method for multi-tenant AI leakage. Services-based testing of applications, not ongoing monitoring of inference providers.

• vLLM prefix cache isolation (cache_salt)

  Built-in per-request salt that splits prefix-cache reuse by tenant. Makes the isolated cache tier a free configuration option for any host, not a defensible product.

• OpenAI prompt caching (organization-scoped)

  OpenAI says caches are shared only within an organization, so per-tenant isolation is already a standard feature at the largest provider.

• Tinfoil

  YC X25 startup offering inference inside hardware secure enclaves with cryptographic attestation checked automatically by its SDK. Gives regulated buyers stronger, provable isolation than a cache partition.

• Phala Confidential AI Inference

  GPU secure-enclave inference that attaches a remote attestation report to each response. Competes for the same regulated buyers with hardware-backed isolation.

• Fireworks AI prompt caching and dedicated deployments

  Caching is on by default and cache hits stay within one replica. Dedicated deployments are the standard isolation upsell, already sold by inference hosts.

Checked 2026-10-27