Leak Probe: canary tests for cross-request cache leaks at hosted model APIs

Tool-call conformance and provider-quality scoreboards are already taken. Moonshot gives away its Kimi Vendor Verifier, Artificial Analysis publishes a per-provider accuracy index against a reference setup across 9 to 20 hosts per model, and OpenRouter benchmarks hosts and routes around bad ones automatically. A new entrant would struggle to sell that as a standalone product. The open space is cache isolation. Research shows real cross-tenant prompt-cache leakage, including global cache sharing at 7 API providers in the Stanford audit and prompt-reconstruction attacks on shared KV caches in 2026 papers. Yet I found no packaged CLI, hosted monitor or public scoreboard that tests hosts for whether their caches are isolated per account, per org or globally. A leakprobe CLI plus a nightly isolation scoreboard across open-model hosts, possibly with attestation reports for compliance reviews, looks unclaimed. One technical caution: published leaks are detected through timing (time to first token), not by cached text coming back in a reply, so a canary-echo design alone will likely report nothing. The product should use a timing-based statistical test like the Stanford method. Willingness to pay is also unproven. It may depend on making cache isolation a compliance checkbox rather than relying on developer curiosity. The tool-call checks are best offered as an extra that wraps existing suites rather than as the lead product.

• Kimi Vendor Verifier (formerly K2-Vendor-Verifier), Moonshot AI

  Free, open-source tool (reportedly MIT) from Moonshot that checks how closely third-party hosts match the official Kimi API. Its ToolCall test scores a host's tool-call decisions against the official API as an F1 score. This directly covers the Tool-Call Conformance Pack for Kimi, and the model maker gives it away.

• Artificial Analysis Endpoint Accuracy Index

  Launched August 2026. Scores every serverless endpoint for an open-weights model against a self-hosted copy running the official weights and the lab's recommended serving setup. Tool calling is weighted as one third of the score, alongside reasoning and long-context recall. Covers GLM-5.2 (15 providers), gpt-oss-120b (20) and DeepSeek V4 Pro (9), with Kimi K3 next. This is essentially the public scoreboard part of Provider Watch, minus isolation testing.

• OpenRouter Exacto / Auto Exacto

  The model router benchmarks providers all the time, scores them on tool-call error rates and similar signals, and routes traffic to better providers by default. It also publishes per-provider benchmark pages. This takes away much of the reason for router users to buy separate tool-call regression alerts.

• Auditing Prompt Caching in Language Model APIs (Gu et al., Stanford, ICML 2025)

  Academic method, not a product. It uses timing statistics to detect whether a prompt cache is shared per user, per organization or globally, and found global sharing at 7 API providers. This is the closest prior art to leakprobe, and it suggests timing, not text coming back in replies, is what actually detects leaks. I could not confirm a released code repository.

• promptfoo LM Security DB entries on prefix-cache leakage

  promptfoo, an open-source LLM red-teaming tool, lists cross-tenant prefix-cache timing attacks (CacheSolidarity, OptiLeak) in its vulnerability database. That shows the security tooling community knows about this risk, though I found no dedicated scanner for it in promptfoo.

Checked 2026-10-27