Weights Witness: proof your inference provider serves the model you validated

Your two ideas are in very different positions. The public provider fidelity ranking is mostly taken. In August 2026 Artificial Analysis launched an Endpoint Accuracy Index that scores providers of open models against a self-hosted reference. Moonshot keeps a Kimi vendor leaderboard, and OpenRouter has published provider-variance data and sells Exacto routing. A new entrant would compete with free, trusted, already-cited rankings. The only way to stand out is a stricter claim: statistical evidence that the weights are identical, rather than that the provider kept most of the benchmark accuracy. That works better as a content-marketing side project than as the business. The replay sidecar for bank and insurer model-risk teams still looks open commercially. The core statistics exist as free research: Model Equality Testing and the substitution-auditing follow-up. DiFR even ships an open-source vLLM integration. But I found no vendor packaging any of it as continuous production monitoring with SR 11-7 style evidence. Model-risk platforms like Superwise do drift and governance, not checks on which weights are served. Verifiable-inference startups like Inference Labs, Ambient and TOPLOC-based systems need the provider to cooperate and are aimed at crypto or decentralized compute, not banks. A new entrant could win by selling validation evidence rather than the algorithm. That means validator-ready reports, alert thresholds tied to the bank's model inventory, retained evidence for examiners, and integration with existing model-risk tools. Sampling real production prompts is also a real advantage, because the research warns that providers can spot synthetic audit probes and route them to the honest model. There are three risks. Buyers can build it themselves on the open-source code. A bank that cares this much may just self-host or demand contracts that pin model versions. And Artificial Analysis or an incumbent model-risk platform could add a monitoring tier quickly.

• Artificial Analysis Endpoint Accuracy Index

  Launched in August 2026. It scores each serverless provider of GLM-5.2, gpt-oss-120b and DeepSeek V4 Pro against a self-hosted reference of the official weights. It uses tool-calling, reasoning and long-context tests, and one provider kept only 52% of reference accuracy. It is free, already public and widely covered, so it largely takes the 'Provider fidelity report' idea. It measures capability retained, not statistical identity of the weights.

• OpenRouter Exacto

  OpenRouter measured how much tool-calling varies across Kimi K2 providers. It now sells ':exacto' endpoints that route only to high-fidelity providers. This is a router-level fix that a platform team might pick instead of an audit, though it gives no attestation of which weights were served.

• Moonshot Kimi Vendor Verifier (K2-Vendor-Verifier)

  An open-source tool from the model maker. It checks third-party Kimi providers against the official API on about 4,000 tool calls and checks that providers follow the decoding parameters, and Moonshot says it will keep a public vendor leaderboard. It covers only Kimi models.

• Model Equality Testing (Gao, Liang, Guestrin, ICLR 2025)

  An academic two-sample MMD test that answers 'is this API serving these weights'. In 2024 it found that 11 of 31 Llama endpoints differed from Meta's reference weights. It is the statistical core of the replay idea, with published code but no product.

• Auditing model substitution in LLM APIs (2025 follow-up)

  Research on detecting when providers swap in quantized or substitute models. It flags an adversarial weakness: providers can spot audit queries and send them to the real model. That argues for sampling real production prompts, as the sidecar does.

• DiFR (Token-DiFR / Activation-DiFR)

  An ICML 2026 method that checks provider outputs against a trusted reference despite nondeterminism. It detects 4-bit quantization with AUC above 0.999 within about 300 tokens and ships an open-source vLLM integration. Anyone could build a free version of the replay sidecar on it.

• Prime Intellect TOPLOC

  A locality-sensitive hash commitment on activations that detects changes to the model, prompt or precision. It needs the provider to cooperate, so it fits decentralized inference better than auditing an unwilling vendor.

• Inference Labs

  A startup that raised $6.3M in 2025 for zero-knowledge 'Proof of Inference'. It is crypto and agent focused, and I could not confirm any traction with bank model-risk teams.

• Ambient (enterprise verified inference)

  Sells verified-inference 'receipts' for audit and compliance. It is a provider-side attestation offering, not an independent check of a third-party host.

• Superwise model risk management

  A general AI model-risk monitoring platform covering drift, validation governance and SR 11-7 style reporting. It is a likely buyer-side incumbent that could add weight-fidelity checks, but I found no evidence it verifies served weights today.

Checked 2026-10-10