Swap Audit: check that a hosted open model is the one you are billed for

The public scorecard and the marketplace badge are both mostly taken. Artificial Analysis already publishes a reference-anchored accuracy index for each endpoint of popular open models. OpenRouter, the obvious marketplace buyer, has built its own continuous provider scoring and is reportedly working on logprob checks and a verified-provider tier. Moonshot gives away a vendor verifier for its own model. The detection methods are open research with free code, and that research says output-only statistical probes are weak against subtle quantization. A weekly scorecard built on black-box sampling would therefore be both behind the incumbent and technically fragile. The opening that remains is private and continuous: auditing the exact endpoints, account tiers, regions and parameter settings that a given enterprise actually uses. Public indexes test shared serverless endpoints occasionally and cannot see dedicated deployments, time-of-day swaps, or degradation that only appears at non-zero temperature or on the customer's own prompt mix. A fintech product would need to produce audit-grade evidence for model risk management and vendor contracts, such as SR 11-7 style records of ongoing monitoring, SLA breach evidence and alerts on drift. It would also need to combine probing with logprob replay against self-hosted reference weights and with TEE attestation where hosts offer it. That is a narrower compliance and procurement tool, not a ranking business, and its biggest risks are that attestation like Tinfoil's becomes standard or that OpenRouter's verified tier becomes the de facto badge.

• Artificial Analysis Endpoint Accuracy Index

  Launched in August 2026. It scores each serverless endpoint for open-weight models such as gpt-oss-120b, GLM-5.2 and DeepSeek V4 Pro against a self-hosted reference of the official weights and flags endpoints that fall outside parity. Search coverage says it tested about 20 gpt-oss-120b endpoints. This is close to the Host Scorecard idea, it is public, and it comes from the best-known benchmarking brand. Per-provider pages are at artificialanalysis.ai/models/gpt-oss-120b/providers.

• OpenRouter Exacto / Auto Exacto

  OpenRouter, the largest routing marketplace, does provider-quality auditing itself. It has scored every tool-call response since August 2025, runs controlled benchmarks such as TauBench and GPQA-Diamond against providers, and re-ranks providers about every 5 minutes. Its October 2025 post said it would pull providers that fall outside an acceptable range until they are fixed. A July 2026 third-party post says it is working on logprob-based detection and a verified-provider tier. This directly undercuts selling Router Certification to the biggest buyer.

• Moonshot K2 Vendor Verifier

  Moonshot's open-source tool, also packaged in EvalScope. It replays thousands of tool-call requests against third-party Kimi K2 hosts and compares the results with the official API. It shows that model labs themselves publish free verifiers for their own models.

• sunblaze-ucb llm-api-audit (Are You Getting What You Pay For?)

  Berkeley NeurIPS 2025 paper and open-source code that test text classifiers, MMD model-equality tests, benchmarks, logprobs and identity prompts for detecting model substitution. It finds that output-only statistical tests are query-hungry and miss subtle quantization, and that logprob checks are confounded by nondeterminism. Anyone can build on the code for free, and the findings challenge the core technical claim of this product.

• Tinfoil Modelwrap

  Takes the attestation route instead of probing. It uses TEEs and dm-verity to cryptographically prove which exact weights are served on each request. If this spreads, statistical probing is no longer needed for hosts that adopt it.

• DiFR: Inference Verification Despite Nondeterminism

  Research method that lets users verify open-weight model inference despite GPU nondeterminism. It also warns that temperature-zero spot checks can be gamed by providers that degrade only non-zero-temperature traffic. It is more groundwork that commoditizes the detection technique.

Checked 2026-10-28