Model substitution, phone-home and distillation tests turn model attestation into a procurement line

For the served-model audit, the methods are no longer novel. Several papers (DiFR, IMMACULATE, IRIS, Cai et al.) have published them, there are free open-source fingerprint checkers, and OpenRouter and Moonshot already check provider quality themselves. What I didn't find is a neutral third party selling continuous, contract-grade attestation to enterprises. That would be a signed audit trail, tied to SLAs, that holds up against providers who detect audit traffic and reroute it, and that covers closed models behind resellers as well as open weights. The opening is in procurement packaging: audit reports and evidence of breach that a buyer can put in a contract, not the detection algorithm. The risk is that routers and labs make self-verification a free feature, and that one-off checks get commoditized by free tools. For open-weight behaviour certification, incumbents like Protect AI/Palo Alto and HiddenLayer already own the approval gate, but they scan files and weights statically. Public writing on backdoored Qwen variants and the Kimi K3 sandbox escape says static scans miss trigger-based behaviour. I found no recognized certification scheme, and no vendor running per-release sandboxed tests for network egress and unprompted tool use for bank or government model-risk teams. That part looks emptier and is defensible if the entrant becomes the accepted standard. The danger is that a scanning incumbent adds a dynamic sandbox module, or that government evaluators such as the US and UK AI institutes publish free assessments of major releases.

• OpenRouter Exacto / Auto Exacto

  The biggest router now scores its own providers on tool-call telemetry and benchmarks about every 5 minutes, and routes traffic toward the better ones by default. This handles quality variance but isn't independent, because the router grades its own supply and doesn't publish the full dataset. Buyers who don't trust the router get no assurance from it.

• Kimi Vendor Verifier (Moonshot AI)

  An open-source tool from the model maker that checks whether third-party hosts of Kimi reproduce the official API's tool-calling behaviour. It sets an acceptance threshold. It shows model labs are starting to verify their own hosts, but it covers only one model family and the lab runs it.

• Verify LLM API

  A commercial endpoint checker launched on Product Hunt. It fingerprints a model from one-token output distributions and says it covers 161 model IDs. This is the closest direct match to a served-model audit, but it looks like a small launch-stage tool and its accuracy hasn't been checked independently.

• Tosea AI LLM API Fingerprint Checker / llm-fingerprint-detector

  An MIT-licensed open-source tool plus a free browser tool. They check OpenAI-compatible endpoints against a trusted reference to catch resellers swapping models or silently quantizing. Free and easy to copy, so they set a low price floor for basic checks.

• DiFR (MATS research, ICML 2026)

  An inference verification method with an open-source vLLM integration. It detects 4-bit quantization of open-weight models with high confidence after a few hundred tokens, and it was tested on public Llama 3.1 8B providers. This is research code, not a product, but anyone could build on it.

• IMMACULATE and IRIS (academic audit frameworks)

  Research frameworks that detect model substitution, quantization abuse and token overbilling (IMMACULATE), and routing dilution in LLM gateways (IRIS, arXiv 2607.20860). They show the statistical methods are well developed and free to use.

• Cai et al., Auditing Model Substitution in LLM APIs (UC Berkeley)

  A formal study comparing detection methods: classifiers, MMD equality testing, log-probabilities and TEEs. It finds that providers who spot audit traffic and reroute it are the main unsolved problem.

• Protect AI Guardian (Palo Alto Networks)

  Model supply-chain scanning built on the open-source ModelScan. It has policy gates that block non-compliant models from production and runs in air-gapped containers. It sits in the open-weight approval workflow, but it scans files for malicious code and doesn't test runtime behaviour such as phone-home calls.

• HiddenLayer Model Scanner

  Scans more than 35 model formats for malicious code, deserialization exploits and weight backdoors. It maps findings to OWASP, ATLAS and NIST, which helps with model-risk paperwork. This is the incumbent a bank model-risk team would buy first, but it's a static scanner, not a sandboxed behaviour test.

• Microsoft / NVIDIA open-weight provenance guidance

  Large vendors are pushing signed-artifact provenance checks for open-weight models. This may make integrity proofs a free platform feature, which would push an independent certifier toward behavioural testing.

Checked 2026-10-28