Swap Watch: catch LLM hosts that quietly serve a cheaper model or a heavier quant

The free probe CLI and the public host leaderboard are both already taken. At least seven open-source fingerprinting tools on GitHub probe OpenAI-compatible endpoints for swaps and quantization. Artificial Analysis reportedly runs a provider accuracy index next to price that names Kimi K3, Moonshot keeps its own vendor verifier and leaderboard, and OpenRouter benchmarks providers and routes away from weak ones. What is still open is continuous, paid monitoring run from the buyer's own key and routing path. Today's leaderboards test through the evaluator's account, and a 2026 paper notes that hosts can send recognizable probes to the real model while serving everyone else a cheaper one. A service that hides its probes inside real traffic, alerts on changes over time and keeps timestamped evidence for refunds or contract disputes would be new, especially for relays, resellers and small GPU hosts that the big leaderboards don't cover. The catch is that the research shows text-only probes struggle to catch good 8-bit quantized models and hosts that adapt to being tested, so a quant-likelihood score from text alone may be noisy and easy to dispute unless the host exposes log-probabilities. The incumbents could also add a monitoring tier.

• Artificial Analysis Endpoint Accuracy Index

  Reportedly launched August 2026 (seen only in secondary sources). Ranks inference providers of the same open model (GLM-5.2, gpt-oss-120b, DeepSeek V4 Pro, with Kimi K3 next) by whether they match the real model, shown next to price. This is almost exactly the Host Grade idea, from a trusted incumbent.

• Kimi Vendor Verifier (Moonshot)

  Open-source toolkit from the model lab that checks third-party hosts of Kimi models against the official API. Moonshot keeps a public vendor leaderboard with it.

• OpenRouter benchmarks and Auto Exacto

  OpenRouter benchmarks providers continuously, lists each endpoint's quantization level and automatically routes away from underperforming providers. This covers much of the router-buyer use case inside the router itself.

• attest

  Open-source behavioral verification CLI for OpenAI-compatible endpoints. Claims to catch quiet model swaps, quantization degradation and token over-billing.

• llm-fingerprint-detector (ToseaAI)

  TypeScript library and CLI based on the paper One Token Is Enough. Compares one-word answer distributions against a trusted reference using Jensen-Shannon divergence to catch substitution by resellers.

• fpverify

  Behavioral fingerprinting with sequential statistical tests (stated false-positive rate of 1% or less), aimed at API relays and agent IDEs. The author reports detecting quantized downgrades.

• modelprint

  Browser-based fingerprinting of any OpenAI-compatible endpoint, with a logprob-geometry probe for quantized or substituted weights and community-written probes.

• model-fingerprint

  Zero-dependency Python black-box fingerprinting covering serving stack, tokenizer, capabilities and behavior distributions.

• llm-verify

  FastAPI service that runs 32 forensic prompts to tell whether an API reseller is serving a cheaper model than it claims.

• llm-api-audit (Berkeley, Cai et al.)

  Code for the paper Are You Getting What You Pay For? Auditing Model Substitution in LLM APIs (https://arxiv.org/abs/2504.04715v1). Finds that text-only tests struggle against subtle quantization and that log-probabilities work better.

• IMMACULATE

  Research framework that uses verifiable computation to audit substitution, quantization abuse and overbilling. A stronger guarantee than probing, if hosts adopt it.

Checked 2026-10-28