Bill Replay, now with a harness swap: test Argon in your own agent loop before switching

Replaying traffic on a new model is already covered by rightmodeler, Kitaru, Grepture, Doubleword and AgentX. Langfuse and Braintrust also cover it for teams willing to build datasets and evals. Routing is covered by Not Diamond, Martian, RouteLLM and LiteLLM Auto Routing. What none of them lead with is re-counting tokens under each model's tokenizer combined with cost per successful task. That matters because tokenizer differences are large: one pricing analysis found Claude models use about 2x the tokens of GPT-5.4 on tool definitions. The realistic opening is a cheap, local, open-source CLI for small teams spending $500 to $20k a month that reads raw provider logs and Helicone or Langfuse exports with no setup. It would print a migrate or don't-migrate table on true cost per task and write a fixed per-route LiteLLM or OpenRouter config. No config generator of that kind turned up, but it is a thin feature that incumbents could add quickly. The risks are that rightmodeler already holds nearly the same position, that Langfuse or OpenRouter could ship this as a feature, and that replay accuracy is limited when logs store summarized tool outputs and replays run without cache state. Small teams may also see it as a one-time job each time a model launches rather than something worth subscribing to.

• rightmodeler

  The closest match. A directory listing says it is an open-source CLI, listed in August 2026, that replays agent traces through cheaper candidate models. It reads traces from LangSmith, Langfuse, Braintrust and the OpenAI SDK. It scores agreement, cost savings and abstentions, and opens a pull request for approved swaps. Its GitHub repo was not found and the listing shows one upvote, so it is new and unproven.

• ZenML Kitaru

  Saves agent runs so you can replay them on a cheaper model and compare. Its vendor example reports failures and cost per session for gpt-5-mini against gpt-5 across 90 sessions.

• Doubleword migration kit

  A CLI that tests candidate open-weight models against a set of your real traces, comparing accuracy, latency and cost. It is aimed at moving to open models, and its license was not stated.

• Grepture

  Replays production LLM requests and reuses your existing LLM-as-judge scorers to show how quality changes per request.

• Langfuse Experiments

  Run a dataset against several models and compare cost, latency, scores and outputs side by side. Many target customers already have this. It needs a dataset and code setup, and it does not report cost per successful task as a headline number.

• Braintrust

  Combines evals, gateway routing and online scoring to check model swaps and routing for cost before shifting traffic.

• AgentX self-host model portability

  Estimates cost, latency and a quality rating for running a captured trace on another model. It works from the inputs only.

• Not Diamond

  Commercial router that can train custom routers on your data. It claims 30 to 90 percent cost savings. This competes with the routing-rule idea.

• Martian

  A model router that picks a model for each prompt in real time to cut cost. Its claims come only from a third-party listing and were not checked on Martian's own site.

• RouteLLM

  Open-source router framework from LMSYS that can be calibrated on your own query data and uses LiteLLM to call providers.

Checked 2026-10-09