Task Bill: cost per passed task for coding-agent setups, measured on your own repo

Measuring cost per passed task on your own repo is already covered by SelfBench, RepoGauge, the AWS sample framework, agent-bench, the agent-eval skill and in-house builds like Fullscript's. Repeated runs to measure consistency are also already offered. A newcomer cannot win on the core measurement. What is left is trust and packaging for regulated buyers. That means proper statistics with confidence intervals from repeated runs, a fixed and signed report an engineering manager can take to finance or audit, and fully self-hosted runs inside the customer's network for fintechs that cannot send code out. It also means costing the whole job: retries, failed attempts and reviewer time spent fixing passed but low-quality patches, not just API spend. A recurring version that reruns on fresh tickets whenever new models or harness versions ship could justify a subscription, where the free tools are one-off scripts. Prompt Diet's analysis side is largely free through VS Code's Cache Explorer and observability vendors like Datadog and Grafana. I found no tool that proposes prompt and tool-definition cuts and then proves by replay that pass rate holds. That loop is open, but it is stronger as a feature inside a Task Bill-style replay product than as a standalone linter.

• SelfBench

  Builds tasks from a repo's merged pull requests and reports accuracy and cost per task for each agent or model. It is nearly Task Bill's core pitch.

• RepoGauge

  Builds a benchmark from your own commits and reports pass rate and cost per solved bug across coding agents.

• agent-bench (TayHCode)

  Open source. It turns past bug fixes into tasks with hidden regression tests, runs agents on fresh checkouts with deterministic grading, and runs locally without uploading anything.

• AWS sample-agent-cost-bench

  AWS open-source framework that measures cost, quality and duration across any agent CLI and any model on real repos, with your own verification tests. It comes with an AWS blog post on benchmarking cost and quality on your own code.

• agentic-coding-harness-benchmarks

  Compares models across harnesses such as Claude Code and pi on Bedrock or self-hosted setups and plots the cost-versus-quality trade-off. It covers harness and model pairs directly.

• agent-eval skill (ECC)

  Widely copied community skill that compares Claude Code, Aider, Codex and others on your codebase. It tracks pass rate, cost, time and consistency across repeated runs.

• Fullscript internal benchmark

  Write-up of a team that built a SWE-bench-style set of 36 tasks from its own git history and compared models and harnesses by cost. Evidence that capable teams build this in-house.

• SWE-bench collection pipeline

  Open-source harness with a pipeline for mining issue, patch and test tasks from any repository and grading them in containers, which makes the do-it-yourself route cheap.

• VS Code Cache Explorer

  Built into VS Code agent debug logs. It shows cache hit percentage per step and the first divergence between consecutive requests, broken down by system instructions, tool definitions and messages. It overlaps the analysis half of Prompt Diet.

• Datadog LLM Observability prompt caching

  Monitors prompt cache usage and token spend for LLM apps and agents.

• Grafana Cloud agent observability

  Cost-optimization guidance and tracking for agent token and cache usage.

Checked 2026-10-24