Google open-sources self-tuning agent harnesses, and the app layer's hand-built moat thins

The idea's main premise holds up, though the evidence is thin. Self-modifying harnesses are now open source (RRSI and ReasoningBank from Google), so value is moving toward whoever owns graders and replay data. But both product ideas run into players that are already funded or active. Grader Store has three kinds of rivals. Free academic benchmarks with verifiers already cover parts of finance: BankerToolBench, FORCE-Bench and FrontierFinance. Vendors already sell expert rubric data and environments to labs: Mercor, Deeptune and the broader RL-environments crowd. And Galtea already pitches custom scorers to banks. What nobody I found offers is graders and replay sets built for retail and commercial banking back-office work, such as KYC refresh, payment exceptions, reconciliations, disputes and loan servicing. That work is regulated, operations-heavy and full of proprietary data. Most public finance evals target investment-banking or research deliverables. The hard part is data, not software. Real replay data would have to come from banks under privacy and regulatory limits. A bank like Santander that builds its own lab may want to own its graders in-house, since those graders become its moat. The idea is more defensible as a consortium or a clean-room service that builds and certifies graders on each bank's own logs, without trying to resell shared data. It is weaker as a marketplace of static graders: self-tuning harnesses will overfit to public graders quickly, which makes them a commodity. Independent Scorecard is closer to taken. Vals AI already does private, application-level evaluation for enterprise buyers. Epoch publishes independent re-runs, and Armilla sells third-party verification reports to buyers. The remaining opening is narrow. It would mean running a lab's claimed benchmark on the buyer's own harness and tasks, quickly, at launch time, and framing the gap between the claim and the buyer's result as a procurement artifact. Self-tuning harnesses make that more relevant, because scores now depend on the harness as much as on the model. One caution: an independent scorecard tends to get pulled toward being a consulting service or a feature inside existing eval platforms. Braintrust, Patronus and Scale were not covered by these searches and could add this capability easily. Winning probably depends on regulatory pull, for example bank model-risk management or EU AI Act conformity, more than on technology.

• Google RRSI (Regularized Recursive Self-Improvement)

  This is the trigger event rather than a competitor. A single third-party blog reports that Google Cloud AI Research, with academic partners, released an Apache-2.0 framework in which an agent rewrites its own prompts, tools, memory and sub-agents while the model weights stay fixed. The post dates it 21 September 2026 and claims gains on six of six held-out benchmarks. I could not confirm any of this against a Google source.

• Google Research ReasoningBank

  A Google open-source framework in which agents improve themselves through memory. It is more evidence that the harness layer is being commoditized.

• Vals AI

  A neutral evaluator that runs private, domain-expert test sets, including finance, and is moving from model scores toward application-level reports for enterprise buyers. It is the closest match to the Independent Scorecard and also overlaps with Grader Store on finance.

• Sacra / application-level AI validation for procurement

  Market research showing that application-level validation for procurement is already treated as a recognized category.

• Armilla Verified

  Third-party verification of AI products. Vendors get a seal, and both vendors and enterprise buyers get an independent expert report. It competes with the Scorecard on the procurement side.

• Epoch AI benchmarking hub

  Independently re-runs and publishes frontier model results, which positions it as the outside check on lab claims. It does not test on a buyer's own harness or tasks, and its results arrive after launches.

• Galtea

  An evaluation and monitoring platform aimed at banking, with custom scorers and thresholds, pre-release testing, and output that supports regulatory processes. It competes directly with Grader Store for bank AI teams.

• Mercor APEX-Agents

  Expert-built agent tasks with binary rubrics, including 8 investment-banking worlds. Mercor already sells expert rubric data to labs.

• BankerToolBench

  An open-source investment-banking agent benchmark with rubrics written by bankers, an agentic verifier and a runnable Harbor environment. It is free and covers front-office analytical work, not back-office operations.

• FORCE-Bench

  An open benchmark, dataset and evaluation harness for agentic AI in enterprise finance operations. Teams can apply its rubric framework to their own data, and it sits close to back-office use cases.

• Prime Intellect Environments Hub

  An open community hub with the Verifiers library and over 1,000 contributed RL and eval environments. It is the open-source alternative to a paid grader store.

• Deeptune

  Sells managed RL environments, mainly to frontier labs, and reportedly raised a $43M Series A led by a16z. Wing's analysis of the environments and verifiers market says it will consolidate around embedded partners.

Checked 2026-10-19