AI-made discoveries outrun referees, and scientific verification becomes a paid layer

The two product ideas are at different stages. For the shepherd network, I found no paid network of human domain referees and formal-proof checkers that labs or companies hire to vouch for AI-derived claims before publication. What does exist splits into three groups. Automated AI manuscript checkers (Reviewer3, q.e.d, Refine) are cheap, aimed at authors, and check against the literature. Automated formal checkers (Axiom's AXLE, Harmonic) only work for math. Replication agents (Inherent Labs, the IFP proposal) are still unproven. None of them sells accountable, signed-off validation by credentialed experts, and that is what a frontier lab or corporate R&D group needs before announcing an "AI discovered X" claim. A new entrant could win by combining AI pre-screening with named human referees and Lean checks where they apply, then issuing an attestation that publishers and buyers recognize. The risks are that it's hard to recruit enough top referees, labs may prefer to run validation in-house, and Axiom or Harmonic could add human review on top of their checkers. The answer-key environments idea is more contested, though still early. Leash has shown the model with Minos but only for DEL binding data. CueBench and others sell science RL environments, mostly simulations rather than measured data. Edison/FutureHouse says it plans to move into the wet lab. Lila and Periodic generate their own ground truth but don't license it. Generic RL-environment vendors (more than 35, per SemiAnalysis) could expand into science. The open space is in large, held-out, pre-measured experimental datasets in other wet-lab domains, licensed as gated evals. Examples are protein expression and stability, CRISPR or perturbation screens, ADMET assays, materials synthesis outcomes, and enzyme activity. The advantage would come from cheap, high-throughput assays that rule out shortcuts, the way Leash's barcode controls do, and from keeping the test sets private so they don't leak into training. The weak point is that it costs a lot of capital to generate the data, and well-funded autonomous labs could start selling their own data as evals.

• Leash Bio (Minos / Strelka dataset)

  This is the template for the answer-key idea. Minos is a sandbox of pre-measured lab worlds built from DNA-encoded library binding data, where agents look for protein binders. It is released as an open dataset with a preprint, and it only covers binding chemistry.

• CueBench (YC S26)

  A startup that sells RL environments for training and evaluating models on scientific reasoning and experiment design. The examples I found are physics simulations, such as a hidden double pendulum, not measured wet-lab data.

• Edison Scientific / FutureHouse (LAB-Bench, LABBench2, Aviary)

  Builds public biology benchmarks and the Aviary framework of scientific RL environments. The authors say future versions should move into the physical wet lab and drug development, so they could become a direct competitor for answer-key environments.

• Lila Sciences

  Raised about $450M in total to build autonomous AI Science Factories that produce their own experimental ground truth. Today it mostly generates data for its own use, but it could license environments or validation later.

• Periodic Labs

  Pairs AI scientists with autonomous labs because physical experiments give a verifiable signal. Its agents work for industrial partners. It is a vertically integrated alternative to buying outside answer-key data.

• Gamow Labs (LabRobotBench)

  A benchmark that tests frontier models on simulated robotic wet-lab tasks. The coverage is thin and I could not fully verify it.

• Axiom Math (AXLE)

  A hosted Lean proof-verification service released in March 2026. It covers the formal-proof-checker part of the shepherd network for math claims, but it is automated, not a human referee network.

• Harmonic (Aristotle)

  A commercial formal-reasoning system that produces Lean-verified proofs. It is a likely supplier or competitor for the formal-checking part of verifying AI math results.

• Reviewer3

  An AI review tool with multiple agents that try to reproduce findings, rerun statistics, check code and cross-check citations. A competitor blog reports self-serve pricing from about $50 per review, which I haven't verified. It is AI checking manuscripts, not paid human domain referees.

• q.e.d Science

  Breaks manuscripts into individual claims and checks them against the literature. bioRxiv and medRxiv plan to integrate with AI review tools of this kind. It works on literature consistency, not experimental or formal ground truth.

• Refine Technologies

  A Harvard Innovation Labs venture building AI peer review for research papers. It is at an early stage.

• Inherent Labs (Faraday)

  An agent for reproducing published findings. The company says it beats frontier models, but that comes from its own benchmark and no outside party has checked it. It points toward automated replication as a way to verify results.

• IFP Replication Engine proposal

  A policy proposal, not a product: AI agents would reproduce computational findings at publication time, starting with a $10M pilot. It shows that funders could become buyers of verification.

Checked 2026-10-27