160 updated first flagged18 Oct 2026

Harness Tune: AGENTS.md edits proposed by Harness2, kept only if replays improve

A CLI that reads failed coding-agent sessions, suggests instruction edits, replays them and keeps only the edits that win. Developers pay.

Competitive · Google open-sources Harness2

Written for: Solo builder

Next step

This week, connect Harness2 to the Claude Code session logs in one of your own repos. Pick five failed sessions and ship a CLI that prints an AGENTS.md diff and a replay score. Post the before-and-after numbers in the r/LocalLLaMA Harness2 thread and on r/ClaudeAI to find your first users.

Why now

Harness Tune was on the watchlist because picking which instruction edit is better without ground truth was unsolved. Harness2 solves exactly that step. Annotrace shows that agent traces from OpenTelemetry GenAI, LangSmith and Langfuse can already be read in a standard way, so getting session data in is a known problem. Taste-Bench found that even the best model picks the right long-term path on just under 60% of simple binary questions. That is why edits have to be checked by replaying sessions, not by asking the model whether it did better. Cognition tying token budgets to performance shows teams already pay for harness quality.

What became possible

Harness2 is open source. It lets an agent change its own harness (system prompt, tools, memory, skills, subagents) for each task. It proposes changes, runs a few candidate versions side by side and keeps the better one, with no answer key or grades needed. That loop is the hard part of Harness Tune, and it is now free to use.

Two ways in

harness-tune CLI
Reads Claude Code or Codex session logs and groups the failed runs. It uses Harness2 to propose diffs to AGENTS.md or a skill file, replays 5 to 20 past tasks in a throwaway worktree with each candidate, and opens a PR with before-and-after pass rates and token cost.

Customer
Individual developers who use Claude Code, Codex or Cursor every day and keep rewriting their rules files by hand
Model
Free MIT CLI. $15 a month for hosted replay runners and history
Harness CI gate
A GitHub Action that runs the team's replay suite whenever someone edits AGENTS.md or a skills folder, and blocks changes that lower the pass rate or raise cost per task.

Customer
Engineering teams of 5 to 50 people that share agent instructions in a monorepo
Model
$49 per repo per month, self-serve

Competition

Competitive

The risks are serious. Anthropic could extend skill-creator's evals to CLAUDE.md and session-derived edits at any time. Tessl already charges in this area. Popular free plugins and DIY CI recipes put a low ceiling on what individual developers will pay. A paid version probably has to sell to teams, built on the changelog and audit trail, rather than to solo developers.

Full competition notes (10 competitors)

Every piece of this product already exists, but I couldn't find one packaged developer CLI that does the whole job. Reflection tools like claude-reflect, revise-claude-md and the ClawHub skill already turn session corrections into proposed AGENTS.md or CLAUDE.md edits. Measurement tools like mdarena, Anthropic's skill-creator benchmark mode and Tessl's skill-optimizer already compare instructions with and without a change. The full mine, propose, replay and accept-only-if-better loop is published as Self-Harness and can be built with GEPA. So the opening is integration and workflow polish, not a new idea. A new entrant could still win in three ways. First, by being the tool that works across agents, reading Claude Code, Codex and OpenCode transcripts natively and replaying cheaply, since Anthropic and Tessl mostly cover their own skill formats. Second, by making replay statistically honest at low cost. Three saved tasks is too few to separate a real gain from noise, and running the agent many times is the expense users already complain about. Third, by owning the instruction changelog with replay evidence attached, which none of the tools above offers and which small teams with a shared setup would value. The risks are serious. Anthropic could extend skill-creator's evals to CLAUDE.md and session-derived edits at any time. Tessl already charges in this area. Popular free plugins and DIY CI recipes put a low ceiling on what individual developers will pay. A paid version probably has to sell to teams, built on the changelog and audit trail, rather than to solo developers.

mdarena (HudsonGri/mdarena)
An open-source MIT tool that benchmarks CLAUDE.md variants against a no-context baseline, using tasks mined from your own merged PRs. This is the replay-measurement half of Harness Tune. It measures but does not write edits from failed sessions or commit them automatically.
Self-Harness (paper and qzzqzzb/Self-Harness repo)
A June 2026 research method that finds recurring failure patterns in clustered traces, proposes small edits to the harness, and keeps an edit only if it improves performance without regressions. It is almost exactly the core loop, but it is aimed at research and benchmarks rather than a developer CLI.
GEPA / gskill / optimize_anything
A text-evolution optimizer. Its gskill pipeline learns skills for a specific repo for coding agents using verifiable tasks from SWE-smith, with a reported jump from 55 to 82 percent resolve rate on Jinja. Practitioners already suggest pointing it at CLAUDE.md plus a test suite.
OptimizeSpec (terminaluse)
A plugin that guides a coding agent to write an eval spec and optimization code, using GEPA as the engine. It is general-purpose agent optimization scaffolding.
Anthropic skill-creator (eval and benchmark mode)
Anthropic's official skill-creator runs test cases with and without a skill, does blind A/B comparisons between skill versions, and tracks pass rate and tokens. It covers the with/without replay gate for skill files but does not draft edits from failed sessions.
Tessl skill-optimizer
A commercial registry tile that generates scenarios, runs an agent with and without a skill, scores the results with a judge, and runs a review-and-fix loop. It is the closest paid product, but it focuses on skills and plugins rather than AGENTS.md or session logs.
claude-reflect
A popular plugin (about 2.5k stars) that captures corrections from Claude Code sessions, can scan past sessions, and applies learnings to CLAUDE.md or AGENTS.md after review. It has no replay validation.
claude-md-management /revise-claude-md
A Claude Code plugin command that reflects on the current session and proposes CLAUDE.md diffs for the user to approve. It has no evidence gate.
ClawHub self-improving-agent skill
Logs errors and corrections to markdown, then promotes them to CLAUDE.md, AGENTS.md or Copilot instructions. It is a logging workflow, not a test-gated one.
Continuous evals for agent harnesses (developertoolkit.ai guide, StackHawk eval harness)
Documented practice of running headless regression evals in CI whenever CLAUDE.md, rules or skills change. StackHawk describes a similar old-versus-new skill A/B harness. These show teams building this themselves.

Checked 2026-10-18

Risks

Replays are not deterministic. Small suites will produce noisy wins unless runs are repeated, and repeats cost tokens.
Coding-agent vendors may build self-tuning memory into the product and make a separate tool pointless.
Harness2 is new. Its API and licence terms may change.
Pairwise selection without grades can reward edits that only look better, such as longer outputs.

Strong team fit; tech readiness and market pull and open space are the weak point (6).

tech 6 · pull 6 · space 6 · fit 9 · build 7

Exposed

GOOGL MSFT

Companies that could benefit or be threatened. News and context, not investment advice.

Momentum of its main topic

Agents AI

168 developments this week +53 vs last week

Stage: Coverage

Sources

Status history

2026-10-18 Added to the watchlist
2026-10-19 Promoted from the watchlist: Today's evidence made it buildable
2026-10-19 Flagged as buildable

Build Brief → Open in Earlyink