Agent reliability engineering forms as research shows bigger models don't handle tool failures better

I'd call this space competitive overall, but each of the three ideas is at a different stage. Model-swap regression gates are the most crowded. Raindrop, which has $50M in funding, replays production traffic against agent changes. FutureAGI, reenact, vouch, DigitalOcean's evaluations API and most eval platforms already sell or support golden-set gates. A new entrant would need fault-based reliability scores to stand out, not just the gate. MCP chaos testing is early. The existing tools are small open-source projects (mcp-drill, RateMyAgent, a hackathon project) and Gravitee's gateway marketing. None of them yet offers fault schedules managed across releases, completion rate under outages as a tracked metric, or compliance-ready reports for regulated buyers such as fintechs whose agents move money. Model-or-harness attribution looks the most open. In these searches I found research (the Model or Harness? taxonomy paper and the WHO&WHEN PRO benchmark) and advice on reading traces, but no product that labels failed runs and points to where the fix belongs. The research also says LLM judges tend to blame the model for harness bugs and get much less accurate on long traces. That leaves a technical edge for whoever solves it well. The best opening is probably one product that combines all three: inject faults, attribute each failure to the model or the harness, then gate releases and model swaps on reliability under faults. The main risk is that observability vendors like Arize, LangSmith and Braintrust, or Raindrop, add fault injection and attribution as features. To defend against that, sell to MCP-heavy platform teams and integrate deeply with their gateways and CI.

• mcp-drill

  An open-source proxy that sits between an MCP client and server. It injects timeouts, malformed JSON-RPC and corrupted-but-valid tool outputs so you can test in CI without a live LLM. It overlaps directly with the chaos-in-CI idea, though it covers the transport layer only and has no completion-rate reporting product.

• RateMyAgent

  An open-source package that combines a load tester, a chaos tool and a test runner to check how agents and MCP tools hold up under latency and dependency failures.

• ChaosMonkey MCP

  A May 2026 hackathon project: an MCP server that lets coding agents run fault injection and crash discovery. It is immature, but it shows that others have the same idea.

• Gravitee

  An API and agent-gateway vendor that publishes material on chaos testing MCP servers and LLM traffic, covering error storms, latency floods, retry storms and policy-invariant checks. It could add this as a gateway feature.

• Raindrop

  A Series A company with $50M in total funding, announced September 2026. Its Simulations product replays production traffic against proposed agent changes, including model changes, and flags anomalies. That is close to the swap regression gate idea.

• Bespoke Labs

  Reported to have raised a $40M Series A to build environments for testing agents before deployment. I only found this in a secondary source and did not verify it.

• Maxim AI

  A platform for agent simulation and evaluation aimed at production reliability. It is part of a crowded eval and observability category that could add fault scenarios.

• reenact

  An open-source tool that records an agent run and replays it offline. CI goes red when a model swap changes the agent's trajectory, which overlaps with the swap gate idea.

• FutureAGI

  An eval vendor that markets golden-set regression gates for model swaps and testing at the level of the agent's full trajectory.

• Model or Harness? (arXiv 2607.28802)

  A research taxonomy of 41 failure modes that assigns each agent failure to the model, the harness or the environment. Its best LLM judge agrees with human labels at kappa 0.76. It is not a product, but anyone can use it as the basis for an attribution tool.

• Arize (harness engineering)

  An observability vendor that publicly describes how to tell model failures from harness failures by reading traces. It could plausibly ship automated attribution in its existing tracing product.

Checked 2026-10-26