Snapshot Worlds for the web: offline site and form replicas for agent evals

The replica-world market is already competitive. Halluminate and Plato sell simulated website environments to labs and agent companies. Academic work (REAL, WARC-Bench, WebHarbor, VeriEnv, WebForge) has made both record-and-replay and site cloning close to standard practice. Safety benchmarks like SafeArena and ST-WebAgentBench already score harmful actions in sandboxed sites. A generic "offline replicas for agent evals" pitch would be up against funded companies and free benchmarks. The narrower angle still looks open. No product I found is framed as a sinkhole: a deny-by-default proxy that records arbitrary sites, replays them, intercepts every side-effecting request (form POSTs, email sends, payments, tip-line submissions) and writes a report of what the agent tried to do in the real world. The existing vendors focus on whether the agent completed the task. The safety benchmarks are fixed academic suites, not maintained products, and they don't cover high-risk targets like government forms, police tip lines, webmail or bank logins. A credible wedge would be an open-source drop-in proxy that works with Playwright or any computer-use harness and turns any recorded site into an exfiltration and side-effect detector. That could lead to paid, versioned "high-risk world packs" for safety teams, red-teamers and pre-deployment audits, a niche that unilateral-action and whistleblowing evals point toward. There are two risks. Halluminate or Plato could add a safety scoring layer cheaply. And the buyers who care most, frontier-lab safety teams, are few and often build in-house.

• Halluminate Westworld (YC S25)

  A simulated internet of synthetic consumer and enterprise apps, used to train and evaluate computer-use agents. It is reported to be deterministic and to run offline, with verifiers that check simulator state. It has paying customers among labs and browser-agent companies, and one report says it raised $30M, which I did not confirm. Its focus is capability on shopping, travel and finance tasks rather than safety or sinkholing.

• Plato (plato.so)

  A startup that builds simulated replicas of real websites, such as Amazon-, Airbnb- and Gmail-style sites, with state tracking, scoring and an SDK, for training and evaluating browser and computer-use agents. The details come from third-party directories and are low-confidence.

• REAL benchmark (AGI Inc / NeurIPS 2025)

  Deterministic, high-fidelity replicas of 11 popular websites with 112 tasks. This is the closest public example of curated replica 'world packs', but it targets capability and has no safety scoring.

• WARC-Bench (Uniphore)

  Records websites as Web ARChive files and replays them as sandboxed interactive environments with programmatic rewards. It is essentially the record-once, replay-offline mechanism of the Sinkhole Proxy idea, built as a benchmark rather than a product.

• VeriEnv / WebForge / WebHarbor research projects

  A run of 2025 and 2026 work on cloning or dockerizing live sites for reproducible offline agent benchmarking. WebHarbor dockerizes 15 live sites, and VeriEnv has a coding agent clone sites with their databases. Together they show the replica-generation step is becoming commoditized.

• SafeArena

  A safety benchmark with 250 harmful web tasks across four realistic sandboxed web environments, graded on risk levels from refusal through completion. It overlaps directly with the idea of scoring attempted harmful real-world actions.

• ST-WebAgentBench (IBM)

  Policy-adherence and safety evaluation built on WebArena environments (GitLab, ShoppingAdmin, SuiteCRM), measuring completion while the agent follows policy.

• mitmproxy / httpr and similar record-replay proxies

  General open-source HTTP record-replay tools. They can serve recorded traffic offline, but they do not block unrecorded or side-effecting requests by default, and they do not report what an agent tried to submit. These gaps are what a Sinkhole Proxy would fill.

• Browser Use RL environments

  A major browser-agent company that publicly argues synthetic clones lose live-web difficulty, and instead builds environments from real task traffic. This is a sign that some buyers will build their own environments or reject replicas.

Checked 2026-10-10