Escalate Only When Lost: small-model drafts with frontier checks for back-office agents

I found no commercial product built for step-level small-to-frontier routing in computer-use agents, and none aimed at mortgage or lending back offices. The idea itself is not protectable: several 2026 papers (AVR, STEPGATE, the event-driven cascade paper, R2V-Agent) publish the exact technique with strong cost numbers. General routers like Not Diamond and Martian pick a model once per request, and browser agent tools like Stagehand and Skyvern lower cost by replaying cached actions rather than routing between models. A new entrant could win by packaging the router for regulated operations teams: audit-ready logs of every handoff, keeping sensitive borrower data on the local model, forcing the frontier model on high-risk actions like payments or record changes, and honest reports of cost per completed task against an always-frontier baseline. It would also need to work alongside replay caching and run inside existing UiPath or Automation Anywhere deployments. The main risk is that UiPath, Skyvern or a router company adds per-step escalation as a feature, so the moat has to come from mortgage-specific workflow benchmarks, compliance packaging and moving fast.

• Adaptive VLM Routing for Computer-Use Agents (arXiv 2603.12823)

  A 2026 research paper. A routing layer between the computer-use agent and a pool of vision-language models estimates how hard each action is and checks the small model's confidence. It reports up to 78% cost reduction, within 2 points of an all-large-model baseline. This is the core technique, published openly.

• STEPGATE: Uncertainty-Aware Step-Level Handoff for Small Language Model Agents

  Listed at NeurIPS 2026. It scores each small-model step and escalates to a cloud model only on hard steps; a 1.5B/7B pair got 82.7% success with 30.8% escalation. These are academic results on small scripted tasks, not a product.

• Event-driven cascade for computer-use agents (arXiv 2604.27151)

  A research paper. A small policy runs by default and escalates when monitors detect stalled progress or drift at milestones. More evidence that the technique is well known.

• R2V-Agent step-level router

  A research paper that trains a router to estimate the small model's failure risk at each step and escalates to a teacher LLM only when needed.

• Not Diamond

  A commercial predictive model router that powers OpenRouter's auto mode and is positioned for coding agents. It picks a model per query, not per screen step. It is well placed to add per-step agent routing.

• Martian

  A well-funded commercial LLM router claiming 20 to 97% cost reduction. General-purpose routing per request, not built for computer-use agents.

• RouteLLM (LMSYS / Berkeley)

  An open-source framework that routes each query between a strong and a weak model. A free starting point anyone could adapt for agent steps.

• Stagehand caching (Browserbase)

  Saves browser agent actions and replays them without model calls, calling the LLM only when the page changes. Lowers cost in a different way and will be compared against step routing.

• Skyvern

  A browser automation agent company. Its explore-then-replay approach reportedly cut average run cost from $0.11 to $0.04. No documented model escalation, but it competes for the same goal of cheaper runs.

• UiPath Healing Agent

  Fixes failing UI automations at runtime and lets customers bring their own LLM. No automatic escalation between models, but UiPath already owns the RPA bots the target buyers run.

• Perplexity hybrid local-cloud inference

  Shown at Computex 2026: a local model decides task by task what stays on the device and what goes to cloud models. Consumer-focused, not back-office agents.

• Windows hybrid intelligence

  Microsoft is building routing into Windows that runs work locally or in the cloud depending on the task. A general platform feature, not built for automation teams.

Checked 2026-10-16