Labs bundle web search for agents, and standalone search APIs lose pricing power

Search Bench looks hard to win head-on. Artificial Analysis already runs a recurring, controlled, in-agent search API leaderboard with cost and speed, and AIMultiple plus the vendors' own reports cover the rest. The opening is in metrics they don't appear to publish: the share of fetches that hit blocked or 402 pages, freshness on content less than a day old, and cost per task in a customer's own workload. A paid tool that replays a team's real traffic against several providers would be a more defensible product than another public leaderboard. Artificial Analysis also doesn't say who funds it, so a benchmark with transparent funding and methodology could add trust. Licensed Fetch is already claimed. TollBit does almost exactly this with thousands of sites and edge partners, Cloudflare and Akamai own the network layer where blocking and payment happen, and Microsoft and ProRata handle the large publisher licensing deals. A newcomer could still try a buyer-side layer: one key, one bill and one rights record that combines TollBit, Cloudflare Pay Per Crawl and direct licences. It could also go after commerce and retail sites, which the current tools, aimed mostly at news publishers, cover less. Both openings are narrow, and Cloudflare or TollBit could close them easily.

• Artificial Analysis Search Index

  Launched August 2026 with a live leaderboard. It benchmarks search APIs inside a fixed agent setup (GPT-5.6 Luna, 25 turns, web_search and web_fetch tools) on DeepSearchQA, BrowseComp and AA-Omniscience, and reports quality, cost and speed. This is almost exactly the Search Bench idea, from a trusted benchmarking brand.

• AIMultiple agentic search benchmark

  An independent, regularly updated comparison of 8 search APIs on agent-style queries, with Brave, Exa, Tavily, Parallel and Firecrawl among those tested.

• Parallel benchmarks

  A vendor-run BrowseComp and agent search comparison of Exa, Brave and Tavily against Parallel. It is not independent, but it shapes how buyers perceive the field.

• Keenable Select

  Summarises published search API benchmarks and compares result quality against cost.

• Arena (LMArena) Search Arena

  Blind human-preference voting on search-augmented LLMs. It ranks finished systems rather than search APIs, but it overlaps with Search Bench for buyers.

• TollBit

  A per-request paywall for bots and agents that publishers price themselves. It returns authorised content as Markdown, publishers keep 100% of their rate, and the AI customer pays a transaction fee. TollBit reports more than 3,000 sites, with Arc XP, Akamai, DataDome and HUMAN as partners. This is close to the Licensed Fetch idea.

• Cloudflare Pay Per Crawl

  Lets sites set a price per fetch, with HTTP 402 payment responses at the network edge. Reports say it may move toward a broader pay-per-use model and default blocking of AI crawlers, though secondary sources disagree on how wide that default is. Stack Overflow adopted it in February 2026.

• Microsoft Publisher Content Marketplace

  Announced in February 2026, built with AP, Business Insider and USA Today. Publishers set licensing terms for AI grounding and are paid on delivered value, with usage reporting.

• ProRata / Gist

  An answer engine built on licensed content from 500+ publishers, sharing revenue 50/50 according to how often each publisher's content is cited. Nieman Lab also names Sphere as a TollBit-style intermediary.

• Akamai AI bot monetization

  An edge security company that routes AI bots to paid access through partners such as TollBit.

Checked 2026-10-18