Gateway Scorecard: independent MCP gateway testing for bank procurement

The opening is Gateway Scorecard as an independent service. Test material already exists: mcpsec covers gateway security and MCPTox covers tool poisoning. What I could not find is anyone who runs these tests and measures tool-routing recall on a copy of a specific buyer's tool catalogue, then packages the results as a vendor-risk report that a bank's third-party risk team and regulators would accept. Vendor comparisons and RFP checklists are written by the gateway companies themselves, and the one independent review leaves security out. Regulatory pressure is building: articles cite an NSA warning on MCP and expect MCP-specific testing to show up in UK bank supervision under DORA. That pressure favours a neutral tester, especially for regional and mid-size banks that lack in-house red teams. To win, a new entrant needs credibility (published methods, repeatable test kits and possibly a link to a standard or audit firm) and test cases based on real banking workflows. The main threats are that mcpsec is free for banks to run themselves, that Big Four firms or AI security vendors could add this to existing assurance work, and that one-off procurement projects may not bring repeat revenue unless the product turns into ongoing retesting. Catalogue Finder has little room: search-and-call layers over large tool catalogues are already a standard gateway feature (LiteLLM, Nexla) and a common open-source pattern (OmniMCP, mcp-mux, toolsieve). It would have to win on proven recall over messy internal APIs, not on the design, and is better used as one input to the Scorecard's routing-recall tests than sold on its own.

• mcpsec

  A free, vendor-neutral (Apache-2.0) security benchmark for MCP gateways: 10 formal security properties, 64 attack test cases across 12 attack classes, and a tiered score from Unsafe to Hardened. It is the closest thing to the scoring engine itself, but it is a tool, not a bank-specific service or report.

• AIMultiple MCP gateway benchmark

  A published latency and security comparison of six MCP gateways using an instrumented backend. It is generic research content, not tailored to any buyer's tool catalogue.

• Tetrate AI gateway benchmarks

  A vendor benchmark of the overhead Envoy AI Gateway adds to MCP tool calls. It covers latency only and is written by the vendor.

• MCPTox

  An academic tool-poisoning benchmark (45 real MCP servers, 1348 attack cases). It tests AI models, not gateways, but a scorecard could reuse its methods.

• Composio MCP gateway RFP checklist

  A buying checklist written by a gateway vendor (credential isolation, policy enforcement, SOC 2 evidence, logging of denied calls). It shows procurement demand, but the evaluation is self-serving and nothing is actually tested.

• Best MCP Gateways in 2026 hands-on comparison (Substack)

  The only independent hands-on comparison found. It explicitly leaves security, production load and failure recovery out of scope.

• Invariant Labs MCP-Scan

  Static and dynamic scanning of MCP connections for prompt injection, tool poisoning and toxic flows. It is a general security tool, not a way to compare gateways for procurement.

• LiteLLM MCP Tool Search

  Two virtual tools, search and call, in front of a large MCP catalogue, with results limited to the tools each API key may access. This directly overlaps Catalogue Finder.

• Nexla MCP Gateway discovery meta-tools

  An enterprise gateway that adds search, describe and call tools ranked by semantic similarity. It overlaps Catalogue Finder.

• OmniMCP

  An open-source single-tool semantic router across many MCP servers that loads tool schemas only when needed. It overlaps Catalogue Finder.

• toolsieve

  Open-source embedding-based tool lookup. It publishes its own recall benchmark: 69% of queries got a ready-to-call right tool across 181 tools and 159 queries.

• mcp-mux

  An open-source MCP router that exposes only search and execute tools. It overlaps Catalogue Finder.

Checked 2026-10-19