Coverage Count: what share of your agents' calls your monitors actually see
Compares network and billing logs with monitoring traces to find agent LLM and tool calls nobody watched. Bank AI governance teams pay.
Competitive · OpenAI audit finds 25% unmonitored
Written for: Small startup team
Next step
Call the head of model risk management at a top-50 US bank that already runs internal agents. Ask what percentage of last month's agent calls they could prove were monitored, and who would own that number.
Why now
If the best-resourced lab misses a quarter of its agent traffic, a bank running agents from several vendors misses more. Model risk teams and examiners can understand one number: the share of calls that were monitored. Monitoring products report what they saw, not what they missed. Our earlier Activation monitors and Session Receipts cards were about the monitors themselves. This measures the gaps between them.
What became possible
METR and Redwood Research spent three weeks on-site at OpenAI. They found up to 25% of agent traffic from GPT-5.5-class and newer models went unmonitored. NVIDIA's OpenShell reference architecture makes the same point: controls have to sit outside the agent.
Two ways in
- Coverage Count
Joins egress proxy logs, LLM provider billing exports and API-key inventories with the traces stored in the monitoring tool. Reports the percentage of agent calls that were monitored, by agent, team and provider, and lists each gap.
- Customer
- Heads of AI governance and model risk at regional banks
- Model
- Annual license by number of agent identities
- Shadow Agent Report
A monthly list of API keys and service accounts calling LLM endpoints with no matching trace, sent to the model inventory owner.
- Customer
- Same buyers, plus CISOs
- Model
- Included in the license
Competition
No search turned up a product that reconciles egress proxy logs, provider billing exports and API-key inventories against the traces stored in an existing monitoring tool to report the share of agent calls that were monitored. The vendors found, Zenity, Noma, Microsoft Agent 365 and Gravitee, discover agents or force traffic through their own control point, so they report coverage only relative to their own sensor.
Full competition notes (8 competitors)
No search turned up a product that reconciles egress proxy logs, provider billing exports and API-key inventories against the traces stored in an existing monitoring tool to report the share of agent calls that were monitored. The vendors found, Zenity, Noma, Microsoft Agent 365 and Gravitee, discover agents or force traffic through their own control point, so they report coverage only relative to their own sensor. A neutral cross-check that counts calls that were billed but never traced, broken down by agent, team and provider, and produces evidence model-risk validators and examiners can use, looks unclaimed. It fits regional banks running several monitoring tools that do not want another inline proxy. The risks are that it is a thin feature that Zenity, Noma, Gravitee or Microsoft could add quickly, especially Microsoft by tying Agent 365 to Azure OpenAI billing. It also depends on getting provider billing exports detailed enough per key and trace data that can be matched to them. The Shadow Agent Report idea overlaps much more with existing shadow AI discovery tools sold to CISOs and is the weaker of the two.
- Zenity
AI agent governance and observability platform. It inventories agents across platforms, scans continuously for unmanaged agents and MCP servers, and attributes each agent to its creator, tools and systems. It is reportedly named a Gartner 'company to beat' in AI agent governance, though that claim comes from a competitor's blog. It discovers agents but does not appear to reconcile billing against traces.
- Noma Security
Automatically discovers AI models, agents, MCP servers and data sources, and flags shadow AI deployments and unmanaged agents. It sells to security teams.
- Microsoft Agent 365
Agent registry built from Entra Agent IDs, manual registration, and shadow agents found through Defender and Intune, with discovery synced to AWS Bedrock and Google Cloud. Parts of it are in preview. It is the biggest platform threat for banks already standardized on Microsoft.
- Gravitee
API and agent gateway vendor that publishes heavily on 'monitoring coverage' as the percentage of LLM, MCP and API traffic passing an observation point, and on shadow AI detection in banking. Its coverage is measured through its own gateway, not by reconciling billing data.
- AgentDiscover (open source)
Open-source scanner that finds agents using static analysis, network monitoring, eBPF/Kubernetes runtime, endpoint and CloudTrail signals, and exports an AI bill of materials (AIBOM). It is a free substitute for the discovery half of the idea.
- Arthur AI
AI monitoring vendor that publishes on finding and inventorying shadow agents across the enterprise. It is moving into agent discovery from the monitoring side.
- Mozilla any-llm-gateway
Open-source LLM gateway that issues, revokes and monitors virtual API keys with budgets. It handles key sprawl by prevention, routing all calls through one gateway, rather than by auditing afterward.
- Entrans (AI model inventory for banking)
Vendor positioning on bank AI and agent model inventories under SR 11-7, built around discovery, a registry and enforcement. It shows that model-risk buyers are being targeted with inventory tools.
Checked 2026-10-24
Risks
- Needs access to network and billing logs, which banks share slowly
- Observability vendors can add a coverage metric
- Many banks have few agents in production yet
- A coverage number can be gamed by monitoring the wrong thing
Strong team fit and buildability; market pull is the weak point (6).
tech 7 · pull 6 · space 7 · fit 8 · build 8
Exposed
Companies that could benefit or be threatened. News and context, not investment advice.
Momentum of its main topic
Sources
Status history
- 2026-10-24 Flagged as buildable