Skill Recall: measure and fix skill-routing collapse as your agent's skill folder grows

Both halves of this idea already exist as free, open-source tools. Measuring trigger recall with headless runs and near-miss negatives is covered by Anthropic's own skill-creator eval loop, Google's skill-reach (which also does collision detection), skillcaller (cross-harness pass-rate gates) and several smaller harnesses. Embedding-based top-k routing exists as skill-router-mcp, as community UserPromptSubmit hooks, and in research systems like SkillRouter that report stronger approaches than metadata-only top-k. The one angle I didn't find packaged anywhere is a scaling curve. That would mean measuring how recall for a fixed set of target skills decays as you inject growing numbers of realistic distractor skills, and turning that into a regression budget in CI. Something like "adding these 5 skills dropped recall of the deploy skill from 92 to 71 percent" would be useful to teams with 50 to 200 skills. A hosted CI product would need its edge to come from paying the cost of the LLM runs, comparing results across harnesses and models, tracking history, and suggesting description fixes that resolve collisions. Even then, Anthropic or Google could absorb that capability easily. The router would only stand out if it beats the native progressive disclosure with measured gains, ideally ranking on skill bodies and diversity rather than descriptions alone. Developers' willingness to pay for this, as opposed to running the free tools, looks weak.

• google/skill-reach

  Open-source eval suite from Google for skill triggering and collision detection. It measures discovery and selection F1 when competing descriptions are present, optimizes descriptions, and lints description length. It is very close to the skill-recall CLI idea.

• Anthropic skill-creator trigger eval loop

  The official skill-creator plugin ships run_loop.py, which runs claude -p headless against labeled should-trigger and should-not-trigger queries. It reports recall and accuracy and iteratively rewrites descriptions. A headless-triggering bug in it was closed in June 2026, but the search results don't show what the fix was. This is the platform vendor covering the core measurement for free.

• skillcaller

  Trigger-reliability harness with pass-rate gates and repeated runs. It works across Claude Code, Cursor, Codex and Antigravity, so it is already CI-gate oriented.

• open-skill-standard trigger eval (PR #32)

  Runs Claude Code headless and parses Skill tool calls from the stream. It scores precision, recall and false alarms on labeled sets that are about half near-miss negatives.

• Skill-Lab

  Auto-generates explicit, implicit, contextual and negative trigger tests from skills and runs them through the Claude CLI. This overlaps with generating probe tasks from descriptions.

• skill-eval-harness (adewale)

  Python CLI for evaluating skills, with should-fire and should-not-fire trigger checks, trace artifacts and runner adapters.

• addyosmani/agent-skills evals

  Evals that check skills trigger when they should and stay distinct from one another.

• skill-router-mcp

  Open-source MCP server with match_skill(query, top_k) using local Ollama embeddings plus get_skill for on-demand loading. It is explicitly motivated by selection degrading as libraries grow, so it is essentially the top-k router idea.

• SkillRouter (arXiv 2603.22455)

  Research retrieve-and-rerank router benchmarked on about 80K skills. It finds that the skill body, not just the description, is the decisive routing signal, which suggests a metadata-only router underperforms.

• Diversity-aware skill routing (arXiv 2609.05824)

  Recent paper critiquing pointwise top-k skill retrieval and proposing diversity-aware selection.

• Skill activation hook (claudefa.st)

  Community UserPromptSubmit hook that matches prompts against skill rules and injects skill recommendations. Similar write-ups use ChromaDB semantic search in a hook.

• Scaling Claude Code Skills Without Burning Context

  Write-up of a vector-search hook that surfaces only relevant skills per prompt, which is the router-as-hook pattern.

Checked 2026-10-14