Skill Check: does your Agent Skill work the same on every platform that claims support?

I found no product that does exactly this pitch: a hosted GitHub Action that runs one skill headlessly on several agent CLIs and reports, for each platform, whether it loaded, triggered and followed the skill, with a compatibility badge. But the parts are already covered. Skillgrade is free and already runs skill evals on Claude Code, Codex and Gemini CLI, and vendors like NativeScript run it in CI. Tessl scores activation and runs with-vs-without evals in its public registry. Skill Lab, skillvitals, skill-lens and agent-exam cover trigger and output testing on one platform. The opening is narrow. A new entrant would need to own the cross-platform comparison itself: a maintained fleet of agent CLIs at pinned and latest versions, each loading skills from its own paths; repeated trials, since triggering is random; a diff of trigger and output behavior across platforms; and a public badge or registry that buyers trust. Version drift is a selling point, because silent breakage when Cursor, Codex or Copilot changes how it loads skills is something a free one-off script won't catch. Paid demand is the weak point. The Agensi test found that 8 of 10 skills behaved identically everywhere, and the failures came from predictable frontmatter features a linter can flag. So most authors may be fine with free skillgrade, and the real buyers are likely a small group of devtool companies shipping official skills. Tessl or a platform vendor could also add this as a feature.

• Skillgrade (mgechev)

  Open-source CLI for writing unit-test-style evals for Agent Skills. You configure tasks, graders, a Docker provider, trial counts and pass thresholds in one eval.yaml. It runs on Claude Code, Codex and Gemini CLI. NativeScript already uses it in CI with a discovery check for each skill, so it is the closest existing match to running one skill on several agents. It does not appear to produce a side-by-side platform comparison or a badge.

• NativeScript/skills (skillgrade in CI)

  A vendor skill repo whose CI lints, validates and runs a skillgrade eval suite. It uses headless graders and checks that the agent discovered each skill. This shows skill publishers are already building this themselves with free tools.

• Tessl skill review and evals

  Commercial registry that scores each skill when it is published. The score combines deterministic validation, an LLM-judged activation (trigger) score and an implementation score. It also runs scenario evals comparing agent success with and without the skill, and shows the results publicly in the registry. This is a well-funded neighbor that could add cross-agent compatibility.

• Skill Lab (sklab)

  Skill validation CLI with a documented GitHub Actions workflow and LLM-generated trigger tests. Trigger tests depend on Claude CLI, so it covers a single platform.

• skillvitals (ContextJet)

  Measures trigger accuracy (recall, false-fire rate, F1) from labeled prompts, plus efficacy with and without the skill. It has a free heuristic mode and can also use a model backend. It is not a multi-agent-CLI runner.

• skill-lens

  Python tool that turns skill edits into a score and an exit code a CI pipeline can gate on, using assertions on the output. It covers output checks, not triggering across platforms.

• agent-exam

  Eval framework with a dedicated trigger eval type. It stops after the first skill invocation to check which skill fired.

• Agensi cross-agent compatibility test

  A one-off editorial test, not a product. It ran 10 skills on Claude Code, Codex CLI, Gemini CLI, Cursor, Copilot and OpenClaw, and 8 behaved identically everywhere. Divergences came from agent-specific frontmatter such as allowed-tools and context: fork. This is evidence of real but fairly narrow divergence.

• agent-skills-compat-matrix (porex-bot)

  A static, self-reported matrix of which agents support which skill features and paths. These are not executed tests.

• promptfoo agent skills guide

  The general eval tool promptfoo has a guide for testing agent skills. It supports many providers and could be set up to compare agents.

Checked 2026-10-23