Harness Fit: find which agent harness works with your local model

I didn't find any product or open-source tool that runs several installed coding-agent harnesses (Claude Code, Codex CLI, OpenCode, Pi, Aider, Qwen-Code, Cline) headless against the same local llama-server model, sorts their failures by cause, and recommends settings. The pieces exist separately. localagentbench is a reproducible local eval, but only through OpenCode. HarnessTax is an academic cross-harness comparison on hosted models. Blog posts and gists document the actual failure modes by hand: tool-call parser mismatches, Codex needing the Responses API, 27k-token system prompts overflowing small contexts, and retry loops. A new entrant could win by being the first to diagnose rather than just score. That means answering why a given harness fails with a given local model and outputting concrete fixes: the right llama.cpp chat template or --jinja flag, context size, a smaller system prompt profile, and API mode. Shipping a community-shared results matrix by model and harness would also help. The risks are significant. The space is moving fast, so localagentbench or Raschka-style scripts could add multi-harness support quickly. Harnesses and llama.cpp change weekly, so the task pack needs constant upkeep. And willingness to pay among hobbyist local-LLM developers is doubtful. Free open source with a paid hosted leaderboard, or selling to the harness vendors and model labs, may be more realistic than charging individual developers.

• localagentbench

  This is the closest match I found. It's an open-source, local-first eval harness for coding agents that uses Ollama with OpenCode. It runs 22 tasks with held-out oracles, gates each model with a 30-second tool-call canary, and reports pass@k plus tool-call counts and failures. It tests a single harness (OpenCode), not several harnesses side by side, and it doesn't recommend settings.

• HarnessTax (UC Berkeley / Arena)

  This is a study published in September 2026, not a tool you can run. It tested 21 model–harness pairs across Claude Code, Codex CLI and Pi on SWE-bench Lite and Terminal-Bench 2.0. Harness choice mostly changed cost (up to 5x) and context overhead, not success rate. The models were hosted frontier models, not local llama.cpp.

• harness-bench-fast

  A Python package with 351 file-operation tasks for LLM-backed coding agents. Its results table includes local models such as Qwen3-Coder-30B-A3B. The focus is benchmarking, not diagnosing why an installed harness fails.

• Sebastian Raschka's local coding agent evals

  A newsletter article plus a repo of Ollama benchmark scripts with tool-selection checks. Its finding is that the same local model (Qwen3.6) can do better in Codex than in its native Qwen-Code harness. That's the core insight harness-fit would build on, but it's an ad hoc study, not a product.

• little-coder

  A harness built on Pi and tuned for small local models on llama.cpp or Ollama. It reports 40% on Terminal-Bench with a 35B Qwen model. It competes as a harness, not as a tool for comparing harnesses.

• Gemma 4 local agentic coding benchmarks (gist)

  A hand-run write-up of exactly the kind of failures harness-fit would sort: a Codex CLI streaming bug that misrouted tool calls, Codex's roughly 27k-token system prompt hanging Ollama on Apple Silicon, and retry loops. It shows the pain is real but there's no tooling here.

• NVIDIA NemoClaw tool-calling reliability guide

  Troubleshooting docs for local models that emit text-looking tool calls with no structured tool_calls field. The fix is matching the server's parser to the model family. It's documentation, not a test runner.

Checked 2026-10-22