Schema Lint: catch the structure tax in JSON-mode prompts
Linter and A/B harness that reorders output schemas to reasoning-first and measures the accuracy gain; LLM app developers pay.
Early · Paper pins accuracy loss on schema design
Written for: Solo builder
Next step
This weekend, implement three Zod/Pydantic lint rules from the paper's findings, reproduce one benchmark result in the README, and post to r/LocalLLaMA and the Latent Space Discord.
Why now
Nearly every production LLM call now uses JSON mode, and most schemas put the answer field first. The fix is mechanical and testable, which suits a lint rule plus a small eval runner. Honestly, it is a feature, not a company, but it is a quick open-source wedge into eval tooling.
What became possible
The Structure Tax paper finds that the accuracy loss from structured output depends on how the schema is designed, not on structure itself. Putting reasoning fields before answer fields matches or beats free-form output.
Two ways in
- schemalint
ESLint/ruff-style rules for Pydantic, Zod and JSON Schema: answer-before-reasoning order, over-deep nesting, enum fields without descriptions.
- Customer
- Developers shipping LLM features
- Model
- Free open-source
- Reorder A/B in CI
Runs your existing eval set against the original and the reordered schema, and comments the accuracy and token difference on the PR.
- Customer
- Small AI product teams
- Model
- $19/month per repo
Competition
No tool I found lints LLM output schemas for answer quality, as opposed to provider compatibility, or automatically creates a reasoning-first variant of a schema and measures accuracy and token changes against a team's own eval set. SchemaFit and agent-schema-lint only check whether a provider will accept a schema.
Full competition notes (5 competitors)
No tool I found lints LLM output schemas for answer quality, as opposed to provider compatibility, or automatically creates a reasoning-first variant of a schema and measures accuracy and token changes against a team's own eval set. SchemaFit and agent-schema-lint only check whether a provider will accept a schema. Promptfoo and Braintrust can post A/B results on pull requests, but users have to write the variants themselves. A new entrant could own the category of schema quality rules (answer before reasoning, overly deep nesting, enums without descriptions) plus one-click reorder experiments that settle the field-order question with each team's own numbers. That matters because public evidence is mixed: one controlled pydantic-evals test found negligible gains, while posts on how strict structured output works argue order is a hard constraint. The main risks are that SchemaFit or promptfoo could add a field-order rule or variant test cheaply, that simple lint rules tend to become free open-source software rather than something developers pay for, and that if measured gains are often near zero the paid A/B product loses its value. Paid value would more likely come from hosted benchmarking across models and automatic schema rewriting than from the linter itself.
- SchemaFit
Free MIT-licensed linter that runs in CI and checks JSON Schema, tool definitions and response_format schemas against each provider's structured-output rules (OpenAI, Anthropic, Gemini, and per its project page Mistral and Cohere). Has a repair command and is actively developed. Checks whether the provider will accept a schema, not answer accuracy or field order, but it is the obvious incumbent that could add such rules.
- agent-schema-lint
Offline linter for schemas used in LLM and agent structured output, written in MoonBit, with numbered rule codes. Focused on compatibility with a supported subset of JSON Schema, not reasoning order or accuracy.
- promptfoo GitHub Action
Runs evals on prompt changes before and after the edit and posts pass/fail counts as a pull request comment. Covers the generic CI A/B and PR-comment workflow, though it does not create reordered schema variants on its own.
- Braintrust eval-action
Commercial eval platform whose GitHub Action posts experiment comparisons and score changes on pull requests. Overlaps with the CI harness half of the idea.
- pydantic-evals field order experiment
Not a product, but a public controlled experiment finding little accuracy difference between answer-first and reasoning-first schemas on GPT models. Undercuts the size of the gain the product would promise.
Checked 2026-10-10
Risks
- Promptfoo and similar eval tools could add this as a single check
- Gains vary by model and may shrink as models improve
- Low willingness to pay for a linter
Strong team fit and buildability; market pull is the weak point (4).
tech 8 · pull 4 · space 7 · fit 9 · build 9
Momentum of its main topic
Sources
What happened next
- +7d · 17 Oct 2026 nothing yet. No launches, funding, acquisitions, or significant press coverage of schema linting tools or reasoning-first schema reordering products were found after October 10, 2026. SchemaFit was already a known competitor at flag time. Academic literature on schema field order exists, but no qualifying product events were detected.
Status history
- 2026-10-10 Flagged as buildable