Policy Compile: plain-English agent rules turned into cheap trace checks

The core idea, turning plain-English rules into deterministic checks on tool calls, is already a product feature at AWS (AgentCore Policy), is open source from IBM (ToolGuard in Langflow), and is the subject of at least three 2026 papers (NOMOS, VIGIL, AgentLTL). Invariant/Snyk and Microsoft AgentPex already check recorded traces after the fact. The pieces are all taken. What still looks open is narrower. First, an offline CLI that works with any vendor's logs: it reads arbitrary JSON tool-call logs (OpenAI, Anthropic, LangSmith, OTel exports) without needing a gateway or a specific framework. AWS needs its gateway and ToolGuard sits inside Langflow and runtime guarding. Second, rules that span several steps of a trace and are readable, so non-engineers can review them, such as "a refund over 100 dollars must be preceded by a manager_approval call in the same session." Most shipping products judge one call at a time. Third, debugging help: highlighting the step that decided the result and explaining why. Fourth, trust in the compiled rules, which NOMOS shows is a real problem: checking whether different compiler models produce the same rule, letting users replay a new rule against past logs to see how many decisions would change, and generating test traces for each rule. A small company can win with support and ops teams that already have written SOPs and log files but don't run on AWS or Langflow. It would need to be the CI and audit tool for trace compliance, not a runtime gate. The risks are that eval platforms (LangSmith, Braintrust, Maxim) could add this as a feature, and that teams may settle for LLM-as-judge checks or hand-written Invariant rules.

• Amazon Bedrock AgentCore Policy

  Turns plain-English rules into Cedar and checks them against the tool schema. Its CLI has a command, agentcore add policy --generate, that does this. The policies are enforced on every tool call at the AgentCore Gateway. It only works if you have a deployed AWS gateway, and it checks one call at a time before the call runs. It is not built for checking multi-step traces after the fact.

• IBM ToolGuard (via Langflow Policies)

  Developers write business policies in plain language. At build time, ToolGuard turns them into a specification and then into executable guard code. At runtime, that code checks tool calls. It is open source and very close to the core idea, but it guards calls before they run rather than auditing JSON logs.

• NOMOS (NeurIPS 2026)

  Research that compiles written policies into deterministic gates for tool calls, with static verification. It found that when different models compile the same policy, 18 to 29 percent of recorded calls can get different decisions. That makes it a known risk for any product built on this idea.

• VIGIL

  Research that compiles a policy into the same set of facts it extracts from an agent's tool-call trace, then checks temporal safety properties with SMT. This is almost exactly the policy-compile CLI checker, but as an academic paper.

• AgentLTL

  A framework for verifying traces with first-order temporal logic, aimed at procedural compliance in tool-using agents. It can also gate calls online. It deliberately skips translating from natural language, so its rules are written by hand.

• Microsoft Research AgentPex (Willful Disobedience)

  Pulls behavioral rules out of agent prompts and checks recorded traces for compliance after the fact. This overlaps directly with offline log checking.

• Invariant Guardrails (Snyk)

  An open-source (Apache 2.0) rule language in Python syntax that runs over agent traces and tool-call sequences, either locally on a trace or through a gateway. It includes Explorer, a trace viewer that can suggest rules. Snyk acquired it in 2025. Rules are written as code, not compiled from English.

• Superagent Guard

  A startup product that enforces near-English rules on tool and API calls, such as only allowing read operations on the users table.

• frenum

  A small open-source SDK. You write policies in YAML, then lint them, evaluate tool calls, test them, measure coverage and keep an audit log. It shows that small teams already have cheap deterministic tools, but without English compilation.

• Microsoft Agent Governance Toolkit / agentgateway

  Open-source MCP and tool-call guardrails from Microsoft and the agentgateway project. Another free option for enforcing policy on tool calls.

Checked 2026-10-23