Boundary Bench: test your coding-agent permissions against SWE-MISBEHAVIOR

The benchmark side looks open. SWE-MISBEHAVIOR came out in October 2026 and was only tested on research agents. Existing guardrails like Fence and denylist skills publish no benchmark results, and GuardFall shows most guardrails can be bypassed. Claude Code's own docs say its permission rules can be bypassed. Nobody I found sells a tool that takes the permission config and hooks a team actually uses and gives a repeatable pass/fail score in CI. A new entrant could win by testing the user's real setup, adding the GuardFall bypass tricks to the scenarios, and tracking whether a config gets better or worse across agent versions. The risks are that the paper authors or Anthropic/OpenAI ship something similar, that replaying with live agents makes CI runs slow and costly, and that the SWE-MISBEHAVIOR data may not be openly available. The undo shim is much less open: AgentFS, Container Use, Kairos and cleveragents-core already cover it, so it works better as a feature of the bench than as its own product.

• SWE-MISBEHAVIOR paper and CODE-KIT

  The source benchmark itself, from October 2026, with 7.4K boundary-violation samples across code generation, file management and system interaction. Its CODE-KIT mitigation was tested on smolagents and OpenHands. There is no product for running it against Claude Code or Codex permission configs, and I didn't check whether the data is public.

• GuardFall (Adversa AI, via Cloud Security Alliance)

  A research report that tested five shell bypass techniques against eleven open-source coding agents and found ten of them vulnerable. It is the closest thing to a guardrail test suite, but it is a one-off report, not a product you run in CI.

• Fence (hoop.dev)

  An open-source hook-based blocker for Claude Code and Codex that looks at what a command is trying to do. It has no published benchmark, and coverage notes it is unclear how it handles disguised or multi-step commands. It could be a customer or a thing being tested rather than a competitor.

• Global agent guardrails skill (Tessl registry)

  A shared denylist for Claude Code, Codex and other agents, with a small regex test script. Its author says disguised commands can slip past it.

• Claude Code permissions and sandbox (Anthropic)

  Built-in allow, ask and deny rules and an OS-level sandbox. The docs warn that Bash rules match prefixes and can be bypassed, and a sandbox escape (CVE-2026-25725) was patched in version 2.1.2. Anthropic is the platform owner and could add its own config testing.

• AgentFS (Turso)

  The agentfs run command holds agent writes in a copy-on-write overlay stored in SQLite and blocks writes outside the working directory. This covers most of the undo shim idea.

• Container Use (Dagger)

  Gives each coding agent its own container and git branch. Works with Claude Code, Codex and other agents and keeps a history of what each agent did. An alternative way to contain agent writes.

• Kairos

  An October 2026 research design that runs the agent in a microVM, holds file writes in an overlay layer with a log for rollback, and commits or aborts after a policy decision. This is the design the undo shim is named after.

• cleveragents-core overlay sandbox

  An open-source agent framework with an OverlayFS sandbox, all-or-nothing commits after the user approves, and rollback.

Checked 2026-10-14