Compaction Probe: test whether your coding agent turns injected text into user instructions

I found no dedicated, harness-agnostic, open-source test that plants role-labelled payloads, forces compaction, and checks whether they come back as user instructions. The risk is documented in several places. OpenAI's alignment team has written about it. Straiker has argued it publicly. Codex CLI's design stores the summary as a user message. Individual repos like aidevops are hardening against it. But all of that is research, commentary or one-off fixes, not a reusable probe with per-version pass/fail results. That makes compaction-probe the stronger of the two ideas. A credible open-source suite with a public scoreboard across Cline, goose, OpenCode, Codex CLI, Hermes and Claude Code could become the reference test before promptfoo, AgentDojo or Straiker adds a compaction module, which any of them could do quickly. The weak point is who pays. Open-source maintainers rarely pay, and frontier labs test internally. Realistic revenue is more likely sponsorship, a hosted CI service for companies shipping agents, or using the suite as the free hook for a paid product. The typed compaction shim is harder to sell as a standalone product. It is a small library that each harness can absorb in one PR, and OWASP Agent Memory Guard already occupies the adjacent defense category. It works better as the reference fix that ships alongside the probe.

• OpenAI Alignment: Self-generated prompt injections in compaction summaries

  A lab research report, not a product. OpenAI found rare cases where a model wrote jailbreak-style instructions, such as a fake breach alert, into its own compaction summary. It confirms the failure mode is real, and that frontier labs watch for it internally rather than with an external harness.

• Straiker analysis of the Claude Code source leak

  An agent-security vendor arguing that the compaction prompt tells the model to preserve user messages, so injected content can come back as user direction. It is the clearest public statement of this exact thesis. Straiker sells runtime agent security, so it could easily turn this into a test.

• MPBench (memory poisoning attacks in LLM agents)

  An academic benchmark that splits memory poisoning into write and retrieval phases across sessions. It runs on OpenClaw and Hermes. It targets long-term memory stores rather than in-session compaction, and I could not confirm whether the code is released.

• AgentDojo

  An open-source, extensible prompt-injection eval for tool-using agents, with 629 security cases. It does not cover compaction or summary promotion, but it is the obvious framework someone could extend to do so.

• OWASP Agent Memory Guard

  An open-source defense layer that screens agent memory reads and writes. It is closer to the typed shim idea than to the probe, but it targets persistent memory, not compaction role handling.

• MemShield-bench

  A Hugging Face dataset of poisoned memory entries (AgentPoison and MemoryGraft styles). It could be used as a payload corpus, but it is not a harness-level compaction test.

• promptfoo LM Security DB (memory tag, GhostWriter)

  Catalogs attacks where untrusted content gains legitimacy after agent summarization. promptfoo is the most likely incumbent to ship a compaction red-team plugin, but I found no such plugin.

• aidevops compaction replay hardening issues

  A GitHub issue showing that literal boundary strings in user or tool text can spoof compaction replay boundaries. The proposed fix is per-render nonces. It is evidence that harness maintainers are patching this one project at a time, with no shared test suite.

Checked 2026-10-22