Guard Polyglot: test whether your agent guard changes its verdict by language

No product I found takes a team's own labelled agent-step test set, translates it, and reports where the guard's verdict flips against English along with the worst examples. Promptfoo and DeepTeam translate attack prompts and score each language, and MrGuard defines the consistency metric in research, but none treats flip rate as the main output for agent guards or controls translation quality. A new entrant could win with a narrow, developer-friendly tool. It would run on real agent traces and tool calls, use back-translation or a judge to throw out bad translations so they don't count as guard failures, show flips per language and per category, and give a simple CI threshold. The main risk is that promptfoo adds a flip report as a small feature, and a standalone CLI may be hard to charge for. Revenue more likely comes from hosted history and drift tracking, stronger translation for languages with little training data, and compliance reports for teams in regulated markets outside English.

• promptfoo multilingual strategy

  Open-source eval and red-team tool already used in CI. Its multilingual strategy translates test inputs into chosen languages (default Bengali, Swahili, Javanese) and reports metrics for each language. It doesn't report verdict flips against English or check translation quality, but it is the most likely to absorb this idea.

• DeepTeam Multilingual attack

  Confident AI's open-source red-teaming library has a Multilingual attack that translates attacks into less-spoken languages to get past filters. It is attack-focused and doesn't compare verdicts on a labelled set.

• garak-multilingual (third-party garak plugin)

  Community plugin that adds multilingual prompt-injection and jailbreak probes to NVIDIA's garak scanner. It is a probe library, not a tool for checking a guard's consistency.

• MrGuard (academic)

  Paper with code that defines cross-lingual consistency as how often a guard's verdict agrees between English and other languages on the XSafety parallel set. This is almost exactly the flip-rate metric, but as research rather than a product.

• Benchmarking LLM Guardrails in Handling Multilingual Toxicity

  Academic study that machine-translated guard test sets into 8 languages and measured Llama Guard and others per language. It is a one-off benchmark, not a tool.

• Mozilla.ai any-guardrail and multilingual evaluation

  Open-source single interface for many guard models, used to compare English and Farsi guard behavior. It also warns that judges can make cross-language results look misleadingly consistent.

• Lakera PINT benchmark

  Prompt-injection detector benchmark with about 1,300 non-English inputs. It is a fixed dataset, not a tool for translating your own test set.

• ML-Bench / ML-Guard

  May 2026 benchmark of 14 languages built from regional regulations, comparing 11 guardrail baselines. It shows research interest in multilingual guard evaluation.

Checked 2026-10-17