Phish Drill: adaptive phishing tests for agents that read your email

I didn't find a product that does exactly this: a hosted service you can sign up for that sends phishing to a real test inbox connected to your email agent and logs its replies, forwards and link fetches. The nearby options go wrong ways. Straiker and Promptfoo red-team AI agents in general and aren't built around email. Cimento and Penligent are moving toward testing agents, but they sell to enterprise security teams. KnowBe4, Hoxhunt and Adaptive Security still phish only humans. The academic benchmarks (LLMail-Inject, AgentPhishing, LivePI, AgentRedBench) show the problem is real; in one benchmark, four of nine agents fell for most phishing. But they're research tools, not services. The hosted run is where a newcomer could win. It would need fast self-serve onboarding for n8n, Zapier and Gmail or Outlook agents, real external behavior captured through tracked links and monitored forward addresses, and business-fraud scenarios like invoice bank-detail changes and wire requests that the benchmarks mostly skip. It should also produce a plain pass/fail report a small firm can understand before switching on an assistant. The open-source CI scenario pack is weaker as a business. Microsoft's LLMail-Inject data and the academic benchmarks already give away a lot of attack content, and Promptfoo already has the CI test harness. It works better as a free funnel into the hosted product than as something to charge for. The main risks are that Promptfoo or a phishing-simulation vendor adds an email-agent mode, and that small firms may not pay before an incident.

• Promptfoo (indirect prompt injection red-team plugin)

  An open-source red-teaming CLI that runs in CI. Its indirect injection plugin points generated attacks at a variable you choose, such as an email body. It is a general tool, though: it has no phishing scenarios for email, does not use a real inbox, and has no business-fraud cases like changed bank details. It is the most likely substitute for the CI scenario pack.

• Straiker Ascend AI

  A well-funded enterprise product that red-teams agentic apps. Its attack agents map prompts, tool calls and data flows, and it plugs into CI/CD. It is aimed at large enterprises, not n8n or Zapier builders, and I found nothing about email or phishing specifically.

• Cimento AI

  A human-risk and phishing simulation startup that came out of stealth in 2026. A third-party directory says it also keeps an inventory of a company's AI agents and tests them with authority impersonation and boundary probing. It is the closest example of a phishing-simulation vendor moving toward agents, but I could not confirm this on the vendor's own site.

• Penligent

  An AI pentesting vendor. It markets safe re-enactment of email-borne prompt injection and social engineering against AI agents in a customer's environment to find data-exfiltration paths. The claims come from its own marketing.

• LLMail-Inject (Microsoft dataset and challenge)

  An open, MIT-licensed set of over 200,000 adaptive attack emails against an LLM email assistant, labeled with outcomes for retrieval, detection and tool calls, including data exfiltration. It is free raw material that overlaps with the scenario-pack idea.

• AgentPhishing benchmark

  An academic benchmark of LLM email agents under adaptive phishing. It scores detection rate and compliance (clicking links, supplying information, writing unsafe drafts). In one evaluation of nine agents, four detected under 10% of phishing and complied over 45% of the time. It is close to the proposed scoring method but is research, not a product.

• LivePI / AgentRedBench / DTap

  Academic benchmarks and red-team platforms from 2026. LivePI plants attacks in a monitored Gmail inbox. AgentRedBench (arXiv 2606.02240) covers Gmail among 24 integrations. DTap (arXiv 2605.04808) includes fake approval threads for payouts. More open scenario sources.

• huntoai/phishing-ai-agent

  An open-source agent that writes phishing emails and sends them over SMTP. It targets human employees, not AI agents, and seems inactive, but it could be reused as an attack generator.

Checked 2026-10-23