Measured against
the guardrails you use today

Attacks on a tool-using agent

AgentDojo (ETH Zurich) is the standard benchmark for prompt injection against agents that call tools. We replayed its 97 user tasks, and its 588 attacks that try to make the agent write, send or pay something it shouldn't.

LLM Guard reads each tool output and decides whether it holds an injection. Most of its 86 misses are attacks past the first 512 tokens of an output, which is all its model reads. AgentFox doesn't have to spot the injection. It tracks where each argument of a tool call came from, and a payment or email whose details came from a tool output goes to a person first.

LLM Guard can close the gap. In its long-input mode, with a threshold picked on this test set, it stops all 588 at the same 24 tasks, but flags 38% of harmless tool outputs on the way.

With every detector switched off

Once attackers can adapt, published defences fail more than 90% of the time (Nasr, Carlini et al., 2025). So we tested what happens after the detectors have already lost: eight attacks, one for each way AgentFox stops a call, with detection disabled.

What containment costs

Stopping every attack has a price. By default, three in four normal AgentDojo tasks needed a person to approve a step. That is a setting, and these are the trade-offs on the same replay.

Session-level taint treats everything after the agent reads a tool output as untrusted. It stops every attack, and most legitimate payments and messages with them. Argument-level taint checks each argument. It lets more work through and misses 61 attacks: short values like a file id of "13", and attacker text buried inside a longer argument.

Most of what's left can't be fixed by provenance alone. Paying a bill means copying an IBAN out of a document, which is exactly what an attack that redirects a payment does.

Detection accuracy

The detectors are the second layer. These are their numbers on held-out data that nothing was tuned on.

Held-out attacks are 116 examples from deepset/prompt-injections, scored with no false alarms on that split. Harmless look-alikes are NotInject's 339 prompts, written to use the same words attacks do. Prompt Guard 2 86M does better on SPML and flags less harmless text. PIGuard catches the most, so it stays the default.

A second model can run beside PIGuard. With it on, 41% of the look-alikes are flagged, so its hits are recorded and never block: an enforcing rule acts on PIGuard alone.

Attacks a text scanner can't see

A prompt-injection scanner checks one string at a time. These attacks only show up across messages, inside tool arguments, or in what the agent is allowed to do. LLM Guard was installed and called for real.

LLM Guard is more precise on attacks hidden in tool output, with 2 false alarms on the harmless half against our 5. The permission scenario AgentFox got wrong was an ordinary ticket update held for approval, because the demo declares that updating a ticket sends an email.

An attacker who adapts

Here the attacker sees our verdict and rewrites the attack, up to 50 times. It starts only from attacks our detectors already stop, so every success is a real bypass.

With rewrites a model could still read (paraphrases, other languages, no encoding), three in four attacks got past the detectors within 50 tries. With encodings allowed, all of them did within 25. Detectors alone don't hold, which is why the next check happens at the tool call.

Rerun it yourself

Each benchmark is one script with its own throwaway database, and it writes the results file quoted here. A number that doesn't match is a bug.

The LLM Guard comparisons need LLM_GUARD_VENV_PYTHON pointing at a separate environment, because LLM Guard pins an older transformers. None of these runs needs a model API key. Each figure is a single run, and the run dates, hardware and LLM Guard version weren't recorded.

Latency: what a check adds to your agent What AgentFox doesn't catch yet