Each repository was mapped with agentfox init, and its own agents, tools and handoffs drove the traffic: single turns, five-turn sessions and multi-agent handoffs, through the same guard endpoints an SDK calls.
1. Per check and per agent turn
Tool calls are the cheapest step in every layer, 2.4 ms at the median with the default detectors. Text the classifier must read (inputs, tool results, replies) carries the cost. A turn is every guard call it makes, summed: four for a single turn.
| Layer | What runs | Per check, p50 / p95 (ms) | Per single-agent turn, p50 (ms) |
|---|---|---|---|
| Deterministic | Patterns, PII, secrets, schema, unicode and code checks | 2.1 / 6.0 | 8.0 |
| Default | Deterministic + ML prompt-injection classifier | 49.6 / 260 | 156 |
| Extended | Default + similarity search against known attacks | 9.4 / 262 | 73.9 |
| With an LLM judge | Extended + injection and PII judged by gpt-4.1-mini | 68.7 / 949 | 1,017 |
Messages under 200 characters take 49.0 ms at the median and 60.3 ms at p95 with the default detectors, and none runs out of time. Between 200 and 600 characters, 22.3% reach the 300 ms check budget before the classifier finishes on a CPU with no GPU; those checks return without its verdict and say so. With judges on, a judge waits inline only when an enforcing rule relies on it, the default, so most checks do not wait for it; replies pay the commitment check, about 0.85 s.
Source: benchmarks/latency/results/headline.json, from benchmarks/latency
2. Head to head on one machine
Every system ran on the same 880 messages, on the same machine, on CPU, under the same Python, one request at a time, with AgentFox at each of its levels. Base-size models (AgentFox's classifier, the one in LLM Guard's scanner, Prompt Guard 2 86M) all take about 50 ms; AgentFox adds an HTTP round trip and its deterministic checks, 6.9 ms on their own.
| System | Detects | Short messages, p50 / p95 (ms) | Long messages, p50 / p95 (ms) |
|---|---|---|---|
| AgentFox deterministic | PII, secrets, injection patterns, unsafe content, schema, unicode, insecure code | 6.9 / 9.7 | 10.1 / 15.7 |
| AgentFox default | The above, plus injection and jailbreaks by classifier | 58.8 / 72.1 | 107 / 275 |
| AgentFox default, cache on | Same | 55.3 / 61.9 | 103 / 273 |
| AgentFox extended | The above, plus similarity to known attacks | 59.4 / 70.5 | 111 / 274 |
| AgentFox with LLM judges, default mode | The above, plus injection and PII judged by gpt-4.1-mini when a rule relies on it | 64.9 / 94.9 | 114 / 274 |
| AgentFox with LLM judges always inline | Same, judged on every check | 838 / 1,057 | 846 / 1,122 |
| LLM Guard, prompt-injection scanner | Prompt injection | 52.2 / 59.7 | 87.2 / 199 |
| Meta Prompt Guard 2 86M | Prompt injection and jailbreaks | 51.7 / 60.5 | 85.9 / 213 |
| Meta Prompt Guard 2 22M | Prompt injection and jailbreaks | 23.9 / 30.2 | 45.9 / 102 |
| Guardrails AI, DetectJailbreak | Jailbreaks | 99.4 / 107 | 149 / 325 |
| LLM Guard, nine default scanners | Injection, PII, secrets, toxicity and five more | 249 / 316 | 348 / 650 |
| OpenAI omni-moderation (hosted) | Harm categories, not injection | 410 / 547 | 413 / 521 |
| NeMo Guardrails, self-check (gpt-4.1-mini) | One LLM policy check | 738 / 908 | 735 / 908 |
On long messages AgentFox reads the first 4,000 characters, where the others stop at 512 tokens, so it does more work: on CPU 55 of 150 long inputs ran past the 300 ms check budget before the classifier finished. In its default mode a judge waits only when a rule relies on it; with every judge inline, AgentFox is about 100 ms slower than NeMo's single self-check, mostly because its judge answers three questions where NeMo asks one.
Source: benchmarks/latency/head_to_head/results/summary.json
3. What moves latency, and what does not
- Message length moves it. The classifier reads text in 256-token windows, so a 500-character tool result costs several times a short question.
- Session length does not. Per-check latency is flat from the first turn of a session to the fifth.
- Framework and repository shape barely do. Single-agent, independent-agent and multi-agent repositories, and LangGraph, CrewAI, AutoGen, the OpenAI Agents SDK, ADK and the rest, land within a few milliseconds of each other per check.
- Where it runs does. A hosted guard pays the network every call; from Bengaluru to a US region that is about 260 ms before the guard does anything. Run the gateway beside the agent.
4. What other vendors publish
Few vendors publish latency, and fewer say under what conditions. Azure Prompt Shields, AWS Bedrock Guardrails, Google Model Armor, OpenAI Moderation, Llama Guard and Guardrails AI publish none. These are the figures we found, as their authors state them:
| Product | Published latency | Conditions |
|---|---|---|
| Protect AI LLM Guard, injection scanner | 104 ms mean / 107 ms p95 (ONNX) | AWS m5.xlarge CPU, 384-token input |
| Meta Prompt Guard 2 | 86M: 92.4 ms; 22M: 19.3 ms | A100 GPU, 512 tokens |
| Lakera Guard | p95 50 ms to 300 ms | 1 to 16 detectors, about 1k to 5k characters |
| NVIDIA NeMo Guardrails | about 0.5 s added per chat turn | Three rails, GPU microservices |
| Palo Alto Prisma AIRS | about 700 ms added | Network firewall intercept mode |
5. Method and limits
- 119 agent repositories were examined; 109 produced an agent map and were tested, 76 public, 26 held out from the mapper work and 7 local, four of them private and anonymised.
- Apple M1 Pro, CPU only, one gateway worker on SQLite, Python 3.12 as in the production image. A server with more workers and a server database will do better; read these as a floor.
- Engine time is the gateway's own time per check. The agent's model and the network are excluded, except where a row says otherwise.
- The default layer is reported with the score cache off: the attack suite repeats text, and the cache would flatter it.
- This measures speed. What each guard catches is on the security benchmarks page.
6. How to reproduce it
python -m benchmarks.latency.fetch_repos var/latency/repos python -m benchmarks.latency.map_repos var/latency/repos var/latency python -m benchmarks.latency.run_latency var/latency/maps var/latency/run python -m benchmarks.latency.report var/latency var/latency/results run
Every recorded call, the scripts and the head-to-head harness are in benchmarks/latency.