Coverage
What an agent can get wrong, and what we catch
116 ways an agentic request can fail, built from the architecture of the request rather than from our own feature list. 105 of them are executed against the running product every night; the rest say plainly that they were assessed by reading the code.
Where the request can fail
Each layer is a stage the request actually travels through. The score counts a partial control as half, because most of the honest answers are partial.
What we do not catch
Published for the same reason the rest is: a coverage page that only listed successes would not be evidence of anything. 3 scenarios have no control at all, and 15 have a partial one.
- Invalid logical inferenceL0 model-intrinsic
- Context stuffing to push out the system promptL1 input and prompt
- An instruction in an image, or a scanned PDFL2 retrieval and context
What it costs, and what happens when it fails
Latency and false positives are the most-cited reasons guardrails get switched off. Every control here runs inside a declared budget and records its own degradation per decision, so "the check was down" is a fact in the record rather than a thing nobody noticed.
Four controls may not be configured to fail open
Not a setting with a safe default — a refusal. Their failure mode is a disclosure rather than an outage, and "we allowed it because the check was down" is not a defensible answer.
- Tenant isolation
- One customer's data reaching another is not an outage.
- Entitlement filter
- Same.
- Data access scope
- A query across every customer's rows reads as ordinary.
- Audit chain
- A gap in the record is the thing the record exists to prevent.
Generated by scripts/probe/run.py --json from the taxonomy the nightly harness executes. A row marked verified has fired at least once against the real product; a row that is not executable says so rather than borrowing the others' credibility. The full table, with the control behind each row, is in docs/coverage-map.md.
Try to break it before you trust it
No account, no install, and the same enforcement code as the product.
pip install agentfox agentfox init && agentfox demo
Offline: no API key, no downloaded weights, no network egress.