In CI and in production

Fail a pull request when an agent gets weaker, and keep checking it once it is live.

Before you start

Suites and baselines live in the AgentFox database, so the CI job sets AGENTFOX_DATABASE_URL (a secret) to the database you created them in.

Fail the build when an attack gets through
test redteam exits 1 when any attack got through. With --adaptive it also retries blocked attacks in disguised forms (encodings, role-play, look-alike tool names) and compares with the last run: WEAKER, STRONGER or UNCHANGED.
bash
```
agentfox test redteam support-triage --adaptive
```
The first adaptive run is the baseline. Keep the probe list and --budget the same between runs, or each run starts a new baseline.
Keep a test suite
A suite is a set of prompts and what a good answer contains. Create one on Testing → Test suites, then add cases on its page. To turn a real failure into a case, paste its trace id (trc_…) under Promote a trace.
Pin a baseline, then gate on it
Run the suite once with your real model and save that run as the baseline. In CI, test gate runs it again and exits 1 if a score drops by more than 0.05, or if any case errored.
.github/workflows/agent-tests.yml (steps)
```
- run: pip install "agentfox[postgres]"
- run: agentfox test redteam support-triage --adaptive
- run: agentfox test gate support-quality --provider openai --model gpt-4o-mini --junit reports/junit.xml --sarif reports/agentfox.sarif
```
Save the baseline with agentfox test baseline RUN_ID. The CI tab on Testing shows these commands with your server filled in. A real provider needs AGENTFOX_ALLOW_EGRESS=true and its key.
Score real traffic against a target
On Testing → Reliability, Score real traffic grades recent runs with the same scorers as a suite. Add target sets a goal for one agent and one measure, such as 95% of answers grounded over 7 days, and shows how much budget is left.
bash
```
agentfox test online support-triage
agentfox report drift support-triage --scorer groundedness
```

Probe a live agent

AgentFox can send a small, fixed set of attacks (9 attacks and 2 ordinary questions) to a deployed agent's endpoint on a schedule. When an attack that was stopped last time gets through, it opens an issue; when a later run stops it again, it closes the issue.

Set it up over the API: register the endpoint (POST /api/probes/targets, created off), then opt in with the exact warning text from GET /api/probes/warning. The opt-in is recorded on the audit trail. See the probes routes.
Your endpoint takes {"message": …} and answers with the reply, any tool_calls, and blocked.
At most one scheduled run an hour and one manual run every ten minutes. Only the host you opted in for is contacted. AGENTFOX_LIVE_PROBES_ENABLED=false stops every target.

If something goes wrong

You see	Do this
`unknown suite` in CI	`AGENTFOX_DATABASE_URL` points at a different database from the one you created the suite in.
`GATE FAIL` with errored cases	The model was not reached. Check egress and the provider key.
"No previous adaptive campaign" on every run	The probe list or `--budget` changed. Keep them fixed.
A target reads "No data"	Press Score real traffic for that agent first.

Before you start

Suites and baselines live in the AgentFox database, so the CI job sets AGENTFOX_DATABASE_URL (a secret) to the database you created them in.

Fail the build when an attack gets through

test redteam exits 1 when any attack got through. With --adaptive it also retries blocked attacks in disguised forms (encodings, role-play, look-alike tool names) and compares with the last run: WEAKER, STRONGER or UNCHANGED.

bash

agentfox test redteam support-triage --adaptive

The first adaptive run is the baseline. Keep the probe list and --budget the same between runs, or each run starts a new baseline.

Keep a test suite

A suite is a set of prompts and what a good answer contains. Create one on Testing → Test suites, then add cases on its page. To turn a real failure into a case, paste its trace id (trc_…) under Promote a trace.

Pin a baseline, then gate on it

Run the suite once with your real model and save that run as the baseline. In CI, test gate runs it again and exits 1 if a score drops by more than 0.05, or if any case errored.

.github/workflows/agent-tests.yml (steps)

- run: pip install "agentfox[postgres]"
- run: agentfox test redteam support-triage --adaptive
- run: agentfox test gate support-quality --provider openai --model gpt-4o-mini --junit reports/junit.xml --sarif reports/agentfox.sarif

Save the baseline with agentfox test baseline RUN_ID. The CI tab on Testing shows these commands with your server filled in. A real provider needs AGENTFOX_ALLOW_EGRESS=true and its key.

Score real traffic against a target

On Testing → Reliability, Score real traffic grades recent runs with the same scorers as a suite. Add target sets a goal for one agent and one measure, such as 95% of answers grounded over 7 days, and shows how much budget is left.

bash

agentfox test online support-triage
agentfox report drift support-triage --scorer groundedness

Probe a live agent

Set it up over the API: register the endpoint (POST /api/probes/targets, created off), then opt in with the exact warning text from GET /api/probes/warning. The opt-in is recorded on the audit trail. See the probes routes.

Your endpoint takes {"message": …} and answers with the reply, any tool_calls, and blocked.

At most one scheduled run an hour and one manual run every ten minutes. Only the host you opted in for is contacted. AGENTFOX_LIVE_PROBES_ENABLED=false stops every target.

If something goes wrong

You see

Do this

unknown suite in CI

AGENTFOX_DATABASE_URL points at a different database from the one you created the suite in.

GATE FAIL with errored cases

The model was not reached. Check egress and the provider key.

"No previous adaptive campaign" on every run

The probe list or --budget changed. Keep them fixed.

A target reads "No data"

Press Score real traffic for that agent first.