Guide

Retrieval and answers

Filter what retrieval returns down to what the person asking may see, tell AgentFox which sources it can trust, and make the agent say it does not know instead of guessing.

When to use this

Use it when an agent answers people from documents or records it reaches with its own service identity. That identity can usually read far more than any one caller is entitled to, so every permission check passes and the answer still contains something the caller should not see. The same agent will also answer from a stale wiki page as confidently as from the system of record, and will invent a number for a question its data does not cover.

Three controls address those three failures. You can adopt them one at a time:

The examples below were run in order against one fresh state directory. The agent is support-triage; it answers from a support knowledge base, and its index also happens to contain finance and HR documents.

Who may see what

Entitlement is decided per request, for the person asking, not for the agent. You register that person as a principal, grant groups (or single subjects) access to resource patterns, and pass retrieved chunks through a filter before the model sees them. Anything without a matching grant is withheld: the native engine is default-deny.

1. Register the people the agent acts for

   The subject is whatever stable identifier your identity provider gives you, such as an OIDC sub or an employee id. Groups are what grants refer to.

   agentfox declare principal ana@example.com --groups support --display "Ana Ruiz"
   agentfox declare principal raj@example.com --groups finance,hr

   ✓ principal ana@example.com
     groups: support
   ✓ principal raj@example.com
     groups: finance,hr

   --clearances lists the restricted classes a person may see (below). Running the command again for the same subject updates it.

2. Grant resource patterns

   A resource is matched against each chunk's source (or its id when there is no source). Patterns are shell-style globs. A grant is for a group unless you pass --kind subject.

   agentfox permit user "kb/support/*" support
   agentfox permit user "finance/*" finance
   agentfox permit user "finance/board/*" finance --classes mnpi
   agentfox permit user "hr/*" hr --purposes payroll

   ✓ support → kb/support/*
   ✓ finance → finance/*
   ✓ finance → finance/board/*
     carries mnpi — needs a matching clearance
   ✓ hr → hr/*

   • --classes marks the resources as a restricted class. The restricted classes are mnpi, legal_hold, blackout, insider and pii_sensitive. A grant alone never discloses them; the principal also needs that class in --clearances. The class applies to every grant pattern that matches, so the broader finance/* grant does not open up board minutes.
   • --purposes limits a resource to the purposes you list. A request that states a different purpose is refused. A request that states no purpose is not checked for it.

3. Filter retrieval before the model sees it

   POST /api/entitlement/filter takes the subject, the candidate chunks and, optionally, the purpose, agent and trace id. It returns the chunks this person may see and records what it withheld. Call it between retrieval and generation: once the answer is written, the only option left is to refuse to send it.

   {
     "subject": "ana@example.com",
     "agent": "support-triage",
     "chunks": [
       {"source": "kb/support/password-reset.md", "text": "To reset a password, open Settings > Security."},
       {"source": "finance/board/q3-minutes.md", "text": "The board approved the acquisition of Northwind."},
       {"source": "hr/salaries.csv", "text": "employee,band,salary"}
     ]
   }

   curl -s -X POST http://127.0.0.1:8080/api/entitlement/filter \
     -H "Authorization: Bearer $AGENTFOX_TOKEN" \
     -H "Content-Type: application/json" \
     -d @ana.json

   {
       "chunks": [
           {
               "source": "kb/support/password-reset.md",
               "text": "To reset a password, open Settings > Security."
           }
       ],
       "principal": "ana@example.com",
       "candidates": 3,
       "visible": 1,
       "withheld": 2,
       "reasons": {
           "not_entitled": 2
       },
       "over_permission": 0.6667,
       "withheld_sources": [
           "finance/board/q3-minutes.md",
           "hr/salaries.csv"
       ]
   }

   The same three chunks for Raj, who is in finance and hr but has no mnpi clearance, with "purpose": "support" added to the body:

   {
       "chunks": [],
       "principal": "raj@example.com",
       "candidates": 3,
       "visible": 0,
       "withheld": 3,
       "reasons": {
           "not_entitled": 1,
           "restricted:mnpi": 1,
           "purpose_limitation": 1
       },
       "over_permission": 1.0,
       …
   }

   Use the chunks field as your model context. The other fields say why each chunk was removed. The token is an operator token; see the gateway guide for how to start the server and mint one.

4. Check the answer as well (Python)

   If you use agentfox.auto(), pass the caller and the retrieved chunks on the model call. These keyword arguments are removed before the provider sees them. After the model answers, AgentFox runs the same filter and raises a critical finding if the answer quotes a chunk the caller was not entitled to. When the decision enforces, the answer is also withheld. This is the safety net for a retriever that skipped the pre-filter.

   import agentfox
   import mock  # an offline stand-in for the OpenAI client
   
   agentfox.auto(agent="support-triage", mode="observe")
   client = mock.client()  # in your code: OpenAI()
   
   chunks = [
       {"source": "kb/support/password-reset.md", "text": "To reset a password, open Settings > Security."},
       {"source": "finance/board/q3-minutes.md", "text": "The board approved the acquisition of Northwind."},
   ]
   reply = client.chat.completions.create(
       model="gpt-4o-mini",
       messages=[{"role": "user", "content": "Any news about Northwind?"}],
       agentfox_principal="ana@example.com",  # who is asking
       agentfox_chunks=chunks,                # what retrieval returned
   )
   print(reply.choices[0].message.content)

   agentfox findings

    id         severity  type                    what
    …yq8rmjpd  critical  entitlement_disclosure  the answer contains content from
                                                 'finance/board/q3-minutes.md', which
                                                 'ana@example.com' is not entitled to see
   
     1 open finding(s).

5. Read the over-permission number

   agentfox report entitlement

   3 request(s) checked · 3 withheld something · 2 principal(s) · 75.0% of retrieved content was withheld
     not_entitled          4
     restricted:mnpi       1
     purpose_limitation    1
   
     That share is what the agent could reach and the caller could not. It is the oversharing number,
   not an error rate.

   This is the share of what retrieval returned that the caller could not see. Only requests where something was withheld are recorded, so a filter call that removed nothing does not appear in the count or the ratio. --days sets the window (default 7). With no decisions recorded yet, the command says so and points you at agentfox declare principal.

In the web app Access control: principals, grants and withheld content

Which sources to trust

A source tier says how far an answer from that source can be trusted. Tiers, from most to least authoritative: system_of_record, approved, unverified, external. A source you have not registered is treated as unverified, never as approved. The key must be exactly what your retriever emits as a chunk's source. Keys are matched exactly, not as globs.

1. Register sources one at a time

   agentfox declare source kb/support/password-reset.md --tier system_of_record \
     --owner support-ops --domain support --sla-hours 720 --updated now
   agentfox declare source wiki/notes/reset-tips.md --tier unverified --domain support

   ✓ kb/support/password-reset.md → system_of_record
   ✓ wiki/notes/reset-tips.md → unverified

   --sla-hours is a freshness SLA. --updated records when the source last changed, as an ISO date or now. A source with an SLA and no update time counts as stale on purpose, because its age is unknown. --domain names the corpus it belongs to; an agent answering from another domain's source is flagged.

2. Register many from a JSON file

   The file is either a list of objects or an object with a sources list. Each entry takes key (required), title, tier (default unverified), owner, domain, freshness_sla_hours and deprecated.

   {
     "sources": [
       {"key": "kb/support/billing-faq.md", "title": "Billing FAQ", "tier": "approved", "owner": "support-ops", "domain": "support"},
       {"key": "https://status.example.com", "tier": "external", "domain": "support"},
       {"key": "kb/support/legacy-billing.md", "tier": "approved", "domain": "support", "deprecated": true},
       {"key": "finance/pricing.md", "tier": "approved", "owner": "finance", "domain": "finance", "freshness_sla_hours": 24}
     ]
   }

   agentfox declare import-sources sources.json
   agentfox declare list sources

   ✓ registered 4 source(s) from sources.json
     tier                source                          owner          domain     state
     external            https://status.example.com      —              support    ok
     unverified          wiki/notes/reset-tips.md        —              support    ok
     approved            finance/pricing.md              finance        finance    age unknown
     approved            kb/support/billing-faq.md       support-ops    support    ok
     approved            kb/support/legacy-billing.md    —              support    deprecated
     system_of_record    kb/support/password-reset.md    support-ops    support    ok

   The import does not read an update time, so finance/pricing.md reads age unknown. Set it with a single declare source:

   agentfox declare source finance/pricing.md --tier approved --owner finance \
     --domain finance --sla-hours 24 --updated now

   ✓ finance/pricing.md → approved

   GET /api/sources/health lists what is stale, deprecated or unowned. Before that fix it returned:

   {
       "registered": 6,
       "stale": [
           {
               "source": "finance/pricing.md",
               "reason": "source has a freshness SLA but no recorded update time"
           }
       ],
       "deprecated": [
           "kb/support/legacy-billing.md"
       ],
       "unowned": [
           "wiki/notes/reset-tips.md",
           "https://status.example.com",
           "kb/support/legacy-billing.md"
       ],
       "healthy": 4
   }

3. Dry-run an answer against its sources

   POST /api/sources/assess takes an answer, the chunks it was built from, and optionally the agent's domain. It changes nothing. It reports the weakest tier used, deprecated, stale or off-domain sources, and citations to documents that were not retrieved.

   {
     "answer": "Open Settings > Security to reset it [kb/support/legacy-billing.md]. Billing moved to a new provider in 2024 [kb/support/billing-v2.md].",
     "chunks": [
       {"source": "kb/support/legacy-billing.md", "text": "Open Settings > Security to reset it."},
       {"source": "wiki/notes/reset-tips.md", "text": "Try clearing cookies first."}
     ],
     "agent_domain": "support"
   }

   curl -s -X POST http://127.0.0.1:8080/api/sources/assess \
     -H "Authorization: Bearer $AGENTFOX_TOKEN" \
     -H "Content-Type: application/json" \
     -d @assess.json

   {
       "weakest_tier": "unverified",
       "breaches": [
           {
               "kind": "deprecated_source",
               "source": "kb/support/legacy-billing.md",
               "reason": "'kb/support/legacy-billing.md' is marked deprecated and should not be answered from"
           }
       ],
       "fabricated_citations": [
           {
               "citation": "kb/support/billing-v2.md",
               "kind": "unknown_source",
               "reason": "'kb/support/billing-v2.md' was not among the retrieved sources"
           }
       ],
       "conflicts": [],
       "uncited_claims": [],
       "sources": [ … ],
       "clean": false
   }

   At run time the same checks run on the model's answer when agentfox.auto() is given agentfox_chunks (with or without agentfox_principal). An answer grounded in the deprecated page above produced:

    …qdjh27k7  high      source_authority        'kb/support/legacy-billing.md' is marked deprecated
                                                 and should not be answered from

4. Check a document before it enters the index

   POST /api/sources/context-check is a dry run of the ingestion and chunking quality gate. Send text (one extracted document), chunks (a list of strings as they will be indexed), or both.

   {
     "source_key": "kb/support/password-reset.md",
     "chunks": [
       "To reset a password, open Settings > Security and choose Reset.",
       "and the account stays locked for thirty",
       "minutes after five failed attempts."
     ]
   }

   curl -s -X POST http://127.0.0.1:8080/api/sources/context-check \
     -H "Authorization: Bearer $AGENTFOX_TOKEN" \
     -H "Content-Type: application/json" \
     -d @chunks.json

   {
       "chunks": {
           "count": 3,
           "findings": [
               {
                   "code": "orphan-chunk",
                   "detail": "chunk 1 is 39 characters \u2014 too short to answer anything on its own, and it will still be retrieved",
                   "severity": "warn",
                   …
               },
               {
                   "code": "split-sentence-start",
                   "detail": "chunk 1 opens mid-sentence; the subject of the clause is in the previous chunk and will not be retrieved with it",
                   "severity": "degraded",
                   …
                   "verdict": "abstain"
               },
               …
           ]
       }
   }

   It reports; it does not repair. Re-chunking or dropping the document is your call.

In the web app Verified sources: tiers, owners and freshness

When to say "I don't know"

A knowledge boundary says which systems an agent answers from, how far back its data goes, which kinds of question it answers, and which topics are out of scope. The check runs on the question, before the model is called, so an unanswerable question never produces an invented answer.

1. Declare the boundary

   Question types are fact, aggregate, prediction, opinion and procedure. --answerable defaults to fact,aggregate,procedure. A boundary starts in observe mode.

   agentfox declare boundary support-triage --systems help-center,tickets \
     --coverage-months 12 --answerable fact,procedure \
     --out-of-scope "legal advice,medical advice"

   ✓ boundary declared for support-triage
     answerable: fact, procedure
     coverage:   last 12 months
     observe mode — refusals are recorded, not applied. Re-run with --mode enforce when the dry runs
   look right.

2. Try real questions against it

   agentfox test boundary runs nothing and changes nothing. It shows the refusal a person would get. Replay real questions through it before you enforce: a false positive here is a customer being told no.

   agentfox test boundary support-triage "How do I reset my password?"
   agentfox test boundary support-triage "Will ticket volume go up next quarter?"
   agentfox test boundary support-triage "What caused the outage in 2019?"
   agentfox test boundary support-triage "Can you give me legal advice about my contract?"
   agentfox test boundary support-triage "How many tickets were closed last week?"

   answerable  (procedure)
   ╭─ would abstain — unknowable ─────────────────────────────────────────────────────────────────────╮
   │ That asks for a projection rather than a recorded fact. I can only report what is in             │
   │ help-center, tickets, so I don't have an answer for it.                                          │
   ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
     recorded only (observe mode)
   ╭─ would abstain — out_of_coverage ────────────────────────────────────────────────────────────────╮
   │ I hold data from 2025-10-10 onwards, and that question is about 2019. Data not available for     │
   │ that period.                                                                                     │
   ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
     recorded only (observe mode)
   ╭─ would abstain — out_of_domain ──────────────────────────────────────────────────────────────────╮
   │ That topic is outside what this agent is set up to cover (legal advice).                         │
   ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
     recorded only (observe mode)
   ╭─ would abstain — unsupported_question_type ──────────────────────────────────────────────────────╮
   │ That's an aggregate question, and this agent is set up to answer fact or procedure questions     │
   │ from help-center, tickets.                                                                       │
   ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
     recorded only (observe mode)

   The title of each panel is the abstention kind. The fifth kind, out_of_scope_entity, only applies when the caller supplies the identifiers it holds (known_entities on the HTTP check, below) and the question names an identifier-shaped token, such as TKT-991, that is not among them. Without that list, entity scope is never guessed.

3. Enforce, and handle the abstention

   agentfox declare boundary support-triage --systems help-center,tickets \
     --coverage-months 12 --answerable fact,procedure \
     --out-of-scope "legal advice,medical advice" --mode enforce

   Re-declare with every option, not only --mode: the command writes the whole boundary. In agentfox.auto()'s default "policy" mode, an enforced abstention raises agentfox.Blocked before the provider is called. The verdict is abstain, and result.content is the sentence to show the user.

   import agentfox
   import mock
   
   agentfox.auto(agent="support-triage", quiet=True)  # mode="policy", the default
   client = mock.client()
   
   try:
       client.chat.completions.create(
           model="gpt-4o-mini",
           messages=[{"role": "user", "content": "Will ticket volume go up next quarter?"}],
       )
   except agentfox.Blocked as exc:
       print("verdict:", exc.result.verdict)
       print("rule:   ", exc.result.rules_fired[0]["rule_id"])
       print("say:    ", exc.result.content)

   verdict: abstain
   rule:    answerability.unknowable
   say:     That asks for a projection rather than a recorded fact. I can only report what is in help-center, tickets, so I don't have an answer for it.

   Over HTTP, POST /api/answerability/check gives the same verdict as JSON, and GET /api/answerability/report puts abstentions next to over-refusals:

   curl -s -X POST http://127.0.0.1:8080/api/answerability/check \
     -H "Authorization: Bearer $AGENTFOX_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"agent": "support-triage", "question": "What caused the outage in 2019?"}'

   {
       "agent": "support-triage",
       "question_type": "fact",
       "boundary_declared": true,
       "answerable": false,
       "abstention_kind": "out_of_coverage",
       "reasons": [
           {
               "check": "temporal",
               "in_scope": false,
               "asked_about": "2019",
               "earliest": "2025-10-10",
               "reason": "asked about 2019; coverage starts 2025-10-10"
           }
       ],
       "response": "I hold data from 2025-10-10 onwards, and that question is about 2019. Data not available for that period.",
       "mode": "enforce",
       "should_abstain": true
   }

   With the identifiers the caller holds:

   curl -s -X POST http://127.0.0.1:8080/api/answerability/check \
     -H "Authorization: Bearer $AGENTFOX_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"agent": "support-triage", "question": "What is the status of ticket TKT-991?", "known_entities": ["TKT-100", "TKT-200"]}'

   {
       …
       "answerable": false,
       "abstention_kind": "out_of_scope_entity",
       …
       "response": "I can't find TKT-991 in help-center, tickets. Rather than guess, I'd rather tell you it isn't there.",
       "mode": "enforce",
       "should_abstain": true
   }

In the web app Agents → an agent → Knowledge boundary

Where these show up

AgentFox checks content per surface (see Detectors and findings). Retrieved content enters on the retrieved surface and is tainted, so text from a document cannot authorise a tool call however it is phrased. The controls on this page attach at these points:

• Before generation (input): the knowledge boundary. An abstention is the abstain verdict, with rule id answerability.<kind>.
• Between retrieval and generation: the entitlement filter, which you call. Withheld chunks are recorded for report entitlement.
• After generation (output): entitlement_disclosure (critical) when the answer quotes a withheld chunk; source_authority for deprecated, stale or off-domain sources; boundary_breach when an answer goes outside the declared boundary anyway.

What can go wrong

• unknown principal 'zoe@example.com'. Register it first. The HTTP filter refuses a subject it does not know. Register every caller with declare principal, or from your sign-in path with PUT /api/entitlement/principals.
• unknown user 'admin@example.com'. Send X-AgentFox-User or a nom_api_ bearer token. The request had no operator token. See the gateway guide.
• No finding from auto(). Pass the passages as agentfox_chunks: the source checks run on them with or without a principal. An agentfox_principal that is not registered is evaluated as that subject with no groups or clearances, so it sees only what is granted to the subject directly, and what it could not see is recorded against it. Register it (agentfox declare principal) to give it its groups.
• A source reads age unknown or stale right after import. It has an SLA and no update time. Run declare source … --updated now when it changes.
• tier must be one of: system_of_record, approved, unverified, external. No other tier names exist.
• unknown question type(s): ['rumour']. Use fact, aggregate, prediction, opinion or procedure.
• unknown agent 'research-bot'. The boundary commands need a registered agent. An agent is registered the first time it runs under agentfox.auto(agent=…).
• A question you expected to be refused is answerable. Question typing is lexical: "Which plan is better for me?" was read as fact, not opinion. It is precise when it fires and misses many questions. Test with your own traffic, and see Benchmarks.

Limits

• You register principals yourself. There is no Okta, Entra or other IdP integration. Principals and their groups are whatever you send.
• Only the native engine works. The OpenFGA adapter is a declared seam, not an implementation; configured, it raises instead of filtering. Grants live in AgentFox.
• No catalog ingestion. Sources are not read from DataHub, OpenMetadata or Unity Catalog. You tier them with the commands above.
• Context integrity is partial. The ingestion and chunk checks report problems. They do not repair chunk boundaries or re-extract a corrupt document.
• The output-side entitlement check matches quotes, not paraphrases. In enforce mode it withholds an answer that quotes a withheld chunk; an answer that restates it in other words is not caught. The pre-filter is what prevents the disclosure.
• Current status for each area is in Limits.