Guide

Traces and integrations

Connect AgentFox to the tools your team already watches: OpenTelemetry spans in, Langfuse and LangSmith runs joined to governance decisions, Prometheus metrics, SIEM export, and signed webhooks when a finding is raised or resolved.

When to use this

• Your agents already emit OpenTelemetry and you want them in the agent registry, with shadow-agent detection, without changing code.
• An engineer is looking at a Langfuse or LangSmith trace and needs to know which AgentFox decision shaped it.
• Your on-call alerts come from Prometheus, your security team works in a SIEM, or a ticketing system should open an issue when AgentFox finds something.

Before you start: the server and its authentication

Every integration here goes through the control-plane API. Start it with agentfox serve api (default 127.0.0.1:8080; see the gateway guide and Self-hosting), then check how it authenticates:

agentfox serve api --port 8080
agentfox admin auth status

╭─ Authentication: development mode ───────────────────────────────────────────────╮
│ The X-AgentFox-User header is accepted.                                          │
│                                                                                  │
│ environment = development · auth_mode = auto                                     │
│ Anyone who can reach this port is any user they name. That is fine for local     │
│ work and unacceptable anywhere else.                                             │
│                                                                                  │
│ Set AGENTFOX_ENVIRONMENT=production, or AGENTFOX_AUTH_MODE=token, to require API │
│ tokens.                                                                          │
╰──────────────────────────────────────────────────────────────────────────────────╯

In development mode, an X-AgentFox-User: you@example.com header is enough for /api/*. With AGENTFOX_AUTH_MODE=token (or AGENTFOX_ENVIRONMENT=production) that header is refused and you send Authorization: Bearer nom_api_…, a token from agentfox admin auth issue. The examples below show the header that worked in each case.

Send OpenTelemetry spans in

POST /v1/traces accepts an OTLP/HTTP trace export in JSON. Each span is stored on an AgentFox trace, and every agent it names goes into the registry. An agent nobody registered raises a shadow_agent finding.

{
  "resourceSpans": [{
    "resource": {"attributes": [
      {"key": "service.name", "value": {"stringValue": "research-bot"}},
      {"key": "deployment.environment", "value": {"stringValue": "staging"}}
    ]},
    "scopeSpans": [{
      "scope": {"name": "opentelemetry.instrumentation.langchain"},
      "spans": [
        {
          "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
          "spanId": "00f067aa0ba902b7",
          "name": "chat gpt-4o-mini",
          "startTimeUnixNano": "1791200000000000000",
          "endTimeUnixNano": "1791200000850000000",
          "attributes": [
            {"key": "gen_ai.system", "value": {"stringValue": "openai"}},
            {"key": "gen_ai.request.model", "value": {"stringValue": "gpt-4o-mini"}}
          ]
        },
        {
          "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
          "spanId": "00f067aa0ba902b8",
          "name": "tool web.fetch",
          "startTimeUnixNano": "1791200000900000000",
          "endTimeUnixNano": "1791200001300000000",
          "attributes": [
            {"key": "gen_ai.tool.name", "value": {"stringValue": "web.fetch"}}
          ]
        }
      ]
    }]
  }]
}

curl -s -X POST http://127.0.0.1:8080/v1/traces \
  -H 'Content-Type: application/json' --data @span.json | python3 -m json.tool

{
    "spans_ingested": 2,
    "traces": [
        "trc_01m469h1q532xt106f"
    ],
    "agents_seen": [
        "research-bot"
    ],
    "frameworks": {
        "research-bot": "langchain"
    },
    "shadow_agents": [
        …
        {
            "slug": "research-bot",
            "environment": "production",
            "first_seen": "2026-10-05T15:04:45.034507",
            "last_seen": "2026-10-05T15:04:45.038338",
            "calls": 1,
            "models": [
                "gpt-4o-mini"
            ],
            "providers": [
                "openai"
            ],
            "framework": "langchain",
            "suggested_registration": {
                …
            }
        }
    ]
}

How the span is read:

• Agent: the agentfox.agent attribute, else service.name, else gen_ai.agent.name, else unknown. Resource attributes and span attributes are merged.
• Trace: one AgentFox trace per agent and OTel trace id. The OTel trace id becomes its session id; deployment.environment, gen_ai.request.model and gen_ai.system fill in the environment, model and provider.
• Span kind: tool if it has gen_ai.tool.name or its name starts with tool; llm for other gen_ai.* spans; retrieval or agent from the span name. Status code 2 marks it an error.
• Framework: detected from attribute and scope names (LangGraph, LangChain, LlamaIndex, CrewAI, AutoGen, the Claude Agent SDK, Google ADK, the OpenAI Agents SDK, Semantic Kernel).

In the web app Traces → the trace, with each span and its attributes

The route accepts both OTLP/HTTP encodings: protobuf (application/x-protobuf, what the OpenTelemetry exporters send by default) and JSON (application/json), either one gzip-compressed (Content-Encoding: gzip). A protobuf request gets the empty protobuf ExportTraceServiceResponse the OTLP spec asks for; a JSON request gets the summary above. So the stock exporter works as it is:

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http import Compression
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider(resource=Resource.create({"service.name": "research-bot"}))
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(
            endpoint="http://127.0.0.1:8080/v1/traces",
            compression=Compression.Gzip,  # optional
        )
    )
)
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("research-bot")
with tracer.start_as_current_span("tool db.query") as span:
    span.set_attribute("gen_ai.tool.name", "db.query")
provider.shutdown()

Run with OpenTelemetry SDK 1.44, the span arrives as a tool span named tool db.query on a research-bot trace, with or without gzip. An OpenTelemetry Collector's otlphttp exporter can point at the same URL. Decoding protobuf needs the opentelemetry-proto package (part of agentfox[otel]); a server without it answers a protobuf body with 415 and a message saying so, and JSON still works. With a token-mode server, add the Authorization: Bearer header through the exporter's headers= argument.

What ingest does not do: it runs no policy and no detectors over the spans (the trace reads verdict: allow because nothing evaluated it), it does not keep the parent-child structure between spans, and the trace's start time is when it arrived, not the span's timestamp. To enforce on calls, route them through agentfox.auto() or the gateway.

Sending AgentFox events out as OpenTelemetry

AgentFox has no exporter that pushes spans or logs to a collector. The OpenTelemetry output it has is a pull: GET /api/export/siem?format=otlp returns decisions and findings as OTLP/JSON log records (see SIEM export). Fetch it on a schedule and forward it yourself.

Join decisions to Langfuse and LangSmith

When a model call goes through the AgentFox proxy (/v1/chat/completions or /v1/messages), AgentFox reads the join key off the request headers and stores it with its own trace. Nothing has to be installed and nothing leaves your network for this part. The headers it reads:

When the LangSmith or Langfuse SDK is installed in the same process as an in-process AgentFox call, AgentFox also reads the current run from it.

1. Send a call with the join key

   curl -s -X POST http://127.0.0.1:8080/v1/chat/completions \
     -H 'Content-Type: application/json' \
     -H 'X-AgentFox-Agent: support-triage' \
     -H 'langfuse-trace-id: lf-7d1c2e90' \
     -H 'langfuse-observation-id: obs-55a1' \
     -H 'traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01' \
     -d '{"model":"echo-1","messages":[{"role":"user","content":"Summarise ticket 4182"}]}'

   The response carries x-agentfox-trace: trc_01m469nhx6q4d17k22 and x-agentfox-verdict: allow.

2. Resolve their id to the decision

   GET /api/traces/resolve takes system (langfuse, langsmith or otel) and external_id, which can be either the trace id or the run/observation id.

   curl -s "http://127.0.0.1:8080/api/traces/resolve?system=langfuse&external_id=obs-55a1" \
     -H 'X-AgentFox-User: priya@example.com' | python3 -m json.tool

   {
       "matches": [
           {
               "trace_id": "trc_01m469nhx6q4d17k22",
               "agent": "support-triage",
               "verdict": "allow",
               "started_at": "2026-10-05T15:07:12.678900",
               "system": "langfuse",
               "external_trace_id": "lf-7d1c2e90",
               "external_run_id": "obs-55a1",
               "url": "https://cloud.langfuse.com/trace/lf-7d1c2e90",
               "detail": "/api/traces/trc_01m469nhx6q4d17k22"
           }
       ]
   }

   The same trace resolves from system=otel and the W3C trace id 0af7651916cd43dd8448eb211c80319c. An id with no match returns 404 with {"detail":"no governed trace correlates with that id"}. GET /api/traces/{trace_id} lists the same links under links.

The url is a deep link built from AGENTFOX_LANGFUSE_HOST (default https://cloud.langfuse.com) or, for LangSmith, AGENTFOX_LANGSMITH_UI_URL and AGENTFOX_LANGSMITH_PROJECT. Set them for a self-hosted deployment, or the link points at the cloud service.

Write the verdict back onto their run (optional)

AgentFox can tag the external run with its decision, so the verdict shows up where the engineer is already looking. This is off by default and needs egress:

export AGENTFOX_ALLOW_EGRESS=true
export AGENTFOX_CORRELATION_PUSH=true
# Langfuse
export AGENTFOX_LANGFUSE_HOST=https://cloud.langfuse.com
export AGENTFOX_LANGFUSE_PUBLIC_KEY=pk-lf-…
export AGENTFOX_LANGFUSE_SECRET_KEY=sk-lf-…
# LangSmith
export AGENTFOX_LANGSMITH_API_KEY=lsv2-…
export AGENTFOX_LANGSMITH_PROJECT=support

Checked against a local stand-in for each service, after a call through the proxy:

• Langfuse: POST /api/public/traces with basic auth (public key, secret key) and the body {"id":"lf-91b0aa12","metadata":{"agentfox_trace_id":"trc_01m469pg0nm74k2n2a","agentfox_verdict":"allow","agentfox_effective_verdict":"allow","agentfox_rules":[],"agentfox_agent":"support-triage"},"tags":["agentfox:allow"]}.
• LangSmith: PATCH /runs/<run id> (the trace id when no run id was sent) with an x-api-key header and the same metadata under extra.metadata, tagged agentfox:allow.

The push is best-effort with a 2-second timeout (AGENTFOX_CORRELATION_TIMEOUT_SECONDS). A failure is logged at debug level and never affects the request. It was not run against the real Langfuse or LangSmith services.

Prometheus metrics

GET /metrics serves the Prometheus text format. Point a scrape job at it; it needs no token.

curl -s http://127.0.0.1:8080/metrics

# HELP agentfox_decisions_total Governance decisions by verdict and policy mode.
# TYPE agentfox_decisions_total counter
agentfox_decisions_total{mode="enforce",verdict="allow"} 11
agentfox_decisions_total{mode="observe",verdict="allow"} 4
agentfox_decisions_total{mode="enforce",verdict="block"} 8
agentfox_decisions_total{mode="enforce",verdict="escalate"} 1
agentfox_decisions_total{mode="observe",verdict="escalate"} 1
# HELP agentfox_decisions_by_mode_total Decisions split by whether the policy was enforcing.
# TYPE agentfox_decisions_by_mode_total counter
agentfox_decisions_by_mode_total{mode="observe"} 5
agentfox_decisions_by_mode_total{mode="enforce"} 20
# HELP agentfox_detector_runs_total Detector executions.
# TYPE agentfox_detector_runs_total counter
agentfox_detector_runs_total{detector="injection.heuristic"} 11
agentfox_detector_runs_total{detector="pii.native"} 25
…
# HELP agentfox_detector_duration_ms_max Slowest detector run in the window.
# TYPE agentfox_detector_duration_ms_max gauge
agentfox_detector_duration_ms_max{detector="injection.heuristic"} 0.56825
…
# HELP agentfox_detector_degraded_total Detector runs that timed out, errored or were shed for budget. A control that quietly stops running while reporting effective is the failure mode that makes compliance products worthless.
# TYPE agentfox_detector_degraded_total counter
agentfox_detector_degraded_total 0
# HELP agentfox_open_findings Open findings by type and severity.
# TYPE agentfox_open_findings gauge
agentfox_open_findings{severity="critical",type="containment"} 9
agentfox_open_findings{severity="high",type="containment"} 4
…
agentfox_open_findings{severity="high",type="shadow_agent"} 2
…
# HELP agentfox_missed_escalation_rate Share of qualifying conversations that never handed off. PRD target < 0.05.
# TYPE agentfox_missed_escalation_rate gauge
agentfox_missed_escalation_rate 0
# HELP agentfox_handoffs Hand-offs by status.
# TYPE agentfox_handoffs gauge
agentfox_handoffs{status="pending"} 1
# HELP agentfox_circuit_breaker_state Circuit breaker state per provider: 0 closed, 1 open, 2 half-open.
# TYPE agentfox_circuit_breaker_state gauge
# HELP agentfox_traces_total Governed traces in the window.
# TYPE agentfox_traces_total counter
agentfox_traces_total 6
…

Worth alerting on: agentfox_detector_degraded_total (a detector timed out or errored, and in fail-open mode the request went through unchecked), agentfox_detector_duration_ms_max, agentfox_open_findings by severity, agentfox_missed_escalation_rate, and agentfox_circuit_breaker_state (1 means a provider's breaker is open). Observe-mode decisions are labelled mode="observe", so a dashboard can keep would-have-blocked apart from blocked.

SIEM export

GET /api/export/siem returns recent decisions and findings, oldest first, in the format your SIEM reads.

With token authentication on, issue a token for the export job and use it:

agentfox admin auth issue aisha@example.com --name siem-export --days 1
curl -s "http://127.0.0.1:8080/api/export/siem?format=cef&since_days=1&limit=1" \
  -H "Authorization: Bearer $AGENTFOX_TOKEN"

Without a token, the same request gets 401 and an authentication required message. The response is text/plain in every format. One line of each, from the same recorded block of a payments.transfer call:

{"event_type": "agent.decision", "event_id": "dec_01m469a9nrffpr535q", "timestamp": "2026-10-05T15:01:03.800641+00:00", "agent": null, "environment": null, "trace_id": null, "surface": "tool_args", "tool": "payments.transfer", "verdict": "block", "mode": "enforce", "policy_version_id": "pvr_01m4699d5spyd2ryxk", "rules_fired": ["eu.art14.human_oversight", "capability.constraint_violated"], "reasons": ["EU AI Act Art. 14 \u2014 irreversible action by a high-risk system requires human oversight.", "agent:payments-ops holds a grant for 'payments.transfer', so this is not a missing permission. The grant allows amount below 1000, but this call passed 25000."], "controls": […], "entities": [], "taint": "none", "latency_ms": 5.023792036809027, "severity": 8}

CEF:0|AgentFox|ControlPlane|0.1.0|agent.decision|block|8|rt=2026-10-05T15:01:03.800641+00:00 externalId=dec_01m469a9nrffpr535q act=block deviceCustomString1=payments.transfer deviceCustomString1Label=tool deviceCustomString2=eu.art14.human_oversight,capability.constraint_violated deviceCustomString2Label=rules … msg=EU AI Act Art. 14 — irreversible action by a high-risk system requires human oversight.; agent:payments-ops holds a grant for 'payments.transfer', so this is not a missing permission. The grant allows amount below 1000, but this call passed 25000.

LEEF:2.0|AgentFox|ControlPlane|0.1.0|agent.decision|	devTime=2026-10-05T15:01:03.800641+00:00	cat=agent.decision	sev=8	tool=payments.transfer	verdict=block	rules=eu.art14.human_oversight,capability.constraint_violated	…

{
  "resourceLogs": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": {
              "stringValue": "agentfox"
            }
          },
          …
      "scopeLogs": [
        {
          "scope": {
            "name": "agentfox.governance"
          },
          "logRecords": [
            {
              "timeUnixNano": "1791212464033230080",
              "severityNumber": 16,
              "severityText": "BLOCK",
              "body": {
                "stringValue": "agent.decision"
              },
              "attributes": [
                {
                  "key": "event_id",
                  …

Findings arrive as governance.finding.<type> events with a title, severity, status and the finding's evidence. Severity runs 0–10 in CEF and LEEF (block 8, escalate 6, a critical finding 10).

Webhooks for findings

AgentFox POSTs a JSON body to one URL whenever a finding at or above a severity is committed, and again when one changes status. Configure it with environment variables (see Configuration):

export AGENTFOX_ALLOW_EGRESS=true                 # nothing is sent without this
export AGENTFOX_WEBHOOK_URL=https://hooks.example.com/agentfox
export AGENTFOX_WEBHOOK_SECRET=whsec_…              # signs every request
export AGENTFOX_WEBHOOK_MIN_SEVERITY=high           # critical | high | medium | low; default high
export AGENTFOX_WEBHOOK_TIMEOUT_SECONDS=3

Events:

• finding.created: a new finding was committed.
• finding.resolved, finding.suppressed, finding.reopened: a finding's status changed.
• webhook.test: a test delivery you send yourself (below).

Nothing is sent for work that was rolled back. Delivery runs on a background thread behind a 1,000-item queue, so it never slows or fails the code that raised the finding. A timeout, connection error or 5xx is retried once after half a second; a redirect counts as a failure. Failures are logged at WARNING and dropped: there is no persistent retry queue.

What a delivery looks like

The headers and body of a real delivery, as the receiving server saw them:

Accept-Encoding: identity
Content-Length: 107
Host: 127.0.0.1:18449
Content-Type: application/json
User-Agent: agentfox/0.3.1
X-AgentFox-Event: webhook.test
X-AgentFox-Delivery: 0b73737568964955a66901e87c890733
X-AgentFox-Timestamp: 1791212816
X-AgentFox-Signature: sha256=0f5cc905c3c7fea77d4dd355f611cebfa15c66a8f13f679260a92adddd6422b6
Connection: close

{"event":"webhook.test","finding":null,"org_id":"org_default","sent_at":"2026-10-05T15:06:56.357567+00:00"}

For finding events, finding has the same field names as GET /api/findings/{id}: id, type, severity, status, title, subject_type, subject_id, evidence, occurrences, the resolution and suppression fields, and timestamps.

The signature is the hex HMAC-SHA256 of the raw body, keyed with your secret. You can check one by hand:

printf '%s' '{"event":"webhook.test","finding":null,"org_id":"org_default","sent_at":"2026-10-05T15:06:56.357567+00:00"}' \
  | openssl dgst -sha256 -hmac whsec_test_7f3a9c

SHA2-256(stdin)= 0f5cc905c3c7fea77d4dd355f611cebfa15c66a8f13f679260a92adddd6422b6

Verify it in your receiver

Compute the HMAC over the bytes you received, before parsing them; compare in constant time; then reject stale and repeated deliveries. Check freshness with sent_at inside the body, which the signature covers. The X-AgentFox-Timestamp header is not signed, so anyone replaying a captured request can change it. A retry reuses the same body, signature and X-AgentFox-Delivery id, so de-duplicate on that id. This receiver uses only the standard library:

import datetime as dt
import hashlib
import hmac
import json
import os
from http.server import BaseHTTPRequestHandler, HTTPServer

SECRET = os.environ["AGENTFOX_WEBHOOK_SECRET"]
MAX_AGE = dt.timedelta(minutes=5)
seen_deliveries = set()  # use a shared store (Redis, a table) in production


def verify(raw_body: bytes, headers) -> dict:
    """Return the event if the request is authentic and fresh; raise otherwise."""
    expected = "sha256=" + hmac.new(SECRET.encode(), raw_body, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(headers.get("X-AgentFox-Signature", ""), expected):
        raise ValueError("bad signature")
    event = json.loads(raw_body)
    # sent_at is inside the signed body; the X-AgentFox-Timestamp header is not signed.
    sent_at = dt.datetime.fromisoformat(event["sent_at"])
    if abs(dt.datetime.now(dt.timezone.utc) - sent_at) > MAX_AGE:
        raise ValueError("stale delivery")
    delivery = headers.get("X-AgentFox-Delivery")
    if delivery in seen_deliveries:
        raise ValueError("duplicate delivery")
    seen_deliveries.add(delivery)
    return event


class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        raw = self.rfile.read(int(self.headers["Content-Length"]))
        try:
            event = verify(raw, self.headers)
        except ValueError as exc:
            print("rejected:", exc, flush=True)
            self.send_response(401)
            self.end_headers()
            return
        finding = event.get("finding") or {}
        print("accepted", event["event"], finding.get("severity"), finding.get("title"), flush=True)
        self.send_response(204)
        self.end_headers()

    def log_message(self, *args):
        pass


HTTPServer(("127.0.0.1", 18448), Handler).serve_forever()

Test it end to end

1. Start the receiver

   AGENTFOX_WEBHOOK_SECRET=whsec_test_7f3a9c python3 receiver.py

2. Send a test event

   There is no CLI command for this yet; call the function the product uses, with the same environment the server runs with:

   export AGENTFOX_ALLOW_EGRESS=true
   export AGENTFOX_WEBHOOK_URL=http://127.0.0.1:18448/agentfox
   export AGENTFOX_WEBHOOK_SECRET=whsec_test_7f3a9c
   python -c "from agentfox.core.webhooks import send_test_event; print(send_test_event())"

   (True, 'HTTP 204')

3. Raise a real finding

   With agentfox serve api running under the same variables, an OTLP span from an agent nobody registered raises a high-severity shadow_agent finding (here, span.json from above with the service renamed to triage-experimental). The receiver prints:

   accepted webhook.test None None
   accepted finding.created high Ungoverned agent 'triage-experimental' observed in production

4. Check that a wrong secret is refused

   With AGENTFOX_WEBHOOK_SECRET=wrong-secret on the sending side, send_test_event() returns (False, 'HTTP 401') and the receiver prints rejected: bad signature.

What can go wrong

• No webhook arrives. Egress is off. send_test_event() says so: (False, 'egress is disabled (AGENTFOX_ALLOW_EGRESS=false); nothing sent'). The message uses the older variable name; AGENTFOX_ALLOW_EGRESS is the one to set. Also check the finding's severity against AGENTFOX_WEBHOOK_MIN_SEVERITY, and the server log for finding webhook: delivery of … failed.
• /v1/traces returns 415 or 400. 415: the body is neither protobuf nor JSON, it uses a compression other than gzip or deflate, or it is protobuf and the server lacks opentelemetry-proto (install agentfox[otel], or set OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/json). 400: the body does not decode as what its headers say. The detail field names which.
• /api/traces/resolve returns 404. The call did not go through the proxy, or did not carry one of the headers in the table. OTLP ingest does not create these links.
• /api/* returns 401. Token mode is on and the request sent X-AgentFox-User. Send Authorization: Bearer nom_api_…; check with agentfox admin auth status.
• A shadow agent shows the wrong environment. The trace records deployment.environment, but the registry entry and the finding say production regardless.

Limits

• OTLP ingest is JSON only and records spans; it does not enforce anything.
• There is no push exporter for OpenTelemetry or Prometheus; both are pull.
• Metrics cover a rolling 24-hour window and carry no per-agent labels.
• Webhooks go to one URL, with one retry and no persistent queue. A receiver that is down for longer than that misses events; the SIEM export is the way to backfill.
• Langfuse and LangSmith correlation needs the call to go through the proxy (or the vendor SDK to be in the same process). Pushing the verdict back needs egress and their API keys.