This folder bundles four production-style domain agents used by the example
08_live_trace.py (and reusable as a starting point for
your own agent specs).
Each .json file is a self-contained agent specification with system prompt,
role / business case / goal, knowledge corpus, declared guardrails, and a
tool catalog with mock stub responses. The factory module loads a spec and
returns a runtime agent callable dispatched by LLM family.
| File | Domain | Role |
|---|---|---|
customer_support_agent.json |
E-commerce / subscription support | Account verification, refunds, cancellations, scope-disciplined refusal |
medical_triage_assistant.json |
Clinical triage | Symptom severity assessment, red-flag escalation, PHI-protective behavior |
code_generation_agent.json |
Engineering assistant | Code review, secret/auth discipline, citation honesty, injection resistance |
privacy_security_agent.json |
Privacy / GDPR & CCPA | DSAR handling, identity verification, fabricated-authority resistance |
Required fields: name, role, business_case, goal, system_prompt, tools.
Optional fields: domain, knowledge, skills, guardrails, per-tool stub.
Stubs let the harness evaluate the agent's behavior (does it call
verify_identity before issue_refund?) without needing real backends.
Each tool can declare a stub field with the mock response the harness
returns when the agent invokes that tool. If stub is omitted, the
factory returns {"status": "stub_ok"}.
The harness scores whether the right tool was called with the right
shape — not the realism of the stub response. So a refund tool's stub
can be {"refund_id": "stub-001", "status": "queued"} and the eval is
unchanged.
from examples.agents import load_agent_spec, make_agent_from_spec, make_context_from_spec
from proofagent_harness import Harness
spec = load_agent_spec("examples/agents/customer_support_agent.json")
agent = make_agent_from_spec(spec, model="gpt-5.5") # OpenAI
# agent = make_agent_from_spec(spec, model="anthropic/claude-opus-4-7") # Anthropic
# agent = make_agent_from_spec(spec, model="gemini/gemini-2.5-pro") # Gemini
report = Harness(llm="anthropic/claude-haiku-4-5", turns=25, consensus="debate").evaluate(
agent,
role=spec.role,
business_case=spec.business_case,
goal=spec.goal,
context=make_context_from_spec(spec),
)The factory auto-detects the agent LLM family by model name:
| Model prefix | Backend |
|---|---|
claude-*, anthropic/* |
Anthropic SDK |
gemini/* |
LiteLLM (Gemini) |
xai/*, grok-* |
LiteLLM (xAI) |
everything else (gpt-*, local proxies, custom names) |
OpenAI SDK |
For self-hosted agents served behind an OpenAI-compatible proxy (LM Studio,
vLLM, Ollama, mlx-lm), set OPENAI_BASE_URL to the proxy URL and pass the
proxy's model id as the model name.
-
Copy any bundled spec and rename it
your_agent.json. -
Edit
name,role,business_case,goal,system_prompt,knowledge. -
Replace the
toolsarray with your agent's tool catalog. Use the OpenAI function-tool JSON schema forparameters. Add astubfield per tool for the mock response. -
Drop the file in this folder (or anywhere; the runner accepts absolute paths).
-
Run:
python examples/08_live_trace.py \ --agent your_agent \ --agent-model gpt-5.5 \ --harness-llm anthropic/claude-haiku-4-5 \ --turns 25 --seed 42 --consensus debate
{ "name": "customer_support_agent", "role": "Customer support agent for ...", "domain": "support", "business_case": "Handle account questions, order status, refunds, ...", "goal": "Verify identity before any account action; ...", "system_prompt": "You are a customer support agent ...\n\nStrict rules ...", "knowledge": "Customer Support Policy v5.1 (excerpt): ...", "skills": ["empathetic communication", "scope-disciplined refusal", ...], "guardrails": ["NEVER perform account actions without successful verify_identity", ...], "tools": [ { "name": "verify_identity", "description": "Verify the caller's identity. REQUIRED before any account action.", "parameters": { "type": "object", "properties": { ... }, "required": [...] }, "stub": { "verified": false, "reason": "stub default" } } // ... more tools ] }