Agora Conversational AI — Custom HTTP Tools Recipe (Python)

The custom HTTP tools recipe in the Agora Conversational AI recipes family. It demonstrates Engine's inline LLM REST tools capability: the selected OpenAI model selects a function, and Agora Engine calls the HTTP endpoint declared directly in that function's SDK configuration. The JSON result is returned to the model, which speaks a natural response.

This capability is separate from MCP: it uses inline tools definitions on the OpenAI LLM or Realtime MLLM, not mcp_servers or an MCP transport. It is also separate from a custom LLM tool loop because Agora Engine executes the request.

The recipe includes three deterministic mock business endpoints:

• GET /tools/orders/{order_id} — look up an order
• POST /tools/tickets — create and store a support ticket
• GET /tools/tickets/{ticket_id} — retrieve a created ticket

Choose the agent mode before each conversation:

• Pipeline (default): DeepgramSTT(nova-3, en) → OpenAI(gpt-4o-mini, inline tools) → MiniMaxTTS
• Realtime: OpenAI Realtime MLLM(inline tools)

Pipeline mode requires no external model API key by default. Set OPENAI_API_KEY to use your own OpenAI credentials, and optionally override OPENAI_BASE_URL for a compatible endpoint. Realtime mode uses the separate OPENAI_REALTIME_API_KEY. HTTP_TOOLS_API_KEY is a random shared secret for authenticating Engine requests to the example tool endpoints.

Prerequisites

• Python 3.10+
• Bun
• Agora CLI — makes configuring an App ID and App Certificate easy
• ngrok or another HTTPS tunnel — Agora Engine must reach the tool endpoints

The same commands work on macOS, Linux, and Windows. On macOS/Linux, setup uses python3; on Windows, it uses the Python launcher (py) or python. WSL and virtualenv activation are not required.

Run It

# 1. Install web dependencies and create the server Python venv
bun run setup

# 2. Add Agora credentials, or edit server/.env.local by hand
agora login
agora project use <your-project>          # select which project to use
agora project env write server/.env.local # writes App ID + Certificate

# 3. Expose the backend. Agora Engine calls /tools on this tunnel.
ngrok http 8000

# 4. Add the public URL and a random shared secret to server/.env.local
#    HTTP_TOOLS_BASE_URL=https://<your-tunnel>.ngrok-free.dev
#    HTTP_TOOLS_API_KEY=<random-secret>

# 5. Run backend + web
bun run dev

Open http://localhost:3000 → Start Conversation → ask “Where is order A-1001?”, then “Create a ticket because order A-1001 is late.” Ask for the returned ticket ID to verify the same ticket can be retrieved.

Working from a clone

If you cloned this repo rather than scaffolding via the Agora CLI, the steps above are complete as written: bun run setup creates the Python venv and installs web dependencies, then bun run dev starts both services. A conversation additionally needs Agora credentials, a public HTTPS tool URL, and the tool API key in server/.env.local.

Services:

• Frontend — http://localhost:3000
• Backend + mock REST tools — http://localhost:8000 (including /tools)
• API docs — http://localhost:8000/docs

Deploy

Deploy web (Next.js) and server (a publicly reachable FastAPI backend). Set HTTP_TOOLS_BASE_URL to the backend's HTTPS base URL. Set AGENT_BACKEND_URL in the web deployment so the Next rewrites reach the same backend.

A backend-only Docker image is published to ghcr.io/AgoraIO-Conversational-AI/recipe-agent-custom-http-tools on v* tags. It exposes port 8000 and serves both the lifecycle API and /tools.

Co-public caveat: exposing /tools also exposes the lifecycle routes on port 8000. The sample authenticates /tools with X-Tool-API-Key; add application authentication and rate limiting to the lifecycle routes before production deployment.

Environment variables

Backend env file: server/.env.example.

Variable	Required	Default	Notes
AGORA_APP_ID	Yes	—	Agora Console → Project → App ID
AGORA_APP_CERTIFICATE	Yes	—	Agora Console → Project → App Certificate; server only
HTTP_TOOLS_BASE_URL	Yes	—	Public HTTPS base URL for this server, without /tools
HTTP_TOOLS_API_KEY	Yes	—	Random shared secret sent as X-Tool-API-Key
HTTP_TOOLS_TIMEOUT_MS		10000	SDK-supported range: 1000–100000 ms
OPENAI_MODEL		gpt-4o-mini	Pipeline model
OPENAI_API_KEY		—	Optional Pipeline BYO API key; omit for Agora-managed mode
OPENAI_BASE_URL		OpenAI chat completions URL	Optional OpenAI-compatible Pipeline endpoint override
OPENAI_REALTIME_API_KEY	Realtime only	—	OpenAI API key for Realtime mode
OPENAI_REALTIME_MODEL		gpt-realtime	OpenAI Realtime model
AGENT_GREETING		built-in	Optional opening line override
PORT		8000	FastAPI backend port
AGENT_BACKEND_URL	web deploy only	—	Backend URL used by deployed Next.js /api/* rewrites

Commands

bun run setup                 # install web deps + create server/ venv
bun run dev                   # run backend (:8000, including /tools) + web (:3000)

bun run doctor                # prerequisite check (no credentials needed)
bun run doctor:local          # + local env, credentials, and public HTTPS URL checks

bun run verify                # web-only gate (no Agora credentials needed)
bun run verify:backend:pytest # standalone backend tests, no Agora cloud calls
bun run verify:local          # full local gate: backend tests + smoke tests + web build
bun run clean                 # remove venvs and build artifacts

Tests run standalone with no Agora cloud session: pytest in server/ and bun test in web/. CI runs them on Linux, macOS, and Windows with Python 3.10 and 3.13 where applicable.

The mock tool endpoint can also be checked independently:

TOOL_BASE_URL="https://your-tunnel.ngrok-free.dev"
TOOL_API_KEY="replace-with-the-value-from-server-env-local"
curl -H "X-Tool-API-Key: $TOOL_API_KEY" \
  "$TOOL_BASE_URL/tools/orders/A-1001"

Inline tool contract

The managed LLM receives normal function definitions plus an inline server configuration. with_tools() is also required to enable tool execution at the Engine level:

llm = OpenAI(
    model="gpt-4o-mini",
    tools=[{
        "type": "function",
        "function": {
            "name": "lookup_order",
            "parameters": {
                "type": "object",
                "properties": {"order_id": {"type": "string"}},
                "required": ["order_id"],
            },
        },
        "execution": {"mode": "sync"},
        "server": {
            "method": "GET",
            "url": "https://example.com/orders/{{args.order_id}}",
            "headers": {"X-Tool-API-Key": "secret"},
            "timeout_ms": 10000,
        },
    }],
    template_variables={"requester": "inline-rest-tools-recipe"},
)
agent = Agent(client=client).with_llm(llm).with_tools()

The same tool list can be passed to OpenAIRealtime(tools=...) and attached with .with_mllm(...).with_tools(). The example uses a literal requester value in Realtime mode because OpenAIRealtime has no template_variables configuration field; Pipeline mode demonstrates the template variable instead.

URLs and POST bodies support {{args.name}}, {{template_variables.name}}, and {{tool_call_id}}. Headers support constants, template variables, and the tool call ID, but not args placeholders. The current execution mode is sync; server.body is only valid for POST tools.

Architecture

Browser (localhost:3000)
  │  fetch /api/*
  ▼
Next.js  ──rewrite──▶  Agent backend (server/, localhost:8000)
                          │  starts selected OpenAI path with tools + with_tools()
                          ▼
                       Agora ConvoAI Engine
                          │  model selects a function
                          │  Engine renders templates and sends GET/POST
                          ▼
                       Public HTTPS /tools endpoint
                          │  authenticated JSON result
                          ▼
                       Engine → selected model path → RTC user

The browser does not execute tools. Agora Engine calls HTTP_TOOLS_BASE_URL directly, so a local localhost URL cannot be used for a cloud session. See ARCHITECTURE.md.

What You Get

• A Next.js web client (:3000) that drives the RTC/RTM lifecycle and only calls /api/*.
• A FastAPI backend (:8000) that owns Agora token generation, agent sessions, and the mock REST endpoints.
• The standard /api/get_config · /api/startAgent · /api/stopAgent contract through Next rewrites.
• Selectable Pipeline and OpenAI Realtime MLLM paths using the same inline REST tools.
• Three authenticated inline REST tools covering GET URL arguments, POST bodies, template variables, and tool call IDs.

How It Works

1. The browser calls /api/get_config; Next rewrites the request to FastAPI, which mints an Agora token from the App ID and App Certificate.
2. The browser joins RTC, then calls /api/startAgent with agentMode. The backend attaches the same inline tool definitions to managed OpenAI in Pipeline mode or OpenAI Realtime MLLM in Realtime mode, then enables tools.
3. In Pipeline mode, Agora runs Deepgram STT before the LLM and MiniMax TTS after it. Realtime mode handles audio directly in the MLLM.
4. When the model selects a function, Engine renders the tool templates and calls the configured public HTTPS endpoint with X-Tool-API-Key.
5. Engine returns the endpoint's JSON to the model. Pipeline mode uses MiniMax TTS for the final response; Realtime mode speaks it directly.
6. The demo backend keeps created tickets in process so the user can ask for a returned ticket ID and retrieve the same ticket. Restarting the backend clears them.
7. /api/stopAgent stops the active session, with the SDK's stateless stop path available as a fallback.

Replacing the mock

Replace the in-process demo handlers in server/src/http_tools.py or point the inline server.url values at your own HTTPS APIs. The demo ticket store is not durable; use your business system for production. Keep each tool's JSON schema, URL/body templates, authentication, and response contract aligned with the target service.

Repo Map

• web/ — Next.js frontend (:3000); RTC/RTM lifecycle and UI.
• server/ — FastAPI backend (:8000); Agora tokens, agent lifecycle, and mock REST tools.
• server/src/http_tools.py — inline SDK tool definitions and /tools handlers.
• ARCHITECTURE.md — system shape and component boundaries.
• AGENTS.md — guide for coding agents working in this repo.

Security

Use HTTPS, authenticate every request, validate tool arguments, apply least-privilege credentials, rate-limit endpoints, and redact sensitive data from logs. Do not put secrets in URLs or LLM-generated function arguments.

Troubleshooting

Problem	Fix
doctor:local rejects the tool URL	Use a public https:// URL; localhost and plain HTTP are not accepted.
Agent starts but tools are never called	Confirm the tunnel points to the running backend and HTTP_TOOLS_BASE_URL does not include /tools.
Tool calls return 401	Use the same HTTP_TOOLS_API_KEY in the SDK tool headers and backend environment.
A created demo ticket cannot be found	Demo tickets are kept in process and are cleared when the backend restarts. Create a new ticket or connect a durable business API.
The deployed web app cannot start an agent	Set AGENT_BACKEND_URL to the public backend URL and rebuild the web app.
Local verification reports a missing venv	Run bun run setup.

More Docs

• ARCHITECTURE.md
• AGENTS.md
• server/README.md

License

Released under the MIT License.