🥷 Try it live — no local setup. ICP Ninja is a web-based IDE that builds and deploys this project to the mainnet for free, right in your browser. Click the badge above, or hit Deploy if you're already in Ninja. To build and run it locally instead, follow the steps below.
This example demonstrates how an ICP canister can interact with a large language model (LLM) to generate text. The user can input a prompt and the canister will use the LLM to generate a response. Follow-up prompts continue the conversation with the full message history.
The backend canister calls the LLM canister's v1_chat endpoint directly (see backend/lib.rs), without a helper crate. It reads the LLM canister's principal from the PUBLIC_CANISTER_ID:llm environment variable. Locally, icp deploy deploys a copy of the LLM canister (backed by Ollama) and injects this variable automatically. On mainnet the shared LLM canister already exists, so icp.yaml sets the variable to its principal (w36hm-eqaaa-aaaal-qr76a-cai) for the ic environment.
- Node.js
- icp-cli:
npm install -g @icp-sdk/icp-cli @icp-sdk/ic-wasm
The LLM canister delegates inference to Ollama. Install it and then run:
ollama serveIn a separate terminal, download the model (about 4 GiB, one-time) and load it into memory:
ollama run llama3.1:8b "hi"ollama run pulls the model if needed and warms it in memory. This matters: the
LLM canister's HTTP outcall to Ollama has a ~30 s deadline, and a cold model
load alone can take longer than that — so the first call after ollama serve
starts may time out (SysFatal: Timeout expired) if the model isn't warm yet.
Warming it first avoids this; ollama serve then keeps it loaded. If you do hit
a timeout on the first call, simply retry — the model stays resident afterwards.
git clone https://github.com/dfinity/examples
cd examples/rust/llm_chatboticp network start -d
icp deployOpen the frontend URL printed by icp deploy to use the chatbot in the browser. Make sure Ollama is running with the model warmed (see above) so the first message does not time out.
For hot-reload frontend development:
npm run dev --prefix frontendicp deploy -e icNo Ollama setup is needed — mainnet calls go directly to the LLM canister at w36hm-eqaaa-aaaal-qr76a-cai.
icp build backend && candid-extractor target/wasm32-unknown-unknown/release/backend.wasm > backend/backend.didIf you base your application on this example, familiarize yourself with the security best practices for developing on ICP. This example may not implement all best practices.