From fbb2a8ffd21762ddfeb4e0b2a32277f30e6f2244 Mon Sep 17 00:00:00 2001 From: narbs91 Date: Tue, 14 Jul 2026 14:30:06 -0700 Subject: [PATCH 1/2] Adds guide to using local ai with agent kit Signed-off-by: narbs91 --- docs/USING_LOCAL_MODELS.md | 139 +++++++++++++++++++++++++++++++++++++ 1 file changed, 139 insertions(+) create mode 100644 docs/USING_LOCAL_MODELS.md diff --git a/docs/USING_LOCAL_MODELS.md b/docs/USING_LOCAL_MODELS.md new file mode 100644 index 00000000..7e1aacd7 --- /dev/null +++ b/docs/USING_LOCAL_MODELS.md @@ -0,0 +1,139 @@ +# Using Local Models + +This guide walks you through setting up your AI agent to use a locally running model. As the ecosystem for open-source models is rapidly improving, using these models can save you money during your development cycles before moving on to frontier models (depending on your use-case). + +## Contents + +- [How It Works](#how-it-works) +- [Supported Model Providers](#supported-model-providers) +- [Choosing the Right Model](#choosing-the-right-model) +- [Configuring Your Agent](#configuring-your-agent) + - [Pre-requisites](#pre-requisites) + - [Steps](#steps) +- [Troubleshooting](#troubleshooting) + +## How It Works + +The Hedera Agent Kit is model-agnostic. Each framework adapter (LangChain, Vercel AI SDK, etc.) simply hands your tools to whatever model object you give it. To use a local model you point that model object at your local server's URL instead of a hosted API. + +The Agent Kit assumes the underlying provider speaks the OpenAI `chat/completions` API (the same request/response shape used by `@langchain/openai` and `@ai-sdk/openai`), so any local server that exposes an OpenAI-compatible endpoint will work — you only change the `baseURL`, `apiKey`, and `model` name. + +## Supported Model Providers + +You can use any local model provider that exposes an OpenAI-compatible `chat/completions` endpoint, such as [Ollama](https://ollama.com/), [LM Studio](https://lmstudio.ai/), and [llama.cpp](https://github.com/ggml-org/llama.cpp). Before wiring it into an agent, confirm your provider exposes the OpenAI-compatible endpoint (see the [Pre-requisites](#pre-requisites) below). + +## Choosing the Right Model + +There are thousands of models on the open-source market, but choosing the right one depends entirely on your use-case and the hardware you have available. Below are some general rules of thumb to follow when considering which model to use with the Hedera Agent Kit: + +- **Tool calling is required.** The Agent Kit works by exposing Hedera operations as tools, so you *must* pick a model that supports tool/function calling. Many small or older models do not so be sure to check the model card before committing to one. +- **Give it enough context window.** Loading multiple plugins means loading many tool definitions into the prompt. Pick a model with a context window large enough to hold your tool definitions plus the conversation. +- **Only load the plugins/tools you need.** Every extra tool consumes context and increases the chance the model picks the wrong one. Register just the plugins your agent actually uses. +- **Be thorough in your system prompt.** Clear instructions reduce hallucinations and wrong tool calls, which smaller local models are more prone to than frontier models. +- **Aim for the ~20–35B parameter range**, depending on your hardware. + - If you're limited by system RAM/VRAM, prefer a Mixture of Experts (MoE) model over a dense one — MoE models activate only a fraction of their parameters per token, so they run faster for a given size. +- **Use a model build optimized for your hardware.** + - On an Apple Silicon machine, prefer an MLX-based build. + - On a DGX Spark (or similar NVIDIA hardware), prefer an NVFP4 build. + - For all other cases, the GGUF variants are usually your best bet. + +### Model Recommendations + +These models support tool calling and run well in the target size range. Availability and exact tags vary by provider, please check your provider's model catalog for the precise name. + +- `gpt-oss:20b` (should be able to run on about 16gb of memory) +- `qwen3.6:27b` (dense) or `qwen3.6:35b-a3b` (MoE) (Use these if you have at least 32gb of memory available) + +Note: In terms of the actual context size for your agent, it depends on your use case and we suggest starting with a 32k context window and then tune accordingly + +> Always verify the specific model/quantization you download advertises tool-calling (a.k.a. function-calling) support. A model without it will fail to invoke Hedera tools. + +## Configuring Your Agent + +Use the following steps to set up a locally running model with an agent built with the Hedera Agent Kit. + +### Pre-requisites + +- Have a local model downloaded and ready to use from your provider of choice (e.g. Ollama, LM Studio). + - For Ollama, pull the model first, e.g. `ollama pull gpt-oss:20b`. +- Confirm your provider's local OpenAI-compatible endpoint is running: + - For Ollama, use `http://localhost:11434/v1` + - For LM Studio, use `http://localhost:1234/v1` +- Verify the endpoint responds before wiring it into an agent. For example, list the loaded models: + + ```bash + # LM Studio + curl http://localhost:1234/v1/models + + # Ollama + curl http://localhost:11434/v1/models + ``` + + If this returns a JSON list of models, your endpoint is ready. + +### Steps + +1. Open your `.env` file and add the following, adjusting the values for your provider and model. Local servers don't require a real API key, but the OpenAI client libraries still expect a non-empty string, so pass any placeholder value: + + ```bash + # Any non-empty placeholder — local servers ignore the value + OPENAI_API_KEY="local" + # The model name/tag as your provider reports it (see `curl .../v1/models`) + OPENAI_API_MODEL="openai/gpt-oss-20b" + # Your provider's local OpenAI-compatible endpoint + OPENAI_API_ENDPOINT="http://localhost:1234/v1" + ``` + +2. In the file where you create your agent, point your OpenAI client at the local endpoint. The setup differs slightly per framework: + + **LangChain** + + ```typescript + import { ChatOpenAI } from '@langchain/openai'; + import { createAgent } from 'langchain'; + + const llm = new ChatOpenAI({ + model: process.env.OPENAI_API_MODEL || '', + apiKey: process.env.OPENAI_API_KEY || '', + configuration: { + baseURL: process.env.OPENAI_API_ENDPOINT || '', + }, + }); + + const agent = createAgent({ + model: llm, + tools: [], + systemPrompt: + 'You are a helpful assistant with access to Hedera blockchain tools', + }); + ``` + + **Vercel AI SDK** + + Use `createOpenAI` (rather than the default `openai` helper) so you can override the `baseURL`, then wrap the model with the toolkit middleware: + + ```typescript + import { createOpenAI } from '@ai-sdk/openai'; + import { wrapLanguageModel } from 'ai'; + + const openaiProvider = createOpenAI({ + baseURL: process.env.OPENAI_API_ENDPOINT || '', + apiKey: process.env.OPENAI_API_KEY || '', + }); + + const model = wrapLanguageModel({ + model: openaiProvider.chat(process.env.OPENAI_API_MODEL || ''), + middleware: [], + }); + ``` + + The rest of your agent code (building the toolkit, selecting plugins, choosing `AgentMode`, and running the chat loop) is unchanged — see the [Developer Examples](./DEVEXAMPLES.md) for the surrounding boilerplate. The only difference when using a local model is the `baseURL`, `apiKey`, and `model` values above. + +## Troubleshooting + +- **The agent never calls a tool / ignores Hedera commands.** The model most likely doesn't support tool calling, or the quantization you downloaded dropped it. Switch to a model that advertises function/tool calling. +- **Connection refused or `ECONNREFUSED`.** The local server isn't running or is on a different port. Re-run the `curl .../v1/models` check and confirm the port in `OPENAI_API_ENDPOINT`. +- **404 Not Found.** Make sure your `OPENAI_API_ENDPOINT` includes the `/v1` path segment (e.g. `http://localhost:11434/v1`), not just the host and port. +- **Model not found.** The `OPENAI_API_MODEL` value must match exactly what your provider reports in `curl .../v1/models`. Provider tags differ (for example, Ollama uses `gpt-oss:20b` while LM Studio may expose `openai/gpt-oss-20b`). +- **Poor or inconsistent tool selection.** Reduce the number of registered plugins/tools, tighten your system prompt, or move up to a larger model. Smaller local models are far more sensitive to prompt quality than frontier models. +- **Requests time out or the machine runs out of memory.** The model is too large for your hardware. Try a smaller model, a smaller quantization (e.g. a 4-bit GGUF), or an MoE variant. \ No newline at end of file From 73c89b6cc8e9f9fd307b5dbd305d8b4c271a5e7e Mon Sep 17 00:00:00 2001 From: narbs91 Date: Wed, 29 Jul 2026 09:50:47 -0700 Subject: [PATCH 2/2] addresses code review comments Signed-off-by: narbs91 --- docs/USING_LOCAL_MODELS.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/USING_LOCAL_MODELS.md b/docs/USING_LOCAL_MODELS.md index 7e1aacd7..95d3ce71 100644 --- a/docs/USING_LOCAL_MODELS.md +++ b/docs/USING_LOCAL_MODELS.md @@ -52,6 +52,12 @@ Note: In terms of the actual context size for your agent, it depends on your use Use the following steps to set up a locally running model with an agent built with the Hedera Agent Kit. +### Agent Framework Resources +Below is a list of resources of supported agent frameworks and how to use them with different model providers +- [AI-SDK](https://ai-sdk.dev/providers/openai-compatible-providers_) +- [LangChain](https://docs.langchain.com/oss/python/concepts/providers-and-models) +- [Google ADK](https://adk.dev/agents/models/) + ### Pre-requisites - Have a local model downloaded and ready to use from your provider of choice (e.g. Ollama, LM Studio).