This software is provided “as is,” without any express or implied warranties. The authors and contributors shall not be held liable for any damages arising from its use. The code may be incomplete or insufficiently tested. Users are solely responsible for evaluating its suitability and assume all associated risks.
Note: Contributions are welcome. Please ensure thorough testing before deploying in critical systems.
This sample helps developers use QAI AppBuilder + Python to build Genie based Open AI compatibility API service on Windows on Snapdragon (WoS) platform.
We suggest using the C++ version of GenieAPIService provided by us.
• Support both stream and none stream mode
• Support tools call [NEW!]
• Support enable/disable thinking mode [NEW!]
• Support switching between models
• Support to extend the use of your own model
Refer to python.md on how to setup x64 version Python environment.
Run following commands in Windows terminal:
pip install uvicorn==0.34.0 pydantic_settings==2.10.1 fastapi==0.115.8 langchain==0.3.19 langchain_core==0.3.45 langchain_community==0.3.18 sse_starlette==2.2.1 pypdf==5.3.0 python-pptx==1.0.2 docx2txt==0.8 openai==1.63.2 json-repair==0.47.4
Download the files from the AI-Hub LLM models list at the end of this page, save them to following path.
qai-appbuilder\samples\genie\python\models\<model name>
- Select 'Snapdragon® X Elite' as the device for WoS platform while downloading the model.
- You need to unzip the 'weight_sharing_model_N_of_N.serialized.bin' files from model package and copy them to following path. Download and copy the corresponding 'tokenizer.json' file to the following directory path too.
- Please be careful not to mix 'tokenizer.json' file of different models. Ensure that the' tokenizer.json' file corresponding to the IBM Granite model is placed in the "samples\genie\python\models\IBM-Granite-v3.1-8B" directory, and the' tokenizer.json' file corresponding to the Phi 3.5 model is placed in the "samples\genie\python\models\Phi-3.5-mini" directory.
If you want to modify the relative path of the directory where the model file is located, you need to modify the "config.json" file in the corresponding directory of the model to ensure that the 'tokenizer.json', 'htp_backend_ext_config.json' and model files set in the configuration file can be found correctly.
qai-appbuilder\samples\genie\python\models\<model name>\config.json
- You can also use your own LLM Genie model (if you have one). Refer to setup custom model part for detailed steps.
Run following commands in Windows terminal:
cd qai-appbuilder\samples
Run the following commands to launch Genie API Service (Do not close this terminal window while service is running)
python genie\python\GenieAPIService.py --modelname "IBM-Granite-v3.1-8B" --loadmodel --profile
The service prints the following log, indicating that GenieAPIService started successfully.
INFO: loading model <<< IBM-Granite-v3.1-8B >>>
[INFO] "Using create From Binary"
[INFO] "Allocated total size = 353404160 across 10 buffers"
INFO: model <<< IBM-Granite-v3.1-8B >>> is ready!
INFO: model init time: 4.71 (s)
INFO: Started service process [7608]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8910 (Press CTRL+C to quit)
--all-text: Output all text includes tool calls text. Disabled by default.
--enable-thinking: Enable thinking mode. Disabled by default.
*. It is recommended to disable thinking mode when using the tools call function.
*. You can refer to GenieAPIClientTools.py on how to use tools call.
python genie\python\GenieAPIService.py --modelname "Qwen3.0-8B-v31" --loadmodel --profile --all-text --enable-thinking
The default IP address for this API is: '127.0.0.1:8910', you can access this IP address in the client app.
You can run the following Python in a new terminal window.
The following command to generate text from text prompt:
python genie\python\GenieAPIClient.py --prompt "How to fish?" --stream
GenieAPIClient.py is command compatible with the C++ GenieAPIClient and only needs the Python standard library. It supports --prompt, --system, --model, --stream, --img, --audio, --host, --key, --raw_file, --bench and --v2 (send the request to /v2/chat/completions instead of /v1/chat/completions).
*. Text requests work with both the C++ and the Python version of the service. --img, --audio, --bench and --v2 need the C++ GenieAPIService, the Python 'GenieAPIService.py' only accepts plain text messages on /v1.
The following command to generate image from text prompt:
python genie\python\GenieAPIClientImage.py --prompt "spectacular view of northern lights from Alaska"
- When you run the client, you can see the current status of processing client requests from the service terminal window.
- When you run the request of image generation for the first time, the service may have to download the Stable Diffusion model from AI-Hub, and it will take a long time.
| Sample | Description |
|---|---|
| GenieAPIClient.py | A client sample code to call GenieAPIService to generate text, command compatible with the C++ GenieAPIClient. |
| GenieAPIClientTools.py | A client sample code to make tool call to GenieAPIService. |
| GenieAPIClientImage.py * | A client sample code to call GenieAPIService to generate image. |
| GenieSample.py | Use GenieContext to load and run the LLM model in local process. |
*. Image generation is only supported in the Python version of GenieAPIService.
| Model | model file | tokenizer file * |
|---|---|---|
| IBM Granite v3.1 8B | IBM-Granite-v3.1-8B-Instruct | tokenizer.json |
| Phi 3.5 mini ** | Phi-3.5-mini-instruct | tokenizer.json |
*. Tokenizer files will be downloaded automatically while you running 'GenieAPIService.py'.
**. For 'Phi-3.5-Mini-Instruct' model, to see appropriate spaces in the output, remove lines 192-197 (Strip rule) in the 'tokenizer.json' file if you download it manually. Remember to delete the extra commas on line 191. (If you download it through 'GenieAPIService.py', the Python script will modify the 'tokenizer.json' file automatically.)
The following is the correct change for the 'tokenizer.json' file of 'Phi-3.5-Mini-Instruct' model:
- },
- {
- "type": "Strip",
- "content": " ",
- "start": 1,
- "stop": 0
- }
+ }
***. Refer to setup Stable Diffusion v2.1 models before run 'GenieAPIService.py' (Our Python version 'GenieAPIService.py' support generating image, it depends on Stable Diffusion v2.1 sample code.)
| Model | model file |
|---|---|
| Qwen2-7B-SSD | Qwen2.0-7B-SSD-8380-2.34.zip |
| Qwen2.5-VL-3B | qwen2.5vl3b-8380-2.42.zip |
| Llama 3.2 3B | llama3.2-3b-8380-qnn2.37.zip |
| Llama 3.1 8B | llama3.1-8b-8380-qnn2.38.zip |
You can create a subdirectory in the path "qai-appbuilder\samples\genie\python\models" for your own model and customize the "config.json", "prompt.conf" or "prompt.json" files for your model. Both files should be stored in the same directory as your model files. Then use the new directory which you've created as your model name, the name can be used in the client application.
Note: "prompt.conf" is for old version GenieAPIServcie(< 2.0). Since GenieAPIServcie 2.0, we use "prompt.json" to replace it.
-
config.json : Model configuration file. It includes key parameters for the model. You can get several template configuration files for popular models such as Llama 2 & 3, Phi 3.5, Qwen 2 from here:
https://github.com/quic/ai-hub-apps/tree/main/tutorials/llm_on_genie/configs/genie
You need to make sure that the following parameters in the configuration file point to the correct file path.
a. tokenizer: The path to the model 'tokenizer.json' file.
b. extensions: The path to Genie extension configuration file. You can find it from path: 'qai-appbuilder\samples\genie\python\config\htp_backend_ext_config.json'
c. ctx-bins: The path to model context bin files(ctx-bins). Usually a model has 3~5 context bin files.
d. forecast-prefix-name:The path to the directory where SSD model 'kv-cache.primary.qnn-htp' file is stored. Only SSD model needs this parameter.
e. other parameters: Set other parameters according to your model. -
prompt.json : model prompt format configuration file. It defines the input format template for how the model organizes system prompts, user inputs, model responses and other information when receiving user requests.
Following is a format template for the prompt.json file of the QWEN model:
{
"prompt_system": "<|im_start|>system\n string <|im_end|>\n",
"prompt_user": "<|im_start|>user\n string <|im_end|>\n",
"prompt_assistant": "<|im_start|>assistant\n string <|im_end|>\n",
"prompt_tool": "<|im_start|>tool\n string <|im_end|>\n",
"prompt_start": "<|im_start|>assistant\n"
}
Field description:
a. prompt_system: the format template of the system role prompt, string will be replaced with the actual system prompt.
b. prompt_user: the format template of user input, string will be replaced with the user's actual problem.
c. prompt_assistant: the format template of the model's reply, string is replaced with the historical reply that the model has generated.
d. prompt_tool: the format template of return content of tool/function call, string is the placeholder for tool's retrun result.
e. prompt_start: generate a start tag that indicates "it's the model's turn to reply", and does not contain an end tag at the end, from where the model will start generating text.
The following link contains the prompt format of some other models.
https://github.com/quic/ai-hub-apps/tree/main/tutorials/llm_on_genie#prompt-formats
Note: Some models (such as Llama) do not require additional starting tokens, which can be left blank or set to a special token based on the tokenizer.
-
prompt.conf : (deprecated from GenieAPIService v2.0.0).Model prompt format configuration file. There are two lines in this file, which are used to set prompt_tags_1 and prompt_tags_2 parameters respectively. A complete prompt consists of the following contents:
prompt = prompt_tags_1 + < user questions > + prompt_tags_2
Taking the QWen 2 model as an example, a complete prompt example is as follows:
<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nHow to fish?<|im_end|>\n<|im_start|>assistant\n
In this example, the user's question is "How to fish?". Therefore, we extract the content before the question as the content of prompt_tags_1 and extract the content after the question as the content of prompt_tags_2. We set the content of 'promtp.conf' as follows:
prompt_tags_1: <|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n
prompt_tags_2: <|im_end|>\n<|im_start|>assistant\n
The following link contains the prompt format of some models.
https://github.com/quic/ai-hub-apps/tree/main/tutorials/llm_on_genie#prompt-formats -
For more information about LLM Genie model, you can refer to following link:
https://github.com/quic/ai-hub-apps/tree/main/tutorials/llm_on_genie