Skip to content

Latest commit

 

History

History
185 lines (160 loc) · 13.4 KB

README.md

File metadata and controls

185 lines (160 loc) · 13.4 KB

Run the large language model on the local NPU.

OpenAI Compatible API Service (Python)


Disclaimer

This software is provided “as is,” without any express or implied warranties. The authors and contributors shall not be held liable for any damages arising from its use. The code may be incomplete or insufficiently tested. Users are solely responsible for evaluating its suitability and assume all associated risks.
Note: Contributions are welcome. Please ensure thorough testing before deploying in critical systems.

Introduction

This sample helps developers use QAI AppBuilder + Python to build Genie based Open AI compatibility API service on Windows on Snapdragon (WoS) platform.
We suggest using the C++ version of GenieAPIService provided by us.

Features

• Support both stream and none stream mode
• Support tools call [NEW!]
• Support enable/disable thinking mode [NEW!]
• Support switching between models
• Support to extend the use of your own model

Setting Up Environment For Service:

Step 1: Install Dependencies

Refer to python.md on how to setup x64 version Python environment.

Step 2: Install basic Python dependencies for service

Run following commands in Windows terminal:

pip install uvicorn==0.34.0 pydantic_settings==2.10.1 fastapi==0.115.8 langchain==0.3.19 langchain_core==0.3.45 langchain_community==0.3.18 sse_starlette==2.2.1 pypdf==5.3.0 python-pptx==1.0.2 docx2txt==0.8 openai==1.63.2 json-repair==0.47.4

Step 3: Download models and tokenizer files

Download the files from the AI-Hub LLM models list at the end of this page, save them to following path.

qai-appbuilder\samples\genie\python\models\<model name>
  • Select 'Snapdragon® X Elite' as the device for WoS platform while downloading the model.
  • You need to unzip the 'weight_sharing_model_N_of_N.serialized.bin' files from model package and copy them to following path. Download and copy the corresponding 'tokenizer.json' file to the following directory path too.
  • Please be careful not to mix 'tokenizer.json' file of different models. Ensure that the' tokenizer.json' file corresponding to the IBM Granite model is placed in the "samples\genie\python\models\IBM-Granite-v3.1-8B" directory, and the' tokenizer.json' file corresponding to the Phi 3.5 model is placed in the "samples\genie\python\models\Phi-3.5-mini" directory.

If you want to modify the relative path of the directory where the model file is located, you need to modify the "config.json" file in the corresponding directory of the model to ensure that the 'tokenizer.json', 'htp_backend_ext_config.json' and model files set in the configuration file can be found correctly.

qai-appbuilder\samples\genie\python\models\<model name>\config.json
  • You can also use your own LLM Genie model (if you have one). Refer to setup custom model part for detailed steps.

Step 4: Switch to samples directory

Run following commands in Windows terminal:

cd qai-appbuilder\samples

Step 5: Run service

Run the following commands to launch Genie API Service (Do not close this terminal window while service is running)

python genie\python\GenieAPIService.py --modelname "IBM-Granite-v3.1-8B" --loadmodel --profile

The service prints the following log, indicating that GenieAPIService started successfully.

INFO:     loading model <<< IBM-Granite-v3.1-8B >>>
[INFO]  "Using create From Binary"
[INFO]  "Allocated total size = 353404160 across 10 buffers"
INFO:     model <<< IBM-Granite-v3.1-8B >>> is ready!
INFO:     model init time: 4.71 (s)
INFO:     Started service process [7608]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8910 (Press CTRL+C to quit)

More parameters:

--all-text: Output all text includes tool calls text. Disabled by default.
--enable-thinking: Enable thinking mode. Disabled by default.
*. It is recommended to disable thinking mode when using the tools call function.
*. You can refer to GenieAPIClientTools.py on how to use tools call.

python genie\python\GenieAPIService.py --modelname "Qwen3.0-8B-v31" --loadmodel --profile --all-text --enable-thinking

Step 6: Now you can access the API service

The default IP address for this API is: '127.0.0.1:8910', you can access this IP address in the client app. You can run the following Python in a new terminal window.
The following command to generate text from text prompt:

python genie\python\GenieAPIClient.py --prompt "How to fish?" --stream

GenieAPIClient.py is command compatible with the C++ GenieAPIClient and only needs the Python standard library. It supports --prompt, --system, --model, --stream, --img, --audio, --host, --key, --raw_file, --bench and --v2 (send the request to /v2/chat/completions instead of /v1/chat/completions).
*. Text requests work with both the C++ and the Python version of the service. --img, --audio, --bench and --v2 need the C++ GenieAPIService, the Python 'GenieAPIService.py' only accepts plain text messages on /v1.
The following command to generate image from text prompt:

python genie\python\GenieAPIClientImage.py --prompt "spectacular view of northern lights from Alaska"
  • When you run the client, you can see the current status of processing client requests from the service terminal window.
  • When you run the request of image generation for the first time, the service may have to download the Stable Diffusion model from AI-Hub, and it will take a long time.

Sample List:

Sample Description
GenieAPIClient.py A client sample code to call GenieAPIService to generate text, command compatible with the C++ GenieAPIClient.
GenieAPIClientTools.py A client sample code to make tool call to GenieAPIService.
GenieAPIClientImage.py * A client sample code to call GenieAPIService to generate image.
GenieSample.py Use GenieContext to load and run the LLM model in local process.

*. Image generation is only supported in the Python version of GenieAPIService.

AI-Hub LLM models:

Model model file tokenizer file *
IBM Granite v3.1 8B IBM-Granite-v3.1-8B-Instruct tokenizer.json
Phi 3.5 mini ** Phi-3.5-mini-instruct tokenizer.json

*. Tokenizer files will be downloaded automatically while you running 'GenieAPIService.py'.
**. For 'Phi-3.5-Mini-Instruct' model, to see appropriate spaces in the output, remove lines 192-197 (Strip rule) in the 'tokenizer.json' file if you download it manually. Remember to delete the extra commas on line 191. (If you download it through 'GenieAPIService.py', the Python script will modify the 'tokenizer.json' file automatically.)
The following is the correct change for the 'tokenizer.json' file of 'Phi-3.5-Mini-Instruct' model:

-      },
-      {
-        "type": "Strip",
-        "content": " ",
-        "start": 1,
-        "stop": 0
-      }
+      }

***. Refer to setup Stable Diffusion v2.1 models before run 'GenieAPIService.py' (Our Python version 'GenieAPIService.py' support generating image, it depends on Stable Diffusion v2.1 sample code.)

AI-Dev LLM models:

Model model file
Qwen2-7B-SSD Qwen2.0-7B-SSD-8380-2.34.zip
Qwen2.5-VL-3B qwen2.5vl3b-8380-2.42.zip
Llama 3.2 3B llama3.2-3b-8380-qnn2.37.zip
Llama 3.1 8B llama3.1-8b-8380-qnn2.38.zip

Setup custom model:

You can create a subdirectory in the path "qai-appbuilder\samples\genie\python\models" for your own model and customize the "config.json", "prompt.conf" or "prompt.json" files for your model. Both files should be stored in the same directory as your model files. Then use the new directory which you've created as your model name, the name can be used in the client application.
Note: "prompt.conf" is for old version GenieAPIServcie(< 2.0). Since GenieAPIServcie 2.0, we use "prompt.json" to replace it.

  1. config.json : Model configuration file. It includes key parameters for the model. You can get several template configuration files for popular models such as Llama 2 & 3, Phi 3.5, Qwen 2 from here:
    https://github.com/quic/ai-hub-apps/tree/main/tutorials/llm_on_genie/configs/genie

    You need to make sure that the following parameters in the configuration file point to the correct file path.
    a. tokenizer: The path to the model 'tokenizer.json' file.
    b. extensions: The path to Genie extension configuration file. You can find it from path: 'qai-appbuilder\samples\genie\python\config\htp_backend_ext_config.json'
    c. ctx-bins: The path to model context bin files(ctx-bins). Usually a model has 3~5 context bin files.
    d. forecast-prefix-name:The path to the directory where SSD model 'kv-cache.primary.qnn-htp' file is stored. Only SSD model needs this parameter.
    e. other parameters: Set other parameters according to your model.

  2. prompt.json : model prompt format configuration file. It defines the input format template for how the model organizes system prompts, user inputs, model responses and other information when receiving user requests.
    Following is a format template for the prompt.json file of the QWEN model:

{
  "prompt_system": "<|im_start|>system\n string <|im_end|>\n",
  "prompt_user": "<|im_start|>user\n string <|im_end|>\n",
  "prompt_assistant": "<|im_start|>assistant\n string <|im_end|>\n",
  "prompt_tool": "<|im_start|>tool\n string <|im_end|>\n",
  "prompt_start": "<|im_start|>assistant\n"
}

Field description:
a. prompt_system: the format template of the system role prompt, string will be replaced with the actual system prompt.
b. prompt_user: the format template of user input, string will be replaced with the user's actual problem.
c. prompt_assistant: the format template of the model's reply, string is replaced with the historical reply that the model has generated.
d. prompt_tool: the format template of return content of tool/function call, string is the placeholder for tool's retrun result.
e. prompt_start: generate a start tag that indicates "it's the model's turn to reply", and does not contain an end tag at the end, from where the model will start generating text.

The following link contains the prompt format of some other models.
https://github.com/quic/ai-hub-apps/tree/main/tutorials/llm_on_genie#prompt-formats
Note: Some models (such as Llama) do not require additional starting tokens, which can be left blank or set to a special token based on the tokenizer.

  1. prompt.conf : (deprecated from GenieAPIService v2.0.0).Model prompt format configuration file. There are two lines in this file, which are used to set prompt_tags_1 and prompt_tags_2 parameters respectively. A complete prompt consists of the following contents:
    prompt = prompt_tags_1 + < user questions > + prompt_tags_2

    Taking the QWen 2 model as an example, a complete prompt example is as follows:
    <|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nHow to fish?<|im_end|>\n<|im_start|>assistant\n

    In this example, the user's question is "How to fish?". Therefore, we extract the content before the question as the content of prompt_tags_1 and extract the content after the question as the content of prompt_tags_2. We set the content of 'promtp.conf' as follows:
    prompt_tags_1: <|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n
    prompt_tags_2: <|im_end|>\n<|im_start|>assistant\n

    The following link contains the prompt format of some models.
    https://github.com/quic/ai-hub-apps/tree/main/tutorials/llm_on_genie#prompt-formats

  2. For more information about LLM Genie model, you can refer to following link:
    https://github.com/quic/ai-hub-apps/tree/main/tutorials/llm_on_genie