Skip to content

README.md

Latest commit

 

History

History
123 lines (88 loc) · 4.04 KB

README.md

File metadata and controls

123 lines (88 loc) · 4.04 KB
Raw

OpenRefine OpenAI API Integration

This repository contains a Python script that integrates with the OpenAI API to process data from OpenRefine. The script reads a cell value, builds a prompt, and sends a request to the OpenAI API based on configuration settings provided in a JSON file. The response, containing processed arguments, is returned diectly to OpenRefine in the new added column.

Features

  • Reads configuration settings (API key, model, max tokens, temperature, and endpoint) from a JSON file (config.json).
  • Loads a custom system prompt from a text file (system_prompt.txt).
  • Supports external tools specified in a tools.json file.
  • Automatically handles errors and returns them in a readable format.
  • Includes timeout and rate-limiting mechanisms.

Prerequisites

  • Python 3.6+
  • An OpenAI API key
  • OpenRefine (optional, if integrating with OpenRefine)

Installation

  1. Clone this repository:

    git clone https://github.com/dvnr/openrefine-api-integration.git
cd openrefine-api-integration

  2. Create the required configuration files:

    • config.json: Contains API and model configurations.
    • system_prompt.txt: Contains the custom system prompt text.
    • tools.json: Lists the tools to be included in the request.

    Example config.json:

    {
  "model": "gpt-4",
  "max_tokens": 500,
  "temperature": 0,
  "endpoint": "https://api.openai.com/v1/chat/completions",
  "api_key": "your_openai_api_key"
}

    Example tools.json:

    {
    "type": "function",
    "function": {
        "name": "format_dates",
        "description": "Return formatted dates in JSON format with 'start' and 'end' following ISO 8601 format.",
        "parameters": {
            "type": "object",
            "required": ["start", "end"],
            "properties": {
                "start": {
                    "type": "string",
                    "description": "The start date in ISO 8601 format."
                },
                "end": {
                    "type": "string",
                    "description": "The end date in ISO 8601 format."
                }
            },
            "additionalProperties": false
        },
        "strict": true
    }
}

Usage with OpenRefine

To use this script within OpenRefine, you can execute it by calling a subprocess from OpenRefine’s Python/Jython environment. This approach passes the current cell’s value to the Python script and returns the API’s response.

  1. Choose Edit column → Add column based on this column.

  2. In the expressions editor window, select "Clojure & Jython" as epxression language.

  3. Run this code in OpenRefine

import subprocess

output = subprocess.check_output(["/path/to/python", "/path/to/main.py", value.encode("utf-8")])

return output.decode("utf-8")
  • Adjust "/path/to/python" to the location of your Python executor and "/path/to/main.py" to the path of the main script (main.py).

Parameters

  • cell_value: The value of the cell to be processed by the script.

Configuration

Modify the settings in config.json to specify:

  • model: The model to use for the API call.
  • max_tokens: The maximum tokens to generate in the API response.
  • temperature: Controls the creativity of the response.
  • endpoint: The API endpoint URL.
  • api_key: Your OpenAI API key (required).

Example Files

  • system_prompt.txt: Contains the system prompt text, which sets the context for API requests.
  • tools.json: Defines external tools available to the model.

Error Handling

The script handles potential errors, including:

  • HTTP Errors: Displays the HTTP status code and message if the API request fails.
  • URL Errors: Handles issues with connecting to the API endpoint.
  • Rate Limiting: Includes a sleep delay to avoid exceeding API rate limits.

License

This project is licensed under the MIT License. See the LICENSE file for details.