Repository navigation
Description
Is your feature request related to a problem?
Assessment configs can declare audio input columns only when the assessment and pre-filter provider is Google (google-aistudio / google-gcp). Configs on OpenAI are rejected at save, so clients who assess voice submissions have to move the whole config to Gemini.
What we verified against the OpenAI API
Live calls on api.openai.com, with a 2-second mp3/wav clip:
- GPT-5 text models (
gpt-5throughgpt-5.6-luna/terra/sol,gpt-6-luna) reject audio on both the Responses API (Audio input is not available.) and Chat Completions (Content blocks are expected to be either text or image_url type.). gpt-audio,gpt-audio-1.5,gpt-audio-miniacceptinput_audioon Chat Completions only. They reject it on the Responses API, reject bothjson_schemaandjson_objectoutput, and are refused by the Batch API (model ... is not supported by the Batch API).- OpenRouter documents
input_audioon its own/api/v1/responses; that is OpenRouter's gateway, not OpenAI's API.
So "attach audio to an OpenAI text model" is not possible today, and the audio models cannot run BATCH or structured-output assessments.
Describe the solution you'd like
Support audio columns for OpenAI configs on the API-client paths (BATCH, RESPONSE and their pre-filter), with the audio attached directly to GPT-5 and GPT-6 family models:
- Each audio cell is sent as an
input_audiocontent part in the same request as the text prompt and the other attachments (images, PDFs), the way Gemini receives audio today. No separate transcription step. - OpenAI takes audio as base64 (
mp3/wav), not by URL, so the request builder downloads the cell URL, converts other formats tomp3, and inlines it. - Batch,
json_output_schemaand the existing Responses request builder keep working unchanged for these models. - A row whose audio cannot be downloaded or converted is reported per row in
stage_errors; the rest of the stage still runs. - Legacy RUN stays out of scope and keeps rejecting OpenAI + audio.
- Blocked until OpenAI enables audio input on GPT-5 / GPT-6 models (see above). Re-run the live check before starting; the builder side (download, base64, per-row errors) can be prepared in the meantime.
Describe alternatives you've considered
gpt-audiomodels through Chat Completions: blocked for BATCH (no Batch API support) and for structured output.- Keep audio Google-only: no work, but OpenAI configs stay unable to take voice submissions.
Additional context
- Gate:
UNSUPPORTED_COLUMN_TYPES_BY_PROVIDERinbackend/app/models/config/assessment_blob.py. - Google audio support landed in feat(assessment): Add audio input column #1317.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsTo Do
Activity