Skip to content

Assessment: OpenAI audio integration #1344

Open

Description

@vprashrex

Is your feature request related to a problem?
Assessment configs can declare audio input columns only when the assessment and pre-filter provider is Google (google-aistudio / google-gcp). Configs on OpenAI are rejected at save, so clients who assess voice submissions have to move the whole config to Gemini.

What we verified against the OpenAI API
Live calls on api.openai.com, with a 2-second mp3/wav clip:

  • GPT-5 text models (gpt-5 through gpt-5.6-luna / terra / sol, gpt-6-luna) reject audio on both the Responses API (Audio input is not available.) and Chat Completions (Content blocks are expected to be either text or image_url type.).
  • gpt-audio, gpt-audio-1.5, gpt-audio-mini accept input_audio on Chat Completions only. They reject it on the Responses API, reject both json_schema and json_object output, and are refused by the Batch API (model ... is not supported by the Batch API).
  • OpenRouter documents input_audio on its own /api/v1/responses; that is OpenRouter's gateway, not OpenAI's API.

So "attach audio to an OpenAI text model" is not possible today, and the audio models cannot run BATCH or structured-output assessments.

Describe the solution you'd like
Support audio columns for OpenAI configs on the API-client paths (BATCH, RESPONSE and their pre-filter), with the audio attached directly to GPT-5 and GPT-6 family models:

  • Each audio cell is sent as an input_audio content part in the same request as the text prompt and the other attachments (images, PDFs), the way Gemini receives audio today. No separate transcription step.
  • OpenAI takes audio as base64 (mp3 / wav), not by URL, so the request builder downloads the cell URL, converts other formats to mp3, and inlines it.
  • Batch, json_output_schema and the existing Responses request builder keep working unchanged for these models.
  • A row whose audio cannot be downloaded or converted is reported per row in stage_errors; the rest of the stage still runs.
  • Legacy RUN stays out of scope and keeps rejecting OpenAI + audio.
  • Blocked until OpenAI enables audio input on GPT-5 / GPT-6 models (see above). Re-run the live check before starting; the builder side (download, base64, per-row errors) can be prepared in the meantime.

Describe alternatives you've considered

  • gpt-audio models through Chat Completions: blocked for BATCH (no Batch API support) and for structured output.
  • Keep audio Google-only: no work, but OpenAI configs stay unable to take voice submissions.

Additional context

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions