Skip to content

Add structured transcription results#6642

Draft
wskr00 wants to merge 2 commits into
spring-projects:mainfrom
wskr00:feat/structured-transcription
Draft

Add structured transcription results#6642
wskr00 wants to merge 2 commits into
spring-projects:mainfrom
wskr00:feat/structured-transcription

Conversation

@wskr00

@wskr00 wskr00 commented Jul 18, 2026

Copy link
Copy Markdown

Summary

  • Add AudioTranscriptionResult, AudioTranscriptionSegment, and AudioTranscriptionWord to expose structured transcription data.
  • Preserve language, duration, segments, words, and speaker annotations from OpenAI verbose and diarized responses.
  • Preserve diarized segment data for streaming responses.
  • Update the transcription reference documentation.
  • Close audio resource input streams and simplify redundant option checks.

Breaking Change

AudioTranscription.getOutput() now returns AudioTranscriptionResult instead of String.

Use .text() when only the transcription text is needed:

String text = response.getResult().getOutput().text();

Testing

  • ./mvnw process-sources
  • ./mvnw javadoc:javadoc
  • ./mvnw clean package
  • ./mvnw -pl spring-ai-docs antora

Closes #6640

Expose structured transcription data from OpenAI responses, including
duration, language, segments, words, and diarized speaker annotations.

Close audio resource input streams and simplify redundant option checks.

Closes spring-projects#6640

Signed-off-by: Lucas Christian <lucaschgf7@gmail.com>
@wskr00

wskr00 commented Jul 18, 2026

Copy link
Copy Markdown
Author

Converted this PR to draft because the OpenAI Java SDK currently treats diarized_json as a string response and wraps the raw JSON in Transcription.text.

I've found a upstream issue and a PR. Once the SDK issue is resolved, I will update this PR accordingly.

Signed-off-by: Lucas Christian <lucaschgf7@gmail.com>
@wskr00

wskr00 commented Jul 19, 2026

Copy link
Copy Markdown
Author

I found another upstream issue affecting the current diarized implementation.

I tested the OpenAI API directly with gpt-4o-transcribe-diarize and response_format=diarized_json. Despite calling OpenAI itself, the response did not contain duration.

However, duration is modeled as required by the OpenAI Java SDK. When Spring AI accesses TranscriptionDiarized.duration(), the SDK throws an OpenAIInvalidDataException because the field is not set.

I have opened an upstream issue and PR to address the SDK deserialization problem.

I am waiting for feedback from the OpenAI team before continuing with the diarized integration, since merging the current implementation without handling this case would cause a runtime error for a response returned by the OpenAI API.

My current expectation is that the field may remain required in the generated SDK model; the OpenAI API schema and SDKs in other languages currently describe duration as non-optional. However, the API behavior observed in practice conflicts with that contract, so the Java SDK needs a way to handle the omitted value safely.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TranscriptionModel returns plain text for all response formats, discarding structured data (diarized, verbose, JSON)

2 participants