Skip to content

Fix VibeVoice-ASR streaming via generate() and server mapping#483

Open
JM27616 wants to merge 3 commits into
Blaizzy:mainfrom
JM27616:pc-fix-vibevoice-asr
Open

Fix VibeVoice-ASR streaming via generate() and server mapping#483
JM27616 wants to merge 3 commits into
Blaizzy:mainfrom
JM27616:pc-fix-vibevoice-asr

Conversation

@JM27616

@JM27616 JM27616 commented Feb 5, 2026

Copy link
Copy Markdown

Context

Streaming transcriptions for VibeVoice‑ASR were not incremental via the OpenAI-compatible endpoint (stream=true), so users only received output after completion. This aligns VibeVoice with the generate-centric streaming pattern used in #474 and fixes #482.

Description

  • Make Model.generate(stream=True) the single entrypoint for streaming and non‑streaming, returning an iterator of StreamingResult objects when streaming is enabled.
  • Update stream_transcribe() to emit structured streaming results with token timing estimates and final token counts.
  • Keep server-side compatibility by mapping stream -> generation_stream for models that accept generation_stream but not stream.
  • Improve iterator handling in the server to stream NDJSON chunks correctly for any generator.

Changes in the codebase

  • Added StreamingResult dataclass in VibeVoice‑ASR and exported it in __init__.py.
  • Updated VibeVoice‑ASR generate() to accept stream and return a generator when enabled.
  • Updated stream_transcribe() to yield StreamingResult objects instead of raw strings.
  • Adjusted server streaming to detect iterators and handle chunk types more robustly; kept legacy mapping for generation_stream.

Changes outside the codebase

None.

Additional information

Manual validation with curl -N confirms multiple NDJSON lines are emitted over time when stream=true, and non‑streaming behavior remains unchanged.

Checklist

@Blaizzy Blaizzy left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @JM27616

Thanks for the PR, this is a good start!

Please revert all the server.py changes.

This PR should be focused on adding streaming to VibeVoice-ASR.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a couple nits:

  1. This model is multilingual, so hardcoding language to "en" is not optimal.
  2. This model outputs structured outputs, so we need to think about streaming properly because I know users won't know how to decode the answers and even if they were they would need to instantiate the model on the client side to access the method to decode (see generate).

This is why I was creating the spec for this feature throughout the week.

@Blaizzy

Blaizzy commented Feb 5, 2026

Copy link
Copy Markdown
Owner

Could you also provide the Word Error rate for streaming vs generate on audios of 5min, 10min, 30min and if possible 1h.

So we can see if the implementation is accurate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Streaming doesn’t work for VibeVoice-ASR

2 participants