Skip to content

Support known speaker counts in MSDD diarization - #379

Open
ciklois-neo wants to merge 1 commit into
MahmoudAshraf97:mainfrom
ciklois-neo:feature/known-speaker-count
Open

ciklois-neo wants to merge 1 commit into
MahmoudAshraf97:mainfrom
ciklois-neo:feature/known-speaker-count

Conversation

@ciklois-neo

Copy link
Copy Markdown

Summary

  • add an optional num_speakers argument to MSDDDiarizer.diarize
  • include the known count in the NeMo manifest
  • pass it to NeuralDiarizer._initialize_configs, which enables NeMo's oracle speaker-count path
  • validate the supported 1–8 speaker range

Motivation

Automatic clustering can under-count speakers when voices are similar or some participants are quieter or farther from the microphone. Allowing downstream applications to provide a known count avoids that failure mode while preserving the existing automatic behavior when the argument is omitted.

Validation

Validated through a downstream Buzz integration with focused tests covering the manifest and NeMo configuration path. It was also tested successfully on a 30-minute Hungarian room recording with three participants, where automatic detection previously found only two speakers.

@MahmoudAshraf97

Copy link
Copy Markdown
Owner

Hello and thank you for your contribution

  1. This config parameter is only used for evaluation
  2. MSDD is scheduled for deprecation soon

@ciklois-neo

Copy link
Copy Markdown
Author

Thanks for the clarification. I checked the NeMo 2.8.0rc2 execution path again. NeuralDiarizer._initialize_configs() sets diarizer.clustering.parameters.oracle_num_speakers = num_speakers is not None, and perform_clustering() passes the manifest's num_speakers to LongFormSpeakerClustering.forward_infer(oracle_num_speakers=...). That also matches the behavior we observed in an actual 3-speaker recording. I understand from your note that this is intended for oracle/evaluation use rather than supported production configuration.

Given the planned MSDD deprecation, would you prefer that I close this PR and explore known-speaker-count support on Sortformer instead? If so, is there a supported way to constrain or otherwise guide the speaker count in the current Sortformer path?

@MahmoudAshraf97

Copy link
Copy Markdown
Owner

Yes exploring it on sortformer is more realistic, since it will eventually be replaced by another sortformer model that supports up to 8 speakers

@ciklois-neo

Copy link
Copy Markdown
Author

Thanks, that makes sense.

I will treat the current MSDD implementation as a transitional downstream compatibility path in Buzz rather than relying on it as a long-term upstream production API. Since it is opt-in and has solved a real under-counting case, Buzz can keep it while the MSDD backend remains available, with automatic detection still the default. In parallel, I will explore known-speaker guidance in Sortformer and the future Sortformer model you mentioned that supports up to eight speakers. The MSDD-specific option can be removed when that backend is retired.

Separately, I have completed the backend-independent speaker workflow in Buzz in PR #1587: speaker labels are stored separately from transcript text, can be edited or assigned to multiple rows, shown in a colored and filterable Speakers view, and exported to a colored Word document with optional timestamps. That layer will remain useful regardless of whether the diarizer is MSDD, Sortformer, or a future model.

Thank you for pointing me toward the realistic long-term direction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants