Skip to content

Commit 02a5dde

Browse files
Shinji Watanabeclaude
authored and
Shinji Watanabe
committed
Separate a Libri2Mix mixture, and score it
The separation section mixed two ESPnet test recordings by hand, because wsj0-2mix is built on WSJ0 and cannot be published. Libri2Mix is built on LibriSpeech, which is CC BY 4.0, so this uses the real thing instead: mixture 7729-102255-0031_2094-142345-0028 of the Libri2Mix test set, built here from its two sources the way LibriMix builds it - the gains the official metadata gives for that pair, resampled to 8 kHz, truncated to the shorter source. The model moved with the data: espnet/anogkongda_librimix_enh_train_raw, a Conv-TasNet trained on Libri2Mix, and the ASR after it is a LibriSpeech transformer, since that is where both recordings come from. Having the two sources is the point. The separation is now scored rather than only listened to, by the notebook itself: speaker 1: 3.24 dB in the mixture -> 16.21 dB separated speaker 2: -3.28 dB in the mixture -> 12.88 dB separated Mix Speech to Text: HE CONTINUED ON HIS GROUND AND SIR HOWDY HUNDRED HE OF WET COLLARED TO HIS LATE ERRAND CARRYING TWO AROUND Separated Speech 1: HE CONTINUED HIS PRETENDED SEARCH AND TO GIVE COLOR TO HIS ERRAND MADE TO ARREST Separated Speech 2: YOU KNOW NO NO TOTTY HAD AT HER FEET WET SAID MISSUS POYSER CARRYING AWAY HER IRON Two and ten characters from the references, against a mixture that comes out as nonsense. That model could not be loaded at all until today: a TCN checkpoint from before May 2023 cannot be built by current espnet, because TCNSeparator defaults to a layout its weights predate. Its config on the Hub now records the layout it was trained with, and so do four others in the organisation that were unloadable for the same reason. The two source recordings are published beside it, so this notebook needs neither the corpus nor a Drive link. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 0f40d38 commit 02a5dde

2 files changed

Lines changed: 149 additions & 61 deletions

File tree

‎Courses/CMUSpeechTechnology26S/README.md‎

Lines changed: 25 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -36,13 +36,31 @@ something the notebook can fetch on its own:
3636
built on WSJ0. The page's sample is single-channel where the corpus has six;
3737
the enhancement in the notebook is single-channel either way, so the
3838
demonstration is unchanged.
39-
- **wsj0-2mix.** The separation model is trained on mixtures built from WSJ0,
40-
which cannot be published, and the course's mixture came from Drive. The
41-
notebook now builds one the same way at runtime — two ESPnet test recordings
42-
of different speakers, summed at equal level and normalised. It is a
43-
different mixture from the course's, and an easier one: two clean utterances
44-
with no shared channel. The separation and the ASR scoring after it still
45-
show what they were there to show.
39+
- **wsj0-2mix → Libri2Mix.** The separation section used a mixture from
40+
wsj0-2mix, which is built on WSJ0 and cannot be published, fetched from
41+
Drive. It now separates a **Libri2Mix** mixture instead, and the model
42+
changed with the data:
43+
[`espnet/anogkongda_librimix_enh_train_raw_valid.si_snr.ave`](https://huggingface.co/espnet/anogkongda_librimix_enh_train_raw_valid.si_snr.ave),
44+
a Conv-TasNet trained on Libri2Mix. Libri2Mix is built on LibriSpeech, which
45+
is CC BY 4.0, so the two sources could be published beside the model; the
46+
notebook builds the mixture from them the way LibriMix does — the gains the
47+
official metadata gives for that pair, resampled to 8 kHz, truncated to the
48+
shorter source. It is the real test mixture
49+
`7729-102255-0031_2094-142345-0028`, not an improvisation.
50+
51+
Having the sources means the separation is **scored** rather than only
52+
listened to: SI-SNR 3.24 → 16.21 dB for one speaker and −3.28 → 12.88 dB for
53+
the other. The ASR that follows moved to a LibriSpeech model for the same
54+
reason, and it reads the separated streams almost perfectly while the
55+
mixture comes out as nonsense, which is the point of the section.
56+
57+
The noisy half of Libri2Mix adds WHAM! noise, CC BY-NC 4.0, and is not used.
58+
59+
That model did not load at all before this: a checkpoint from before May
60+
2023 cannot be built by current espnet, because `TCNSeparator` defaults to a
61+
layout its weights predate. Its config now records the layout it was trained
62+
with, and so do four others in the organisation that were unloadable for the
63+
same reason.
4664

4765
## Running them outside Colab
4866

‎Courses/CMUSpeechTechnology26S/speech_enhancement.ipynb‎

Lines changed: 124 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -311,11 +311,17 @@
311311
"source": [
312312
"### Model selection\n",
313313
"\n",
314-
"The separation models trained on wsj0_2mix are published in the [ESPnet\n",
315-
"organisation](https://huggingface.co/espnet) and load by tag. The course used\n",
316-
"a TF-GridNet checkpoint kept on Google Drive; the Hub copy of that one is an\n",
317-
"empty repository, so this uses DPTNet instead, which is published properly and\n",
318-
"separates the same mixtures.\n"
314+
"Separation is a different task from the enhancement above: one microphone, two\n",
315+
"people talking at once, and one output per speaker rather than one clean\n",
316+
"output.\n",
317+
"\n",
318+
"The model is\n",
319+
"[`espnet/anogkongda_librimix_enh_train_raw_valid.si_snr.ave`](https://huggingface.co/espnet/anogkongda_librimix_enh_train_raw_valid.si_snr.ave),\n",
320+
"a Conv-TasNet trained on Libri2Mix at 8 kHz and published in the ESPnet\n",
321+
"organisation. The course used a TF-GridNet checkpoint kept on Google Drive,\n",
322+
"and a wsj0-2mix mixture from the same place; wsj0-2mix is built on WSJ0 and\n",
323+
"cannot be published, while Libri2Mix is built on LibriSpeech, which is CC BY\n",
324+
"4.0 — so this notebook can fetch both the model and the audio it separates."
319325
]
320326
},
321327
{
@@ -327,16 +333,10 @@
327333
"from espnet2.bin.enh_inference import SeparateSpeech\n",
328334
"\n",
329335
"separate_speech = SeparateSpeech.from_pretrained(\n",
330-
" model_tag=\"espnet/Wangyou_Zhang_wsj0_2mix_enh_train_enh_dptnet_raw\",\n",
331-
" # for segment-wise process on long speech\n",
332-
" segment_size=2.4,\n",
333-
" hop_size=0.8,\n",
334-
" normalize_segment_scale=False,\n",
335-
" show_progressbar=True,\n",
336-
" ref_channel=None,\n",
336+
" model_tag=\"espnet/anogkongda_librimix_enh_train_raw_valid.si_snr.ave\",\n",
337337
" normalize_output_wav=True,\n",
338338
" device=\"cuda\" if torch.cuda.is_available() else \"cpu\",\n",
339-
")\n"
339+
")"
340340
],
341341
"execution_count": null,
342342
"outputs": []
@@ -352,48 +352,104 @@
352352
"cell_type": "markdown",
353353
"metadata": {},
354354
"source": [
355-
"#### Separate the example in wsj0_2mix testing set\n"
355+
"#### Separate one mixture from the Libri2Mix test set"
356356
]
357357
},
358358
{
359359
"cell_type": "markdown",
360360
"source": [
361-
"### Task2 )\n",
361+
"The mixture is built here rather than downloaded, the way LibriMix builds its\n",
362+
"own: take the two gains the official metadata gives for this pair, resample to\n",
363+
"the model's 8 kHz, truncate to the shorter source — LibriMix's `min` mode —\n",
364+
"and add them together.\n",
362365
"\n",
363-
"Run inference of pretrained speech seperation model based on TF-GRIDNET.\n"
366+
"Building it from the two sources is worth the six lines: it means the\n",
367+
"references are in hand, so the separation can be **scored** further down\n",
368+
"rather than only listened to."
364369
],
365370
"metadata": {}
366371
},
367372
{
368373
"cell_type": "code",
369374
"metadata": {},
370375
"source": [
371-
"# Two people talking at once. The model was trained on wsj0-2mix, whose\n",
372-
"# mixtures are built from WSJ0 and so cannot be published; this makes one the\n",
373-
"# same way - two recordings of different speakers, summed at equal level.\n",
374-
"!wget -q -O spk1.wav https://github.com/espnet/espnet/raw/master/test_utils/ctc_align_test.wav\n",
375-
"!wget -q -O spk2.wav https://github.com/espnet/espnet/raw/master/test_utils/st_test.wav\n",
376+
"# Mixture 7729-102255-0031_2094-142345-0028 of the Libri2Mix test set, named in\n",
377+
"# metadata/Libri2Mix/libri2mix_test-clean.csv of the LibriMix repository. The\n",
378+
"# two sources are LibriSpeech test-clean utterances (CC BY 4.0), kept beside\n",
379+
"# the model so that this notebook does not need the corpus.\n",
380+
"!wget -q -O spk1.flac https://huggingface.co/espnet/anogkongda_librimix_enh_train_raw_valid.si_snr.ave/resolve/main/samples/7729-102255-0031.flac\n",
381+
"!wget -q -O spk2.flac https://huggingface.co/espnet/anogkongda_librimix_enh_train_raw_valid.si_snr.ave/resolve/main/samples/2094-142345-0028.flac\n",
376382
"\n",
377-
"import os\n",
378383
"import numpy as np\n",
379384
"import soundfile\n",
380385
"from IPython.display import display, Audio\n",
386+
"from scipy.signal import resample_poly\n",
381387
"\n",
382-
"first, sr = soundfile.read(\"spk1.wav\")\n",
383-
"second, _ = soundfile.read(\"spk2.wav\")\n",
384-
"length = min(len(first), len(second))\n",
385-
"mixwav = first[:length] + second[:length]\n",
386-
"mixwav = mixwav / np.abs(mixwav).max() * 0.9\n",
387-
"soundfile.write(\"two_speakers.wav\", mixwav, sr)\n",
388-
"waves_wsj = separate_speech(mixwav[None, ...], fs=sr)\n",
388+
"# the gains that row of the metadata gives, and the model's sampling rate\n",
389+
"GAINS = {\"spk1.flac\": 0.5159350719675272, \"spk2.flac\": 0.22360240150798563}\n",
390+
"FS = 8000\n",
391+
"\n",
392+
"refs = []\n",
393+
"for name, gain in GAINS.items():\n",
394+
" source, fs = soundfile.read(name, dtype=\"float32\")\n",
395+
" refs.append(resample_poly(source * gain, FS, fs))\n",
396+
"length = min(len(ref) for ref in refs) # LibriMix's \"min\" mode\n",
397+
"refs = [ref[:length] for ref in refs]\n",
398+
"mixwav = sum(refs)\n",
399+
"sr = FS\n",
400+
"\n",
401+
"waves_librimix = separate_speech(mixwav[None, ...], fs=sr)\n",
389402
"\n",
390403
"print(\"Input mixture\", flush=True)\n",
391404
"display(Audio(mixwav, rate=sr))\n",
392-
"print(f\"========= Separated speech with model =========\", flush=True)\n",
393-
"print(\"Separated spk1\", flush=True)\n",
394-
"display(Audio(waves_wsj[0].squeeze(), rate=sr))\n",
395-
"print(\"Separated spk2\", flush=True)\n",
396-
"display(Audio(waves_wsj[1].squeeze(), rate=sr))"
405+
"for i, wave in enumerate(waves_librimix, start=1):\n",
406+
" print(f\"========= Separated speaker {i} =========\", flush=True)\n",
407+
" display(Audio(wave.squeeze(), rate=sr))"
408+
],
409+
"execution_count": null,
410+
"outputs": []
411+
},
412+
{
413+
"cell_type": "markdown",
414+
"metadata": {},
415+
"source": [
416+
"#### How much of each speaker came back\n",
417+
"\n",
418+
"Both sources are in hand, so this is measurable. SI-SNR asks how much of the\n",
419+
"signal is the speaker you wanted and how much is everything else, after the\n",
420+
"best scaling — one number per speaker, in dB, for the mixture going in and for\n",
421+
"the separated stream coming out."
422+
]
423+
},
424+
{
425+
"cell_type": "code",
426+
"metadata": {},
427+
"source": [
428+
"from itertools import permutations\n",
429+
"\n",
430+
"\n",
431+
"def si_snr(estimate, reference):\n",
432+
" \"\"\"Scale-invariant signal-to-noise ratio, in dB.\"\"\"\n",
433+
" estimate = estimate - estimate.mean()\n",
434+
" reference = reference - reference.mean()\n",
435+
" scale = np.dot(estimate, reference) / np.dot(reference, reference)\n",
436+
" target = scale * reference\n",
437+
" noise = estimate - target\n",
438+
" return 10 * np.log10(np.dot(target, target) / np.dot(noise, noise))\n",
439+
"\n",
440+
"\n",
441+
"estimates = [wave.squeeze() for wave in waves_librimix]\n",
442+
"# separation has no speaker order, so the scoring picks the better pairing\n",
443+
"order = max(\n",
444+
" permutations(range(len(estimates))),\n",
445+
" key=lambda p: sum(si_snr(estimates[p[i]], refs[i]) for i in range(len(refs))),\n",
446+
")\n",
447+
"estimates = [estimates[i] for i in order]\n",
448+
"\n",
449+
"for i, (estimate, reference) in enumerate(zip(estimates, refs), start=1):\n",
450+
" before = si_snr(mixwav[: len(reference)], reference)\n",
451+
" after = si_snr(estimate[: len(reference)], reference)\n",
452+
" print(f\"speaker {i}: {before:6.2f} dB in the mixture -> {after:6.2f} dB separated\")"
397453
],
398454
"execution_count": null,
399455
"outputs": []
@@ -454,14 +510,15 @@
454510
"\n",
455511
"fig, axes = plt.subplots(3, 2, figsize=(18, 9))\n",
456512
"for row, (wav, name) in enumerate(\n",
457-
" [(mixwav, \"Mixture\"), (waves_wsj[0], \"Separated (spk1)\"),\n",
458-
" (waves_wsj[1], \"Separated (spk2)\")]\n",
513+
" [(mixwav, \"Mixture\"), (waves_librimix[0], \"Separated (spk1)\"),\n",
514+
" (waves_librimix[1], \"Separated (spk2)\")]\n",
459515
"):\n",
460516
" plot_spectrogram(axes[row][0], wav, f\"{name} spectrogram\")\n",
461517
" plot_waveform(axes[row][1], wav, f\"{name} waveform\")\n",
462518
"axes[2][1].set_xlabel(\"Time (s)\")\n",
463519
"plt.tight_layout()\n",
464-
"plt.show()\n"
520+
"plt.show()\n",
521+
""
465522
],
466523
"execution_count": null,
467524
"outputs": []
@@ -470,15 +527,17 @@
470527
"cell_type": "markdown",
471528
"metadata": {},
472529
"source": [
473-
"## Evaluate separated speech with pretrained ASR model\n",
530+
"## Evaluate separated speech with a pretrained ASR model\n",
474531
"\n",
475-
"The ground truths are:\n",
532+
"SI-SNR says how much of each speaker came back. This asks the question the\n",
533+
"other way round: can something else use the result? The transcripts of the two\n",
534+
"sources are\n",
476535
"\n",
477-
"`text_1: SOME CRITICS INCLUDING HIGH REAGAN ADMINISTRATION OFFICIALS ARE RAISING THE ALARM THAT THE FED'S POLICY IS TOO TIGHT AND COULD CAUSE A RECESSION NEXT YEAR`\n",
536+
"`text_1: HE CONTINUED HIS PRETENDED SEARCH AND TO GIVE COLOR TO HIS ERRAND MADE TWO ARRESTS`\n",
478537
"\n",
479-
"`text_2: THE UNITED STATES UNDERTOOK TO DEFEND WESTERN EUROPE AGAINST SOVIET ATTACK`\n",
538+
"`text_2: NO NO NO TOTTY UD GET HER FEET WET SAID MISSUS POYSER CARRYING AWAY HER IRON`\n",
480539
"\n",
481-
"(This may take a while for the speech recognition.)\n"
540+
"(This may take a while for the speech recognition.)"
482541
]
483542
},
484543
{
@@ -493,9 +552,12 @@
493552
{
494553
"cell_type": "markdown",
495554
"source": [
496-
"### Task3 )\n",
555+
"### Recognise the mixture, and the separated streams\n",
497556
"\n",
498-
"Show inference of pre-trained ASR model (trained on WSJ corpus) on mixed and seperated speech.\n"
557+
"The model is a transformer trained on LibriSpeech, which is where both of\n",
558+
"these recordings come from. It runs at 16 kHz while separation ran at 8 kHz,\n",
559+
"so each waveform is resampled on the way in — the interesting comparison is\n",
560+
"the mixture against the separated streams, and that is unaffected."
499561
],
500562
"metadata": {}
501563
},
@@ -506,22 +568,30 @@
506568
"import torch\n",
507569
"from espnet2.bin.asr_inference import Speech2Text\n",
508570
"\n",
509-
"# a WSJ model from the organisation, so nothing here depends on a Google\n",
510-
"# Drive copy staying up\n",
511571
"speech2text = Speech2Text.from_pretrained(\n",
512-
" model_tag=\"espnet/kamo-naoyuki_wsj_transformer2\",\n",
572+
" model_tag=(\n",
573+
" \"espnet/Shinji_Watanabe_librispeech_asr_train_asr_transformer_e18\"\n",
574+
" \"_raw_bpe_sp_valid.acc.best\"\n",
575+
" ),\n",
513576
" ngram_weight=0.0,\n",
514577
" lm_weight=0.0,\n",
515578
" device=\"cuda\" if torch.cuda.is_available() else \"cpu\",\n",
516579
")\n",
517580
"\n",
518-
"text_est = [None, None]\n",
519-
"text_est[0], *_ = speech2text(waves_wsj[0].squeeze())[0]\n",
520-
"text_est[1], *_ = speech2text(waves_wsj[1].squeeze())[0]\n",
521-
"text_m, *_ = speech2text(mixwav)[0]\n",
581+
"ASR_FS = 16000\n",
582+
"\n",
583+
"\n",
584+
"def recognise(wav):\n",
585+
" \"\"\"Transcribe one 8 kHz waveform with a 16 kHz model.\"\"\"\n",
586+
" text, *_ = speech2text(resample_poly(wav, ASR_FS, sr).astype(\"float32\"))[0]\n",
587+
" return text\n",
588+
"\n",
589+
"\n",
590+
"text_est = [recognise(estimate) for estimate in estimates]\n",
591+
"text_m = recognise(mixwav)\n",
522592
"print(\"Mix Speech to Text: \", text_m)\n",
523593
"print(\"Separated Speech 1 to Text: \", text_est[0])\n",
524-
"print(\"Separated Speech 2 to Text: \", text_est[1])\n"
594+
"print(\"Separated Speech 2 to Text: \", text_est[1])"
525595
],
526596
"execution_count": null,
527597
"outputs": []
@@ -580,8 +650,8 @@
580650
"\n",
581651
"\n",
582652
"text_ref = [\n",
583-
" \"SOME CRITICS INCLUDING HIGH REAGAN ADMINISTRATION OFFICIALS ARE RAISING THE ALARM THAT THE FED'S POLICY IS TOO TIGHT AND COULD CAUSE A RECESSION NEXT YEAR\",\n",
584-
" \"THE UNITED STATES UNDERTOOK TO DEFEND WESTERN EUROPE AGAINST SOVIET ATTACK\",\n",
653+
" \"HE CONTINUED HIS PRETENDED SEARCH AND TO GIVE COLOR TO HIS ERRAND MADE TWO ARRESTS\",\n",
654+
" \"NO NO NO TOTTY UD GET HER FEET WET SAID MISSUS POYSER CARRYING AWAY HER IRON\",\n",
585655
"]\n",
586656
"\n",
587657
"print(\"=====================\" , flush=True)\n",

0 commit comments

Comments
 (0)