Repository navigation
The course's fine-tuning notebook, on the API it is run against - #60
Merged
Merged
Conversation
They were held back from #59 by `recipe_dir must be set when data_src is None`, reported as espnet/espnet#6724. Masao's answer there - that it is the notebooks that need fixing rather than ESPnet3 - is right: a dataset is no longer an object the notebook hands over. DataOrganizer resolves every entry through load_dataset_module(), which imports <recipe_dir>/dataset/__init__.py and expects a class called Dataset; the config passes its arguments as data_src_args. So each notebook writes its dataset to that file with %%writefile and points recipe_dir at it, which is how egs3/mini_an4/asr is laid out - what the notebook teaches is now what a recipe does. Running them found two more drifts past the three in the issue: `optim:` is `optimizer:` now, so the old key was ignored and the run died in configure_optimizers; and `step300.ckpt` names a step number that follows limit_train_batches, so both now load the last checkpoint the trainer wrote. Also here, and unrelated to the API: - both cloned espnet from master and installed it, which is a build from source inside a notebook against an API that moves. They pin espnet[train]==202610.post1 now, like the other five. - owsm_finetuning_ctc loaded the model twice, as Speech2Text and as the deprecated Speech2TextCTCGreedySearch, to compare beam search against CTC. s2t.best_path() is the CTC head of the model already loaded, so one object does both and the comparison cannot quietly be between two sets of weights. - text_raw, which S2TPreprocessor does not read, is gone. The weekly run is a smaller run and says so. A free runner cannot fine-tune at full size, so both notebooks read LIMIT_TRAIN_BATCHES and a few others from the environment and their workflows set them; unset, which is what a reader has, each is the real thing. _run_notebook.yml grew an env_json input for that, passed to the job as a map through fromJSON rather than interpolated into a shell line. Their badge means the notebook installs, finds its data and still agrees with ESPnet3 - not that the numbers printed in it were reproduced, and the README says so. Both were run end to end against espnet[train]==202610.post1 in a venv with nothing else in it, which is what their CI installs: 33 cells and 50 cells, RESULT ok. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Tick the box to add this pull request to the merge queue (same as
|
A green badge reads as "this worked", and for these two it means something narrower: the notebook installed, found its data and agreed with ESPnet3's API for two steps on a handful of utterances. Nothing in CI checks that the fine-tuning helped, or that the word error rates in the notebook come out again. That was in the README, at the bottom, where nobody reads it against a green tick. It is now in the three places the tick is: - the badge itself. A badge shows its workflow's name, so the workflows are named "owsm_finetuning (short run)" and "owsm_finetuning_ctc (short run)" and the label carries the caveat wherever the badge is pasted. - the first cell of each notebook, under the badges. - both README tables, in the row, and then in full under the course table. Asked for by a reviewer on #60, and a fair thing to ask: the distinction is invisible at the moment someone forms the impression. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Author
|
Agreed, and thank you — the distinction was in the README, at the bottom,
Pushed as 74fd32a. The other five keep their plain labels, so the difference |
Two fine-tuning notebooks is one more than is worth keeping working, and `owsm_finetuning` was the odd name in a directory where every other notebook is called after its task. It is `speech_recognition.ipynb` now; what it fine-tunes is the first line of the notebook. The other one - fine-tuning on spoken digits, then CTC decoding against the beam search - is dropped, with a line in MAINTAINING.md saying it existed. The README had grown a changelog and a runbook, neither of which is for someone opening a notebook in Colab. It is the course, the table, one paragraph about what the badges mean, and credit: 35 lines. The rest moved to MAINTAINING.md, including which recordings had to be replaced and why, which is the licence reasoning and worth keeping somewhere. Re-run after the rename, against espnet[train]==202610.post1 and the same environment variables its workflow sets: 33 cells, RESULT ok. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two paragraphs were still written for whoever maintains these rather than whoever opens one: that they are "kept here so that they keep working", and the whole explanation of what a badge covers, what (short run) means and what green does not promise. Someone opening a notebook in Colab needs none of it. It is in MAINTAINING.md now, under a heading of its own, and the README is the course, the notebooks, and who wrote them: 27 lines. The (short run) marker is gone from both tables with it. The badge label still says it, which is where it is useful - beside a green tick, in whatever page someone pasted it into. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"They run on CPU" was true and unhelpful: it reads as advice, and someone following it fine-tunes OWSM on a free CPU runtime. The GPU is the recommendation; the CPU is what CI proves, which is a different claim and belongs in the same sentence rather than instead of it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The course's fine-tuning notebook, updated to the ESPnet3 API it is run
against, executed before this was opened, and named
speech_recognition.ipynblike every other notebook in that directory.
It was held back from #59 by
AssertionError: recipe_dir must be set when data_src is None, reported in espnet/espnet#6724. Masao's answer there — thatit is the notebook that needs fixing, not ESPnet3 — is right: a dataset is no
longer an object the notebook hands over.
The course's second fine-tuning notebook, which fine-tuned on spoken digits
and then compared CTC decoding against the beam search, is not here. Two
fine-tuning notebooks is one more than is worth keeping working.
What had drifted
from espnet3.utils.logging import configure_loggingespnet3.utils.logging_utilsfrom espnet3.utils.stages import run_stagesespnet3.utils.stages_utilsBaseSystem(train_config=cfg)BaseSystem(training_config=cfg)dataset: {_target_: __main__.FLEURSDataset, ...}recipe_dir+dataset/__init__.pywith aDatasetclass, anddata_src_argsin the configoptim:optimizer:The dataset one is the only interesting change.
DataOrganizerresolves everyentry through
load_dataset_module(), which imports<recipe_dir>/dataset/__init__.pyand expects a class calledDataset; theconfig passes that class's arguments as
data_src_args. So each notebookwrites its dataset to that file with
%%writefileand pointsrecipe_diratit — which is how a recipe in
egs3/is laid out, so what the notebookteaches is now what a recipe does.
Also in these two
espnetfrom master and installed it,which is a build from source in a notebook and a different API every week.
It installs
espnet[train]==202610.post1now, like the other five. The pinmatters more here than in an inference notebook: fine-tuning reads a config
that the trainer's API has to agree with.
step300.ckpt), which is rightonly until someone changes
max_epochs. It loads the last checkpoint thetrainer wrote.
text_raw, whichS2TPreprocessordoes not read, is gone.The weekly run is a smaller run, and says so
A free runner cannot fine-tune at full size. The notebook reads a few
environment variables —
LIMIT_TRAIN_BATCHESand the FLEURS splits — and itsworkflow sets them: two steps instead of three hundred, eight utterances
instead of a split. Unset, which is what a reader has, each is the real thing.
_run_notebook.ymlgrew anenv_jsoninput for this, passed to the job as amap through
fromJSONrather than interpolated into a shell line.So its badge means less than the other five, and its label says so. A
badge shows its workflow's name, so the workflow is called
speech_recognition (short run)and the caveat travels with the badgewherever it is pasted. The notebook's first cell and both README tables say
the same thing: green means it installs, finds its data and still agrees with
ESPnet3's API, not that the fine-tuning was reproduced. (Asked for in review,
and a fair thing to ask.)
Checked
Both were run end to end in a venv holding
espnet[train]==202610.post1andnothing else - what CI installs, rather than a checkout:
with the same environment variables its workflow sets.
check_layout.pyisclean at six course notebooks, and
tests/passes.The README is 35 lines now: the course, the table, what the badges mean, and
credit. What was changed from the course originals, how to run these outside
Colab, and which recordings had to be replaced and why are in
MAINTAINING.md, which is where someone maintaining them will look and wherenobody opening a notebook has to.
🤖 Generated with Claude Code