Skip to content

The course's fine-tuning notebook, on the API it is run against - #60

Merged
sw005320 merged 5 commits into
masterfrom
course/finetuning
Sep 21, 2026
Merged

sw005320 merged 5 commits into
masterfrom
course/finetuning

Conversation

@sw005320

@sw005320 sw005320 commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

The course's fine-tuning notebook, updated to the ESPnet3 API it is run
against, executed before this was opened, and named speech_recognition.ipynb
like every other notebook in that directory.

It was held back from #59 by AssertionError: recipe_dir must be set when data_src is None, reported in espnet/espnet#6724. Masao's answer there — that
it is the notebook that needs fixing, not ESPnet3 — is right: a dataset is no
longer an object the notebook hands over.

The course's second fine-tuning notebook, which fine-tuned on spoken digits
and then compared CTC decoding against the beam search, is not here. Two
fine-tuning notebooks is one more than is worth keeping working.

What had drifted

What the notebooks did What ESPnet3 does now
from espnet3.utils.logging import configure_logging espnet3.utils.logging_utils
from espnet3.utils.stages import run_stages espnet3.utils.stages_utils
BaseSystem(train_config=cfg) BaseSystem(training_config=cfg)
dataset: {_target_: __main__.FLEURSDataset, ...} a module: recipe_dir + dataset/__init__.py with a Dataset class, and data_src_args in the config
optim: optimizer:

The dataset one is the only interesting change. DataOrganizer resolves every
entry through load_dataset_module(), which imports
<recipe_dir>/dataset/__init__.py and expects a class called Dataset; the
config passes that class's arguments as data_src_args. So each notebook
writes its dataset to that file with %%writefile and points recipe_dir at
it — which is how a recipe in egs3/ is laid out, so what the notebook
teaches is now what a recipe does.

Also in these two

  • It pinned nothing. It cloned espnet from master and installed it,
    which is a build from source in a notebook and a different API every week.
    It installs espnet[train]==202610.post1 now, like the other five. The pin
    matters more here than in an inference notebook: fine-tuning reads a config
    that the trainer's API has to agree with.
  • The checkpoint was named by step number (step300.ckpt), which is right
    only until someone changes max_epochs. It loads the last checkpoint the
    trainer wrote.
  • text_raw, which S2TPreprocessor does not read, is gone.

The weekly run is a smaller run, and says so

A free runner cannot fine-tune at full size. The notebook reads a few
environment variables — LIMIT_TRAIN_BATCHES and the FLEURS splits — and its
workflow sets them: two steps instead of three hundred, eight utterances
instead of a split. Unset, which is what a reader has, each is the real thing.

_run_notebook.yml grew an env_json input for this, passed to the job as a
map through fromJSON rather than interpolated into a shell line.

So its badge means less than the other five, and its label says so. A
badge shows its workflow's name, so the workflow is called
speech_recognition (short run) and the caveat travels with the badge
wherever it is pasted. The notebook's first cell and both README tables say
the same thing: green means it installs, finds its data and still agrees with
ESPnet3's API, not that the fine-tuning was reproduced. (Asked for in review,
and a fair thing to ask.)

Checked

Both were run end to end in a venv holding espnet[train]==202610.post1 and
nothing else - what CI installs, rather than a checkout:

running 33 cells ... RESULT ok Courses/CMUSpeechTechnology26S/speech_recognition.ipynb

with the same environment variables its workflow sets. check_layout.py is
clean at six course notebooks, and tests/ passes.

The README is 35 lines now: the course, the table, what the badges mean, and
credit. What was changed from the course originals, how to run these outside
Colab, and which recordings had to be replaced and why are in
MAINTAINING.md, which is where someone maintaining them will look and where
nobody opening a notebook has to.

🤖 Generated with Claude Code

They were held back from #59 by `recipe_dir must be set when data_src is
None`, reported as espnet/espnet#6724. Masao's answer there - that it is the
notebooks that need fixing rather than ESPnet3 - is right: a dataset is no
longer an object the notebook hands over.

DataOrganizer resolves every entry through load_dataset_module(), which
imports <recipe_dir>/dataset/__init__.py and expects a class called Dataset;
the config passes its arguments as data_src_args. So each notebook writes its
dataset to that file with %%writefile and points recipe_dir at it, which is
how egs3/mini_an4/asr is laid out - what the notebook teaches is now what a
recipe does.

Running them found two more drifts past the three in the issue: `optim:` is
`optimizer:` now, so the old key was ignored and the run died in
configure_optimizers; and `step300.ckpt` names a step number that follows
limit_train_batches, so both now load the last checkpoint the trainer wrote.

Also here, and unrelated to the API:

- both cloned espnet from master and installed it, which is a build from
  source inside a notebook against an API that moves. They pin
  espnet[train]==202610.post1 now, like the other five.
- owsm_finetuning_ctc loaded the model twice, as Speech2Text and as the
  deprecated Speech2TextCTCGreedySearch, to compare beam search against CTC.
  s2t.best_path() is the CTC head of the model already loaded, so one object
  does both and the comparison cannot quietly be between two sets of weights.
- text_raw, which S2TPreprocessor does not read, is gone.

The weekly run is a smaller run and says so. A free runner cannot fine-tune
at full size, so both notebooks read LIMIT_TRAIN_BATCHES and a few others
from the environment and their workflows set them; unset, which is what a
reader has, each is the real thing. _run_notebook.yml grew an env_json input
for that, passed to the job as a map through fromJSON rather than
interpolated into a shell line. Their badge means the notebook installs,
finds its data and still agrees with ESPnet3 - not that the numbers printed
in it were reproduced, and the README says so.

Both were run end to end against espnet[train]==202610.post1 in a venv with
nothing else in it, which is what their CI installs: 33 cells and 50 cells,
RESULT ok.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@mergify

mergify Bot commented Sep 20, 2026

Copy link
Copy Markdown

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

A green badge reads as "this worked", and for these two it means something
narrower: the notebook installed, found its data and agreed with ESPnet3's
API for two steps on a handful of utterances. Nothing in CI checks that the
fine-tuning helped, or that the word error rates in the notebook come out
again.

That was in the README, at the bottom, where nobody reads it against a green
tick. It is now in the three places the tick is:

- the badge itself. A badge shows its workflow's name, so the workflows are
  named "owsm_finetuning (short run)" and "owsm_finetuning_ctc (short run)"
  and the label carries the caveat wherever the badge is pasted.
- the first cell of each notebook, under the badges.
- both README tables, in the row, and then in full under the course table.

Asked for by a reviewer on #60, and a fair thing to ask: the distinction is
invisible at the moment someone forms the impression.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sw005320

Copy link
Copy Markdown
Contributor Author

Agreed, and thank you — the distinction was in the README, at the bottom,
which is exactly where nobody reads it against a green tick. It is now in the
three places the tick actually appears:

  • The badge itself. A badge shows its workflow's name, so the workflows are
    named owsm_finetuning (short run) and owsm_finetuning_ctc (short run).
    The label travels with the badge wherever it is pasted, including into a
    table someone copies elsewhere.
  • The first cell of each notebook, under the badges: two steps on a
    handful of utterances, green means it installs, finds its data and agrees
    with ESPnet3's API, not that the fine-tuning below was reproduced.
  • Both README tables — in the row beside the badge, and then in full under
    the course table, where it also says what nothing in CI checks.

Pushed as 74fd32a. The other five keep their plain labels, so the difference
is visible in the table rather than only explained in prose.

Two fine-tuning notebooks is one more than is worth keeping working, and
`owsm_finetuning` was the odd name in a directory where every other notebook
is called after its task. It is `speech_recognition.ipynb` now; what it
fine-tunes is the first line of the notebook.

The other one - fine-tuning on spoken digits, then CTC decoding against the
beam search - is dropped, with a line in MAINTAINING.md saying it existed.

The README had grown a changelog and a runbook, neither of which is for
someone opening a notebook in Colab. It is the course, the table, one
paragraph about what the badges mean, and credit: 35 lines. The rest moved to
MAINTAINING.md, including which recordings had to be replaced and why, which
is the licence reasoning and worth keeping somewhere.

Re-run after the rename, against espnet[train]==202610.post1 and the same
environment variables its workflow sets: 33 cells, RESULT ok.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sw005320 sw005320 changed the title The two fine-tuning notebooks, on the API they are run against The course's fine-tuning notebook, on the API it is run against Sep 20, 2026
Shinji Watanabe and others added 2 commits September 20, 2026 20:37
Two paragraphs were still written for whoever maintains these rather than
whoever opens one: that they are "kept here so that they keep working", and
the whole explanation of what a badge covers, what (short run) means and what
green does not promise.

Someone opening a notebook in Colab needs none of it. It is in MAINTAINING.md
now, under a heading of its own, and the README is the course, the notebooks,
and who wrote them: 27 lines.

The (short run) marker is gone from both tables with it. The badge label
still says it, which is where it is useful - beside a green tick, in whatever
page someone pasted it into.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"They run on CPU" was true and unhelpful: it reads as advice, and someone
following it fine-tunes OWSM on a free CPU runtime. The GPU is the
recommendation; the CPU is what CI proves, which is a different claim and
belongs in the same sentence rather than instead of it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sw005320
sw005320 merged commit 671ef21 into master Sep 21, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant