Skip to content

Run every demo weekly, two new demos, and a front page that matches - #58

Merged
sw005320 merged 12 commits into
masterfrom
ci/run-every-demo
Sep 20, 2026
Merged

sw005320 merged 12 commits into
masterfrom
ci/run-every-demo

Conversation

@sw005320

Copy link
Copy Markdown
Contributor

#57 never reached master. It was stacked on #55, and #55 merged into master
first — so when #57 merged two hours later it went into
notebooks/pin-released-espnet, a branch master had already taken and stopped
following. GitHub only retargets a stacked pull request when its base branch is
deleted. My mistake for stacking; this is the same work, aimed at master.

It is why sds_demo.ipynb and asr_streaming_demo.ipynb are not in
Demos/, and why the
front page has no badge.

What is in here

Every demo runs weekly. The workflow ran one course notebook, and its
pull_request paths did not mention Demos/ at all, so changing a demo ran
nothing. All nine notebooks are in the matrix, and what to install is asked of
each notebook — run_notebook.py --print-install reads its own pip lines, so a
demo that starts needing another extra says so in the cell a reader runs.

Two more demos, each run end to end before committing:

  • asr_streaming_demo.ipynb — the old 2021 streaming notebook made to work, on
    espnet/tedlium2_streaming_transformer, the weights Keqi Deng trained, copied
    into the organization with credit so the link cannot rot.
  • sds_demo.ipynb — spoken dialogue with the parts showing: OWSM-CTC hears,
    Qwen2.5-0.5B answers, VITS speaks. It replaces the one row in espnet's demo
    table that was a link to a recipe rather than a notebook.

Badges, and the checks that stop them being decoration. Each demo carries
the workflow's status beside its Colab badge. check_layout.py fails if a demo
has no badge, if it is not in the workflow matrix, if it is not listed in the
README, or if a README links to a notebook that is not there — which caught the
root README pointing at eight files that the flattening had moved.

Tests for the parsing CI trusts (from review): seventeen cases, three of
which were bugs — a trailing comment became four packages, a line-continued
command raised ValueError, and drop_installs left half of one behind as
invalid Python.

What green means is written down: every cell ran on a clean CPU runner last
Sunday; it does not mean the output was right; and a table of what a demo
depends on that lives outside this repository.

The front page now lists the nine demos, says unmaintained/ is unchecked,
and stops presenting four courses as though they were alike — one is maintained,
the other three have not been checked in years and are kept as a record.

🤖 Generated with Claude Code

Shinji Watanabe and others added 9 commits September 20, 2026 08:10
The six demos were checked by hand and by nothing else: the workflow ran
one course notebook, and its pull_request paths did not even mention
Demos/, so changing a demo ran nothing. All eight are in the matrix now.

What to install is asked of the notebook rather than written in the
workflow: `run_notebook.py --print-install` reads its own pip lines, so a
demo that starts needing another extra says so in the cell a reader runs
and this installs it without being told twice. Only unindented lines
count - the model chooser's S3PRL install sits inside an `if`, and what
it asks for depends on a choice made when the notebook runs.

asr_streaming_demo.ipynb is the old Demos/ASR/streaming_asr_demo.ipynb
made to work: same idea, same 40 ms slices, on the release rather than on
whatever pip resolved in 2021. It prints the hypothesis as it grows,
which is the thing worth seeing and the thing no other demo shows.

That needed the naming rule to admit a variant. It admits one, and gains
a check that every demo is listed in Demos/README.md with a line saying
what it does - so a second page cannot arrive without someone saying how
it differs. That check immediately found that the table listed one demo
while six existed: an edit lost to a careless `git checkout` earlier
today, which nothing would otherwise have noticed.

Two more things the work turned up. drop_installs deleted an indented
install and left the `if` above it with an empty body. And the course
notebook installed S3PRL whichever front-end was chosen, though only the
two WavLM ones import it - and those two do not currently work at all,
because S3PRL calls a torchaudio function removed in 2.1. The cell now
installs it only when it is wanted and says why it will fail there.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Each demo carries the workflow's status badge beside its Colab badge, so
somebody looking at the page in Colab can see it is run rather than hoped
for. One badge for the set: red means one of them broke, not all of them.

A badge is a claim, and a claim wants a check. check_layout.py now fails
when a demo has no badge, and when a demo is not in the workflow's matrix
- which is the way the badge could quietly become a lie, a notebook
sitting in the directory wearing the status of runs that never included
it. Verified by taking codec_demo out of the matrix and watching it fail.

unmaintained/ has no badge, which is the contrast that makes the badge
mean something.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
espnet/tedlium2_streaming_transformer, the weights Keqi Deng trained and
published, copied into the organization with a card that says whose they
are and links the original. Nothing about the model changed; what changed
is that the notebook now points at something the organization keeps
alive, and the paragraph explaining that it did not is gone.

Loaded from the new tag and decoded with it before this commit - same
transcript - and the notebook ran again end to end.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The badge belongs where someone arrives, so it is at the top of the root
README rather than only inside Demos/.

Putting it there showed that page was describing the repository as it was
before yesterday: every one of its eight demo links pointed into
Demos/ASR, Demos/SE, Demos/SLU, Demos/TTS, Demos/Others. Flattening the
directory broke all eight at once, I updated Demos/README.md and espnet's
README and not this one, and nothing said so - a moved notebook is a 404
on GitHub and "Notebook not found" in Colab, neither of which reaches
anybody here. It now lists the seven demos that exist, and says that
unmaintained/ carries no badge and nothing checks it.

check_layout.py now resolves every notebook link in both READMEs. Run
against the page as it stood a minute ago, it names all eight.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The espnet README's demo table had a row pointing at egs2/TEMPLATE/sds1 -
a Gradio app you clone the repository to run. It was the one row in a
table of Colab notebooks that was not one.

sds_demo.ipynb is the cascade with its parts showing: OWSM-CTC hears a
real recording, a 360M instruct model answers it, VITS reads the answer
back. Twelve cells, CPU throughout.

The recording is a real one and not synthesised. I tried having the
notebook speak its own question first, which is tidier: VITS says "What
is the capital of France?", OWSM hears "What tis the capital of Frax",
and the model duly answers about Frax. A short synthetic utterance is not
what a dialogue system meets, and a demo that looks broken is worse than
one cell longer.

The LLM is plain transformers rather than espnet2.sds.llm, and the
notebook says why: that wrapper takes access_token as a required str, so
an ungated model needs a token it does not need. Worth fixing in espnet;
a demo is not the place to work around it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four ungated instruct models, the same turn, three samples each:

  SmolLM2-360M   "The hotel sale will be a key component of Holiday's
                  strategy to focus on property management."
  Qwen2.5-0.5B   "Yes, that seems like an effective strategy for Holiday
                  Hotels Group."
  SmolLM2-1.7B   "The sale of the hotels is part of Holiday's strategy to
                  sell off assets and concentrate on property management."

Two of the three hand the sentence back reworded, which in a dialogue
demo reads as broken. Qwen2.5-0.5B replies to it - twice in three
samples, against never for the other two. It is 494M against 362M, so
about 1 GB to download instead of 720 MB, and 7 seconds to answer on a
CPU instead of 2.

The five-times-larger SmolLM2-1.7B is the clearest result: it paraphrases
every time, takes 75 seconds to load and 3.4 GB to fetch. Size was not
the axis, which is worth knowing before reaching for a bigger model.

Qwen3-0.6B was tried too and left out: it writes its reasoning into the
reply unless thinking mode is turned off, and a demo should not need that
explained.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both from review, and the tests found three bugs on their first run.

`--print-install` decides what the weekly run installs, so a wrong answer
there is a green run of the wrong thing. Seventeen cases, and three of
them failed:

  - a trailing comment became three packages, because shlex was not
    told about comments;
  - a command split over lines raised ValueError on the backslash, and
    only its first line was read;
  - drop_installs removed the first line of such a command and left the
    rest in the cell as Python, which does not parse.

Fixed by reading logical lines rather than physical ones, in both
functions, and by splitting with comments=True. The eight notebooks
report the same dependencies as before.

The tests run in pre-commit, so pull requests see them.

And Demos/README.md now says what green means: every cell ran to the end
on a clean CPU runner last Sunday. It does not mean the output was right
- nothing checks a transcript against a reference. Most of what a demo
depends on is outside this repository, and the table lists each: the
pinned release does not move, its dependency tree is not pinned, the
checkpoints are on the Hub, the LLM in sds_demo is not ESPnet at all. A
red badge is a question about which of those changed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Courses section listed four courses as though they were alike. One is
maintained and one notebook of it runs weekly; the other three have not
been checked in years and nobody runs them. They are worth keeping - they
are the record of how these things were taught - and worth saying that
about, in a paragraph rather than as three more lists that read like
offers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@mergify

mergify Bot commented Sep 20, 2026

Copy link
Copy Markdown

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

Shinji Watanabe and others added 2 commits September 20, 2026 10:40
GitHub's badge is per workflow and cannot show one job of a matrix, so
one shared badge was the most the matrix could offer: red meant
"something broke" and you had to go and look. Nine small workflows now,
each naming its notebook and calling one reusable file that holds the
job. A table with a badge a row, and a pull request that touches one
demo runs that one.

The tables carry them: Demos in both READMEs, and the course notebooks
too, where four of the five say "by hand" instead - which is the honest
column and reads better than a footnote.

check_layout.py follows: a demo needs a workflow of its own, that
workflow has to pass that notebook, and the notebook has to carry that
workflow's badge. The second check wanted tightening - the file names
its notebook in a comment and in `paths:` as well, so a looser match
passed while the workflow ran something else. It matches the input now,
and was checked by pointing tts_demo.yml at another notebook.

CMUSpeechTechnology26S reads "CMU Speech Technology 26S" in the prose.
The directory keeps its name: it is in every link here, in espnet's
documentation, and in the workflow paths, and a space in a path is a
week of small breakages for a nicer heading.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
From review: --print-install is the interface a notebook author writes
against, since the workflow reads the install cell rather than repeating
it, so what it supports should be stated rather than inferred from the
tests.

tools/README.md now has it as two tables - what is read as a dependency,
and what is not and why. The `git+` exclusion gets its reason in full: a
demo pins a release so that it does the same thing in April, and
installing espnet from git in CI would test something other than what a
reader gets, which would make a pass mean nothing.

And what to do when a notebook needs something the contract cannot
express: it is a Courses/ document rather than a demo, because the weekly
badge would be promising something it cannot keep.

Each row of both tables is a case in tests/test_run_notebook.py.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sw005320

Copy link
Copy Markdown
Contributor Author

Right — it is an interface, and it was only written down in the tests.

tools/README.md now states it, in two tables, under "What CI installs: the
install cell is the contract"
. Demos/README.md links there from the rule
about pinning a release, which is where an author is standing when they write
the cell.

Read as a dependency: !pip install and %pip install at the start of a
line; flags passed through as they are; quoting undone (the shell would
otherwise hand pip a name with quotation marks in it); # comments dropped; a
command continued over lines read as the one command it is; every install line,
in one cell or many.

Not read, each with its reason:

why
git+https://… deliberate. A demo pins a release — check_layout.py fails one that does not — so it does the same thing in April as today, and a break on master the night before someone opens it is not their problem. Installing espnet from git in CI would also test something other than what a reader gets, which would make a pass here mean nothing.
an indented install, inside an if what it asks for depends on a choice made when the notebook runs. The course notebook installs S3PRL only for the WavLM front-ends; CI cannot know which branch a reader takes.
!apt-get, !cd repo && pip install . removed before execution, because the runner is not Colab, but not treated as a Python dependency.
anything in a markdown cell it is prose.

Two more things the section says. How to see what CI will install, which is to
ask rather than to read the regex:

$ python tools/run_notebook.py Demos/enh_demo.ipynb --print-install
-q espnet[enh]==202610.post1 espnet_model_zoo librosa

And what to do when a notebook needs something the contract cannot express: it
is a Courses/ document rather than a demo, because the weekly badge would be
promising something it cannot keep.

Every row of both tables is a case in tests/test_run_notebook.py, so the
documentation and the behaviour are checked against each other rather than
drifting.

text_to_speech runs. It was listed as blocked on building a vocoder from
source, which turned out to be ParallelWaveGAN's setup.py importing pip -
a package an isolated build does not have, so `--no-build-isolation` is
the whole fix. It has a workflow and a badge like the demos.

Getting there needed CI to install what a notebook clones. VERSA and
ParallelWaveGAN are on nobody's index, so the course notebooks fetch them
with `git clone` and `pip install .`, two shell lines the runner drops.
`--print-git-install` reads that pair and returns `git+<url>`, keeping
the notebook the single source of truth for its dependencies, as
--print-install already does for the rest. espnet is never returned that
way whatever form it is written in.

And the stale pins go: numpy==1.23.5 with tensorflow==2.12.0, from when
Colab ran Python 3.8. numpy 1.23.5 has no wheel for 3.12, so those lines
could only ever have failed - they passed unnoticed because the runner
skips install cells and a reader on Colab gets a resolver error they were
told to ignore.

The three that still do not run say what stops each, which is worth more
than "by hand":

  speech_enhancement  its noisy sample comes from Google Drive via gdown;
                      everything else installs and runs
  neural_codec        VERSA's scorer exits without writing its output
  speech_translation  not tried since the pins came out

Six more tests for the clone-and-install reading, and tools/README.md
documents it beside the rest of the contract.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sw005320
sw005320 merged commit 47ac076 into master Sep 20, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant