Repository navigation
Conversation
An on-demand pass over a finished job's vocals.wav, like the lead/backing split and mutually exclusive with it in the UI. Two models, because one is not enough. The BS-Roformer chorus model splits by vocal weight and rejects the idle stem by 30-40 dB when the singers sit in different ranges. It does nothing when they share one: two sopranos measured 93/7 energy, which is one voice with gaps. A MedleyVox Conv-TasNet handles that case at about 14.5 dB, and the cascade reaches it only when the register split collapses into a single stem. Which of the two is right cannot be decided from the audio. Five blind measures were tried and every one scored a split that had not happened as healthy, because a wrong separation is confident and uncorrelated in exactly the way a right one is. So the automatic path turns only on the unambiguous case, and STEMDECK_DUET_SAME_REGISTER is how someone who has listened says which they want. docs/models.md has the measurements and the dead ends. Marked EXPERIMENTAL in the toggle and the tooltip. It holds up while the singers trade lines and leaves bleed wherever they sing together. A chorus is beyond both models, since two outputs cannot hold an ensemble. The model code is vendored from asteroid (MIT) into app/pipeline/_convtasnet.py rather than depended on: the package costs 23 transitive packages including pytorch-lightning and pandas for a handful of classes. Weights are Carson Evans' CC BY 4.0 training run, fetched on first use the way Demucs already is. Two things this uncovered, fixed here: peaks.json was served immutable for a year, which held until on-demand splits began rewriting it. A browser takes that header literally and will not revalidate even on a hard reload, so running a split left the new stems playing under the old waveform with no way for the user to refresh out of it. It is now must-revalidate, and the fetch carries a cache buster, since a changing URL is the only thing that reaches an entry already cached under the old header. The third mode button took the composer past its width budget. The URL zone was the only flexible item in that row, so it absorbed the whole overflow and collapsed to its own padding at 1280px: no room for the input, and none for the refusal message that shares the space, so declining a dropped file answered into a zero-width box. The stem chips give the space up instead, ellipsising their labels beside the colour dot that already identifies them.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
|
|
||
| env.setdefault("SSL_CERT_FILE", certifi.where()) | ||
| env.setdefault("REQUESTS_CA_BUNDLE", certifi.where()) | ||
| except ModuleNotFoundError: |
| msg = line[len("@@ERROR@@") :] | ||
| try: | ||
| msg = json.loads(msg) | ||
| except json.JSONDecodeError: |
|
|
||
| import functools | ||
| import inspect | ||
| from typing import List, Optional |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #638
Closes #639
An on-demand pass over a finished job's
vocals.wav, sitting beside the lead/backing split and mutually exclusive with it in the UI. Marked EXPERIMENTAL in the mode toggle and in its tooltip, because it is.Two models, because one is not enough
The BS-Roformer chorus model splits by vocal weight. When the two singers sit in different ranges it is clean: 30 to 40 dB of rejection on the idle stem, measured on "Shallow". When they share a range it does nothing useful. Two sopranos ("What Is This Feeling") came out 93/7, which is one voice with gaps rather than a duet.
So there is a second path. A MedleyVox Conv-TasNet, trained on multi-singer mixtures, gets the other singer about 14.5 dB down on the same-register case. That is a real split rather than an isolation, and it is the only open model I measured that separates simultaneous same-register singing at all.
The cascade reaches the second model only when the first one collapses into a single stem.
Why the choice is not automatic
Picking between them from the audio turns out not to be possible. I tried five blind measures: energy share, share spread, which stem leads, stem correlation, and mask decisiveness. Every one of them scored a split that had not happened as healthy. A wrong separation is confident and uncorrelated in exactly the way a right one is, and the bad split actually scored better on correlation than the model's own correct output, because it is confidently wrong rather than confused.
Rather than ship a sixth guess, the automatic path turns only on the unambiguous case, one stem essentially silent.
STEMDECK_DUET_SAME_REGISTERis how someone who has listened says which they want.docs/models.mdhas the measurements and the dead ends, including a run of mine that was wrong and why.What it does not do
Choruses. The model has two outputs, so once an ensemble joins, both stems contain everyone. That cannot be detected from the audio before or after, so it is documented as a limit rather than guarded against. The tooltip says where it is weak, not only what it does.
Licensing
The model code is vendored from asteroid (MIT) into
app/pipeline/_convtasnet.pywith its copyright notice, rather than depended on. The package pulls 23 transitive packages including pytorch-lightning and pandas, for a handful of classes. Verified that all 347 tensors load withasteroidimports blocked.Weights are Carson Evans' CC BY 4.0 training run, fetched on first use the way Demucs already is. The upstream MedleyVox inference repo carries no licence at all and is deliberately not used.
Two fixes this uncovered
peaks.jsonwas servedimmutablefor a year (#639). True until on-demand splits started rewriting it, after which a split left the new stems playing under the old waveform with no way for the user to refresh out of it. Nowmust-revalidate, plus a cache buster on the fetch, since a changing URL is the only thing that reaches an entry already cached under the old header. Two regression tests, negative-controlled.The third mode button took the composer past its width budget. The URL zone was the only flexible item in that row, so it absorbed the whole overflow and collapsed to its own padding at 1280px: no room for the input, and none for the refusal message that shares the space, so declining a dropped file answered into a zero-width box. The stem chips give the space up instead, ellipsising their labels beside the colour dot that already identifies them. This one never shipped, so it has no issue of its own. Caught by the existing file-drop tests, and there is now a test that asserts the invariant directly at 1024px rather than catching it by luck at Playwright's default.
Before merging
uv.lockchanges, forasteroid-filterbanks. That sends every existing desktop install to a full download instead of an in-app update. Unavoidable if this ships, but worth deciding deliberately rather than discovering after the release.Testing
172 Playwright, 1081 pytest, ruff check and format, bandit, i18n coverage across all ten tables. No Rust changes. One pre-existing pytest failure (
test_section_warmup_loads_cpu_model) is environmental:allin1_inferis not installed here and is not a declared dependency. It fails the same way onmain.