Split a duet into one track per singer

README.md

@@ -92,6 +92,8 @@ StemDeck is free and **does not accept any money, sponsorship, or funding**  fro
 
 **Song analysis** including BPM (librosa beat tracker), key, scale, and confidence (Albrecht-Shanahan profiles), integrated LUFS (BS.1770), and sample peak in dBFS.
 
+**Duet split.** On a finished job, split the vocals into two singers. Where the two sit in different vocal ranges this is clean. Where they share a range, a second model handles it and puts the other singer roughly 14 dB down, which is a real split rather than an isolation. A chorus is beyond both, and `docs/models.md` has the measurements behind all of that.
+
 **Cancellable jobs.** Cancel mid-pipeline and the runner terminates the active subprocess immediately, deletes the partial job dir, and returns to ready.
 
 **Library panel** with folder-based track organisation, drag-and-drop, search, and trash.
@@ -160,7 +162,7 @@ Extract the zip anywhere, run `StemDeck.exe`. FFmpeg, the Demucs model, config,
 
 <br>
 
-StemDeck is built on **[Python 3.12](https://python.org)** managed via **[uv](https://github.com/astral-sh/uv)**, with a **[FastAPI](https://fastapi.tiangolo.com)** backend serving REST and Server-Sent Events. Stem separation uses **[Demucs](https://github.com/facebookresearch/demucs)** (`htdemucs_6s`), Meta AI's open-source 6-stem neural network. The optional on-demand lead/backing vocal split runs the UVR-MDX-NET Karaoke 2 model via **[audio-separator](https://github.com/nomadkaraoke/python-audio-separator)**, trained as part of the **[Ultimate Vocal Remover](https://github.com/Anjok07/ultimatevocalremovergui)** project by Anjok07. YouTube audio is fetched via **[yt-dlp](https://github.com/yt-dlp/yt-dlp)**; transcoding and mixing use **[FFmpeg](https://ffmpeg.org)**. BPM detection and key analysis run on **[librosa](https://librosa.org)**; loudness measurement uses **[pyloudnorm](https://github.com/csteinmetz1/pyloudnorm)** (ITU-R BS.1770). The macOS and Windows desktop shells are **[Tauri v2](https://tauri.app)** (Rust/WKWebView on macOS, Rust/WebView2 on Windows). The frontend is vanilla JS with the Web Audio API, no framework and no build step; waveforms are rendered on `<canvas>` using min/max sample rendering.
+StemDeck is built on **[Python 3.12](https://python.org)** managed via **[uv](https://github.com/astral-sh/uv)**, with a **[FastAPI](https://fastapi.tiangolo.com)** backend serving REST and Server-Sent Events. Stem separation uses **[Demucs](https://github.com/facebookresearch/demucs)** (`htdemucs_6s`), Meta AI's open-source 6-stem neural network. The optional on-demand lead/backing vocal split runs the UVR-MDX-NET Karaoke 2 model via **[audio-separator](https://github.com/nomadkaraoke/python-audio-separator)**, trained as part of the **[Ultimate Vocal Remover](https://github.com/Anjok07/ultimatevocalremovergui)** project by Anjok07. The duet split runs the same project's BS-Roformer chorus model, and for two singers who share a vocal range it falls back to a **[MedleyVox](https://github.com/jeonchangbin49/MedleyVox)** Conv-TasNet, using weights trained and released under CC BY 4.0 by Carson Evans; the model itself is built from **[asteroid](https://github.com/asteroid-team/asteroid)** components (MIT, Pariente Manuel), vendored into `app/pipeline/_convtasnet.py` with its copyright notice rather than pulled in as a dependency. YouTube audio is fetched via **[yt-dlp](https://github.com/yt-dlp/yt-dlp)**; transcoding and mixing use **[FFmpeg](https://ffmpeg.org)**. BPM detection and key analysis run on **[librosa](https://librosa.org)**; loudness measurement uses **[pyloudnorm](https://github.com/csteinmetz1/pyloudnorm)** (ITU-R BS.1770). The macOS and Windows desktop shells are **[Tauri v2](https://tauri.app)** (Rust/WKWebView on macOS, Rust/WebView2 on Windows). The frontend is vanilla JS with the Web Audio API, no framework and no build step; waveforms are rendered on `<canvas>` using min/max sample rendering.
 
 *Thanks to the creators and maintainers of all the open-source libraries that make StemDeck possible.*
 

app/_vendor/LICENSE.asteroid

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2019 Pariente Manuel
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

app/api/config.py

@@ -2,7 +2,7 @@
 
 from fastapi import APIRouter
 
-from app.core.config import EXTRA_STEM_NAMES, STEM_NAMES
+from app.core.config import DUET_STEM_NAMES, EXTRA_STEM_NAMES, STEM_NAMES
 
 router = APIRouter()
 
@@ -14,4 +14,8 @@ def get_config() -> dict:
     # existing clients that assume "every job produces exactly these stems"
     # are unaffected; new clients merge it into their lane vocab (see
     # syncStemNamesFromAPI in static/js/constants.js).
-    return {"stem_names": list(STEM_NAMES), "extra_stem_names": list(EXTRA_STEM_NAMES)}
+    return {
+        "stem_names": list(STEM_NAMES),
+        "extra_stem_names": list(EXTRA_STEM_NAMES),
+        "duet_stem_names": list(DUET_STEM_NAMES),
+    }

app/api/jobs.py

@@ -40,6 +40,7 @@
 from app.pipeline import jobqueue
 from app.pipeline.collect import merge_stem_peaks, presence_for_split
 from app.pipeline.download import InvalidYouTubeURL, validate_youtube_url
+from app.pipeline.duet_split import split_duet
 from app.pipeline.errors import classify_failure
 from app.pipeline.runner import _pipeline_lock
 from app.pipeline.vocal_split import split_vocals
@@ -408,15 +409,15 @@ def cancel_job(job_id: str) -> dict:
     return job.to_state()
 
 
-def _write_vocal_split_error(stems_dir: Path, cause: str, tail: list[str]) -> None:
-    """Best-effort error record for the on-demand vocal split (#275). The job
-    itself stays "done" -- this is diagnostic-only, not the quarantine path
-    (which would delete the job's base stems)."""
+def _write_split_error(stems_dir: Path, filename: str, cause: str, tail: list[str]) -> None:
+    """Best-effort error record for an on-demand split (#275 and the duet
+    split). The job itself stays "done" -- this is diagnostic-only, not the
+    quarantine path (which would delete the job's base stems)."""
     try:
         lines = [f"cause: {cause}", "", "--- stderr tail ---", *tail]
-        (stems_dir / "vocal_split_error.txt").write_text("\n".join(lines) + "\n", encoding="utf-8")
+        (stems_dir / filename).write_text("\n".join(lines) + "\n", encoding="utf-8")
     except OSError:
-        logger.warning("could not write vocal_split_error.txt in %s", stems_dir, exc_info=True)
+        logger.warning("could not write %s in %s", filename, stems_dir, exc_info=True)
 
 
 @router.post("/{job_id}/vocal-split")
@@ -448,7 +449,9 @@ async def start_vocal_split(job_id: str) -> Response:
     except Exception as e:
         cause = classify_failure("\n".join([*(getattr(e, "tail", None) or []), str(e)]))
         logger.warning("[%s] vocal split failed: %s", job_id, e, exc_info=True)
-        _write_vocal_split_error(stems_dir, cause, getattr(e, "tail", None) or [str(e)])
+        _write_split_error(
+            stems_dir, "vocal_split_error.txt", cause, getattr(e, "tail", None) or [str(e)]
+        )
         job.vocal_split = "error"
         _set(job, stage="Done")
         registry_persist(JOBS_DIR)
@@ -471,6 +474,61 @@ async def start_vocal_split(job_id: str) -> Response:
     return JSONResponse(_job_state(job))
 
 
+@router.post("/{job_id}/duet-split")
+async def start_duet_split(job_id: str) -> Response:
+    """Trigger the on-demand duet split for a completed job: a second model
+    pass over the existing vocals.wav, producing voice_1.wav +
+    voice_2.wav. Idempotent once done -- calling again returns 202 with the
+    existing result rather than re-running the (expensive) model.
+
+    Mirrors start_vocal_split above; the two are alternatives, so a job that
+    already ran one can still run the other."""
+    if not JOB_ID_RE.match(job_id):
+        raise HTTPException(status_code=404, detail="job not found")
+    job = registry_get(job_id)
+    if job is None or job.status != "done":
+        raise HTTPException(status_code=404, detail="job not found")
+    if job.duet_split == "running":
+        raise HTTPException(status_code=409, detail="duet split already running")
+    if job.duet_split == "done":
+        return JSONResponse(_job_state(job), status_code=202)
+
+    stems_dir = (JOBS_DIR / job_id / "stems").resolve()
+    if not stems_dir.is_relative_to(JOBS_DIR.resolve()):
+        raise HTTPException(status_code=404, detail="job not found")
+
+    job.duet_split = "running"
+    _set(job, stage="Splitting duet voices...")
+    try:
+        async with _pipeline_lock:
+            new_names = await asyncio.to_thread(split_duet, job, stems_dir)
+    except Exception as e:
+        cause = classify_failure("\n".join([*(getattr(e, "tail", None) or []), str(e)]))
+        logger.warning("[%s] duet split failed: %s", job_id, e, exc_info=True)
+        _write_split_error(
+            stems_dir, "duet_split_error.txt", cause, getattr(e, "tail", None) or [str(e)]
+        )
+        job.duet_split = "error"
+        _set(job, stage="Done")
+        registry_persist(JOBS_DIR)
+        raise HTTPException(status_code=500, detail="duet split failed") from e
+
+    existing = {s["name"] for s in job.stems}
+    for name in new_names:
+        if name not in existing:
+            job.stems.append({"name": name, "url": f"/api/jobs/{job_id}/stems/{name}.wav"})
+    # "vocals" rides along so presence_for_split can recover the scale the base
+    # stems were normalised against (see start_vocal_split).
+    rms_values = merge_stem_peaks(stems_dir, ["vocals", *new_names])
+    extra_presence = presence_for_split(rms_values, job.stem_presence)
+    if extra_presence:
+        job.stem_presence = {**(job.stem_presence or {}), **extra_presence}
+    job.duet_split = "done"
+    _set(job, stage="Done")
+    registry_persist(JOBS_DIR)
+    return JSONResponse(_job_state(job))
+
+
 _SECTION_ID_RE = re.compile(r"^[a-zA-Z0-9_\-]{1,64}$")
 _COLOR_RE = re.compile(r"^#[0-9a-fA-F]{3,8}$")
 _SECTIONS_WRITE_LOCK = threading.Lock()

app/api/stems.py

@@ -20,6 +20,7 @@
 
 from app.core.config import (
     CACHE_DIR,
+    DUET_STEM_NAMES,
     EXTRA_STEM_NAMES,
     JOB_ID_RE,
     JOBS_DIR,
@@ -41,19 +42,26 @@
 # (#275, EXTRA_STEM_NAMES) when a job has requested it. "original" is the
 # re-encoded source song (added when the user picked a strict subset), "mix"
 # is the ffmpeg amix of the user's selected stems.
-_ALLOWED_NAMES = frozenset(STEM_NAMES) | frozenset(EXTRA_STEM_NAMES) | {"original", "mix"}
+_ALLOWED_NAMES = (
+    frozenset(STEM_NAMES)
+    | frozenset(EXTRA_STEM_NAMES)
+    | frozenset(DUET_STEM_NAMES)
+    | {"original", "mix"}
+)
 
 # Lanes the dynamic mixdown may sum: the 6 stems plus "original" (the complement
 # track shown when the user picked a subset) plus lead/backing vocals when a
 # job has split them. "mix" is excluded -- it is the static pre-render this
 # endpoint replaces. Gains are linear; the studio caps a lane at 2.0, so this
 # generous bound just rejects abusive values.
-_MIXDOWN_NAMES = frozenset(STEM_NAMES) | frozenset(EXTRA_STEM_NAMES) | {"original"}
+_MIXDOWN_NAMES = (
+    frozenset(STEM_NAMES) | frozenset(EXTRA_STEM_NAMES) | frozenset(DUET_STEM_NAMES) | {"original"}
+)
 _MIXDOWN_MAX_GAIN = 4.0
 # lead_vocals/backing_vocals are a decomposition of vocals, not an independent
 # signal -- summing vocals alongside either would double-count the vocal
 # energy in the mix (#275).
-_VOCAL_DECOMPOSITION_NAMES = frozenset(EXTRA_STEM_NAMES)
+_VOCAL_DECOMPOSITION_NAMES = frozenset(EXTRA_STEM_NAMES) | frozenset(DUET_STEM_NAMES)
 
 # Output encoders by container/extension, shared by the dynamic mixdown and the
 # stems zip. WAV is lossless PCM, FLAC is lossless compressed, MP3 is VBR ~190 kbps,
@@ -811,7 +819,23 @@ async def _ensure_cached_mp3(src: Path) -> Path:
 
 @router.get("/jobs/{job_id}/stems/peaks.json")
 async def get_stem_peaks(job_id: str) -> Response:
-    """Return pre-computed waveform peaks for all stems."""
+    """Return pre-computed waveform peaks for all stems.
+
+    This used to be served `immutable` for a year, on the reasoning that a job's
+    stems never change once it is done. That stopped being true when the
+    on-demand splits arrived: both the lead/backing split and the duet split add
+    stems to a finished job and rewrite this file through `merge_stem_peaks`.
+
+    `immutable` is a promise the browser takes literally. It will not revalidate,
+    not on a reload and not on a hard reload, so a user who ran a split saw the
+    waveform from before it for as long as the entry survived. The audio has no
+    such header, so the two disagreed: the right stems playing under the wrong
+    picture.
+
+    `must-revalidate` with `max-age=0` keeps the file out of the response body on
+    a repeat visit -- FileResponse's ETag still answers 304 -- while guaranteeing
+    the browser asks.
+    """
     if not JOB_ID_RE.match(job_id):
         raise HTTPException(status_code=404, detail="job not found")
     job = registry_get(job_id)
@@ -823,7 +847,7 @@ async def get_stem_peaks(job_id: str) -> Response:
     return FileResponse(
         path,
         media_type="application/json",
-        headers={"Cache-Control": "public, max-age=31536000, immutable"},
+        headers={"Cache-Control": "public, max-age=0, must-revalidate"},
     )
 
 
@@ -1364,7 +1388,7 @@ async def get_all_stems_zip(
         raise HTTPException(status_code=404, detail="job not ready")
 
     # Resolve the requested subset (whitelisted) or fall back to all stems.
-    all_names = (*STEM_NAMES, *EXTRA_STEM_NAMES)
+    all_names = (*STEM_NAMES, *EXTRA_STEM_NAMES, *DUET_STEM_NAMES)
     if stems:
         requested = {s for s in stems.split(",") if s}
         if not requested <= set(all_names):

app/core/config.py

@@ -65,6 +65,11 @@ def detect_torch_device() -> str:
 # GET /api/config contract all assume "the 6 Demucs stems" and must not change
 # just because a job happened to request this extra pass.
 EXTRA_STEM_NAMES: tuple[str, ...] = ("lead_vocals", "backing_vocals")
+# The on-demand duet split's two stems. Deliberately a separate tuple rather
+# than more EXTRA_STEM_NAMES: the two splits are alternatives, and the lane
+# ordering swaps "vocals" for a pair only when every name in a pair is
+# present, which a single four-name tuple would never satisfy.
+DUET_STEM_NAMES: tuple[str, ...] = ("voice_1", "voice_2")
 JOB_ID_RE = re.compile(r"^[a-f0-9]{12}$")
 
 
@@ -312,6 +317,28 @@ def js_solver_available() -> bool:
 # A first run downloads the checkpoint (hundreds of MB); generous default so a
 # slow connection isn't mistaken for a stall.
 TIMEOUT_VOCAL_SPLIT = _env_int("STEMDECK_TIMEOUT_VOCAL_SPLIT", 1800)
+# On-demand duet split. BS-Roformer trained on male/female vocal separation:
+# it splits a duet by vocal weight/register, which works when the two singers
+# sit in different registers and does nothing when they do not. Measured on
+# "Shallow": 30-40 dB rejection of the idle stem. Measured on two sopranos
+# ("What Is This Feeling"): 93/7 energy, no split. STEMDECK_DUET_MODEL lets a
+# deployment swap the checkpoint without a code change; see docs/models.md.
+DUET_SPLIT_MODEL = (
+    os.environ.get("STEMDECK_DUET_MODEL", "").strip()
+    or "model_chorus_bs_roformer_ep_267_sdr_24.1275.ckpt"
+)
+# Same rationale as TIMEOUT_VOCAL_SPLIT, and this checkpoint is larger (527 MB).
+TIMEOUT_DUET_SPLIT = _env_int("STEMDECK_TIMEOUT_DUET_SPLIT", 1800)
+# Force the same-register duet path (app/pipeline/duet_medleyvox.py) instead of
+# letting the register model try first. The register model is the better answer
+# when the two singers sit in different ranges, and produces a confident, wrong
+# one when they do not -- and which of those happened cannot be measured without
+# knowing the right answer, so it is offered as a choice rather than guessed.
+DUET_SAME_REGISTER = os.environ.get("STEMDECK_DUET_SAME_REGISTER", "").strip().lower() in (
+    "1",
+    "true",
+    "yes",
+)
 # Beat-grid stage decodes the whole drums stem (not the 180 s analyze window),
 # so it gets its own, larger budget.
 TIMEOUT_BEATGRID = _env_int("STEMDECK_TIMEOUT_BEATGRID", 300)

app/core/model_cache.py

@@ -132,3 +132,20 @@ def vocal_split_artifacts(models_dir: Path, model_file: str) -> list[Path]:
         root / "vr_model_data.json",
         root / "mdx_model_data.json",
     ]
+
+
+def duet_split_artifacts(models_dir: Path, model_file: str) -> list[Path]:
+    """The duet-split checkpoint, its yaml config, and the metadata indexes.
+
+    Same reasoning as vocal_split_artifacts, plus one extra file: the MDXC
+    roformer family is distributed as a checkpoint *and* a separate yaml
+    config, and audio-separator needs both. A truncated yaml fails the same way
+    a truncated checkpoint does, so healing has to be able to remove either.
+    """
+    root = models_dir / "audio-separator"
+    return [
+        root / model_file,
+        root / "config_chorus_male_female_bs_roformer.yaml",
+        root / "vr_model_data.json",
+        root / "mdx_model_data.json",
+    ]

app/core/models.py

@@ -98,6 +98,10 @@ class Job:
     # app/pipeline/vocal_split.py) and is recorded in stems/vocal_split_error.txt,
     # not job.error_detail, since the job itself did not fail.
     vocal_split: Literal["none", "running", "done", "error"] = "none"
+    # On-demand duet split -- the same shape of post-hoc action as vocal_split
+    # above, and mutually exclusive with it in the UI, but tracked separately
+    # so a job can have run one, the other, or neither.
+    duet_split: Literal["none", "running", "done", "error"] = "none"
     # When the user put this job in the Trash, or None if they have not.
     #
     # Server-side on purpose. The Trash used to live only in the browser's
@@ -162,6 +166,7 @@ def to_state(self) -> dict[str, Any]:
             "gpu_fallback": self.gpu_fallback,
             "stage_timings": self.stage_timings,
             "vocal_split": self.vocal_split,
+            "duet_split": self.duet_split,
             "trashed_at": self.trashed_at,
             "created_at": self.created_at,
         }

app/core/registry.py

@@ -9,7 +9,7 @@
 import uuid
 from pathlib import Path
 
-from app.core.config import DEMUCS_MODEL, EXTRA_STEM_NAMES, JOB_ID_RE, STEM_NAMES
+from app.core.config import DEMUCS_MODEL, DUET_STEM_NAMES, EXTRA_STEM_NAMES, JOB_ID_RE, STEM_NAMES
 from app.core.models import Job
 
 logger = logging.getLogger("stemdeck.registry")
@@ -311,7 +311,7 @@ def _recover_done_job(job_dir: Path) -> Job | None:
         return None
     stems = [
         {"name": name, "url": f"/api/jobs/{job_dir.name}/stems/{name}.wav"}
-        for name in ("original", *STEM_NAMES, *EXTRA_STEM_NAMES)
+        for name in ("original", *STEM_NAMES, *EXTRA_STEM_NAMES, *DUET_STEM_NAMES)
         if (stems_dir / f"{name}.wav").is_file()
     ]
     if not stems:
@@ -326,6 +326,8 @@ def _recover_done_job(job_dir: Path) -> Job | None:
     # regress (#275).
     has_split = all((stems_dir / f"{name}.wav").is_file() for name in EXTRA_STEM_NAMES)
     vocal_split = "done" if has_split else "none"
+    has_duet = all((stems_dir / f"{name}.wav").is_file() for name in DUET_STEM_NAMES)
+    duet_split = "done" if has_duet else "none"
     meta_path = job_dir / "metadata.json"
     meta: dict = {}
     if meta_path.is_file():
@@ -369,6 +371,7 @@ def _recover_done_job(job_dir: Path) -> Job | None:
         sections_source=meta.get("sections_source"),
         tags=meta.get("tags"),
         vocal_split=vocal_split,
+        duet_split=duet_split,
     )