A model checkpoint that downloaded incompletely is trusted forever

[thcp]

What happens

If a model download is cut short, the partial file stays on disk and is treated as valid on every subsequent run. The feature that depends on it is silently degraded from then on, permanently, with no error the user will ever see.

Confirmed in the wild: a reporter's install has been running this way since setup. Their log shows both models failing 25 seconds into warmup, far too fast to be a download attempt.

WARMUP_FAILED beat_this ('Could not load the checkpoint given the provided name', 'final0')
WARMUP_FAILED vocal_split Unsupported Model File: parameters for MD5 hash 844a5bf2... could not be found
PytorchStreamReader failed reading zip archive: failed finding central directory

Why it never heals

Existence is the entire validity check. In torch.hub, which is what beat_this loads through:

if not os.path.exists(cached_file):
    download_url_to_file(url, cached_file, hash_prefix, progress=progress)

and download_url_to_file reads until the body ends, then moves the result into place with no comparison against Content-Length. check_hash defaults to False and beat_this does not pass it. audio-separator has the same shape, which is why a bad file there surfaces as an MD5 matching nothing.

So a connection that drops mid-body and closes cleanly is promoted to a complete checkpoint.

Why nobody notices

app/pipeline/beat_detect.py caches the failure for the process and falls back to librosa. That fallback is indistinguishable, from the outside, from a machine that never had the model. The user gets a permanently worse beat grid and nothing anywhere connects the two.

Worse, both app/pipeline/warmup.py and desktop/ui/setup.js state that a failed model "falls back to lazy-download-on-first-use". That is true for an empty cache and false for a wrong one. The retry never fires, because the bad file is still there.

Constraints

• check_hash=True is not ours to pass. beat_this calls into the hub itself, and the URL carries no hash.
• Pinning our own checksums means tracking upstream reissues forever.
• Vendoring a patched beat_this is worse than the problem.
• Any retry has to be bounded, and must not fire when the cache was simply empty. A machine that is offline should not pay the same network timeout twice to reach the same answer.

Surfaced by the logs on #502.

[thcp]

Fixed in #606, landed on main as 53a6c04.

Model loads now go through app/core/model_cache.load_or_heal: if a load fails and there was a cached artifact to blame, the artifact is discarded and the load retried once. Only once per artifact per process, and only when something was actually removed, so an empty cache on an offline machine does not pay the same network timeout twice.

Verified end to end against a real poisoned cache: truncating beat_this-final0.ckpt produced exactly the error from the logs here, the loader discarded it, refetched, and the restored file's sha256 matched the original.

Two things review caught that the tests did not. vocal_split_artifacts named model_data.json, which audio-separator never writes (it writes vr_model_data.json and mdx_model_data.json), so the MD5 case the docstring cited as the reason for listing it would never have healed. And the karaoke model's lazy load had no healing at all, which is the only path Docker ever takes.