Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions apps/docs/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
node_modules/
.mintlify/
scripts/docs-*.png
28 changes: 28 additions & 0 deletions apps/docs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# DEMON docs

Public-facing documentation site for DEMON, built with
[Mintlify](https://mintlify.com). Same tooling as the Daydream docs at
`pipelines/apps/docs`.

This directory is the product documentation source: install guides, demo
operator docs, API references, troubleshooting, and deployment notes. The
repo-root `docs/` folder is a separate GitHub Pages project/paper landing page.

## Local development

```bash
cd apps/docs
npm install
npm run dev
# open http://localhost:3033
```

Navigation, theming, and redirects live in `docs.json`. Pages are MDX with
YAML frontmatter. Screenshots live in `images/`.

## Checks

```bash
npm run build
npm run broken-links
```
114 changes: 114 additions & 0 deletions apps/docs/advanced/benchmarking.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
---
title: "Benchmarking & Latency"
description: "Headless throughput benchmarks, the e2e latency suite, and the knob-to-ear path"
icon: "stopwatch"
---

## Headless benchmark

The browser HUD shows `tick_ms` / `dec_ms` live, but for comparing
accelerators, checkpoints, TRT profiles, or VAE settings without browser and
audio-device overhead, use the headless benchmark — it mirrors the backend
inference path exactly (fixture defaults, engine selection, `Session`,
`prepare_source`, `encode_text`, `stream`, repeated ticks, optional decode):

```bash
uv run python -u -m demos.realtime_motion_graph_web.benchmark \
--accel tensorrt --checkpoint acestep-v15-turbo
```

It reports setup timings, per-generation `tick` / `decode` / `tick+decode`
mean, P50/P90/P95, min/max, skip counts, and peak CUDA memory.

Useful variants:

```bash
# Compare torch.compile on the same workload:
uv run python -u -m demos.realtime_motion_graph_web.benchmark --accel compile

# Mixed backends, same flags as the server:
uv run python -u -m demos.realtime_motion_graph_web.benchmark \
--decoder-accel tensorrt --vae-accel eager

# Persist raw samples + summary:
uv run python -u -m demos.realtime_motion_graph_web.benchmark \
--accel tensorrt --json runs/trt-bench.json

# Mirror the runner's decode-skip behavior / decoder-only throughput:
uv run python -u -m demos.realtime_motion_graph_web.benchmark --skip-threshold 1e-3
uv run python -u -m demos.realtime_motion_graph_web.benchmark --no-decode
```

## The knob-to-ear path

A knob change traverses, in order:

| Stage | Typical cost |
|---|---|
| Client knob smoothing + 125 Hz param send | 0–smoothing window |
| WebSocket transport (LAN) | ~0–2 ms |
| Next runner iteration | 0–1 tick |
| **Shared-curve params** (SDE strength, velocity, x0, channel gains…) | **1 tick** (~36–43 ms TRT) |
| **Per-slot params** (ODE denoise, seed) | up to `steps` ticks (~280–460 ms at depth 4 / steps 8) |
| Windowed VAE decode + emit | ~5–8 ms |
| **Playback lead** (decode buffer ahead of the playhead) | **0.12–1.35 s adaptive** |
| Browser AudioWorklet buffer | ~10–30 ms |

Two structural facts dominate: **parameter class decides engine latency**
(shared-curve is an order of magnitude faster than per-slot — see
[Streaming Pipeline](/engine/streaming#the-two-latency-classes)), and **the
playback lead is the dominant audible term** — the engine converges in
~300 ms while the listener waits for the playhead to reach the rewritten
region. The lead floor/ceiling/release are tunable in
[config.json](/demo/configuration#engine).

## Measured numbers

RTX 5090, 2B turbo, all-TRT, depth 4, steps 8 (the README configuration):
~43 ms tick, 4.5 ms windowed decode, **11.3 generations/s**, ~248 ms
parameter convergence.

RTX 4090 (shared GPU), TRT, depth 4, steps 8 — from the latency review in
`docs/LATENCY.md`:

| Metric | Measured |
|---|---|
| tick p50 | ~36 ms |
| `denoise` knob first audible change | 8 ticks / ~282 ms |
| shared-curve first change | 1 tick / ~37 ms |
| knob → fresh generation (full stack) | ~172 ms |
| generations/s | ~7.3 |

## The e2e latency suite

`tests/e2e_latency/` measures the same things reproducibly, black-box at
public API boundaries (GPU required):

```bash
# engine-level: knob -> finished latent (ticks + ms), determinism guard
uv run pytest tests/e2e_latency/test_knob_to_latent.py -v -s

# full stack: headless StreamingSession with a simulated client heartbeat
uv run pytest tests/e2e_latency/test_streaming_session.py -v -s
```

Environment knobs: `DEMON_E2E_ACCEL=tensorrt|eager|compile`,
`DEMON_E2E_GPU`, `DEMON_E2E_DEPTH`, `DEMON_E2E_STEPS`. Reports land in
`runs/latency-reports/e2e-*.json` for build-to-build diffing — **treat the
diff, not the pass/fail, as the output**; the assertions are coarse
architectural ceilings only.

## VRAM: engine duration matters

Per-engine peak workspace measured in isolation on an RTX 5090:

| Component | 60 s engine | 240 s engine | Δ |
|---|---:|---:|---:|
| Decoder (refit) | 13,511 MB | 15,911 MB | +2,400 MB |
| VAE decode | 10,547 MB | 10,814 MB | +267 MB |
| VAE encode | 4,178 MB | 10,614 MB | +6,436 MB |

These are per-engine peaks, not a live-runtime sum — at inference time the
decoder peak dominates, which is why the live demo fits on a 24 GB card. The
takeaway: switching the three engines from 240 s to 60 s frees ~9 GB.
Reproduce with `uv run python -m scripts.benchmarks.bench_vram_60s_vs_240s`.
84 changes: 84 additions & 0 deletions apps/docs/advanced/development.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
---
title: "Development"
description: "The contract-first workflow, tests, and building your own frontend on the SDK"
icon: "wrench"
---

## The contract-first rule

The control surface is **contract-first**: two Python registries are the
single source of truth, and every consumer — the bundled UI, the MCP server,
your frontend — derives from them.

| Registry | Serves |
|---|---|
| `acestep/streaming/knobs.py` | The knob universe → `GET /api/knobs` |
| `demos/realtime_motion_graph_web/protocol.py` | The wire contract → `GET /api/protocol` |

After changing either registry, regenerate the TypeScript projection:

```bash
python demos/realtime_motion_graph_web/scripts/gen_wire_types.py
# writes web/sdk/types/wireContract.gen.ts
```

Contract-drift guards (`tests/unit/test_wire_contract.py`,
`test_client_sdk.py`) fail if the generated types or SDK fall out of sync.

## Tests

```bash
# Unit tests — no GPU required; includes the contract guards
uv run pytest tests/unit -v

# Web front-end checks
cd demos/realtime_motion_graph_web/web
npm run typecheck && npm run build

# Web unit + replay tests, and Playwright e2e
npm run test
npm run test:e2e

# GPU suites
uv run pytest tests/e2e_latency/ -v -s # latency measurements
# tests/golden/ — wire-level regression net (refs captured per GPU class)
```

## Building your own frontend

The demo's web client is built on an importable SDK
(`demos/realtime_motion_graph_web/web/sdk/`) that any frontend can reuse:

- `protocol.ts` + `types/wireContract.gen.ts` — wire format primitives,
generated from the Python source of truth.
- `audio/RemoteBackend.ts` — the WebSocket client (session init + streaming
receive); `audio/AudioPlayer.ts` — playback with crossfade/delta logic;
`assets/audio-worklet.js` — the AudioWorklet ring buffer.
- `fetchKnobManifest.ts` / `fetchWireContract.ts` — runtime contract
discovery; `wsReconnect.ts` — backoff + reconnect.

Two reference implementations to copy from:

- **`DynamicKnobPanel.tsx`** — renders a complete control panel purely from
the `/api/knobs` manifest, zero hardcoded knobs. If your frontend does
this, new server-side knobs appear automatically.
- **`useStartSession.ts`** — the full session bootstrap + reconnect
orchestration.

The SDK ships its own `AGENTS.md` developer guide covering the invariants
(manifest discovery, echo channels, epoch guards, state management).

## Repo gotchas

- **`sys.path` shadowing:** standalone scripts must prepend the repo root
before importing `acestep` — a sibling ACE-Step checkout can shadow the
package.
- **Fixture sidecars:** after editing a fixture's `track.json` (BPM/key),
re-run `uv run python -m scripts.calibration.precompute_fixture_sidecars`
(`--force` to re-derive everything; `--with-stems` to also rip stems).
- **Knob coercion semantics are deliberate:** knob values *clamp*
out-of-range numerics; command-envelope fields *drop* on type mismatch.
The two coercers are intentionally divergent — don't merge them.
- The repo-root `docs/` folder contains internal development notes (TRT
deep-dive, latency review, stem extraction) — useful background, not part
of this site.
95 changes: 95 additions & 0 deletions apps/docs/advanced/mcp.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
---
title: "MCP Control"
description: "Drive the live demo from Claude Code or any MCP client"
icon: "robot"
---

Every user-facing action in the demo — prompt, knobs, LoRA enable/disable,
timbre/structure references, source swaps, even sample-exact audio writes —
is exposed as an MCP tool by the onboard stdio MCP server
(`demos/realtime_motion_graph_web/mcp_server.py`). Point Claude Code (or any
MCP client) at it and the LLM plays the session you have open in your
browser.

## How it works

1. You open the demo in your browser as usual. The backend registers that
session in a process-global registry and starts the **control bus**, a
small HTTP server on `127.0.0.1:1319`.
2. The MCP server speaks HTTP to the control bus. It does **not** open its
own WebSocket and does **not** start a separate GPU session — running it
alongside a browser tab is free.
3. Every MCP command dispatches through the same handler as the browser's own
WebSocket frames. Acks flow back to the browser, so **the UI mirrors
MCP-driven changes automatically** — you watch the knobs move.

<Note>
Open the browser tab first. The MCP server refuses to act when there is no
live session to attach to. Tools take an optional `session_id` and default to
the most recently started session.
</Note>

## Setup

Run the backend the usual way:

```bash
uv run python -u -m demos.realtime_motion_graph_web.run
```

Then register the MCP server with your client — for Claude Code, in
`.mcp.json` or settings:

```jsonc
{
"mcpServers": {
"demon": {
"command": "uv",
"args": [
"run", "python", "-u",
"-m", "demos.realtime_motion_graph_web.mcp_server"
],
"cwd": "/path/to/DEMON"
}
}
}
```

Environment overrides: `DEMON_HOST` / `DEMON_PORT` (backend, default
`127.0.0.1:1318`) and `DEMON_CONTROL_HOST` / `DEMON_CONTROL_PORT` (control
bus, default `127.0.0.1:1319`).

## What an agent can do

The complete tool list with signatures is in the
[MCP tools reference](/reference/mcp-tools). The shape of the surface:

- **Discover** — `list_sessions`, `session_state`, `list_knobs`,
`list_loras`, `list_fixtures`, `describe_protocol` — the same
self-describing manifests the HTTP API serves, so an agent can learn the
control surface at runtime.
- **Steer** — `set_prompt`, `set_prompt_blend`, `set_knob(s)`,
`set_rcfg_mode`, `enable_lora` / `disable_lora`, manual steering slots.
- **Re-source** — `swap_to_fixture` / `swap_to_audio`,
`set_timbre_*` / `set_structure_*` with local audio files (resampled
server-side).
- **Write audio live** — `write_audio` places a clip into the playing source
sample-exactly, as a declicked replace or an overdub (`sum`), optionally
tiled across the track — agent-driven live looping.

## Origin sensitivity

Two commands behave differently when they arrive from the control bus rather
than the browser: `params` (knob values) and `set_prompt_blend` do not apply
directly — the server echoes them to the browser (`params_echo` /
`prompt_blend_echo`), and the browser UI mirrors the value and re-sends it
through its own 125 Hz parameter channel. This keeps the browser the single
writer of the hot parameter path. All other commands apply identically from
any origin.

## Security

The control bus accepts writes into live sessions, which is why it binds to
localhost by default. The `--control-host`, `--control-port`, and
`--no-control` server flags are documented in
[Cloud Deployment](/cloud/deployment#control-bus-security).
Loading