General description

StreamDoctor is a a tool that broadcasts a synthetic stream and measures what happened to it along the way, for teams maintaining video infrastructure, so that e.g. A/V drift and dropped frames get caught by CI rather than by a human clicking through a list of manual tests.

StreamDoctor consists of a couple of elements:

• a "daemon" - server capable of spawning "streamers" and "viewers" and dynamically managing them, controlled via REST API, delivered as a binary, it could be using Boombox and Membrane under the hood
• "core" - a native implementation (e.g. in Rust, with wgpu) of core functions that need to be run both by daemon and in the browser (e.g. via WASM), potentially computationally heavy, like marker encoding/decoding etc.
• a bunch of SDKs (e.g. in Typescript), which spawn daemon processes and use them to execute testing scenarios defined by user

Below there is an example of how the usage of StreamDoctor could look like:

test('stream reaches viewers in sync', async ({ page, stream_doc }) => {
  await page.goto('/login'); // Regular Playwright's login logic

  const rtmpUrl = await page.locator('[data-test=rtmp-url]').innerText();
  const hlsUrl  = await page.locator('[data-test=hls-url]').innerText();

  const session = await stream_doc.session();

  const streamer = session.publish(rtmpUrl, {
    source: { kind: 'synthetic', pattern: 'bars', fps: 60 },
    // {kind: 'file', path: 'my_file.mp4' }
    // {kind: 'file', path: TEST_CORPUS.b_frames_h264}
    marker: true,
  });
  await streamer.waitUntilLive();

  await page.click('button#play');

  const viewer = await session.watch(hlsUrl);

  await viewer.mark('baseline', '30s');
  await streamer.degrade({ packetLoss: 0.3 });
  await viewer.mark('degraded', '15s');
  await streamer.restore();
  await viewer.mark('recovery', '10s');
  
  const m = await viewer.stop();
  
  expect(m.window('baseline').avDriftMs.status).toBe('measured'); // status might be: 'measured', 'missing_marker' or 'invalid'
  expect(Math.abs(m.window('baseline').avDriftMs.value)).toBeLessThan(50);
  expect(m.window('baseline').variants).toEqual(['1080p', '720p', '480p']);

  expect(m.window('baseline').stalls).toHaveLength(0);
  expect(m.window('degraded').video.fps.average).toBeGreaterThan(20);
  expect(m.window('recovery').avDriftMs.value).toBeLessThan(50);
});

StreamDoctor requirements

Below there is a list of features of StreamDoctor
✅ - let's do it in MVP
❌ - not for MVP
⚠️ - technical risk, requires validation

1. Source
   ✅ synthetic video (test pattern, configurable fps/resolution/bitrate)
   ✅ reference marker - visual bars and audio bursts, encoding frame number and session ID ⚠️
   ✅ broadcasting from a file
   ❌ a set of "edge case" streams (strange H.264 profiles, negative timestamps etc.) ⚠️

2. Orchestration
   ✅ session grouping a publisher and its viewers, isolated across parallel runs
   ✅ waitUntilLive()- synchronization with the actual start of broadcasting
   ✅ cleanup on test failure and on process kill
   ❌ named windows (mark()) assigned by marker
   ❌ multiple publishers within a single session
   ❌ session with multiple "hops"

3. Metrics
   Structural (just playlist/container parsing)
   ✅ list of variants with their parameters
   ❌ presence of an audio track
   ✅ container type (MPEG-TS / CMAF)
   ❌ discontinuities
   ❌ PTS monotonicity
   ✅ time to first frame
   Content-based (requires decoding)
   ✅ frozen frames, black frames, silence
   ❌ fps, bitrate
   ❌ flickering ⚠️
   ❌ stalls ⚠️
   Reference (requires a marked stream)
   ❌ A/V synchronization ⚠️
   ✅ frame ordering
   ✅ ingest → output latency
   ❌ missing or duplicated frames
   ❌ VMAF / quality metrics ⚠️

4. Chaos
   ❌ packet loss on the sender side, controlled throughout the test duration (degrade()) ⚠️
   ❌ restore() ⚠️
   ❌ latency, jitter, bandwidth limits ⚠️
   ❌ chaos on the receiver side (throttling, segment delaying) ⚠️

5. Protocols

6. Delivery
   ✅ daemon with REST API as delivered as a binary ⚠️
   ✅ SDK in TypeScript (daemon lazy start, daemon version check during handshake)
   ❌ SDK in Python
   ❌ Docker image
   ❌ daemon hosted "as a service" ⚠️

Blockers/decisions to make

• Should daemon use Membrane/Boombox as a multimedia framework? (alternatively we could use e.g. MediaBunny)