Full-body tracking for VRChat from an Intel RealSense D415, sent over OSC. Works with VRChat on Quest standalone — no PC VR, no SteamVR, no SlimeVR hardware. The machine running this becomes a tracking server on your LAN.
D415 ──► MediaPipe pose ──► depth lift ──► stabilise ──► predict ──► OSC ──► VRChat
(2D landmarks) (real metres) (gate/bones/ (latency) (1 packet
smooth/fill) per frame)
MediaPipe estimates a 3D pose from a single camera, but its depth axis is guessed from limb proportions. This replaces that guess with measured depth from the RealSense, then stabilises and latency-compensates the result.
Read this before relying on it.
Everything up to the network has been measured on real hardware: 100% tracking, 4.0 mm depth jitter, 4.4 mm jitter after stabilisation. What has not been confirmed is the last step — that VRChat's FBT calibration maps our trackers onto the right body parts and interprets our rotations as intended. Desktop VRChat hides all FBT settings without a headset, so that test needs a Quest and has not been run.
Two things could be wrong, and both are one-line fixes:
- Tracker numbering. We follow SlimeVR's convention (hip=1, feet=2/3,
knees=4/5, chest=6, elbows=7/8). VRChat's own docs never state a mapping. If
calibration puts trackers on the wrong limbs, edit
TRACKER_ROLESintransform.py. - Rotations. The maths is proven correct in isolation and the values are
stable, but nothing has confirmed VRChat reads them as intended. Run with
--no-rotationsif limbs orient oddly.
- Intel RealSense D415 (other D400 models likely work, untested)
- Python 3.10–3.12 — 3.12 recommended; MediaPipe has no 3.13+ wheel
- ~2.5 m of clear floor, so the camera sees you head to feet
- A lit room — see Known limitations
Every dependency is a wheel.
py -3.12 -m venv .venv
.venv\Scripts\pip install -e ".[camera]"
.venv\Scripts\python bodytracker.py fetchpython3.12 -m venv .venv
.venv/bin/pip install -e .
.venv/bin/python bodytracker.py fetchpyrealsense2 is the catch: on recent distributions there is no apt package and
no wheel for Python 3.12, so it must be built from source. SETUP.md documents
the exact build — including two flags without which it fails — and the udev rule
the camera needs to be usable without root.
# 1. check your framing -- nothing is sent
python bodytracker.py preview
# 2. send to VRChat on the Quest, with the overlay window
python bodytracker.py run 192.168.1.42 --previewThen in VRChat: Options → OSC → Enabled, set your real height in settings, and Quick Menu → Launch Pad → Calibrate FBT.
If your camera is mounted upside down, add --flip.
| subcommand | what it does |
|---|---|
run [host] |
track and send OSC. --preview adds the overlay in the same process |
preview |
camera + overlay only, sends nothing |
fake [host] |
synthetic trackers, no camera — separates network problems from tracking ones |
listen |
watch VRChat's outgoing OSC on 9001, proving OSC is really enabled |
measure |
record once, replay through each stabilisation stage |
diagnose |
why joints get dropped, and whether depth or landmarks dominate the error |
fetch |
download the pose models |
fetch writes to models/ in a checkout, or to your user data directory if the
package was installed non-editably. It prints the path either way; set
BODYTRACKER_MODEL_DIR to override it.
Measured on a real body. These come from three different recordings, not one run — the source is given per row, because presenting them as a single session implied a consistency the measurements do not have:
| metric | value | recording | reproduce |
|---|---|---|---|
| tracking rate | 100% | 542 frames | diagnose |
| depth jitter / lateral jitter | 4.0 mm / 3.0 mm | 542 frames | diagnose |
| framerate | 28–30 fps | 542 frames | diagnose |
| jitter, stabilised | 4.4 mm | 446 frames | measure |
| limb-length spread, stabilised | 56.8 mm | 446 frames | measure |
| jitter −57%, worst glitch −75% | vs unstabilised | 446 frames | measure |
| positional error at display time | 3.8 mm | 446 frames | not reproducible |
Stabilisation accounts for most of it, and occlusion filling takes ankle availability from 71% to 100%.
The last row is honest but historical: it comes from the lead-time sweep in
commit 4a88d40, whose script was never committed, so no command in this repo
re-derives it. Treat it as a recorded observation rather than a current
measurement. The same applies to the per-role table further down — diagnose
reports per-landmark figures, not per-role.
Occlusion is the real one. A single camera sees one side of you. Turn 90° and your far leg hides behind the near one; turn 180° and it is guessing. IMU trackers (SlimeVR) do not care which way you face — that is why they dominate the market. If you dance or turn constantly this will frustrate you. For sitting, standing and conversation it is fine.
Lighting. Landmarks come from the RGB image, so the default mode does not
work in the dark despite this being a depth camera. The failure is asymmetric and
easy to misdiagnose: the depth stream still looks perfect while tracking has
stopped. In preview, NO POSE DETECTED alongside a good depth image (press d)
means too dark, not broken.
There is a way out: --source ir runs the pose model on the infrared imagers,
which the projector illuminates, so it works with the lights off. Measured at
100% detection and 4.8 mm lateral jitter with --ir-emitter off and a separate IR
source; with the emitter on, the dot pattern is painted over the body and
detection drops to ~48%. See SETUP.md for the full comparison.
Repeated open/close degrades the camera. After several start/stop cycles in one session the stream can collapse to ~0.7 fps with no error at all. The app detects this and tells you to replug. It is a hardware state, not a configuration problem.
Room depth. You need roughly 2.5 m of clear floor. preview shows HEAD in frame / FEET in frame — back up until both are green.
Default is hip + chest + both feet. VRChat's docs warn that fewer trackers often track better, but that is about reliability: its IK compensates well for a missing point and badly for a wrong one. Measured per role over 446 frames:
| role | idx | present | jitter | |
|---|---|---|---|---|
| hip | 1 | 100% | 4.2 mm | core |
| left_foot | 2 | 99% | 7.9 mm | standard FBT |
| right_foot | 3 | 100% | 4.8 mm | standard FBT |
| left_knee | 4 | 100% | 6.7 mm | optional |
| right_knee | 5 | 100% | 5.6 mm | optional |
| chest | 6 | 100% | 3.1 mm | core — our steadiest point |
| left_elbow | 7 | 100% | 5.5 mm | not recommended |
| right_elbow | 8 | 94% | 6.7 mm | not recommended |
Elbows are not recommended: your hands come from the Quest controllers, so an elbow tracker constrains an arm whose endpoint is already authoritative, and a slightly-wrong elbow fights the controller. Knees add little, because VRChat's IK already infers them from hip and foot.
Indices follow SlimeVR's mapping, the de-facto standard (VRCOSCHandler.kt,
which carries the comment "Don't change as third party applications may rely on
this").
bodytracker.py run [host] [--port 9000]
--preview overlay window in the same process
--roles hip,chest,left_foot,right_foot
--flip camera mounted upside down
--lead-ms 50 latency compensation (0 disables)
--send-hz 90 output rate (0 = one send per camera frame)
--flip / --no-flip camera mounted upside down
--model full lite | full | heavy
--min-cutoff 0.5 --beta 0.35 smoothing
--no-bones --no-gate --no-fill disable a stabilisation stage
--no-rotations --no-filter disable rotations / depth post-processing
Every --no-* switch exists so a stage can be measured against its own absence
with bodytracker.py measure, not as a workaround.
Each was chosen by measurement, and several contradict the obvious guess:
- MEDIUM_DENSITY depth preset — ankle availability 96% vs 58% for HIGH_DENSITY. A static-scene benchmark says HIGH_ACCURACY is best; on an actual body it is the worst of the three. Benchmark on a person.
- 848×480 @ 30 fps — 1280×720 carries 50% more depth noise at identical coverage.
fullmodel —heavyis 3× slower with a body in frame (15 fps), and halving the update rate costs more than its per-frame accuracy gains.- Hole filling
nearest, not the SDK defaultfarthest— the default fills a gap on your torso with the wall behind you. - 50 ms prediction lead — cuts display-time error 9.0 → 3.8 mm, and the optimum lands on the independently measured pipeline latency.
Every other default here was settled by measurement. This one was not, and it is called out rather than buried.
VRChat applies tracker data per rendered frame, so a 30 Hz send to a 72 Hz
headset is held for roughly 2.4 frames — up to 33 ms of staleness, 16 ms on
average, stacked on top of the ~50 ms pipeline latency --lead-ms already
cancels. At 90 Hz that ceiling drops to about 11 ms. The poses in between are
re-extrapolations of measurements already held, so the camera and the pose model
do no extra work; it behaves as a predictor rather than a hold, because the
extrapolation horizon grows smoothly between camera frames and resets by only the
prediction error when the next one lands.
What is not known is whether the extra extrapolation between frames costs more jitter than the staleness it removes. Settling that needs a recording denser than the pipeline's own 30 fps — otherwise the ground truth has to be interpolated, and linear interpolation is the same model linear extrapolation assumes, so the yardstick would favour the thing being tested.
Two things make the risk manageable. The sender thread contends for the GIL with
inference, so the achieved rate can fall short of the request — the loop prints
the rate it actually sends at every 150 frames, and that is the number to
watch. And --send-hz 0 restores the previous behaviour exactly: one send per
camera frame, byte-identical output apart from rotation prediction.
--flip if your camera is mounted upside down. On a permanently inverted rig, set
BODYTRACKER_FLIP=1 once instead of remembering the flag — forgetting it fails in
the confusing direction, because an upside-down image still tracks and still sends,
it just puts your feet where your head is. --no-flip overrides the variable.
pip install -e ".[dev]"
pytest # no camera required
ruff check .The tests cover what broke silently during development: the Euler conversion, the axis transform, the OSC wire format, and each stabilisation stage. Every one of those bugs was invisible in the preview window, which is why they are tests rather than something you eyeball.
SETUP.md— build record, every measurement, and the settings auditRESEARCH.md— the original feasibility study and the VRChat OSC contract
MIT