Repository navigation
fix(ingest): contain data-root URIs after symlink resolution - #809
cursor[bot] wants to merge 2 commits into
Conversation
A data-root-relative ingest URI was checked lexically and then re-joined to the data root and resolved later, following symlinks. A directory symlink plus parent segments, or an in-root file symlink, could make the worker open and republish a recording outside the data root. Resolve each local reference once against the real data root, refuse it when the real path leaves the root, and hand that real path to processing, batch sizing, and the media plan filter instead of the original spelling.
[High impact] The PR appears safe to merge; the batch-planning issue from the previous review is fixed.
|
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit ba53d8b. Configure here.
kstonekuan
left a comment
There was a problem hiding this comment.
Batch-mode sizing raises before any episode can run. Could you keep the refusal at the per-episode boundary and include a test in this PR showing that an ordinary sibling recording still processes and the refused item counts as one error?
|
@cursor Could you update this PR so batch planning leaves URI refusals to per-recording processing? Please use benign fixtures to check that one refusal is counted and an ordinary sibling still processes. |
|
In batch mode, a URI that resolves outside the data root no longer fails the whole plan. It is now planned and refused when its recording is processed, where it counts as one error while its siblings still run.
Lint, format and type checks pass. The six #806 tests still pass with this branch's code; I copied the file in temporarily and didn't commit it. The full suite gives 2605 passed and 36 failed. All 36 are the packaging tests that fail the same way on unmodified |
A URI refused for resolving outside the data root no longer fails the batch plan. It is planned at zero bytes without being stat'ed, so process_stage_batch refuses it again and counts it as that recording's error while its siblings process.
kstonekuan
left a comment
There was a problem hiding this comment.
A broken symlink loop raises RuntimeError on Python 3.11, so batch planning still aborts before the other recordings run. Could you normalize path-resolution failures at the input boundary so they count as per-recording errors?
|
The ingest fix landed on main in 27c55e2. The Docker runtime integration passed. |



Summary
Ingest URIs relative to a local data root are now resolved once against the real data root. If the resolved path is outside the root, the URI is refused before the recording is opened. Processing then uses that resolved real path, not the original spelling. A directory symlink plus parent segments (
episodes-in/../neighbor.mcap) and an in-root file symlink pointing outside the root (episodes-in/alias.mcap) are now counted as that episode's error. Before this change, the worker opened and republished those outside recordings into the workspace, where the server served them.Tests that prove the fix: #806. That PR is left unchanged; its test file is not part of this diff.
Why
parse_data_root_relative_urichecks containment on a normalized copy and returns the original spelling, which is intentional (a/../b.mcapstays the conf identity, #314). Later,resolve_episode_referenceandApp._fetch_sourcejoined that spelling to the data root and calledresolve(). That follows symlinks, so a URI could pass the lexical check and still open a file outside the root.What changed:
hflow.uri.resolve_local_data_root_pathresolves the joined path againstdata_root.resolve(), refuses anything outside it, and returns the real path. Callers act on that returned path and never re-join the spelling.resolve_episode_reference(the runtime sync and stage path,hflow ingestin-process, andPOST /api/v1/runtime/ingestworkers) returns the contained real path for local roots. Bucket roots are unchanged because their references are URL joins with no symlinks to follow.App: for a relative reference, the data-root interpretation is only used when it resolves inside the real root. Absolute references and the historical cwd interpretation behave as before (dev API).plan_stage_batchessizes the contained real path, not the lexically normalized key, which pointed at a different file. A URI refused there no longer fails the plan. It is planned at zero bytes and never stat'ed, soprocess_stage_batchrefuses it again and counts it as that recording's error, while its siblings still run._templates.py) used to openPath(data_root) / uridirectly. It now resolves through the same function. A refused URI is kept soprocess_batchrefuses it as a counted error.outstanding_stage_urisand_plan_after_syncpass a refused URI on instead of failing the whole stage plan. The other episodes in the run still complete.docs/RUNTIME.mdnow documents the rule.Other call sites I checked: the server's media and asset endpoints (
hflow_server/_media.py,server.py::_contained_asset_response) already enforce resolved-path containment. No other code joins a caller-supplied path to the data root and opens it.Behaviour change: a symlink inside a local data root that leads outside it is no longer followed for ingest. This includes a directory symlink to another disk. The docs suggest mounting that disk inside the data root, or pointing the data root itself at it (a symlinked data root still works). Source identities for in-root recordings are unchanged, because they were already computed from the resolved path.
Trust boundary: the check binds the resolved real path, and every later step opens that path. Swapping a path component between the check and the open would need write access to directories inside the data root. That access already controls the runtime bundle (the pipeline code the worker runs) and the catalog. A party who can only submit ingest URIs cannot create or swap anything there.
Validation
New in this PR:
tests/test_stage_execution.py::TestLanePlanning::test_batch_lane_leaves_a_refused_uri_to_processing_and_runs_its_sibling. It uses benign synthesized recordings: an ordinary sibling, and an in-root symlink to a harmless recording outside the root. It plans both in batch mode with a realApp, then checks that both URIs are planned, the sync counts total 1 processed and 1 error, the sibling's canonical file exists, and nothing was produced from the linked recording. Without the planner change it fails withValueError: 'episodes-in/linked.mcap' resolves outside the data rootraised from planning.On the PR #806 head with the fix applied, and again on this branch (based on latest
main) with the #806 test file temporarily copied in and not committed:uv run pytest tests/test_symlink_ingest_traversal.py -q: 6 passed. Without the fix, all 6 fail because the outside recording is opened (inotifyIN_OPEN).On this branch, without the #806 tests:
uv run ruff check: passeduv run ruff format --check: passeduv run ty check: passeduv run pytest -q -n 8: 2605 passed, 8 skipped, 36 failed. All 36 failures are intests/test_packaging.pyandtests/test_packaging_manifest_parsing.py. They fail identically on unmodifiedmainin this environment because the native-overlay C compile step (x86_64-linux-gnu-gcc ... labels.c) fails, which is unrelated to this change.uv run pytest tests/test_symlink_ingest_traversal.py tests/test_runtime_bundle.py tests/test_stage_execution.py tests/test_stage_planning.py tests/test_runtime_client.py tests/test_runtime_cli.py packages/hflow-server/tests -n 8: 559 passed.I also retried the technique beyond the tests with a manual probe, using a data root that is itself a symlink:
lnk/../secret.mcap,lnk/deep/x.mcap,lnk/deep/../../secret.mcap, a chained file symlink, and a relative file symlink (../outside/...). This held throughresolve_episode_reference,App.process, batch planning,process_stage_batch,hflow ingestin-process, and the rendered media filter, and the outside files were never opened.inner/good.mcapthrough an in-root directory symlink,inner/../episodes-in/good.mcap,episodes-in/./good.mcap).hflow ingestrun processed the good episode through every stage and counted the refused URIs as sync errors.Checklist
uv run ruff check --fix,uv run ruff format, anduv run ty check.Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.Note
High Risk
This tightens the ingest trust boundary: symlink-based path escape used to allow reading and republishing recordings outside the data root; behavior changes for setups that relied on in-root symlinks to external disks.
Reviewed by Cursor Bugbot for commit 7bfdbc8. Bugbot is set up for automated code reviews on this repo. Configure here.