Skip to content

fix(ingest): contain data-root URIs after symlink resolution - #809

Closed
cursor[bot] wants to merge 2 commits into
mainfrom
cursor/contain-symlink-ingest-uris-27e9
Closed

cursor[bot] wants to merge 2 commits into
mainfrom
cursor/contain-symlink-ingest-uris-27e9

Conversation

@cursor

@cursor cursor Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Summary

Ingest URIs relative to a local data root are now resolved once against the real data root. If the resolved path is outside the root, the URI is refused before the recording is opened. Processing then uses that resolved real path, not the original spelling. A directory symlink plus parent segments (episodes-in/../neighbor.mcap) and an in-root file symlink pointing outside the root (episodes-in/alias.mcap) are now counted as that episode's error. Before this change, the worker opened and republished those outside recordings into the workspace, where the server served them.

Tests that prove the fix: #806. That PR is left unchanged; its test file is not part of this diff.

Why

parse_data_root_relative_uri checks containment on a normalized copy and returns the original spelling, which is intentional (a/../b.mcap stays the conf identity, #314). Later, resolve_episode_reference and App._fetch_source joined that spelling to the data root and called resolve(). That follows symlinks, so a URI could pass the lexical check and still open a file outside the root.

What changed:

  • hflow.uri.resolve_local_data_root_path resolves the joined path against data_root.resolve(), refuses anything outside it, and returns the real path. Callers act on that returned path and never re-join the spelling.
  • resolve_episode_reference (the runtime sync and stage path, hflow ingest in-process, and POST /api/v1/runtime/ingest workers) returns the contained real path for local roots. Bucket roots are unchanged because their references are URL joins with no symlinks to follow.
  • App: for a relative reference, the data-root interpretation is only used when it resolves inside the real root. Absolute references and the historical cwd interpretation behave as before (dev API).
  • plan_stage_batches sizes the contained real path, not the lexically normalized key, which pointed at a different file. A URI refused there no longer fails the plan. It is planned at zero bytes and never stat'ed, so process_stage_batch refuses it again and counts it as that recording's error, while its siblings still run.
  • The media plan filter in the rendered stage DAG (_templates.py) used to open Path(data_root) / uri directly. It now resolves through the same function. A refused URI is kept so process_batch refuses it as a counted error.
  • outstanding_stage_uris and _plan_after_sync pass a refused URI on instead of failing the whole stage plan. The other episodes in the run still complete.
  • docs/RUNTIME.md now documents the rule.

Other call sites I checked: the server's media and asset endpoints (hflow_server/_media.py, server.py::_contained_asset_response) already enforce resolved-path containment. No other code joins a caller-supplied path to the data root and opens it.

Behaviour change: a symlink inside a local data root that leads outside it is no longer followed for ingest. This includes a directory symlink to another disk. The docs suggest mounting that disk inside the data root, or pointing the data root itself at it (a symlinked data root still works). Source identities for in-root recordings are unchanged, because they were already computed from the resolved path.

Trust boundary: the check binds the resolved real path, and every later step opens that path. Swapping a path component between the check and the open would need write access to directories inside the data root. That access already controls the runtime bundle (the pipeline code the worker runs) and the catalog. A party who can only submit ingest URIs cannot create or swap anything there.

Validation

New in this PR: tests/test_stage_execution.py::TestLanePlanning::test_batch_lane_leaves_a_refused_uri_to_processing_and_runs_its_sibling. It uses benign synthesized recordings: an ordinary sibling, and an in-root symlink to a harmless recording outside the root. It plans both in batch mode with a real App, then checks that both URIs are planned, the sync counts total 1 processed and 1 error, the sibling's canonical file exists, and nothing was produced from the linked recording. Without the planner change it fails with ValueError: 'episodes-in/linked.mcap' resolves outside the data root raised from planning.

On the PR #806 head with the fix applied, and again on this branch (based on latest main) with the #806 test file temporarily copied in and not committed:

  • uv run pytest tests/test_symlink_ingest_traversal.py -q: 6 passed. Without the fix, all 6 fail because the outside recording is opened (inotify IN_OPEN).

On this branch, without the #806 tests:

  • uv run ruff check: passed
  • uv run ruff format --check: passed
  • uv run ty check: passed
  • uv run pytest -q -n 8: 2605 passed, 8 skipped, 36 failed. All 36 failures are in tests/test_packaging.py and tests/test_packaging_manifest_parsing.py. They fail identically on unmodified main in this environment because the native-overlay C compile step (x86_64-linux-gnu-gcc ... labels.c) fails, which is unrelated to this change.
  • Targeted run on the PR test: contain symlink-resolved ingest URIs #806 head: uv run pytest tests/test_symlink_ingest_traversal.py tests/test_runtime_bundle.py tests/test_stage_execution.py tests/test_stage_planning.py tests/test_runtime_client.py tests/test_runtime_cli.py packages/hflow-server/tests -n 8: 559 passed.

I also retried the technique beyond the tests with a manual probe, using a data root that is itself a symlink:

  • Refused: lnk/../secret.mcap, lnk/deep/x.mcap, lnk/deep/../../secret.mcap, a chained file symlink, and a relative file symlink (../outside/...). This held through resolve_episode_reference, App.process, batch planning, process_stage_batch, hflow ingest in-process, and the rendered media filter, and the outside files were never opened.
  • Still accepted: in-root spellings (inner/good.mcap through an in-root directory symlink, inner/../episodes-in/good.mcap, episodes-in/./good.mcap).
  • A mixed hflow ingest run processed the good episode through every stage and counted the refused URIs as sync errors.

Checklist

  • I added or updated outcome-focused tests for changed business logic. (Batch planning with a refused sibling is tested here; the attack regression tests live in test: contain symlink-resolved ingest URIs #806.)
  • I updated documentation for changed behavior, flags, formats, or requirements.
  • I ran uv run ruff check --fix, uv run ruff format, and uv run ty check.
  • I ran the relevant pytest suite.
  • I did not add recordings, generated media, credentials, private URLs, or runtime artifacts.
  • I preserved stored-data compatibility or documented an explicit version change.
Open in Web Open in Cursor 

Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.


Note

High Risk
This tightens the ingest trust boundary: symlink-based path escape used to allow reading and republishing recordings outside the data root; behavior changes for setups that relied on in-root symlinks to external disks.

Reviewed by Cursor Bugbot for commit 7bfdbc8. Bugbot is set up for automated code reviews on this repo. Configure here.

A data-root-relative ingest URI was checked lexically and then re-joined to
the data root and resolved later, following symlinks. A directory symlink plus
parent segments, or an in-root file symlink, could make the worker open and
republish a recording outside the data root.

Resolve each local reference once against the real data root, refuse it when
the real path leaves the root, and hand that real path to processing, batch
sizing, and the media plan filter instead of the original spelling.
@greptile-apps

greptile-apps Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[High impact] The PR appears safe to merge; the batch-planning issue from the previous review is fixed.

Summary

Local ingest now checks where symlinks lead and uses the accepted real path.

  • Data-root ingest follows symlinks before it opens a recording.
  • Stage planners leave refused recordings for per-episode errors.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Relative ingest URI] --> B[Resolve against real data root]
    B --> C{Inside root?}
    C -->|Yes| D[Size and process real path]
    C -->|No| E[Keep URI with size zero]
    E --> F[Processing refuses URI]
    F --> G[Count error and record failure]
Loading

Reviews (2) · Last reviewed commit: "fix(ingest): leave batch-plan URI refusa..." · Reviewed by Greptile

Comment thread src/hflow/stage_execution.py Resolved

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit ba53d8b. Configure here.

Comment thread src/hflow/stage_execution.py Resolved

@kstonekuan kstonekuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Batch-mode sizing raises before any episode can run. Could you keep the refusal at the per-episode boundary and include a test in this PR showing that an ordinary sibling recording still processes and the refused item counts as one error?

@kstonekuan

Copy link
Copy Markdown
Contributor

@cursor Could you update this PR so batch planning leaves URI refusals to per-recording processing? Please use benign fixtures to check that one refusal is counted and an ordinary sibling still processes.

@cursor

cursor Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Author

In batch mode, a URI that resolves outside the data root no longer fails the whole plan. It is now planned and refused when its recording is processed, where it counts as one error while its siblings still run.

  • Planner change (src/hflow/stage_execution.py): plan_stage_batches now catches the containment refusal while sizing. It plans that URI at zero bytes without opening or stat-ing it, and process_stage_batch refuses it again as that recording's own error. A URI whose file is simply missing still fails planning, as it did before this PR.

  • New test (test_batch_lane_leaves_a_refused_uri_to_processing_and_runs_its_sibling in tests/test_stage_execution.py): it uses two harmless synthesized recordings, an ordinary sibling and an in-root symlink to one outside the root. It plans them in batch mode and syncs them with a real App, then checks:

    • both URIs are planned;
    • the totals are 1 processed and 1 error;
    • the sibling's canonical file exists;
    • nothing was produced from the linked recording.

    Without the planner change, it fails with the same planning error the reviewers described.

Lint, format and type checks pass. The six #806 tests still pass with this branch's code; I copied the file in temporarily and didn't commit it. The full suite gives 2605 passed and 36 failed. All 36 are the packaging tests that fail the same way on unmodified main here, because a C compile step for native extensions fails in this environment. I pushed the commit and updated the validation and change notes in the PR description of #809, leaving the bot footers others added untouched.

Open in Web Open in Cursor 

A URI refused for resolving outside the data root no longer fails the batch
plan. It is planned at zero bytes without being stat'ed, so
process_stage_batch refuses it again and counts it as that recording's error
while its siblings process.

@kstonekuan kstonekuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A broken symlink loop raises RuntimeError on Python 3.11, so batch planning still aborts before the other recordings run. Could you normalize path-resolution failures at the input boundary so they count as per-recording errors?

@kstonekuan

Copy link
Copy Markdown
Contributor

The ingest fix landed on main in 27c55e2. The Docker runtime integration passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants