Skip to content

fix(snapshot): confine copy-mode artifact reads to the catalog workspace - #808

Closed
cursor[bot] wants to merge 3 commits into
mainfrom
cursor/snapshot-copy-workspace-containment-24a1
Closed

cursor[bot] wants to merge 3 commits into
mainfrom
cursor/snapshot-copy-workspace-containment-24a1

Conversation

@cursor

@cursor cursor Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Summary

hflow export snapshot --media copy (and export_dataset_snapshot(..., media_mode="copy")) now copies only artifacts that belong to the catalog's workspace: the directory or bucket prefix that contains the catalog, which is the data root in the standard <data_root>/catalog layout. Any other artifact URI aborts the export with a ValueError (CLI exit 2) before its bytes are read. Reference mode is unchanged: it never reads artifact URIs and exports them as given.

Why

Before this change, copy mode passed every artifact/* value_text from the catalog's measurements table to fetch_uri, then copied the result into assets/. Those URIs are catalog data. Anyone who supplies a catalog directory can therefore make the export copy any local file the operator can read, or fetch any object-store URL with the operator's ambient credentials, into a directory meant to be shipped.

How the fix works (src/hflow/snapshot.py):

  • Local catalogs. The workspace root is resolved, then opened once per export as a directory descriptor. Each artifact URI is resolved and must lie strictly under that root. The file is then opened by walking the resolved path components with openat(O_NOFOLLOW), starting from the workspace descriptor; directories also get O_DIRECTORY. If a component has become a symlink since the check, the open fails and the export is refused. It does not follow the symlink. The opened descriptor must be a regular file, and it is opened with O_NONBLOCK so a FIFO swapped in can't hang the export. The bytes are copied from that descriptor, and the destination file is created only after it is open. Absolute paths, .. segments, relative paths that escape from the working directory, and symlinks that leave the workspace are all refused. Symlinks that stay inside the workspace still work.
  • Bucket catalogs. A URI must start with <workspace prefix>/, and its remainder must pass _validated_relative_key. The object is fetched through a BucketStorageRoot for the workspace prefix that carries the catalog's own store_options. Any other URL is refused before a request is made, including another bucket or a sibling prefix such as data-other/.
  • Object-store URLs in a local catalog are refused.

Other code that turns catalog URIs into bytes:

  • The remaining fetch_uri callers are operator-supplied paths, not catalog data: snapshot.py's --manifest, doctor, and app._fetch_source sources. The storage.fetch(...) callers use validated storage-relative keys.
  • Still open: media serving in hflow-server (packages/hflow-server/src/hflow_server/server.py:341 and :355, via _media.resolve_served_file). It already refuses URIs that resolve outside the data root. But it checks the resolved path and then lets FileResponse reopen it by name, so a file swapped between the check and the open is not covered there. This PR does not change that component.

Validation

The tests that prove the fix are in #804. They are not included in this branch.

  • uv run pytest -q tests/test_snapshot_artifact_uri_containment.py (from test: confine snapshot copy mode to the workspace #804)
    • Before the fix: 7 failed, 2 passed. Each copy-mode case exported the outside bytes: absolute path, CLI, symlink, .. segments, relative escape, symlink swapped after the stat, and an s3:// URL fetched from a local endpoint.
    • With the fix: 9 passed.
  • An extra local adversarial run (not committed) applied the same technique to the new code path: 13 passed. It covered:
    • Swapping a directory component, the final component, or a FIFO in after the containment check and before the openat walk. All are refused, and no outside bytes appear in the output or the staging directory.
    • file:// URLs.
    • A relative episodes/../../ escape.
    • An inside-to-inside symlink, which still copies.
    • A catalog reached through a symlinked parent.
    • A directory URI.
    • A simulated bucket catalog with a sibling-prefix URL, a .. URL, a foreign s3:// URL, and a local path. All are refused, and nothing is written to the mirror.
    • An in-prefix bucket artifact, which still copies.
  • uv run ruff check, uv run ruff format --check, uv run ty check: all pass.
  • uv run pytest -q -n 4 tests/test_dataset_snapshot.py tests/test_snapshot_verify.py tests/test_lerobot_workflow.py tests/test_storage.py tests/test_cli.py: 155 passed, 1 skipped.
  • uv run pytest -q -n 4 (full suite): 2604 passed, 8 skipped, 36 failed.
    • All 36 failures are in tests/test_packaging.py and tests/test_packaging_manifest_parsing.py. They fail identically on main without this change, because this machine lacks the Python C headers needed for the native-overlay compile.
    • One full run also showed a tests/test_verification.py failure under load. That file passes 3/3 runs in isolation and does not exercise snapshot code.

Checklist

  • I added or updated outcome-focused tests for changed business logic. (Regression tests live in test: confine snapshot copy mode to the workspace #804.)
  • I updated documentation for changed behavior, flags, formats, or requirements. (docs/how-to/export-dataset-snapshot.md, export_dataset_snapshot docstring)
  • I ran uv run ruff check --fix, uv run ruff format, and uv run ty check.
  • I ran the relevant pytest suite.
  • I did not add recordings, generated media, credentials, private URLs, or runtime artifacts.
  • I preserved stored-data compatibility or documented an explicit version change. (No format or behavior-version change; the export layout is identical.)
Open in Web Open in Cursor 

Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.


Note

High Risk
This changes security-sensitive file and object-store access during export; the behavior is intentionally stricter and could break snapshots that relied on out-of-workspace artifact URIs.

Reviewed by Cursor Bugbot for commit e1687a8. Bugbot is set up for automated code reviews on this repo. Configure here.

Copy-mode export read every artifact URI from the catalog's measurements
table through fetch_uri and copied the bytes into assets/, so a catalog
could name any local file or object-store URL the operator can read.

Copy mode now reads only artifacts inside the catalog's workspace (the
catalog root's parent). A local URI must resolve under it, and the file is
opened by walking the resolved components with O_NOFOLLOW from a workspace
directory descriptor opened once per export, so a symlink swapped in after
the containment check is refused instead of followed. Bucket catalogs only
fetch keys under their own prefix with the catalog's store options; other
URLs, including object-store URLs in a local catalog, are refused before
any request. References mode still never reads artifact URIs.
@greptile-apps

greptile-apps Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no actionable new issue was found.

Summary

Copy-mode snapshots now limit artifact reads to the catalog's workspace.

  • Snapshot copy mode only reads artifacts inside the catalog workspace.

Reviews (2) · Last reviewed commit: "fix(snapshot): use the real data root fo..." · Reviewed by Greptile

greptile-apps[bot]
greptile-apps Bot previously approved these changes Nov 20, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit eafb47b. Configure here.

Comment thread src/hflow/snapshot.py Resolved

@kstonekuan kstonekuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A catalog opened through a symlink can reject its own artifacts because copy mode uses the symlink’s parent as the workspace. Could you cover that supported layout and include the copy-mode boundary regression in this PR?

@kstonekuan

Copy link
Copy Markdown
Contributor

@cursor Could you update this PR so copy mode accepts a valid catalog opened through a symlink and include coverage using normal files in that workspace?

@cursor

cursor Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Author

I've updated the PR: copy mode now accepts a valid catalog opened through a symlink, and the PR includes tests for it using normal files.

The bug: copy mode took the parent of the catalog path as typed. A catalog opened through a symlink was therefore boxed into the link's folder, and its own artifacts under the real data root were refused. It now resolves the catalog root first and uses that parent. The fix is one line in _artifact_workspace in src/hflow/snapshot.py; I also updated its docstring and the how-to doc.

Tests now in this PR (tests/test_snapshot_artifact_uri_containment.py):

  • Boundary regressions: the copy-mode cases from PR test: confine snapshot copy mode to the workspace #804, which you asked to bring in.
  • Change to one test: confine snapshot copy mode to the workspace #804 test: the case where a file is swapped for a symlink after its lookup now requires the export to refuse. The original test also accepted a copy of the original bytes, and that assert inside an except broke the repo's ruff rules. The fix always refuses here, so the test is stricter, not weaker.
  • test_copy_mode_accepts_catalog_opened_through_symlink: the catalog is opened through a symlink, and its artifact under the real data root is copied.
  • test_copy_mode_through_catalog_symlink_refuses_files_beside_the_link: a file next to the symlink, outside the real data root, is refused and nothing is written.

Both new tests failed on the previous head of the PR and pass now.

Results:

  • The new test file passes (11 tests), as do my 13 adversarial checks and the 155 related existing tests. ruff and ty pass.
  • The full suite has 36 failures in the packaging tests. They fail the same way on main, because this machine lacks the Python C headers they compile against.
  • One tests/test_verification.py failure showed up once under load, didn't recur on rerun, and passes 3/3 on its own.

Decision for you: if someone keeps the catalog as a symlink inside the data root, pointing elsewhere (data/catalog -> /elsewhere/catalog), artifacts under data/episodes are now refused. The workspace becomes /elsewhere. Accepting both folders as the workspace would fix that layout but widen what copy mode can read, so I left it out.

PR #808

Open in Web Open in Cursor 

Bring the copy-mode containment regressions from #804 into this change:
absolute paths, the CLI, symlinks, parent segments, relative escapes, a
file swapped for a symlink after its status lookup, and an object-store
URL in a local catalog are refused without writing their bytes, while
in-workspace artifacts still copy and references mode never reads URIs.
The swapped-file case now requires refusal outright.
…symlink

Copy mode took the workspace as the parent of the catalog path as given,
so a catalog opened through a symlink was confined to the link's
directory and its own artifacts, recorded as resolved paths under the
real data root, were refused. Resolve the catalog root first and use its
parent, so the workspace is the directory that really contains the
catalog.
@greptile-apps
greptile-apps Bot dismissed their stale review November 20, 2026 13:23

Dismissed because a newer commit was pushed; Greptile will re-review the current head.

@kstonekuan kstonekuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you bind the table query and copy boundary to the same resolved catalog? The query opens the catalog before copy mode resolves the workspace, so the two steps can refer to different directories. Please keep the regression coverage focused on ordinary workspace files.

@kstonekuan

Copy link
Copy Markdown
Contributor

The copy-mode fix landed on main in b465f45.

@kstonekuan kstonekuan closed this Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants