Repository navigation
fix(snapshot): confine copy-mode artifact reads to the catalog workspace - #808
cursor[bot] wants to merge 3 commits into
Conversation
Copy-mode export read every artifact URI from the catalog's measurements table through fetch_uri and copied the bytes into assets/, so a catalog could name any local file or object-store URL the operator can read. Copy mode now reads only artifacts inside the catalog's workspace (the catalog root's parent). A local URI must resolve under it, and the file is opened by walking the resolved components with O_NOFOLLOW from a workspace directory descriptor opened once per export, so a symlink swapped in after the containment check is refused instead of followed. Bucket catalogs only fetch keys under their own prefix with the catalog's store options; other URLs, including object-store URLs in a local catalog, are refused before any request. References mode still never reads artifact URIs.
[Medium impact] The PR appears safe to merge; no actionable new issue was found.
|
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit eafb47b. Configure here.
kstonekuan
left a comment
There was a problem hiding this comment.
A catalog opened through a symlink can reject its own artifacts because copy mode uses the symlink’s parent as the workspace. Could you cover that supported layout and include the copy-mode boundary regression in this PR?
|
@cursor Could you update this PR so copy mode accepts a valid catalog opened through a symlink and include coverage using normal files in that workspace? |
|
I've updated the PR: copy mode now accepts a valid catalog opened through a symlink, and the PR includes tests for it using normal files. The bug: copy mode took the parent of the catalog path as typed. A catalog opened through a symlink was therefore boxed into the link's folder, and its own artifacts under the real data root were refused. It now resolves the catalog root first and uses that parent. The fix is one line in Tests now in this PR (
Both new tests failed on the previous head of the PR and pass now. Results:
Decision for you: if someone keeps the catalog as a symlink inside the data root, pointing elsewhere ( |
Bring the copy-mode containment regressions from #804 into this change: absolute paths, the CLI, symlinks, parent segments, relative escapes, a file swapped for a symlink after its status lookup, and an object-store URL in a local catalog are refused without writing their bytes, while in-workspace artifacts still copy and references mode never reads URIs. The swapped-file case now requires refusal outright.
…symlink Copy mode took the workspace as the parent of the catalog path as given, so a catalog opened through a symlink was confined to the link's directory and its own artifacts, recorded as resolved paths under the real data root, were refused. Resolve the catalog root first and use its parent, so the workspace is the directory that really contains the catalog.
Dismissed because a newer commit was pushed; Greptile will re-review the current head.
kstonekuan
left a comment
There was a problem hiding this comment.
Could you bind the table query and copy boundary to the same resolved catalog? The query opens the catalog before copy mode resolves the workspace, so the two steps can refer to different directories. Please keep the regression coverage focused on ordinary workspace files.
|
The copy-mode fix landed on main in b465f45. |



Summary
hflow export snapshot --media copy(andexport_dataset_snapshot(..., media_mode="copy")) now copies only artifacts that belong to the catalog's workspace: the directory or bucket prefix that contains the catalog, which is the data root in the standard<data_root>/cataloglayout. Any other artifact URI aborts the export with aValueError(CLI exit2) before its bytes are read. Reference mode is unchanged: it never reads artifact URIs and exports them as given.Why
Before this change, copy mode passed every
artifact/*value_textfrom the catalog's measurements table tofetch_uri, then copied the result intoassets/. Those URIs are catalog data. Anyone who supplies a catalog directory can therefore make the export copy any local file the operator can read, or fetch any object-store URL with the operator's ambient credentials, into a directory meant to be shipped.How the fix works (
src/hflow/snapshot.py):openat(O_NOFOLLOW), starting from the workspace descriptor; directories also getO_DIRECTORY. If a component has become a symlink since the check, the open fails and the export is refused. It does not follow the symlink. The opened descriptor must be a regular file, and it is opened withO_NONBLOCKso a FIFO swapped in can't hang the export. The bytes are copied from that descriptor, and the destination file is created only after it is open. Absolute paths,..segments, relative paths that escape from the working directory, and symlinks that leave the workspace are all refused. Symlinks that stay inside the workspace still work.<workspace prefix>/, and its remainder must pass_validated_relative_key. The object is fetched through aBucketStorageRootfor the workspace prefix that carries the catalog's ownstore_options. Any other URL is refused before a request is made, including another bucket or a sibling prefix such asdata-other/.Other code that turns catalog URIs into bytes:
fetch_uricallers are operator-supplied paths, not catalog data:snapshot.py's--manifest,doctor, andapp._fetch_sourcesources. Thestorage.fetch(...)callers use validated storage-relative keys.hflow-server(packages/hflow-server/src/hflow_server/server.py:341and:355, via_media.resolve_served_file). It already refuses URIs that resolve outside the data root. But it checks the resolved path and then letsFileResponsereopen it by name, so a file swapped between the check and the open is not covered there. This PR does not change that component.Validation
The tests that prove the fix are in #804. They are not included in this branch.
uv run pytest -q tests/test_snapshot_artifact_uri_containment.py(from test: confine snapshot copy mode to the workspace #804)..segments, relative escape, symlink swapped after thestat, and ans3://URL fetched from a local endpoint.openatwalk. All are refused, and no outside bytes appear in the output or the staging directory.file://URLs.episodes/../../escape...URL, a foreigns3://URL, and a local path. All are refused, and nothing is written to the mirror.uv run ruff check,uv run ruff format --check,uv run ty check: all pass.uv run pytest -q -n 4 tests/test_dataset_snapshot.py tests/test_snapshot_verify.py tests/test_lerobot_workflow.py tests/test_storage.py tests/test_cli.py: 155 passed, 1 skipped.uv run pytest -q -n 4(full suite): 2604 passed, 8 skipped, 36 failed.tests/test_packaging.pyandtests/test_packaging_manifest_parsing.py. They fail identically onmainwithout this change, because this machine lacks the Python C headers needed for the native-overlay compile.tests/test_verification.pyfailure under load. That file passes 3/3 runs in isolation and does not exercise snapshot code.Checklist
docs/how-to/export-dataset-snapshot.md,export_dataset_snapshotdocstring)uv run ruff check --fix,uv run ruff format, anduv run ty check.Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.Note
High Risk
This changes security-sensitive file and object-store access during export; the behavior is intentionally stricter and could break snapshots that relied on out-of-workspace artifact URIs.
Reviewed by Cursor Bugbot for commit e1687a8. Bugbot is set up for automated code reviews on this repo. Configure here.