Skip to content

fix(verifier): keep image PATH tool dirs on images without python3 and on rust images - #1195

Open
shubham2345 wants to merge 1 commit into
benchflow-ai:mainfrom
shubham2345:fix/1174-verifier-path-no-python3
Open

shubham2345 wants to merge 1 commit into
benchflow-ai:mainfrom
shubham2345:fix/1174-verifier-path-no-python3

Conversation

@shubham2345

Copy link
Copy Markdown

Fixes #1174.

Problem

The verifier rebuilds its PATH from the image PATH, keeping only root-owned, non-group/world-writable directories. Two things made Rust and Java tasks score 0 with cargo: command not found / mvnd: command not found:

  1. The check ran as python3 -c .... On images without python3 it failed, logged at debug level, and every image tool directory was dropped.
  2. Official rust:* images ship /usr/local/cargo/bin, its parent and the binaries inside at mode 777, so the rule rejected cargo even when Python was present.

Change

  • When the python3 check fails, validate with sh + stat and apply the same rule.
  • Before validation, as root, remove group/world write from root-owned image PATH dirs (and their ancestors and the root-owned files inside), skipping the workspace and the agent's home. A dir is kept only if nothing inside is owned by another user. Hardening the ancestors and files matters: otherwise the agent could rename the dir away or overwrite the binaries in place.
  • Directories that stay untrusted are logged at warning level.

Verification

  • tests/test_sandbox_hardening.py: 107 pass (7 new); ruff clean.
  • Docker, oracle agent, crustbench-leftpad (rust:1.83-slim): 0 -> 1.0.
  • Docker, oracle agent, google__gson-2311 (mvnd 755, no python3): 0 on main (mvnd: command not found) -> 1.0. Run on a scratch copy of the task with the #!/bin/bash line moved to line 1, see below.
  • Container checks of _trusted_verifier_path: stock rust 777 trusted and hardened to 755; 777 dir with a world-writable root file trusted; 777 dir with an agent-owned file refused; 755 dir untouched; dir under the agent home refused; non-root-owned dir refused.

Not covered

  • Not run on Daytona.
  • A symlink inside a tool dir is checked by its target's owner; the target's parent dirs are not walked.
  • Unrelated, in the Harbor dataset: the gson tests/test.sh has a comment above #!/bin/bash, so it runs under sh and fails on set -o pipefail before reaching mvnd.

🤖 Generated with Claude Code

…d on rust images (benchflow-ai#1174)

Validate trusted PATH entries with sh+stat when the python3 probe fails,
instead of silently dropping every image tool directory (cargo, mvnd).

Official rust:* images ship /usr/local/cargo/bin, its parent and the
binaries inside at mode 777, which the root-owned/not-writable rule
rejected. Remove group/world write from such root-owned directories, their
ancestors and the files inside before verification, and keep them when
nothing inside is owned by another user. Log directories that stay
untrusted at warning level.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Images without python3 lose every tool directory on their PATH at verification

1 participant