| description | Run each Coder Eval task in its own fresh Docker container — strong host isolation, a pinned reproducible agent runtime, and custom images for task-specific dependencies. |
|---|
Run each evaluation task inside its own fresh container. Strong host isolation and a pinned, reproducible agent runtime.
Set sandbox.driver: docker on a task (or pass --driver docker on the CLI —
a thin alias for -D sandbox.driver=docker) when you want:
- Isolation from the host filesystem/network — agent-generated code can't reach files outside the sandbox.
- A pinned toolchain — the image bakes in Python 3.13, Node 22 LTS,
@anthropic-ai/claude-code, thepiCLI (@earendil-works/pi-coding-agent), the codex/antigravity/litellm agent SDKs,uv, and the matchingcoder_evalversion, so results don't drift with host upgrades.
Aggregation (P/R/F1, suite thresholds, reports) always stays on the host. Each container is a sealed "run one task → emit one task.json" worker.
make docker-image # core + both built-in agents (default; no credentials)
# opt in to the UiPath extra (resolves from public PyPI; no credentials):
make docker-image-fullBoth build coder-eval-agent:<pkg-version> and tag it :latest.
make docker-imageinstalls the core package plus both built-in agents — claude-code (baked above) and Codex (--extra codex, public PyPI). It needs no credentials and covers the common case: claude-code or Codex tasks scored withrun_command/file_contains(incl. converted skillsbench tasks).llm_judge/agent_judgework here too (they route through the run's Anthropic/Bedrock backend).make docker-image-fulladditionally installs theuipathextra. TheuipathSDK resolves from public PyPI (peruv.lock), so the build needs no credentials. Use this only for tasks that shell out to the in-hostuipathCLI. (Codex is already in the default image — no extra needed.)
Codex sandbox under Docker. Codex's Landlock-backed
read-only/workspace-writesandboxes can't initialize inside the eval container — their writes/execs fail silently and the agent produces no artifacts (ascore=0FAILURE with no loud error). The docker runner setsCODER_EVAL_IN_CONTAINER=1, and the Codex agent honors it by falling back tofull-access: the container itself is the trust boundary. Host runs (tempdir) are unaffected — Landlock works there and the marker is unset. So Codex tasks run under--driver dockerwith their naturalacceptEditspermission mode; no need to setbypassPermissionsby hand.
# Single task
coder-eval run path/to/task.yaml --driver docker
# All tasks
coder-eval run --driver docker
# Or in the task YAML
sandbox:
driver: docker
docker:
network: bridge # bridge | llm_only | none (see Network modes)
image: my-custom:tag # override the default imagesandbox.docker.network selects what the task container can reach. The same mode applies to
the grading container of a detached coder-eval evaluate.
| Mode | The container reaches | Use it for |
|---|---|---|
bridge (default) |
the full network | tasks that install packages or call other services |
llm_only |
only the model APIs and the hosts in egress_allowlist, through a proxy sidecar |
agent tasks that must not use general internet |
none |
nothing | agent: {type: none} tasks only: a real agent cannot reach its model API |
sandbox:
driver: docker
docker:
network: llm_only
egress_allowlist: [pypi.org, files.pythonhosted.org] # optional, appended across layersYou can also set it per run: -D sandbox.docker.network=llm_only and
-D 'sandbox.docker.egress_allowlist=["pypi.org"]'.
For each task the host creates:
- A per-task
docker network create --internal --ipv6=falsenetwork withcom.docker.network.bridge.inhibit_ipv4=true. The network has no route out and no IPv4 gateway address on the host, so the task container cannot reach the internet or a host service. - An egress-proxy sidecar from the framework image
coder-eval-agent:<version>. It joins the internal network (DNS aliascoder-eval-egress) and the defaultbridge. It runs the host's ownegress_proxy.py, bind-mounted read-only, as uid 65534 with all capabilities dropped and a read-only root file system. It opens TCP connections only to exacthost:porttargets. - The task container, on the internal network only, with
HTTPS_PROXY/HTTP_PROXY(both cases) set tohttp://coder-eval-egress:3128,NO_PROXY=localhost,127.0.0.1,::1,NODE_USE_ENV_PROXY=1andLITELLM_LOCAL_MODEL_COST_MAP=True. A host value of a proxy variable (*_PROXY,NO_PROXY,NODE_USE_ENV_PROXY) is never forwarded. Its upstream DNS server is192.0.2.1, an address that is never routed: Docker's own DNS still resolves the sidecar, but an external name does not resolve. It also runs withoutNET_RAW, so it cannot send crafted packets onto the internal bridge.
Before the task container starts, the host sends one CONNECT per allowlisted target through
the sidecar. If one fails, the task is an ERROR row that names the failing targets. The host
removes the sidecar and the network on every path (success, error, Ctrl-C). If the host process
is killed, the sidecar stops by itself when the host heartbeat goes stale; see
Troubleshooting egress to remove what is left.
A tool that ignores the proxy variables has no route, so it fails closed. It cannot bypass the allowlist.
You do not list the model APIs yourself. The host derives them from the parts of the task that call a model, and adds nothing for the parts that do not:
| Source | Hosts added |
|---|---|
API backend (API_BACKEND), only when a claude-code agent, an enabled simulation: with a simulated user (not scripted turns), or an llm_judge / agent_judge criterion uses it |
direct: api.anthropic.com:443, platform.claude.com:443 (Claude Code OAuth refresh). bedrock: bedrock-runtime.<AWS_REGION>.amazonaws.com:443, bedrock.<AWS_REGION>.amazonaws.com:443. litellm: the host of the forwarded LITELLM_BASE_URL (a localhost value becomes host.docker.internal) |
| Agent | codex: the host of the forwarded CODEX_BASE_URL, else api.openai.com:443. antigravity: generativelanguage.googleapis.com:443. pi: the host of the provider/ prefix of agent.model (openrouter, anthropic, openai, google; anything else or no model gives openrouter.ai:443). claude-code, opencode, delegate, none: nothing beyond the backend row |
system_one_judge criteria |
the host of each base_url (default api.typesafe.ai:443) |
egress_allowlist |
your entries |
A judge with its own checker_context.api_route.route uses the hosts of that backend. A
route: litellm judge configures its endpoint through params, so add that host to
egress_allowlist. Other forwarded *_URL variables, such as UIPATH_URL, add no host: a task
whose tools call that service lists the host in egress_allowlist. A Pi provider other than the
four above, and every OpenCode or Delegate provider, also needs its host in egress_allowlist.
egress_allowlist takes host or host:port entries; a bare host means port 443. A host is a DNS
name or an IPv4 address. Schemes, paths, wildcards and IPv6 addresses are refused when the task
loads. Entries are appended across the config layers, like env_passthrough_extra. Under bridge
or none the field is ignored.
Typical additions:
| Need | Entries |
|---|---|
sandbox.python.env_packages, pip, uv |
pypi.org, files.pythonhosted.org |
npm, npx -y MCP servers |
registry.npmjs.org |
git clone over https from GitHub |
github.com |
apt (Debian) |
deb.debian.org:80 |
apt (Ubuntu) |
archive.ubuntu.com:80, security.ubuntu.com:80 (amd64) or ports.ubuntu.com:80 (arm64) |
apk (Alpine) |
dl-cdn.alpinelinux.org |
| A remote MCP server or a run-time plugin install | its host |
uv downloading a managed Python not in the image |
github.com, objects.githubusercontent.com (or bake the Python into the image) |
A host in egress_allowlist is open to the agent too. When only the criteria need it, for example a
test script that installs its test dependencies, give it to grading alone:
sandbox:
driver: docker
docker:
network: llm_only
grading_egress_allowlist: [pypi.org, files.pythonhosted.org, astral.sh] # grading only
# or: grading_network: bridge # grading reaches any host| Field | Grading reaches |
|---|---|
| neither (default) | the agent's allowlist |
grading_egress_allowlist |
the agent's allowlist plus these hosts (same entry rules as egress_allowlist, appended across layers) |
grading_network: bridge |
any host, through the sidecar |
Both fields need network: llm_only. grading_network: bridge with network: bridge changes
nothing; any other combination is refused when the task loads, because the container joins one
network for its whole life and grading can only widen it. A task with an enabled simulation is
refused too, because its criteria run between dialog turns.
The container runs the agent under the agent's allowlist. Before the first criterion runs, the
run inside the container asks the host to switch, the host tells the sidecar through a control
port that only the sidecar itself can reach, and the criteria run after the sidecar confirms. The
switch is one-way and adds a PHASE grading line to egress.log. If it fails, the row is an
ERROR that names the failure. Before the agent starts, the host also opens one TCP connection
from the sidecar to each grading_egress_allowlist host, so a host this machine cannot reach
fails the row before any agent run is paid for.
Every graded llm_only run marks the phase, also with neither field set (then the switch adds
no host, and a failed switch is only a warning). After the run, the host records the hosts the
sidecar refused during grading in environment_info.egress_denied_in_grading, for example
"deb.debian.org:80, pypi.org:443", and logs a warning. The row's status does not change, because
a criterion can be refused on purpose (one that checks that a host is blocked). If a criterion
needed a host, add it to grading_egress_allowlist in the task file and grade the run again with
coder-eval evaluate <task.yaml> <task_run_dir>; the new grade replaces the recorded hosts. The
judge CLI hosts in Expected DENY lines
(http-intake.logs.us5.datadoghq.com:443, api.anthropic.com:443) are not recorded.
coder-eval execute keeps the agent's allowlist. A detached grade (coder-eval evaluate,
run --resume) runs no agent, so its container gets the grading network from the start; with
grading_network: bridge it joins the plain bridge network.
The task image needs nothing new: its tools must honour the proxy variables.
| Tool | Under llm_only |
|---|---|
| curl, GNU wget, git over https, pip, uv, npm/npx, apt, apk | honour the proxy; a denied host fails fast (CONNECT tunnel failed, response 403, npm error 403; pip and uv retry for about 8–12 s first) |
| Python urllib / requests / httpx | honour the proxy (trust_env is on by default) |
| Python aiohttp | only with trust_env=True (litellm sets it); otherwise no route |
Node fetch / https |
only on Node ≥ 22.21 or ≥ 24.5, through NODE_USE_ENV_PROXY=1; older Node has no route. Node prints an UNDICI-EHPA warning on stderr |
Go net/http default client |
honours the proxy; a custom transport without Proxy has no route |
busybox wget (Alpine) |
plain http works; https always fails 400, because it sends GET https://… instead of CONNECT. Use curl |
| dnf (Rocky, Fedora) | its metalink picks random mirrors, so an exact-host list cannot work. Pin a baseurl or bake the packages into the image |
| git over ssh, raw sockets, DNS lookups, Java without proxy flags | no route |
Every built-in harness in the framework image honours the proxy (Claude Code, Codex, Antigravity, Pi). See Run-Limit Parity § Network modes.
The sidecar always runs the Debian framework image, so the task image's distribution does not
change the boundary. Debian, Ubuntu, Rocky Linux, Fedora and Alpine task images were tested as
clients. A runtime-kit image needs the framework image too (make docker-images builds both).
Docker versions. Docker 20.10 or later (host-gateway first shipped there). Docker 26.0,
25.0.5 or 23.0.11 or later is recommended: older daemons forward the DNS queries of an internal
network from the host (CVE-2024-29018). The black-hole upstream DNS server above stops that
forward, but a patched daemon removes the cause. Tested on Docker
Desktop 29 (macOS) and on Linux dockerd 20.10, 24 and 29, with both the iptables and the
nftables firewall backends. On Linux, host.docker.internal resolves to the docker0 address,
so a LiteLLM proxy on the host must listen on docker0 or 0.0.0.0, as under bridge.
Some clients call hosts they do not need. These DENY lines are harmless:
| Client | Denied host |
|---|---|
Claude Code with API_BACKEND=direct |
http-intake.logs.us5.datadoghq.com:443 (telemetry) |
| Codex | chatgpt.com:443, github.com:443, api.github.com:443 (update check, remote config). Codex without CODEX_API_KEY falls back to a ChatGPT login whose model host is chatgpt.com, so that setup needs it in egress_allowlist |
litellm without LITELLM_LOCAL_MODEL_COST_MAP |
raw.githubusercontent.com:443 (cost map) |
| OpenCode | models.opencode.ai:443 (model catalog refresh), registry.npmjs.org:443 (update check) |
Claude Code with API_BACKEND=litellm |
api.anthropic.com:443 (startup calls the CLI does not need on this route) |
Claude Code WebFetch on Bedrock |
api.anthropic.com:443 (the domain safety check before a fetch; WebFetch then fails) |
- Parallel tasks. Each task uses one Docker network. A default Docker Desktop has address
pools for about 29 user networks, so keep
--max-parallelunder that. When the pools are exhausted, the row error names--max-parallel, the prune command and the daemon'sdefault-address-poolssetting. - A killed run leaks until the next one. A SIGTERM or SIGKILL of coder-eval leaves the
task's sidecar (stopped) and network behind. Each
llm_onlysetup first removes egress sidecars that stopped and egress networks that no container uses, once they are 10 minutes old. - No upstream proxy. The sidecar connects directly. A host that can reach the internet only
through a corporate proxy cannot use
llm_only. - Not exfiltration-proof. The agent can still send data to an allowlisted host.
- Server-side tools are outside the boundary. A tool that the model provider runs, such as
Claude Code's
WebSearchon the direct API, fetches from the internet on the provider's side. The container network cannot block it; remove it withdisallowed_toolsif the task needs no internet at all. - An exact host is a TCP destination, not a site. A host behind a shared CDN front (for
example
files.pythonhosted.orgordeb.debian.org) can serve other sites that the agent names in itsHostheader or TLS SNI. extra_mountscan defeat the boundary. Mounting the Docker socket, for example, gives the agent the daemon.- Exact hosts only. No wildcards and no CIDR ranges.
- podman and rootless Docker are not tested.
- Harbor export refuses an
llm_onlytask. - The grading switch trusts the container. The request to switch comes from inside the
container, so an agent that knows the request file can ask for it before its turn ends. A
process the agent leaves running in the background also keeps running during grading, with the
grading network. Use the grading fields for test dependencies, not to hide hosts from an agent
that is trying to defeat the boundary. A host such a process is refused during grading is also
recorded in
egress_denied_in_grading. - The sidecar is also on the default
bridge. Abridgecontainer of another task can reach it. It gets no extra reach, because the allowlist is a subset of whatbridgealready reaches, and the sidecar serves at most 256 connections at once (one more gets503). - Plugin agents. A third-party agent gets the API backend's hosts only when its config class
sets
uses_api_backend = True, and its own hosts only throughegress_hosts(); see Extending Coder Eval.
tasks/docker_egress_probe/ tests the mode end to end. The agent tries to reach the internet in
its own ways, and the grading criteria then probe each path out from inside the container: the
proxy, direct IPv4 and IPv6, DNS, raw sockets, the docker host, pip, git and npm. Each
probe passes only when its path is blocked, and the model host must stay reachable:
coder-eval run tasks/docker_egress_probe/docker_egress_probe.yaml # expect 1.000
coder-eval run tasks/docker_egress_probe/docker_egress_probe.yaml \
-D sandbox.docker.network=bridge # control: expect a FAILUREThe task pins Claude Code. To check another harness, replace its agent: block (without the
Claude-only setting_sources). The model-host check accepts the model host of any built-in
harness. Run the probes alone in any llm_only container with python3 egress_probe.py all.
docker_egress_grading.yaml in the same directory checks the grading network without an agent or
a model API: pre_run records that pypi.org is blocked, and the criteria then reach it during
grading while example.com stays blocked.
coder-eval run tasks/docker_egress_probe/docker_egress_grading.yaml # expect 1.000The sidecar log is the task's egress.log, beside docker.log (a grading container's log is
grade.egress.log). It is a separate file so that container output cannot add lines to it.
Each connection is one line:
ALLOW host:port METHOD, DENY host:port METHOD, FAIL host:port <error> (an allowed host the
sidecar could not reach), or BAD … (a request the proxy refused; the line names only its method
and length, because the target can carry credentials). A PHASE grading allow=… line marks the
switch to the grading allowlist: the lines before it are the agent's, the lines after it are
grading's (and any process the agent left running). To see which hosts a task needed:
grep -rhE "^(ALLOW|DENY|FAIL)" --include=egress.log runs/latest | sort | uniq -cAdd each needed DENY host to egress_allowlist. The next llm_only task removes what a killed
run leaked once it is 10 minutes old. To remove it now:
docker rm -f $(docker ps -aq --filter label=org.coder-eval.egress)
docker network prune -f --filter label=org.coder-eval.egressWhen your tasks need extra tools or dependencies, extend the framework image once and point tasks at
the result — no per-task build. The custom image must extend coder-eval-agent:<version> so it
inherits the coder-eval runtime, the entrypoint script, and the org.coder-eval.version label the
host's preflight check reads. (For a task whose Dockerfile can't be rebased onto Debian, use
the runtime kit
instead. To have coder-eval build the image per task rather than pre-building it, see
Building the image from a task Dockerfile.)
FROM coder-eval-agent:<version> # match the version your host runs
RUN apt-get update && apt-get install -y --no-install-recommends custom-tool \
&& rm -rf /var/lib/apt/lists/*docker build -t my-team/image:latest .Select it either in the task YAML:
sandbox:
driver: docker
docker:
image: my-team/image:latest…or from the CLI, which overrides whatever the YAML says:
coder-eval run task.yaml -D sandbox.docker.image=my-team/image:latestThe default when you set nothing is coder-eval-agent:<installed package version>.
A worked example ships in-tree: tasks/byod_smoke_test.yaml runs against
templates/byod_smoke_test/Dockerfile, which extends the framework image and drops a marker file
that the task's success criterion then asserts — proving the custom image was actually used. (The
byod_* names here mean "Bring Your Own Docker"; they are unrelated to the
Bring Your Own Dataset guide, which is about fanning one task out over data rows.)
make docker-image # base image first
docker build -t byod-custom-image:0.1.0 templates/byod_smoke_test/ # then the derived one
coder-eval run tasks/byod_smoke_test.yaml| Symptom | Cause and fix |
|---|---|
docker: Error response from daemon: pull access denied |
The image isn't built locally and isn't pullable. Check docker images, then rebuild it. Docker treats an unknown local tag as a remote reference, which is why the error mentions a pull. |
Image <your-image> runs coder_eval <a> but the host runs <b> |
The run is refused before the container starts: the image carries an org.coder-eval.version label from a different coder-eval than the one installed on the host, usually inherited from a stale framework base. Rebuild the base with make docker-image, then rebuild your derived image with docker build --no-cache. To run a deliberately different image anyway (for example an unreleased build under test), set ALLOW_IMAGE_SKEW=1 in the environment or .env; the mismatch then only warns, and the run gives up the reproducibility guarantee. |
Image <your-image> is not a coder-eval runtime image (missing the org.coder-eval.version label) |
The image doesn't descend from coder-eval-agent (or predates the label). Rebase it on the framework image, or use the runtime kit. ALLOW_IMAGE_SKEW does not bypass this. |
The container returned a result with no container_contract echo or The container did not honor the contract it was sent |
The code inside the image does not match the host's, even if its label agrees — for example an overlay image that reinstalled coder-eval, or a stale image rebuilt under the same tag. The result is refused: the record is moved to task.json.unhonored and a synthetic ERROR task.json takes its place. Rebuild or pull a matching image. ALLOW_IMAGE_SKEW does not bypass this. |
Instead of pointing at a pre-built image, a task can ship its own Dockerfile
and have coder-eval build it before the run:
sandbox:
driver: docker
docker:
dockerfile_path: ./environment/Dockerfile # relative to the task YAML
⚠️ Contract: a task Dockerfile MUST either start withFROM coder-eval-agent:<version>or use the runtime kit (see below). The container runs the coder-eval orchestrator (coder-eval _run-task-internal) via the framework image'sENTRYPOINT. A task Dockerfile extends that image and adds only task-specific layers — extraaptpackages,COPY-ed inputs, etc.Build the framework base first —
make docker-image(tags bothcoder-eval-agent:<version>andcoder-eval-agent:latest).
The FROM coder-eval-agent contract above means a task is rebased onto the
Debian framework image. That breaks tasks whose Dockerfile was written for a
different base image (e.g. a Fedora recipe using dnf, which doesn't exist on Debian). To keep the task's own base image and build successfully, coder-eval's runtime need to be copied into the task's image. Use make coder-eval-runtime first to make the runtime available for copying.
FROM fedora:41 # the task's own base, kept verbatim
RUN dnf -y install ... # the task's native recipe, runs on its own OS
# --- copy coder-eval runtime into a task image ---
COPY --from=coder-eval-runtime:latest /opt/coder-eval /opt/coder-eval
COPY --from=coder-eval-runtime:latest /usr/local/bin/coder_eval_entrypoint.sh /usr/local/bin/coder_eval_entrypoint.sh
LABEL org.coder-eval.version="<ver>"make coder-eval-runtime builds the kit (docker/Dockerfile.runtime): a
standalone CPython + Node + the coder-eval CLI + Claude Code, all under
/opt/coder-eval, plus the entrypoint at the same /usr/local/bin/... path
the host pins. The kit is glibc-only — it runs on debian/ubuntu/fedora/rhel/…
but not musl/Alpine.
Both base images are independent and persistent — build each once and run any mix of rebase
and inject tasks without rebuilding. To build both in one shot (no credentials needed), use
make docker-images (= make docker-image + make coder-eval-runtime); reach for the
individual targets when you only need one.
The kit installs the no-credential set only (core + codex), like
make docker-image— it never installs the[uipath]extra, so there is nomake docker-images-full. An inject-mode task that needs the LLMGW/uipathjudge isn't supported by the kit as built.
# environment/Dockerfile
FROM coder-eval-agent:latest # inherit runtime + entrypoint
RUN apt-get update && apt-get install -y --no-install-recommends poppler-utils
RUN pip install --no-cache-dir PyMuPDF==1.24.10
COPY input/ /root/input/Behavior:
- Path resolution —
dockerfile_pathis resolved relative to the task YAML's directory at load time (with$VAR/${VAR}expansion). A missing file fails fast at load, not mid-run. - Entrypoint check — after building, coder-eval inspects the image's
ENTRYPOINTand aborts with aFROM coder-eval-agenthint if the runtime wasn't inherited. - Overrides
image— whendockerfile_pathis set, it takes precedence over anyimagevalue. - Build context — the build context is the Dockerfile's parent directory,
so relative
COPY ./input/...instructions resolve naturally. In the layout above,environment/is the context. - Caching — the image is tagged deterministically as
coder-eval-task-<task_id>:built, so repeat runs of the same task reuse Docker's layer cache. Edit the Dockerfile and the next run rebuilds the changed layers only. - Version-checked too — the built image inherits
org.coder-eval.versionfrom itsFROM coder-eval-agentbase, so the same preflight applies: a missing label, or a version that differs from the host's, refuses the run (see Troubleshooting custom images).
A build failure aborts the task with a DockerBuildError (a DockerRunError
subclass) carrying docker build's output. Because the build runs before the
run dir, docker.log, or task.json exist, the runner explicitly records the
failure so it is never a silent empty result dir: it creates the run dir, writes
the full build log to docker.log, and writes a synthetic task.json with
final_status: BUILD_FAILED (an error-category status) before re-raising. So
a failed build shows up per-task on the dashboard with its build log, exactly
where you'd look for container output.
The docker build invocation is configurable via sandbox.docker.build:
sandbox:
driver: docker
docker:
dockerfile_path: ./environment/Dockerfile
build:
args: # -> --build-arg KEY=VALUE
PKG_VERSION: "1.2.3"
TOKEN: "${HOST_TOKEN}" # values are $VAR / ${VAR} expanded from the host env
secrets: # -> --secret <spec> (requires BuildKit)
- id=mytoken,env=MY_TOKEN # forward a host env var as a build secret
- id=npmrc,src=~/.npmrc # or a file
extra_args: ["--target", "runtime"] # escape hatch for any other docker build flag
buildkit: true # optional: force DOCKER_BUILDKIT (see below)args→--build-arg KEY=VALUE. Values are environment-expanded against the host. Prefersecretsfor credentials — build-args are recorded in the image history.secrets→--secret <spec>. Useid=NAME,env=VARto forward a host env var orid=NAME,src=PATHfor a file; reference it in the Dockerfile viaRUN --mount=type=secret,id=NAME .... Secrets are exposed only to the mounting RUN step and never baked into layers. Secrets require BuildKit.extra_args→ raw flags inserted before the build context (e.g.--target,--network,--platform). Escape hatch for options without a dedicated field.buildkit→ controls theDOCKER_BUILDKITenv var. Omitted (default), coder-eval inherits the invoker's environment — setDOCKER_BUILDKIT=1before running coder-eval to enable it globally. Setbuildkit: true/falseto force it per task. Ifsecretsare configured but BuildKit isn't enabled, coder-eval logs a warning (the build would otherwise fail).
The build context is always appended last, so extra_args can't displace it.
macOS users — read this first. Claude Code's OAuth tokens live in the macOS Keychain. The container has no path to the Keychain, so the bundled CLI inside will return
Not logged in · Please run /loginand every task will fail at iteration 1. Before running--driver docker, set one of these on the host:
ANTHROPIC_API_KEY=...(direct Anthropic), orCLAUDE_CODE_USE_BEDROCK=1+AWS_BEARER_TOKEN_BEDROCK=...+AWS_REGION=...(Bedrock).Linux hosts where Claude Code stores creds under
~/.claudealready work because that directory is bind-mounted into the container.
Credentials are forwarded via --env VAR (name-only, never embedded in argv) for these vars when set on the host: ANTHROPIC_API_KEY, API_BACKEND, UIPATH_*, AWS_BEARER_TOKEN_BEDROCK, AWS_REGION, CLAUDE_CODE_USE_BEDROCK, ANTHROPIC_MODEL.
To add one or two custom vars to the defaults (recommended), use env_passthrough_extra:
sandbox:
driver: docker
docker:
env_passthrough_extra: ["MY_CUSTOM_TOKEN", "DEBUG_FLAG"] # Keeps all defaults + theseTo completely replace the list, use env_passthrough:
sandbox:
driver: docker
docker:
env_passthrough: ["MY_CUSTOM_TOKEN", "ANTHROPIC_API_KEY"]The default env_passthrough includes HOME so the in-container ~/.claude lookup resolves at the same path as on the host (the mount lands at $HOME/.claude symmetrically). Practical contract:
Path.home()inside the container returns the host'sHOMEvalue (e.g./Users/youon macOS). The directory exists in the container because Docker auto-creates it as the mount parent for~/.claude.~/.claudeis not the host's real dir — the runner makes a throwaway lean copy in a tmp dir per task and mounts that copy read-write at$HOME/.claude. The copy keeps the small set the container needs (auth via.credentials.json,settings.json,plugins/) and drops heavy or transient per-session state —security/(often hundreds of MB),projects/,cache/,file-history/,backups/,downloads/,sessions/,telemetry/,shell-snapshots/,todos/,session-env/, plus the volatile churn dirs the live CLI rewrites. The skip set is a denylist; the authoritative list isCLAUDE_COPY_IGNOREinsrc/coder_eval/isolation/docker_runner.py(a test asserts this doc and that constant agree, so the list never silently drifts). The container may write anywhere under~/.claude; those writes hit the copy and are discarded when the task ends — the host's real~/.claudeis never modified. Note the copy includes the OAuth token (.credentials.json) and is mounted read-write, so the in-container agent can read and tamper with the token copy — contained, since the copy is discarded at task end and the host's real dir is untouched. Opt out entirely withCODER_EVAL_NO_CLAUDE_MOUNT=1.- Writes under
$HOMEoutside the~/.claudemount land in the container's ephemeral rootfs overlay. Don't expect them to persist or to be visible to the host. - If a tool detects platform from
HOME(e.g. "starts with/Users/→ macOS"), it will draw the wrong conclusion. Vanishingly rare in practice.
Remove HOME from env_passthrough if you don't want this behavior — the container's image-default HOME=/root will win, but then the host's OAuth dir is no longer reachable.
The host's run dir is bind-mounted read-write into the container at the same absolute path (so task.json and artifacts land directly on the host filesystem). This makes --run-dir load-bearing for isolation:
- Do not point
--run-dirat a symlink. Docker resolves the source of a bind mount; following a symlink would silently grant the container RW access to a different host location. - Do not point
--run-dirat a sensitive parent (e.g.$HOMEdirectly,/etc, a repo root). Use a dedicatedruns/subtree. - The default (
runs/<timestamp>/) is safe.
| Layer | Location |
|---|---|
| Agent process (Claude Code SDK) | inside container |
| Sandbox + per-row criterion checking | inside container |
task.json serialization |
container → host bind mount |
Per-criterion aggregate() (P/R/F1, suite thresholds) |
host |
| Reports, run summary, experiment rollups | host |
task.json is the only artifact crossing the boundary. Aggregation reads it via the existing host pipeline unchanged.
- Relative template paths:
template_sources[].pathis resolved to a host absolute path before staging, so it won't exist inside the container unless you also forward the parent dir viasandbox.docker.extra_mounts. - No container reuse across tasks: each task = one fresh container. Adds ~1–3 s startup overhead per task; negligible vs. LLM latency.
- macOS Keychain auth: not reachable from the container; set
ANTHROPIC_API_KEY(direct) or Bedrock credentials instead.
The host's DockerRunner (coder_eval/isolation/docker_runner.py) renders the docker run argv, bind-mounts task inputs at /work/input, allocates an output dir at /work/output, and tails container stdout into docker.log in the task's run dir.
If the task declares a reference: block, a throwaway copy of its directory is bind-mounted read-write at /work/references, and an empty tmpfs is layered over its original location inside the task-directory mount so the agent cannot reach the solution through $TASK_DIR. The mount is read-write on purpose — a :ro bind mount cannot be chmod'd (EROFS), and the orchestrator holds this path at mode 000 for the duration of each agent turn. The container also runs with --cap-drop DAC_OVERRIDE --cap-drop DAC_READ_SEARCH (see the caveat below for why FOWNER/CHOWN are deliberately kept).
What this does and does not guarantee. The mode-000 window stops an agent that passively reads the reference —
ls,cat,grepall fail. Two things it does not stop:
- A deliberate re-chmod.
chmod(2)is gated on owner-or-CAP_FOWNER, and the container runs as root, which owns the bind-mounted copy. Verified:chmod 755 /work/referencessucceeds from inside the container and the solution is then readable. DroppingFOWNER/CHOWNdoes not fix this and is deliberately not done — the in-container orchestrator that applies the window is the same root process with the same capabilities, so droppingFOWNERbreaks the harness's ownchmodwherever the bind mount preserves a non-root owner (native Linux). Verified: container root, uid-1000-owned directory,FOWNERdropped →chmod: Operation not permitted. The drop only bites on the hosts where it also disables the control.- Waiting the window out. The window spans
agent.communicate, so between turns and after the final one the reference is back at its normal mode, and nothing reaps the agent's child processes at turn end — a backgrounded read loop started during a turn succeeds once the window closes.The write half of (2) IS closed: the reference tree is hashed at staging time and re-verified before grading, and a mismatch fails the run with
ReferenceTamperedErrorrather than scoringreference_comparisonagainst a file the agent wrote. Closing the read half fully requires running the agent as a non-root uid that does not own the reference, and holding the window for the agent's whole lifetime — tracked as follow-up. Treat the current control as raising the cost of accidental leakage, not as a boundary against an adversarial agent.
See Reference Solutions.
These are driver: docker only. driver: tempdir shares the host uid and has no
filesystem isolation, so neither applies there (nor can — there is nothing to
mask). Both are defense-in-depth passive-read blocks, consistent with the
reference window's posture above; neither contains an adversarial agent.
-
The staged grading inputs are deleted after load. The host stages the post-override
TaskDefinition(withsuccess_criteria) at/work/input/task.yamland acontext.jsonwhosesource_yamlis the raw task text — criteria verbatim, both at the top level and inside everyconfig_lineageentry — for the in-container orchestrator to load once at startup. The agent runs in the same container, so leaving either readable would hand it the grading answer key (deleting onlytask.yamlleaves the identical criteria one file over incontext.json). The in-container entry point deletes both immediately after they are consumed —context.jsonis parsed into memory in the command body andtask.yamlbyload_task, both before the delete (gated onCODER_EVAL_IN_CONTAINER). They are read exactly once — grading reads criteria from the in-memory task, never from disk. The/work/inputmount is therefore read-write (a:romount rejectsrmwith EROFS).prior.jsonis kept: it is read later on the regrade path, and a regrade runs no agent so it is not a leak. -
Plugins mount at a fixed container path. The
i-thagent.plugins[]entry is resolved on the host (a relative path against the task YAML's directory,$VARand~expanded) and mounted:roat/work/plugins/<i>. The task YAML staged into the container is rewritten to point at that path, so the in-container agent loads the directory the host mounted, whatever form the authored path took and whatever the container's cwd. (Before, the staged YAML kept the authored string: a relative or$VARpath the container could not resolve loaded no skill, with only a warning.) An entry that does not resolve to a host directory is neither mounted nor rewritten. It sits under/work, so asandbox.docker.extra_mountsdestination cannot shadow it (those are refused anywhere under/work/). -
Auto-mounted plugin trees are default-deny masked. An
agent.plugins[].path(at/work/plugins/<i>) or aTemplateDirSource.paththat is itself a plugin root (at its host path) is auto-mounted:roso the plugin loads. Eval material colocated under that tree as siblings of the skills dir — sibling task YAMLs, reference solutions, test fixtures — would otherwise be readable. So the runner keeps the whole root mounted but layers an empty--tmpfsover every child dir OUTSIDE the keep-set (.claude-plugin+ the manifest-declared skill dirs). Everything that is not the plugin surface is masked by default, so an unknown or new eval layout can never leak;tests/,node_modules/, and reference solutions are masked for free. A root agent cannotumounta tmpfs (CAP_SYS_ADMINis not in Docker's default set), so this mask is stronger than the mode-000 reference window. Two residuals the mask cannot cover — an eval def or reference COLOCATED inside a skill dir (masking it would hide the skill), and atask_id:YAML file loose at the plugin root (a tmpfs masks a directory, not a single file) — are caught by lint rule CE068 (keep eval material out of skill dirs and off the plugin root; put it under a siblingtests/).
Inside the container, the entrypoint invokes coder-eval _run-task-internal (hidden subcommand), which loads the staged YAML + context, runs the standard in-process Orchestrator (driver auto-coerced back to tempdir), and writes task.json to the output mount. Host reads it and feeds the existing aggregation pipeline.
A result_kind discriminator on CriterionResult ensures ClassificationCriterionResult subclasses survive the JSON round-trip — without it, host-side aggregation would silently lose observed_label/expected_label.