fix(native-eval): benchmark genuine OpenClaw code mode

[vincentkoc]

What does this PR do?

Makes the OpenClaw native-eval arms represent genuine direct, directory, and
Code Mode execution, then exports and validates their public trajectory bundles
before a run can be scored.

Why?

Fixes #61.

The earlier code arm selected the legacy tool_search_code bridge rather than
OpenClaw Code Mode. The harness also masked setup failures, forced thinking off,
terminated the runtime before lifecycle cleanup settled, and reconstructed
delegated traces from private session files. Those failures made the released
direct/code comparison invalid and disproportionately erased Code Mode's nested
tool calls.

Changes

• Map direct, directory, and code to explicit, mutually exclusive
  OpenClaw tool surfaces.
• Propagate the requested reasoning effort through planning, dispatch,
  OpenClaw defaults, subagents, CLI execution, and manifests.
• Fail setup immediately and let openclaw agent exit naturally.
• Export root and child trajectories through
  openclaw sessions export-trajectory.
• Capture child runs at the canonical pre-cleanup terminal hook, including
  nested and repeated runs, without blocking the Gateway.
• Validate audit initialization, trace/session identity, event counts, terminal
  status, complete Code Mode snapshots, and the exact provider-visible
  exec/wait surface.
• Reconstruct Code Mode nested calls and complete session-tree usage from the
  public export bundles.
• Bound child-export stabilization and reject missing or failed evidence rather
  than publishing partial traces.

Tests

• Blacksmith Testbox full suite: 454 passed, 5 skipped
• Blacksmith Testbox focused runner suite after final fixes: 61 passed
• Ruff clean
• Python compile, git diff --check, generated shell syntax for all three
  modes, and generated audit-plugin node --check
• Fresh Codex autoreview: clean
• Fresh matched direct/Code Mode ShellBench batch

The July 29, 2026 released code result remains legacy Tool Search bridge
evidence only. It is not evidence about genuine OpenClaw Code Mode.

[clawsweeper]

Codex review: needs changes before merge. Reviewed August 9, 2026, 4:44 AM ET / 08:44 UTC.

ClawSweeper review

What this changes

This PR updates ShellBench’s OpenClaw native-evaluation harness to configure direct, directory, and Code Mode runs explicitly and validate exported trajectories before scoring.

Regression provenance

Possible regression — probable (reviewed change; failure trace). No predecessor PR is attributed.

Merge readiness

⛔ Blocked by patch quality or review findings - 5 items remain

Keep this member-authored PR open. Its central benchmark-mode change is still necessary, but the planner creates only one OpenClaw run per model/repetition and never assigns a tool mode, so every planned OpenClaw run defaults to direct mode.

Priority: P2
Reviewed head: f5b31cb1260b631b811bf0ede9f4eba51a94b98c

Review scores

Measure	Result	What it means
Overall readiness	🦪 silver shellfish (2/6)	The patch has meaningful validation work, but the unchanged P1 planner defect prevents the advertised benchmark modes from being exercised.
Proof confidence	🌊 off-meta tidepool	Not applicable: The author association is MEMBER, so the external-contributor proof gate does not apply; a redacted matched batch remains useful validation after the planner repair.
Patch quality	🦪 silver shellfish (2/6)	1 actionable review finding remain.

Verification

Check	Result	Evidence
Real behavior	Not applicable	Not applicable: The author association is MEMBER, so the external-contributor proof gate does not apply; a redacted matched batch remains useful validation after the planner repair.
Evidence reviewed	3 items	Current main does not already implement tool-mode planning: Current main has no openclaw_tool_mode symbols, and its matrix builder creates one RunSpec per harness/model/repetition without a mode field. PR planner omits the new mode: The PR adds the mode field but its planner still appends one run without openclaw_tool_mode; downstream harness code resolves a missing value to direct. Native-evaluation ownership history: The current matrix-runner and subsequent research-runbook work both date to Vincent Koc, establishing current-main ownership of this planning surface.
Findings	1 actionable finding	[P1] Expand the matrix across OpenClaw tool modes
Security	None	None.

How this fits together

ShellBench builds a benchmark matrix, dispatches each planned run to an OpenClaw harness, then exports and validates trajectories before scoring. Tool mode must be encoded in each planned run for distinct benchmark arms to reach dispatch.

flowchart LR
A[Benchmark matrix] --> B[Planned runs]
B --> C[Tool-mode selection]
C --> D[OpenClaw dispatch]
D --> E[Trajectory export]
E --> F[Validation and scoring]

Loading

Before merge

• Expand the matrix across OpenClaw tool modes (P1) - build_matrix_plan still emits exactly one OpenClaw run per model/repetition and never sets openclaw_tool_mode. _openclaw treats that missing value as direct, so directory and Code Mode are never scheduled and the 96-entry assertion preserves the invalid matrix. This remains the prior blocker at the unchanged head.
• Resolve merge risk (P1) - Merging as written would publish a purported direct/directory/Code Mode comparison in which all planned OpenClaw arms resolve to direct mode.
• Complete next step (P2) - The remaining code defect is a narrow, mechanical planner repair; the PR should remain open for the member author or a repair worker to address it.
• Improve patch quality - Expand OpenClaw plan generation and update the 96-run expectation to cover all three tool modes.
• Improve patch quality - Attach a redacted matched benchmark transcript or manifest set showing direct, directory, and Code Mode arms after the repair.

Findings

• [P1] Expand the matrix across OpenClaw tool modes — scripts/native_eval/models.py:169-181

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Patch scope	9 files, +1,815 / -952 lines	The patch changes native-runner behavior and its regression coverage substantially.
Production versus test delta	production +699 / -477, tests +1,116 / -475	The test growth is substantial, but the central planner assertion still preserves the old 96-run matrix.

Merge-risk options

Maintainer options:

1. Expand planned OpenClaw arms (recommended)
   Create a run for each supported tool mode, include that mode in the run identity, and update the plan-count and dispatch assertions before merge.

Copy recommended automerge instruction

@clawsweeper automerge

Special instructions:
Expand OpenClaw matrix planning across direct, directory, and code modes; encode the mode in each run label and update expected counts plus focused planner and dispatch coverage.

Technical review

Best possible solution:

Land a matrix that has distinct, uniquely labeled direct, directory, and Code Mode OpenClaw runs, with exported manifests demonstrating each selected arm before using results for comparison.

Do we have a high-confidence way to reproduce the issue?

Yes. The planner can be inspected directly: it omits openclaw_tool_mode, while the harness maps a missing mode to direct.

Is this the best way to solve the issue?

No. Downstream propagation is not sufficient until the matrix emits one uniquely identified OpenClaw run for each requested tool mode.

Full review comments:

• [P1] Expand the matrix across OpenClaw tool modes — scripts/native_eval/models.py:169-181
build_matrix_plan still emits exactly one OpenClaw run per model/repetition and never sets openclaw_tool_mode. _openclaw treats that missing value as direct, so directory and Code Mode are never scheduled and the 96-entry assertion preserves the invalid matrix. This remains the prior blocker at the unchanged head.
  Confidence: 0.99

Overall correctness: patch is incorrect
Overall confidence: 0.99

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label changes:

• add rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🌊 off-meta tidepool and patch quality is 🦪 silver shellfish.
• remove rating: 🧂 unranked krab: Current PR rating is rating: 🦪 silver shellfish, so this older rating label is no longer current.

Label justifications:

• P2: The defect invalidates internal benchmark comparisons but does not affect the OpenClaw product runtime.
• merge-risk: 🚨 other: A successful merge could produce misleading benchmark results because the advertised arms are not all scheduled.
• rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🌊 off-meta tidepool and patch quality is 🦪 silver shellfish.
• status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Not applicable: The author association is MEMBER, so the external-contributor proof gate does not apply; a redacted matched batch remains useful validation after the planner repair.

Evidence

Acceptance criteria:

• [P1] pytest tests/test_native_eval_runner.py -k matrix_plan.
• [P1] pytest tests/test_native_eval_fleet.py -k openclaw_tool_mode.
• [P1] pytest tests/test_native_eval_runner.py tests/test_native_eval_fleet.py.

What I checked:

• Current main does not already implement tool-mode planning: Current main has no openclaw_tool_mode symbols, and its matrix builder creates one RunSpec per harness/model/repetition without a mode field. (scripts/native_eval/models.py:145, 884dd1bb5511)
• PR planner omits the new mode: The PR adds the mode field but its planner still appends one run without openclaw_tool_mode; downstream harness code resolves a missing value to direct. (scripts/native_eval/models.py:169, f5b31cb1260b)
• Native-evaluation ownership history: The current matrix-runner and subsequent research-runbook work both date to Vincent Koc, establishing current-main ownership of this planning surface. (scripts/native_eval/models.py:140, 69f75c6629c4)

Likely related people:

• Vincent Koc: Introduced the auditable native matrix runner and later reproducible-research work on the same current-main planning surface. (role: native-evaluation feature owner; confidence: high; commits: 69f75c6629c4, 569b5c39c783, 884dd1bb5511; files: scripts/native_eval/models.py, scripts/native_eval/plan.py, scripts/native_eval/fleet.py)

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (59 earlier review cycles; latest 8 shown)

• reviewed 2026-08-08T14:56:23.176Z sha f5b31cb :: needs changes before merge. :: [P1] Generate a run entry for each OpenClaw tool mode
• reviewed 2026-08-08T19:02:19.597Z sha f5b31cb :: needs changes before merge. :: [P1] Expand the matrix into one OpenClaw run per tool mode
• reviewed 2026-08-08T19:49:56.619Z sha f5b31cb :: needs changes before merge. :: [P1] Generate a planned run for each OpenClaw tool mode
• reviewed 2026-08-08T21:01:34.873Z sha f5b31cb :: found issues before merge. :: [P1] Generate a planned run for each OpenClaw tool mode
• reviewed 2026-08-08T21:46:52.672Z sha f5b31cb :: needs changes before merge. :: [P1] Generate a planned run for each OpenClaw tool mode
• reviewed 2026-08-09T01:44:23.632Z sha f5b31cb :: needs changes before merge. :: [P1] Generate a planned run for each OpenClaw tool mode
• reviewed 2026-08-09T04:30:14.197Z sha f5b31cb :: needs changes before merge. :: [P1] Generate a planned run for each OpenClaw tool mode
• reviewed 2026-08-09T07:16:14.818Z sha f5b31cb :: needs changes before merge. :: [P1] Generate a planned run for each OpenClaw tool mode