fix: apply native reasoning effort to harnesses

[vincentkoc]

What does this PR do?

Propagates the native eval plan's reasoning effort into OpenClaw, Hermes,
Codex, and Claude Code client controls.

Why?

Fixes #53. Without this, a run can be labeled high while OpenClaw executes
with thinking disabled, invalidating cross-harness quality and efficiency
comparisons.

Changes

• carry reasoning_effort in RunSpec and the run manifest
• map effort to each native harness's supported control
• translate ShellBench's canonical xhigh to Claude Code's max
• preserve existing harness defaults when effort is unspecified
• test low, medium, high, and xhigh across all four harnesses

Tests

• 114 focused native runner, proxy, and fleet tests pass
• focused Ruff check passes
• fresh Codex autoreview reports no actionable findings
• all GitHub CI and CodeQL checks pass

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed August 4, 2026, 8:18 PM ET / August 5, 2026, 00:18 UTC.

ClawSweeper review

What this changes

This PR carries planned reasoning effort into native OpenClaw, Hermes, Codex, and Claude Code commands while preserving each harness’s default when no effort is planned.

Merge readiness

⚠️ Ready for maintainer review - 2 items remain

Keep open: the patch is functionally sound on the reviewed source; fleet dispatch reads reasoning effort from the run-index entry itself, so omitting it from RUN_SPEC_FIELDS does not drop the native setting. A live compatibility smoke run against the pinned harness clients would still reduce the remaining merge risk.

Likely related people: vincentkoc — recent native-eval contributor (high confidence).

Priority: P1
Reviewed head: 887c2597c1e6ef7ad369cd9a72c800b8fb52ffeb

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	A focused, well-covered repair with no remaining source-level correctness finding; live pinned-client confirmation would strengthen merge confidence.
Proof confidence	🌊 off-meta tidepool	Not applicable: This member-authored PR is exempt from the external-contributor real-behavior-proof gate; a redacted live harness smoke run remains useful compatibility evidence.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Not applicable	Not applicable: This member-authored PR is exempt from the external-contributor real-behavior-proof gate; a redacted live harness smoke run remains useful compatibility evidence.
Evidence reviewed	5 items	Fleet dispatch retains the indexed effort: The dispatcher reconstructs a RunSpec for identity fields, then independently reads reasoning_effort from the run-index entry and exports it as SHELLBENCH_REASONING_EFFORT. The prior RUN_SPEC_FIELDS concern therefore does not suppress the setting during normal dispatch. Patch maps the planned value into native clients: The supplied head diff adds RunSpec propagation and per-harness controls, including OpenClaw thinking mode, Hermes agent configuration, Codex reasoning configuration, and Claude Code effort translation. Focused command-assembly coverage: The supplied head diff adds a four-harness, four-effort parameterized test and verifies default behavior when no effort is specified.
Findings	None	None.
Security	None	None.

How this fits together

ShellBench evaluation plans assign a model, harness, and reasoning level to each run-index entry. Fleet dispatch forwards those settings to a remote job, which builds the native harness command and writes run artifacts and metadata.

flowchart LR
  Plan[Evaluation plan] --> Index[Run index]
  Index --> Fleet[Fleet dispatcher]
  Fleet --> Settings[Remote run settings]
  Settings --> Job[Native run job]
  Job --> Harnesses[Native harness clients]
  Harnesses --> Results[Artifacts and manifest]

Loading

Before merge

• Resolve merge risk (P1) - At non-default effort levels this changes native client command/config inputs; a pinned client that rejects a mapped control could fail an evaluation run, and the supplied checks validate command construction rather than a live four-harness dispatch.
• Complete next step (P2) - No mechanical code repair remains; maintainer review should confirm pinned-client compatibility evidence before merge.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Production and test delta	production +29/-3, tests +92	The implementation is small and has broad command-assembly coverage across four native harnesses.

Merge-risk options

Maintainer options:

1. Verify the pinned harness controls (recommended)
   Capture a redacted native smoke dispatch showing each pinned harness accepts its mapped setting, including Claude Code’s xhigh to max translation, before merge.

Technical review

Best possible solution:

Land the narrow propagation after a redacted smoke dispatch confirms each pinned native client accepts its mapped effort setting and Claude Code receives max for ShellBench xhigh.

Do we have a high-confidence way to reproduce the issue?

Yes—source inspection provides a high-confidence path: create a planned run with an effort level, dispatch it, and inspect the generated native command/config for all four harnesses.

Is this the best way to solve the issue?

Yes—the proposed single RunSpec value plus native-client mappings is the narrowest maintainable repair and preserves unspecified defaults.

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label justifications:

• P1: Incorrect reasoning controls invalidate active cross-harness quality and efficiency comparisons.
• merge-risk: 🚨 compatibility: The patch changes version-sensitive native CLI and configuration controls whenever a plan specifies reasoning effort.
• rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: This member-authored PR is exempt from the external-contributor real-behavior-proof gate; a redacted live harness smoke run remains useful compatibility evidence.

Evidence

What I checked:

• Fleet dispatch retains the indexed effort: The dispatcher reconstructs a RunSpec for identity fields, then independently reads reasoning_effort from the run-index entry and exports it as SHELLBENCH_REASONING_EFFORT. The prior RUN_SPEC_FIELDS concern therefore does not suppress the setting during normal dispatch. (scripts/native_eval/fleet.py:1011, 884dd1bb5511)
• Patch maps the planned value into native clients: The supplied head diff adds RunSpec propagation and per-harness controls, including OpenClaw thinking mode, Hermes agent configuration, Codex reasoning configuration, and Claude Code effort translation. (scripts/native_eval/harnesses.py:532, 887c2597c1e6)
• Focused command-assembly coverage: The supplied head diff adds a four-harness, four-effort parameterized test and verifies default behavior when no effort is specified. (tests/test_native_eval_runner.py:681, 887c2597c1e6)
• Native-evaluation ownership history: Current main’s most recent fleet change is the native-evaluation trace reconstruction commit by Vincent Koc; nearby native-evaluation runbook history is also by Vincent Koc. (scripts/native_eval/fleet.py:26, 884dd1bb5511)
• Canonical linked problem: This PR explicitly targets the still-open reasoning-effort consistency report, which remains the canonical user-problem record until this implementation lands. (887c2597c1e6)

Likely related people:

• vincentkoc: Authored the current-main native-evaluation fleet and harness changes immediately adjacent to this path, in addition to this proposed repair. (role: recent native-evaluation contributor; confidence: high; commits: 884dd1bb5511, 569b5c39c783; files: scripts/native_eval/fleet.py, scripts/native_eval/harnesses.py)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Attach redacted smoke output for the pinned native clients at planned effort levels.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (31 earlier review cycles; latest 8 shown)

• reviewed 2026-08-03T01:48:53.091Z sha 887c259 :: needs maintainer review before merge. :: none
• reviewed 2026-08-03T04:09:25.942Z sha 887c259 :: needs maintainer review before merge. :: none
• reviewed 2026-08-03T09:05:46.797Z sha 887c259 :: needs maintainer review before merge. :: none
• reviewed 2026-08-03T20:14:33.241Z sha 887c259 :: needs maintainer review before merge. :: none
• reviewed 2026-08-04T15:03:36.029Z sha 887c259 :: needs maintainer review before merge. :: none
• reviewed 2026-08-04T18:03:27.270Z sha 887c259 :: needs maintainer review before merge. :: none
• reviewed 2026-08-04T21:07:16.018Z sha 887c259 :: needs maintainer review before merge. :: none
• reviewed 2026-08-04T23:03:06.433Z sha 887c259 :: needs changes before merge. :: [P1] Preserve reasoning effort during fleet reconstruction