Run Hermes adapter through local harness

[afrog33k]

Summary

• Wire registered non-OpenClaw adapters through the generic harness lifecycle instead of rejecting them as known-but-unwired.
• Add Hermes local adapter configuration from environment and a local verification shim for file/execution checks.
• Prepend explicit staged-workspace guidance to Hermes MiniSWERunner prompts so local runs inspect the task workspace first.

Validation

• /Users/reckon/miniconda3/bin/python -m pytest tests/test_harness.py tests/test_hermes_adapter.py -q -> 21 passed
• /Users/reckon/miniconda3/bin/python -m pytest -q -> 318 passed, 5 skipped
• Local Hermes rerun: t2-msg-summarize-thread improved from 0.63704 to 1.000 with the workspace-guidance patch.

Notes

• This keeps OpenClaw behavior intact and routes adapter != "openclaw" through the adapter lifecycle.
• Memory/session/cron/browser checks still need adapter-aware verification before broad non-file task sweeps.

[clawsweeper]

Codex review: needs changes before merge. Reviewed August 3, 2026, 4:14 PM ET / 20:14 UTC.

ClawSweeper review

What this changes

This PR enables registered local adapters, including Hermes, to run through ShellBench’s harness, adds Hermes environment configuration, and prepends task-workspace guidance to Hermes prompts.

Merge readiness

⛔ Blocked by patch quality or review findings - 5 items remain

Needs changes before merge: the generic local-adapter path never invokes the adapter state-verification contract, so Hermes runs that advertise memory, session, or cron support can receive invalid completion scores.

Priority: P2
Reviewed head: 67ff264005ab6b55679299ba4e7fcd7cbb1276bf

Review scores

Measure	Result	What it means
Overall readiness	🦪 silver shellfish (2/6)	The branch has credible local-run evidence and useful focused tests, but the unchanged P1 scoring defect makes it unready to merge.
Proof confidence	🐚 platinum hermit (4/6)	Sufficient (live_output): The PR body reports an after-fix local Hermes rerun with an improved task score; it is valid real-run proof for the workspace-guidance behavior, though it does not cover the remaining state-verification defect.
Patch quality	🦪 silver shellfish (2/6)	1 actionable review finding remain.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (live_output): The PR body reports an after-fix local Hermes rerun with an improved task score; it is valid real-run proof for the workspace-guidance behavior, though it does not cover the remaining state-verification defect.
Evidence reviewed	5 items	Local lifecycle bypasses adapter state verification: The proposed local-adapter lifecycle runs phases, calls teardown, and then sends scoring through a gateway-only verification shim; it does not call adapter.verify_state_query while the live adapter context exists. Adapter contract requires adapter-owned state checks: The established adapter interface specifies that each task StateQuery is resolved through adapter.verify_state_query, before teardown releases adapter-side state. Hermes claims state capabilities in AI-agent mode: The branch tests confirm AI-agent mode advertises memory, cron, and related dynamic capabilities, while Hermes implements a state verifier for memory, session, and cron queries.
Findings	1 actionable finding	[P1] Route state assertions through the live adapter verifier
Security	None	None.

How this fits together

ShellBench converts benchmark tasks into canonical adapter phases, runs an agent in an isolated workspace, and then scores the resulting transcript and environment assertions. The new local-adapter path sits between task setup and scoring, where adapter-owned state checks must be resolved before adapter teardown.

flowchart TD
  A[Benchmark task] --> B[Canonical adapter task]
  B --> C[Local adapter lifecycle]
  C --> D[Hermes workspace and agent]
  D --> E[Adapter state assertions]
  E --> F[Completion scoring]
  F --> G[Benchmark result]

Loading

Before merge

• Route state assertions through the live adapter verifier (P1) - The local path calls teardown and then scores with _LocalVerificationClient, so it never invokes adapter.verify_state_query. Hermes AI-agent mode advertises memory, session, and cron capabilities; those assertions are therefore scored against an unavailable gateway instead of the adapter’s verifier. This is the unchanged blocker from the prior review cycle.
• Resolve merge risk (P1) - Merging this path would let Hermes AI-agent runs advertise stateful capabilities while completion scoring bypasses their live verifier, producing misleading memory, session, or cron benchmark results.
• Complete next step (P2) - A repair worker can address the unchanged P1 with a bounded harness/scorer integration and one focused state-assertion regression test.
• Improve patch quality - Route local-adapter StateQueries through the live adapter before teardown.
• Improve patch quality - Add a harness regression that proves a state assertion uses the adapter verifier.

Findings

• [P1] Route state assertions through the live adapter verifier — clawbench/harness.py:636-650

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Patch surface	production +247/-10, tests +103/-7 across 4 files	Most of the change is new harness lifecycle code, so the missing end-to-end state-verification coverage is material to merge confidence.

Merge-risk options

Maintainer options:

1. Wire state checks into the live lifecycle (recommended)
   Pass the live adapter and context into state-query scoring before teardown, then cover a Hermes state assertion through the harness.
2. Restrict local Hermes runs temporarily
   Limit the executable Hermes path to capabilities the existing scorer can validate until adapter-aware state scoring is implemented.
3. Pause the harness integration
   Keep Hermes registered but non-executable if neither complete state verification nor a deliberately restricted capability scope is acceptable.

Copy recommended automerge instruction

@clawsweeper automerge

Special instructions:
Thread live adapter state verification through local-adapter scoring before teardown and add a state-assertion regression test.

Technical review

Best possible solution:

Preserve the generic adapter lifecycle, but resolve every adapter-owned StateQuery against the live adapter before teardown and add regression coverage for a stateful local-adapter task.

Do we have a high-confidence way to reproduce the issue?

Yes—source inspection gives a high-confidence path: run a local adapter task with a memory, session, or cron StateQuery in Hermes AI-agent mode and observe that scoring uses the local gateway shim rather than HermesAdapter.verify_state_query.

Is this the best way to solve the issue?

No. The current branch is not the best complete solution because it bypasses the existing adapter verification contract; threading the live adapter context into state scoring is the narrow maintainable repair.

Full review comments:

• [P1] Route state assertions through the live adapter verifier — clawbench/harness.py:636-650
  The local path calls teardown and then scores with _LocalVerificationClient, so it never invokes adapter.verify_state_query. Hermes AI-agent mode advertises memory, session, and cron capabilities; those assertions are therefore scored against an unavailable gateway instead of the adapter’s verifier. This is the unchanged blocker from the prior review cycle.
  Confidence: 0.99

Overall correctness: patch is incorrect
Overall confidence: 0.99

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label justifications:

• P2: The patch can produce incorrect benchmark results for stateful local-adapter tasks, but the defect is bounded to the new execution path.
• merge-risk: 🚨 session-state: Stateful Hermes capabilities are advertised while the proposed scorer path does not invoke the adapter’s state verifier.
• rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🐚 platinum hermit and patch quality is 🦪 silver shellfish.
• status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (live_output): The PR body reports an after-fix local Hermes rerun with an improved task score; it is valid real-run proof for the workspace-guidance behavior, though it does not cover the remaining state-verification defect.
• proof: sufficient: Contributor real behavior proof is sufficient. The PR body reports an after-fix local Hermes rerun with an improved task score; it is valid real-run proof for the workspace-guidance behavior, though it does not cover the remaining state-verification defect.

Evidence

Acceptance criteria:

• [P1] python -m pytest tests/test_harness.py tests/test_hermes_adapter.py -q.
• [P1] python -m pytest -q.

What I checked:

• Local lifecycle bypasses adapter state verification: The proposed local-adapter lifecycle runs phases, calls teardown, and then sends scoring through a gateway-only verification shim; it does not call adapter.verify_state_query while the live adapter context exists. (clawbench/harness.py:636, 67ff264005ab)
• Adapter contract requires adapter-owned state checks: The established adapter interface specifies that each task StateQuery is resolved through adapter.verify_state_query, before teardown releases adapter-side state. (clawbench/adapters/base.py:20, 56531fbf432b)
• Hermes claims state capabilities in AI-agent mode: The branch tests confirm AI-agent mode advertises memory, cron, and related dynamic capabilities, while Hermes implements a state verifier for memory, session, and cron queries. (tests/test_hermes_adapter.py:199, 67ff264005ab)
• Finding remains unchanged from the last review: The current head is still 67ff264005ab6b55679299ba4e7fcd7cbb1276bf; a direct comparison with the previous reviewed head contains no changes to the affected harness or test files. (clawbench/harness.py:640, 67ff264005ab)
• Feature-history ownership: The generic adapter canonicalization and state-verification interface appears to date to the adapter-layer introduction commit, while the most recent main-branch harness lifecycle work is the workspace-isolation change. (clawbench/adapters/base.py:20, 56531fbf432b)

Likely related people:

• scoootscooob: Introduced the canonical adapter layer and its adapter-owned state-query contract in the central adapter base surface. (role: adapter-layer introducer; confidence: high; commits: 56531fbf432b; files: clawbench/adapters/base.py, clawbench/adapters/__init__.py)
• Robin Narsingh Ranabhat: Most recently changed the main-branch harness lifecycle around per-run workspaces and cleanup, adjacent to the proposed local execution path. (role: recent harness lifecycle contributor; confidence: high; commits: 9fe22f9233f7; files: clawbench/harness.py, tests/test_harness.py)

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (31 earlier review cycles; latest 8 shown)

• reviewed 2026-08-02T14:58:11.781Z sha 67ff264 :: needs changes before merge. :: [P1] Route local state checks through the adapter verifier
• reviewed 2026-08-02T17:06:20.210Z sha 67ff264 :: needs changes before merge. :: [P1] Route local state checks through the adapter verifier
• reviewed 2026-08-02T19:15:11.302Z sha 67ff264 :: needs changes before merge. :: [P1] Route local state checks through the adapter verifier
• reviewed 2026-08-02T20:37:21.042Z sha 67ff264 :: needs changes before merge. :: [P1] Route local state checks through the adapter verifier
• reviewed 2026-08-02T22:14:48.627Z sha 67ff264 :: needs changes before merge. :: [P1] Route local state checks through the adapter verifier
• reviewed 2026-08-03T01:48:51.671Z sha 67ff264 :: needs changes before merge. :: [P1] Route local state checks through the adapter verifier
• reviewed 2026-08-03T04:09:35.877Z sha 67ff264 :: needs changes before merge. :: [P1] Pass adapter state verification into scoring
• reviewed 2026-08-03T09:05:36.956Z sha 67ff264 :: needs changes before merge. :: [P1] Route state assertions through the live adapter