Skip to content

Run Hermes adapter through local harness - #35

Open
afrog33k wants to merge 2 commits into
openclaw:mainfrom
afrog33k:main
Open

Run Hermes adapter through local harness#35
afrog33k wants to merge 2 commits into
openclaw:mainfrom
afrog33k:main

Conversation

@afrog33k

@afrog33k afrog33k commented Jul 1, 2026

Copy link
Copy Markdown

Summary

  • Wire registered non-OpenClaw adapters through the generic harness lifecycle instead of rejecting them as known-but-unwired.
  • Add Hermes local adapter configuration from environment and a local verification shim for file/execution checks.
  • Prepend explicit staged-workspace guidance to Hermes MiniSWERunner prompts so local runs inspect the task workspace first.

Validation

  • /Users/reckon/miniconda3/bin/python -m pytest tests/test_harness.py tests/test_hermes_adapter.py -q -> 21 passed
  • /Users/reckon/miniconda3/bin/python -m pytest -q -> 318 passed, 5 skipped
  • Local Hermes rerun: t2-msg-summarize-thread improved from 0.63704 to 1.000 with the workspace-guidance patch.

Notes

  • This keeps OpenClaw behavior intact and routes adapter != "openclaw" through the adapter lifecycle.
  • Memory/session/cron/browser checks still need adapter-aware verification before broad non-file task sweeps.

@afrog33k
afrog33k requested a review from a team as a code owner July 1, 2026 11:36
@clawsweeper

clawsweeper Bot commented Jul 1, 2026 •
edited
Loading

Copy link
Copy Markdown

Codex review: needs changes before merge. Reviewed August 7, 2026, 11:14 AM ET / 15:14 UTC.

ClawSweeper review

What this changes

The PR enables registered local adapters such as Hermes to run through ShellBench’s generic harness and adds workspace-first prompt guidance.

Merge readiness

⛔ Blocked by patch quality or review findings - 10 items remain

Keep open: the PR’s central Hermes harness capability is not on current main, but its new execution path still bypasses adapter state verification, can retain passing scores after an adapter failure, and admits browser tasks without proving browser readiness.

Priority: P2
Reviewed head: 67ff264005ab6b55679299ba4e7fcd7cbb1276bf

Review scores

Measure Result What it means
Overall readiness 🧂 unranked krab (1/6) The PR supplies useful real-run evidence, but the harness currently produces unreliable results for part of the capability set it enables.
Proof confidence 🐚 platinum hermit (4/6) Sufficient (live_output): The PR body reports an after-fix local Hermes run improving t2-msg-summarize-thread from 0.63704 to 1.000, alongside focused and full test results; the remaining blockers are correctness gaps in broader claimed capabilities.
Patch quality 🧂 unranked krab (1/6) 3 actionable review findings remain.

Verification

Check Result Evidence
Real behavior Verified Sufficient (live_output): The PR body reports an after-fix local Hermes run improving t2-msg-summarize-thread from 0.63704 to 1.000, alongside focused and full test results; the remaining blockers are correctness gaps in broader claimed capabilities.
Evidence reviewed 5 items Current main still rejects Hermes execution: Current main retains the explicit guard that registered non-OpenClaw adapters are not wired into the end-to-end scoring harness, so this PR is neither implemented nor obsolete.
Adapter state verification is bypassed: The branch scores local runs with a gateway-only shim after teardown, while the adapter contract requires state queries to be resolved through adapter.verify_state_query; memory, session, cron, and gateway assertions therefore cannot use Hermes’s verifier.
Canonical contract provenance: The canonical adapter layer introduced the explicit lifecycle in which state queries are resolved by the active adapter; Hermes implements memory/session/cron query handling in that layer.
Findings 3 actionable findings [P1] Route state assertions through the active adapter
[P1] Mark adapter phase errors as failed runs
[P1] Gate browser tasks on Hermes readiness
Security None None.

How this fits together

ShellBench loads benchmark tasks, runs each through an agent adapter, then verifies artifacts and state before aggregating scores. This PR routes non-OpenClaw adapters into that lifecycle, affecting which tasks run and how their results are scored.

flowchart LR
  Tasks[Benchmark tasks] --> Harness[Benchmark harness]
  Harness --> Capability[Capability gate]
  Capability --> Adapter[Hermes adapter]
  Adapter --> LocalRun[Local agent run]
  LocalRun --> Verification[Completion and state verification]
  Verification --> Scores[Benchmark scores]
Loading

Before merge

  • Route state assertions through the active adapter (P1) - The new local path calls score_task_run with _LocalVerificationClient after adapter.teardown, but the canonical contract requires state queries to use adapter.verify_state_query. Any memory, session, cron, or gateway assertion therefore invokes unavailable gateway methods instead of Hermes’s verifier, producing invalid completion scores.
  • Mark adapter phase errors as failed runs (P1) - Setting result.error after scoring does not change run_score, delivery_outcome, failure_mode, or _is_passing_run; a phase can abort and still be aggregated as a pass if its files satisfy completion checks. Convert the phase error into a failed result or recompute the score and classification before returning it.
  • Gate browser tasks on Hermes readiness (P1) - This capability gate admits any task whose declared capabilities include browser because Hermes AI-agent mode claims BROWSER, but the local path has no browser availability check comparable to the OpenClaw path. A browser task can therefore start in an environment that lacks the necessary tool and be scored as an adapter failure rather than excluded or verified.
  • Resolve merge risk (P1) - Merging can publish incorrect benchmark scores for tasks whose completion depends on adapter-owned memory, session, cron, or gateway state.
  • Resolve merge risk (P1) - An adapter phase failure can still be counted as a passing run when deterministic checks happen to pass.
  • Resolve merge risk (P1) - Hermes AI-agent mode claims browser capability without a readiness check, so browser tasks can be admitted into an unavailable environment.
  • Complete next step (P2) - The three blocking defects have a concrete, shared repair boundary in the new local-adapter harness path.
  • Improve patch quality - Route state queries through the live adapter and cover a Hermes memory or cron assertion.
  • Improve patch quality - Make a returned adapter phase error force a failed aggregate result.
  • Improve patch quality - Add browser readiness proof or remove Hermes browser capability until it is available.

Findings

  • [P1] Route state assertions through the active adapter — clawbench/harness.py:640-650
  • [P1] Mark adapter phase errors as failed runs — clawbench/harness.py:652-656
  • [P1] Gate browser tasks on Hermes readiness — clawbench/harness.py:592-600
Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Patch surface production +247/-10, tests +103/-7 across 4 files Most of the change is new harness control flow, so lifecycle and scoring coverage need to exercise the full adapter boundary.

Merge-risk options

Maintainer options:

  1. Complete adapter-aware scoring (recommended)
    Route canonical state queries through the live adapter, make phase errors fail the run, and gate browser tasks on a verified Hermes capability before merge.
  2. Limit the initial Hermes scope
    Merge only after narrowing the advertised capability set to file and execution tasks that the local verifier can score accurately.
Copy recommended automerge instruction 
@clawsweeper automerge

Special instructions:
Preserve OpenClaw behavior; add regression coverage for adapter state verification, phase-error failure classification, and browser-task rejection or readiness.

Technical review

Best possible solution:

Integrate adapter-owned state verification into scoring before teardown, classify adapter phase failures as failed results, and either prove Hermes browser readiness or stop advertising browser support.

Do we have a high-confidence way to reproduce the issue?

Yes, by source inspection: run a local adapter task with a state assertion or a phase error after producing expected files; the branch sends state checks to the gateway shim and does not let result.error affect pass classification.

Is this the best way to solve the issue?

No: using the existing gateway-oriented scorer with a failing shim does not satisfy the canonical adapter contract; scoring must receive adapter-backed state-query results while the adapter context is still live.

Full review comments:

  • [P1] Route state assertions through the active adapter — clawbench/harness.py:640-650
    The new local path calls score_task_run with _LocalVerificationClient after adapter.teardown, but the canonical contract requires state queries to use adapter.verify_state_query. Any memory, session, cron, or gateway assertion therefore invokes unavailable gateway methods instead of Hermes’s verifier, producing invalid completion scores.
    Confidence: 0.99
  • [P1] Mark adapter phase errors as failed runs — clawbench/harness.py:652-656
    Setting result.error after scoring does not change run_score, delivery_outcome, failure_mode, or _is_passing_run; a phase can abort and still be aggregated as a pass if its files satisfy completion checks. Convert the phase error into a failed result or recompute the score and classification before returning it.
    Confidence: 0.98
  • [P1] Gate browser tasks on Hermes readiness — clawbench/harness.py:592-600
    This capability gate admits any task whose declared capabilities include browser because Hermes AI-agent mode claims BROWSER, but the local path has no browser availability check comparable to the OpenClaw path. A browser task can therefore start in an environment that lacks the necessary tool and be scored as an adapter failure rather than excluded or verified.
    Confidence: 0.93

Overall correctness: patch is incorrect
Overall confidence: 0.98

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label justifications:

  • P2: This is a bounded benchmark-harness feature whose current defects affect score correctness but not the core runtime.
  • merge-risk: 🚨 other: The new adapter path can report invalid benchmark outcomes despite successful unit tests.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🐚 platinum hermit and patch quality is 🧂 unranked krab.
  • status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (live_output): The PR body reports an after-fix local Hermes run improving t2-msg-summarize-thread from 0.63704 to 1.000, alongside focused and full test results; the remaining blockers are correctness gaps in broader claimed capabilities.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body reports an after-fix local Hermes run improving t2-msg-summarize-thread from 0.63704 to 1.000, alongside focused and full test results; the remaining blockers are correctness gaps in broader claimed capabilities.

Evidence

Acceptance criteria:

  • [P1] python -m pytest tests/test_harness.py tests/test_hermes_adapter.py tests/test_adapter_base.py -q.
  • [P1] python -m pytest tests/test_scorer.py -q.
  • [P1] Run a local Hermes file task and one adapter-owned state assertion, retaining a redacted result trace.

What I checked:

  • Current main still rejects Hermes execution: Current main retains the explicit guard that registered non-OpenClaw adapters are not wired into the end-to-end scoring harness, so this PR is neither implemented nor obsolete. (clawbench/harness.py:124, 884dd1bb5511)
  • Adapter state verification is bypassed: The branch scores local runs with a gateway-only shim after teardown, while the adapter contract requires state queries to be resolved through adapter.verify_state_query; memory, session, cron, and gateway assertions therefore cannot use Hermes’s verifier. (clawbench/harness.py:640, 67ff264005ab)
  • Canonical contract provenance: The canonical adapter layer introduced the explicit lifecycle in which state queries are resolved by the active adapter; Hermes implements memory/session/cron query handling in that layer. (clawbench/adapters/base.py:20, 56531fbf432b)
  • Prior blockers remain at the same head: The current PR head is the same SHA as the latest completed ClawSweeper review, so its state-verification, phase-error, and browser-readiness blockers have not been changed. (clawbench/harness.py:640, 67ff264005ab)
  • Branch applies cleanly against current main: A three-way merge inspection showed the branch changes can be applied against current main; the concerns are functional, not stale-base deletion. (67ff264005ab)

Likely related people:

  • scoootscooob: Introduced the canonical adapter contract and Hermes capability/state-query behavior that the proposed harness path must honor. (role: introduced adapter canonicalization; confidence: high; commits: 56531fbf432b; files: clawbench/adapters/base.py, clawbench/adapters/hermes.py, clawbench/canonical/convert.py)
  • Robin Narsingh Ranabhat: Most recent current-main contributor to the benchmark harness’s workspace and run lifecycle. (role: recent harness contributor; confidence: medium; commits: 9fe22f9233f7; files: clawbench/harness.py)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (49 earlier review cycles; latest 8 shown)
  • reviewed 2026-08-05T23:39:13.125Z sha 67ff264 :: needs changes before merge. :: [P1] Route state assertions through the active adapter
  • reviewed 2026-08-06T07:18:19.714Z sha 67ff264 :: needs changes before merge. :: [P1] Pass adapter state verification into scoring
  • reviewed 2026-08-06T10:16:42.347Z sha 67ff264 :: found issues before merge. :: [P1] Route state assertions through the active adapter
  • reviewed 2026-08-06T11:57:31.486Z sha 67ff264 :: needs changes before merge. :: [P1] Route state assertions through the active adapter
  • reviewed 2026-08-06T13:16:11.902Z sha 67ff264 :: needs changes before merge. :: [P1] Route state assertions through the active adapter
  • reviewed 2026-08-07T06:00:32.775Z sha 67ff264 :: needs changes before merge. :: [P1] Route state assertions through the active adapter
  • reviewed 2026-08-07T08:03:32.535Z sha 67ff264 :: needs changes before merge. :: [P1] Route state assertions through the active adapter | [P1] Gate browser tasks until Hermes has a local readiness check
  • reviewed 2026-08-07T09:50:44.489Z sha 67ff264 :: needs changes before merge. :: [P1] Route state assertions through the active adapter | [P1] Fail runs when an adapter phase reports an error | [P1] Gate browser tasks on Hermes readiness

@clawsweeper clawsweeper Bot added proof: sufficient Contributor real behavior proof is sufficient. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. P2 Normal priority bug or improvement with limited blast radius. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. labels Jul 1, 2026
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. labels Jul 1, 2026
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. merge-risk: 🚨 session-state 🚨 Merging this PR could lose, corrupt, stale, or mis-associate session or agent state. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. labels Jul 29, 2026
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. merge-risk: 🚨 other 🚨 Merging this PR has meaningful risk outside the owned taxonomy. merge-risk: 🚨 session-state 🚨 Merging this PR could lose, corrupt, stale, or mis-associate session or agent state. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. merge-risk: 🚨 session-state 🚨 Merging this PR could lose, corrupt, stale, or mis-associate session or agent state. merge-risk: 🚨 other 🚨 Merging this PR has meaningful risk outside the owned taxonomy. labels Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

merge-risk: 🚨 other 🚨 Merging this PR has meaningful risk outside the owned taxonomy. P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@afrog33k