fix(native-eval): reject invalid harness runs - #64
Draft
vincentkoc wants to merge 1 commit into
Draft
Conversation
clawsweeper
Bot
added
rating: 🦪 silver shellfish
|
Codex review: found issues before merge. Reviewed August 3, 2026, 5:05 AM ET / 09:05 UTC. ClawSweeper reviewWhat this changesThe PR records structured native-eval execution outcomes, excludes fully invalid runs from score eligibility, and requires an archived exit status before fleet recovery marks an exported run completed. Merge readiness⛔ Blocked until real behavior proof from a real setup is added - 6 items remain Keep this member-authored PR open: it addresses the linked native-eval bug, but its unchanged head still removes the only recovery path for complete legacy exports without an archived exit status. That turns a previously completed recovery into a failed rerun, so it needs a compatibility repair or an explicit maintainer decision to accept the upgrade break. Priority: P1 Review scores
Verification
How this fits togetherShellBench’s native-eval runtime runs benchmark trials and writes result archives. The aggregation step converts those archives into score reports, while fleet recovery finalizes interrupted exported runs and decides whether to rerun them. flowchart LR
A[Benchmark trials] --> B[Native-eval runtime]
B --> C[Result archive]
C --> D[Execution classification]
D --> E[Score aggregation]
C --> F[Fleet recovery]
F --> G[Completed or rerun state]
Decision needed
Why: The patch deliberately removes an existing compatibility fallback; source cannot establish whether maintainers intend the resulting rerun requirement for already-exported historical archives. Before merge
Findings
Agent review detailsSecurityNone. Review metrics
Merge-risk optionsMaintainer options:
Technical reviewBest possible solution: Keep strict rejection for demonstrably invalid runs, but preserve a narrowly validated legacy recovery path or migrate/backfill legacy archives before making archived exit status mandatory for completion. Do we have a high-confidence way to reproduce the issue? Yes. Current main’s focused fleet test and the PR’s replacement test construct the same complete archive plus final-checkpoint scenario; the PR changes its result from completed to failed when Is this the best way to solve the issue? No. Rejecting conclusively invalid runs is the right fix for the linked report, but unconditional failure for a previously accepted legacy archive shape is not the narrowest compatible solution without an explicit maintainer decision. Full review comments:
Overall correctness: patch is incorrect AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511. LabelsLabel justifications:
EvidenceWhat I checked:
Likely related people:
Rating scale
Overall follows the weaker of proof and patch quality. Workflow
HistoryReview history (19 earlier review cycles; latest 8 shown)
|
clawsweeper
Bot
added
status: ⏳ waiting on author
clawsweeper
Bot
added
rating: 🦪 silver shellfish
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Records structured native-eval execution outcomes and rejects runs where every
trial failed at the harness, infrastructure, or verifier layer.
Why?
Verifier rewards are useful diagnostics, but they are not proof that the
benchmark harness executed correctly. Fully invalid runs were being presented
as completed score data, and fleet recovery could infer success without an
archived exit status.
Fixes #63
Changes
Tests
python -m pytest -qpasses on Blacksmith Testboxpython -m ruff check clawbench app.py scripts testspasses on Blacksmith TestboxFocused proof: 117 tests passed; Ruff clean. Autoreview reported no actionable
findings.