fix: accept Claude native trajectories

[vincentkoc]

What does this PR do?

Marks Claude Code native trajectories as real harness events so completed Claude runs remain eligible for native aggregation.

Why?

The native runner already converts Claude Code stream-json output into trajectory.json, but the harness allowlist omitted claude-code. Aggregation therefore classified otherwise valid Claude runs as trajectory_unavailable.

Changes

• add claude-code to the real-trajectory harness allowlist
• assert every pinned native harness reports real_harness_events

Live proof

A native Claude Code control run completed four tasks with four result files, four valid completed results, and no infrastructure failures. Aggregation classified every trajectory as real; the run remained ineligible only because it was intentionally marked as an exploratory four-task subset.

• coverage: 4/4
• score: 0.75
• exact passes: 3
• trajectory complete: true
• trajectory status: real for all four tasks
• infrastructure failures: 0
• exclusion reason: exploratory_subset

Tests

• python -m pytest -q passes locally (444 passed, 5 skipped)
• python -m ruff check clawbench app.py scripts tests passes locally
• live Claude Code native run aggregates as real harness events

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed August 3, 2026, 4:13 PM ET / 20:13 UTC.

ClawSweeper review

What this changes

The PR classifies the pinned Claude Code native harness as emitting real trajectory events and adds coverage requiring all pinned native harnesses to use that mode.

Merge readiness

⚠️ Ready for maintainer review - 3 items remain

Keep this member-authored PR open for maintainer sign-off. The patch is a narrow, source-supported correction: Claude Code already emits validated real trajectories, but its harness name is omitted from the manifest allowlist that aggregation requires for eligibility.

Priority: P2
Reviewed head: ac1c974c12749dddb5d49af734561cec59910c65
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	A focused, source-aligned correction with credible real-run proof; only the benchmark evidence-classification decision remains for maintainers.
Proof confidence	🐚 platinum hermit (4/6)	Sufficient (live_output): The PR body supplies after-fix native-run evidence for four completed Claude Code tasks, reporting real trajectory status for all tasks, no infrastructure failures, and the expected exploratory-subset exclusion; redact any private run details in future attached evidence.
Patch quality	🦞 diamond lobster (5/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (live_output): The PR body supplies after-fix native-run evidence for four completed Claude Code tasks, reporting real trajectory status for all tasks, no infrastructure failures, and the expected exploratory-subset exclusion; redact any private run details in future attached evidence.
Evidence reviewed	5 items	Claude Code already produces real trajectory metadata: Current main dispatches Claude Code runs to its dedicated converter, and the converter only reports a real trajectory after parsing stream events, observing a successful terminal event, and establishing a runtime model name. Current regression coverage proves the reported mismatch: Current main’s Claude Code trajectory test writes stream-json-like events and asserts trajectory_status=real, canonical model identity, terminal completion, and an emitted trajectory; the missing allowlist entry is therefore inconsistent with existing behavior. Aggregation makes the allowlist consequential: Native eligibility requires manifest trajectory_mode=real_harness_events and real task trajectories; otherwise aggregation assigns trajectory_unavailable. The PR changes the manifest source at the narrow point that controls this outcome.
Findings	None	None.
Security	None	None.

How this fits together

ShellBench’s native runner converts each harness’s execution records into a trajectory and writes the harness classification into the run manifest. Aggregation reads that classification with per-task trajectory status to decide whether a completed run is eligible for native benchmark reporting.

flowchart LR
  A[Claude Code stream-json events] --> B[Claude trajectory converter]
  B --> C[Trajectory metadata]
  C --> D[Native run manifest]
  D --> E[Aggregation eligibility check]
  E --> F[Benchmark summary]

Loading

Decision needed

Question	Recommendation
Should ShellBench classify the pinned Claude Code stream-json adapter as real harness events for native aggregation eligibility?	Approve real-event classification: Merge the focused correction because the existing converter and regression test show Claude Code emits completed, canonical real trajectories.

Why: The source and live-run summary support the correction, but the classification changes what future benchmark data is eligible for longitudinal aggregation and publication.

Before merge

• Resolve merge risk (P1) - Merging changes future Claude Code runs from unsupported trajectory evidence to eligible real native-trajectory evidence, so maintainers should explicitly accept the resulting longitudinal benchmark-cohort interpretation before publishing or comparing aggregates.
• Complete next step (P2) - No mechanical repair is needed; a maintainer must approve the real-event classification because it changes future aggregate eligibility semantics.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Focused patch size	production +1, tests +12 across 2 files	The implementation changes one eligibility set and adds a direct regression guard without widening the native-runner surface.

Merge-risk options

Maintainer options:

1. Approve the validated evidence classification (recommended)
   Merge after accepting that the existing Claude Code converter meets ShellBench’s real native-trajectory standard for future aggregates.
2. Pause pending benchmark-policy clarification
   Keep the PR open if maintainers want additional methodology criteria before changing Claude Code from unsupported to eligible evidence.

Technical review

Best possible solution:

Approve the existing narrow allowlist correction if maintainers accept the validated Claude Code stream-json converter as real native evidence, then retain the all-harness regression test to prevent future manifest/adapter drift.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level: current main converts completed Claude Code stream-json records into real trajectories, while the manifest allowlist still returns unsupported for claude-code; the PR body also reports a four-task after-fix native run classified as real.

Is this the best way to solve the issue?

Yes, conditionally: changing the single manifest allowlist entry is the narrowest repair and the added all-harness assertion prevents recurrence, provided maintainers approve the benchmark evidence-classification policy.

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label justifications:

• P2: This is a bounded native benchmark correctness fix with a meaningful but non-emergency reporting impact.
• merge-risk: 🚨 compatibility: The change alters whether future Claude Code runs qualify for aggregate benchmark reporting, affecting longitudinal comparison semantics.
• rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🦞 diamond lobster.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body supplies after-fix native-run evidence for four completed Claude Code tasks, reporting real trajectory status for all tasks, no infrastructure failures, and the expected exploratory-subset exclusion; redact any private run details in future attached evidence.
• proof: sufficient: Contributor real behavior proof is sufficient. The PR body supplies after-fix native-run evidence for four completed Claude Code tasks, reporting real trajectory status for all tasks, no infrastructure failures, and the expected exploratory-subset exclusion; redact any private run details in future attached evidence.

Evidence

What I checked:

• Claude Code already produces real trajectory metadata: Current main dispatches Claude Code runs to its dedicated converter, and the converter only reports a real trajectory after parsing stream events, observing a successful terminal event, and establishing a runtime model name. (scripts/native_eval/harness_trajectories.py:316, 884dd1bb5511)
• Current regression coverage proves the reported mismatch: Current main’s Claude Code trajectory test writes stream-json-like events and asserts trajectory_status=real, canonical model identity, terminal completion, and an emitted trajectory; the missing allowlist entry is therefore inconsistent with existing behavior. (tests/test_native_eval_runner.py:1705, 884dd1bb5511)
• Aggregation makes the allowlist consequential: Native eligibility requires manifest trajectory_mode=real_harness_events and real task trajectories; otherwise aggregation assigns trajectory_unavailable. The PR changes the manifest source at the narrow point that controls this outcome. (scripts/native_eval/aggregate.py:907, 884dd1bb5511)
• Feature provenance: Native trajectory support and the relevant runtime surfaces appear to originate in the native-evaluation rollout and have been maintained through later trace-stabilization work by the same current-main contributor. (scripts/native_eval/models.py:102, b9acd9f7a010)
• Release context: The repository changelog identifies version 0.4.0 as Unreleased; this open PR is not already included in a released version. (CHANGELOG.md:3, 884dd1bb5511)

Likely related people:

• Vincent Koc: Current main history attributes the native matrix runner, trace stabilization, and later trajectory reconstruction work across the manifest, runtime, aggregate, and test surfaces to Vincent Koc. (role: native evaluation feature owner and recent area contributor; confidence: high; commits: 69f75c6629c4, b9acd9f7a010, 884dd1bb5511; files: scripts/native_eval/models.py, scripts/native_eval/run_job.py, scripts/native_eval/runtime.py)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Record maintainer acceptance of the Claude Code real-event classification before merge.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (22 earlier review cycles; latest 8 shown)

• reviewed 2026-08-02T19:14:46.312Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T20:37:25.866Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T22:18:39.060Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-03T01:48:10.144Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-03T04:09:32.756Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-03T09:05:13.579Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-03T17:35:47.679Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-03T19:02:00.829Z sha ac1c974 :: needs maintainer review before merge. :: none