Skip to content

fix: accept Claude native trajectories - #52

Open
vincentkoc wants to merge 1 commit into
mainfrom
fix/native-claude-trajectory-eligibility
Open

fix: accept Claude native trajectories#52
vincentkoc wants to merge 1 commit into
mainfrom
fix/native-claude-trajectory-eligibility

Conversation

@vincentkoc

@vincentkoc vincentkoc commented Jul 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

What does this PR do?

Marks Claude Code native trajectories as real harness events so completed Claude runs remain eligible for native aggregation.

Why?

The native runner already converts Claude Code stream-json output into trajectory.json, but the harness allowlist omitted claude-code. Aggregation therefore classified otherwise valid Claude runs as trajectory_unavailable.

Changes

  • add claude-code to the real-trajectory harness allowlist
  • assert every pinned native harness reports real_harness_events

Live proof

A native Claude Code control run completed four tasks with four result files, four valid completed results, and no infrastructure failures. Aggregation classified every trajectory as real; the run remained ineligible only because it was intentionally marked as an exploratory four-task subset.

  • coverage: 4/4
  • score: 0.75
  • exact passes: 3
  • trajectory complete: true
  • trajectory status: real for all four tasks
  • infrastructure failures: 0
  • exclusion reason: exploratory_subset

Tests

  • python -m pytest -q passes locally (444 passed, 5 skipped)
  • python -m ruff check clawbench app.py scripts tests passes locally
  • live Claude Code native run aggregates as real harness events

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P2 Normal priority bug or improvement with limited blast radius. labels Jul 29, 2026
@clawsweeper

clawsweeper Bot commented Jul 29, 2026 •
edited
Loading

Copy link
Copy Markdown

Codex review: needs maintainer review before merge. Reviewed August 6, 2026, 9:16 AM ET / 13:16 UTC.

ClawSweeper review

What this changes

Adds Claude Code to the native harnesses whose converted trajectories count as real events, with regression coverage for all pinned native harnesses.

Merge readiness

⚠️ Ready for maintainer review - 3 items remain

Keep open for maintainer review: this member-authored PR makes a narrow, source-supported correction to native trajectory eligibility, with after-fix control-run evidence.

Likely related people: vincentkoc (high confidence; recent native-evaluation contributor).

Priority: P2
Reviewed head: ac1c974c12749dddb5d49af734561cec59910c65
Owner decision: Required. See Decision needed.

Review scores

Measure Result What it means
Overall readiness 🦞 diamond lobster (5/6) A focused, validated eligibility correction with concrete after-fix native-run evidence; only evidence-standard acceptance remains.
Proof confidence 🦞 diamond lobster (5/6) Sufficient (live_output): The PR body supplies after-fix native Claude Code control-run results showing all four converted trajectories aggregated as real; no sensitive runtime details are exposed.
Patch quality 🦞 diamond lobster (5/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Verified Sufficient (live_output): The PR body supplies after-fix native Claude Code control-run results showing all four converted trajectories aggregated as real; no sensitive runtime details are exposed.
Evidence reviewed 5 items Current main behavior: Claude Code is a pinned native harness and has a dedicated trajectory converter, but current main's real-trajectory allowlist omits it; the manifest therefore reports an unsupported trajectory mode.
Aggregation boundary: Native aggregation accepts completed trajectories only when the manifest mode is real_harness_events and every required trajectory is real; otherwise it excludes the run as trajectory_unavailable.
Claude conversion validation: Current tests already validate that Claude stream-json converts to a real trajectory, preserves canonical identity, and records a terminal event.
Findings None None.
Security None None.

How this fits together

ShellBench's native evaluator runs agent harnesses and aggregates result and trajectory artifacts into benchmark eligibility. The harness classification determines whether completed runs contribute real trajectory evidence to aggregation.

flowchart LR
A[Native agent run] --> B[Claude Code stream output]
B --> C[Trajectory conversion]
C --> D[Harness classification]
D --> E[Native aggregation]
E --> F[Benchmark eligibility]
Loading

Decision needed

Question Recommendation
Should completed Claude Code runs whose stream-json output passes the existing converter validation count as real native trajectory evidence? Accept converted Claude trajectories: Merge the classification change so completed validated Claude Code runs participate in native aggregation as real evidence.

Why: Source proves the allowlist and converter are inconsistent, but accepting converted traces for benchmark eligibility is a maintainer-owned evidence-standard decision.

Before merge

  • Resolve merge risk (P1) - Merging changes benchmark-evidence eligibility: completed Claude Code runs with converted stream-json traces will no longer be excluded solely by the harness-mode gate.
  • Complete next step (P2) - A benchmark owner should confirm the evidence-eligibility standard before merge; no mechanical repair remains.
Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Patch scope 2 files affected; production +1/-1, tests +12 The PR is limited to the classification gate and an all-harness regression assertion.

Merge-risk options

Maintainer options:

  1. Confirm eligibility standard before merge (recommended)
    Merge after a benchmark owner confirms that the present Claude stream-json conversion is sufficient for real native trajectory aggregation.
  2. Keep the current gate
    Pause the PR if maintainers require a stronger Claude-specific trajectory validation boundary before changing aggregate eligibility.

Technical review

Best possible solution:

Merge the narrow allowlist correction after a benchmark owner confirms that the existing Claude trajectory converter meets the intended real-evidence standard.

Do we have a high-confidence way to reproduce the issue?

Yes—current source shows Claude Code produces validated real trajectories while its manifest mode remains unsupported, and the PR reports an after-fix four-task native control run.

Is this the best way to solve the issue?

Unclear—the implementation is the narrowest consistency repair, but maintainers must decide whether the existing converter validation is sufficient for benchmark-evidence eligibility.

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label justifications:

  • P2: This is a bounded benchmark-evaluation correctness change with limited user blast radius.
  • merge-risk: 🚨 other: The PR changes which completed runs qualify as real benchmark evidence, a policy-sensitive aggregation outcome not settled by CI alone.
  • rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body supplies after-fix native Claude Code control-run results showing all four converted trajectories aggregated as real; no sensitive runtime details are exposed.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body supplies after-fix native Claude Code control-run results showing all four converted trajectories aggregated as real; no sensitive runtime details are exposed.

Evidence

What I checked:

  • Current main behavior: Claude Code is a pinned native harness and has a dedicated trajectory converter, but current main's real-trajectory allowlist omits it; the manifest therefore reports an unsupported trajectory mode. (scripts/native_eval/models.py:102, 884dd1bb5511)
  • Aggregation boundary: Native aggregation accepts completed trajectories only when the manifest mode is real_harness_events and every required trajectory is real; otherwise it excludes the run as trajectory_unavailable. (scripts/native_eval/aggregate.py:906, 884dd1bb5511)
  • Claude conversion validation: Current tests already validate that Claude stream-json converts to a real trajectory, preserves canonical identity, and records a terminal event. (tests/test_native_eval_runner.py:1783, 884dd1bb5511)
  • Proposed patch and proof: The PR adds only Claude Code to the allowlist and asserts every pinned harness has real events; its body reports a four-task native control run with four completed results, no infrastructure failures, and real trajectory status for all tasks. (scripts/native_eval/models.py:102, ac1c974c1274)
  • Area provenance: Recent native-evaluation implementation and stabilization commits covering this path were authored by the same contributor, including the current main native-evaluation history. (scripts/native_eval/models.py:93, b9acd9f7a010)

Likely related people:

  • vincentkoc: Authored the native matrix, trajectory stabilization, runbook, and proposed eligibility work on the central files. (role: recent native-evaluation contributor; confidence: high; commits: 69f75c6629c4, b9acd9f7a010, ac1c974c1274; files: scripts/native_eval/models.py, scripts/native_eval/harness_trajectories.py, tests/test_native_eval_runner.py)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (36 earlier review cycles; latest 8 shown)
  • reviewed 2026-08-05T08:36:32.063Z sha ac1c974 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-05T12:08:04.368Z sha ac1c974 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-05T12:31:14.847Z sha ac1c974 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-05T19:41:07.847Z sha ac1c974 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-05T20:51:51.907Z sha ac1c974 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-05T23:38:36.142Z sha ac1c974 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-06T07:18:36.065Z sha ac1c974 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-06T10:16:21.129Z sha ac1c974 :: needs maintainer review before merge. :: none

@vincentkoc
vincentkoc marked this pull request as ready for review July 29, 2026 17:17
@vincentkoc
vincentkoc requested a review from a team as a code owner July 29, 2026 17:17
@clawsweeper clawsweeper Bot added proof: sufficient Contributor real behavior proof is sufficient. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. merge-risk: 🚨 other 🚨 Merging this PR has meaningful risk outside the owned taxonomy. and removed proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. labels Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

merge-risk: 🚨 other 🚨 Merging this PR has meaningful risk outside the owned taxonomy. P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@vincentkoc