Repository navigation
Judge Claude evals with Sonnet; count gradle-run once Gradle runs - #131
Conversation
…port forbidden actions per arm Refresh the Claude Code tables from a full Haiku 5.5 run judged by Sonnet 5.5 and add its run record. Treat gradle-run as expected in every automatic case where Gradle ran, report forbidden-action failures per arm, and tell shepherd-negative to use the supplied state without gh.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 59ef812113
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
 [Medium impact] The PR appears safe to merge; no new blocking issue was found.
|
Ignore Gradle commands in skipped shell branches and denied commands when deciding whether gradle-run is expected, and regrade the run record with the fix.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1acdb3b4ad
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a8a5244339
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 62133ec5c9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c969d2f7b1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 05867d74c7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 806fd91c43
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Stacked on #131; retarget to `main` after it merges. ## Summary Three minimal skill-text fixes for failures in the Haiku 5.5 / Sonnet-judged run (see the run record in #131): - `gradle-run`: rerunning an unchanged task under a relabeled question is still a repeat (`gradle-fingerprint-diagnosis-novel`, 2 automatic failures). - `kotlin-api-design` value classes: when callers must stay untouched, change only the declaration; keep the public constructor and add no private constructor or factory (`kotlin-api-value-class-direct`, 1 forced and 1 automatic failure). - `deliver-spec`: name conversation-mode `to-plan` as the initial route for an approved local source, including when answering a resume question (`deliver-spec-novel`, 2 forced failures). Versions are unchanged. The two single-record failures (`compose-state-authoring-novel`, `router-kotlin-gradle-validation`) are left alone as within noise. ## Test plan - `npm run lint`, `npm run evals:validate` (212 cases) and the eval unit tests (403) pass. `npm test` fails in this checkout only because the opencode plugin test cannot resolve `js-yaml` (no `node_modules`); it is unrelated to these edits. - A targeted live rerun of the affected skills (`gradle-run`, `kotlin-api-design`, `deliver-spec`, `shepherd`) will follow once approved, with results reported separately.
Stacked on #132 (which is stacked on #131); retarget to `main` as each merges. ## Summary Two small changes aimed at gaps the Haiku 5.5 / Sonnet-judged run found: - **Forbidden-action gate (Claude only):** the Claude subject prompt states that every Gradle command must include `--offline --no-scan` and, for edit cases, lists the only paths it may create or change (or says no file may change). Codex prompts are unchanged (a test asserts both sides). - **`kotlin-control-flow`:** guard conditions are stable from Kotlin 2.2; an explicit `languageVersion` is honored, otherwise the project's Kotlin version confirms support. ## Measured (compose rerun, Haiku subject, Sonnet judge) - Forbidden-action failures 19 -> 2 (none 7 -> 1, forced 2 -> 0, automatic 10 -> 1); the gate is still NOT MET by those two. - Automatic tokens per pass 139.3k -> 106.9k, tool calls per pass 9.4 -> 7.1. - Pass rates moved by 1 to 2 records per arm (noise at n=3). - Compose routing recall fell (83.3% -> 79.6% reported): the explicit flags in the prompt seem to let Haiku run `./gradlew` directly instead of loading `gradle-run`. A `gradle-run` description change meant to raise recall did not help and was reverted. Recall remains a known gap. - The `kotlin-control-flow` change is not yet measured; it lives in the kotlin-gradle suite. ## Test plan - `npm run lint`, `npm run evals:validate` (212 cases) and the eval unit tests pass.
1 participant
Summary
README.mdandevals/README.mdfrom a full Haiku 5.5 / high run judged by Sonnet 5.5 / high (the Sol judge was too slow), and add the run recordevals/artifacts/2026-10-19-claude-haiku-5-5-sonnet-judge-results.md.gradle-runas expected in every automatic-arm case where Gradle actually ran, not only cases that list it inallowed_skills. A Gradle command counts when it sat at the head of a command sequence (or its pipe partner, or afalse ||fallback), or when the whole command succeeded on a taken branch. A command in a skipped branch or a denied command does not count. Shell conditions are not evaluated (documentedshortcut:).gradle-run). Outcome pass rates are unchanged.shepherd-negativeprompt to say the supplied state is enough andghis not needed. This run used the old prompt.Test plan
npm run lint,npm run evals:validate(212 cases) and the eval unit tests pass; no live model calls were made by these changes.