Skip to content

Bump claude-agent-acp pin to 0.81.0 for claude-opus-5-5 - #1184

Open
shubham2345 wants to merge 1 commit into
benchflow-ai:mainfrom
shubham2345:feature/acp_versioning
Open

shubham2345 wants to merge 1 commit into
benchflow-ai:mainfrom
shubham2345:feature/acp_versioning

Conversation

@shubham2345

@shubham2345 shubham2345 commented Oct 15, 2026 •

Copy link
Copy Markdown

Fixes #1137

Problem

bench eval run --agent claude-agent-acp --model claude-opus-5-5 fails on every turn with ACP error -32603 ... API Error: 400 Claude Code 2.1.257 does not support this model; version 2.1.280 or newer is required.

The _CLAUDE_AGENT_ACP_PACKAGE pin (0.73.0) bundles claude-agent-sdk 0.3.257, i.e. Claude Code 2.1.257, which is too old for claude-opus-5-5.

Change

  • src/benchflow/agents/registry.py: raise the pin to @agentclientprotocol/claude-agent-acp@0.81.0 (sdk 0.3.280, Claude Code 2.1.280) and update the explanatory comment.
  • tests/test_agent_registry.py: update the pin assertion.
  • CHANGELOG.md: add a Fixed entry.

Verification

  • tests/test_agent_registry.py: 31 passed.
  • Direct ACP check on 0.81.0 with a valid key: selecting claude-opus-5-5 and claude-fable-5-1 succeeds and a one-word prompt returns end_turn. On 0.73.0, selecting claude-opus-5-5 fails with Invalid value for config option model.
  • Not run: a full bench eval run through the runner and a sandbox.

Notes

  • The gated live guard (tests/test_acp_pinned_protocol_guard.py, RUN_ACP_DEP_GUARD=1) is unchanged. On 0.81.0 its Fable check needs ANTHROPIC_API_KEY, because the adapter validates a concrete model against the API when it is selected.
  • Not included: the optional env override (BENCHFLOW_CLAUDE_AGENT_ACP_VERSION) mentioned in the issue; this PR only raises the pin.

@shubham2345
shubham2345 force-pushed the feature/acp_versioning branch 2 times, most recently from 20209fe to 5e2f4ac Compare October 15, 2026 07:11
0.73.0 bundles Claude Code 2.1.257, which claude-opus-5-5 rejects (needs
>= 2.1.280). 0.81.0 bundles sdk 0.3.280 / Claude Code 2.1.280.

@Benjamin-eecs Benjamin-eecs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-binding review from the #1137 reporter: this matches what FrontierPhysics ran.

  • FrontierPhysics has run its claude-opus-5-5 cells on claude-agent-acp 0.81.0 since 2026-09-23 through a local re-pin of this constant (FrontierPhysics-Infra 3e20674, FPINFRA_CLAUDE_AGENT_ACP_VERSION=0.81.0), through the full bench eval run path on Daytona and on docker. agent/runtime.json records sandbox.claude_code_version = 2.1.280, and the model and effort config options took effect. That covers the "Not run" item in the description.
  • 0.81.0 does not change #1138: a large image read still produces multi-MB ACP lines, as on 0.73.0.
  • release/0.8 solves #1137 differently (6bd05f2: adapter 0.81.2 plus @anthropic-ai/claude-code@2.1.280 through CLAUDE_CODE_EXECUTABLE). For main, this one-line bump is enough; PR #1139 is the same bump and could be closed in favour of this one.
  • Optional follow-up, not needed here: an env override for the adapter (or CLI) version, so the next model that needs a newer Claude Code does not wait for a release.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

claude-agent-acp pin 0.73.0 bundles Claude Code 2.1.257; claude-opus-5-5 requires 2.1.280 (0.81.0)

2 participants