feat(benchmark): add release evaluation pipeline

[DennisYu07]

Context

Qwen Code needs a reproducible and reviewable path from a published version to a public benchmark result. Before this change, the GitHub trigger, ECS execution prototype, and result display logic lived on separate branches and still included an unused HTTP control plane.

This PR consolidates them into one ECS-based release benchmark pipeline. It deliberately keeps the POC small: one self-hosted runner, one systemd worker, SQLite as the local queue/source of truth, Harbor as the Docker execution framework, and the triggering GitHub Release as the only public result surface.

End-to-end flow

stable GitHub release.published
        or
manual workflow_dispatch against an existing prerelease
        │
        ▼
GitHub Actions on [self-hosted, Linux, X64, qwen-benchmark]
  1. resolve the release tag through annotated tags to a 40-char commit SHA
  2. validate the allowlisted suite and release metadata
  3. call the root-owned local dispatcher through a narrow sudo rule
        │
        ▼
SQLite (WAL mode)
  - idempotent run creation
  - run / instance / event records
  - heartbeat, attempts, terminal state
        │
        ▼
systemd qwen-benchmark-worker
  - freeze the suite manifest
  - validate npm release availability
  - run Harbor + Docker testbed + requested Qwen Code npm version
  - collect the agent result and grader output
  - build summary and SHA-256 artifact manifest
        │
        ▼
GitHub Release publisher
  - verify release tag == benchmarked qwen_ref
  - require prerelease for manual publication
  - replace only the paired benchmark marker block
  - preserve all existing release notes

The GitHub job intentionally ends after durable enqueue. A successful Action means that the immutable version was validated and the request was accepted by SQLite; it does not claim that the multi-minute or multi-hour benchmark has already finished. The ECS worker owns execution, heartbeat, retry, completion validation, artifacts, and asynchronous Release publication.

Trigger behavior

Trigger	Version	Suite	Publication behavior
release.published	published stable Release tag	default swebench_verified_harbor_smoke in the POC	automatically eligible to update that Release
workflow_dispatch	explicit existing prerelease tag	explicit allowlisted suite	may update only the matching prerelease when release_id is supplied
prerelease release.published	prerelease tag	none	skipped to prevent an automatic public run

The production workflow is .github/workflows/benchmark-dispatch.yml. The manual job in stale.yml is a POC test trampoline because GitHub only registers a newly added workflow_dispatch workflow after it exists on the default branch. It allowed the exact PR implementation to be exercised before merge. This PR remains a draft while that POC-only entry is reviewed.

What is included

GitHub orchestration

• stable release.published trigger plus a manual prerelease path
• annotated-tag dereferencing to an immutable commit
• ten-minute dispatch timeout; the Action does not hold a runner during evaluation
• idempotency from repository, GitHub run ID, and run attempt
• a dedicated runner label and a single allowlisted sudo entry for /usr/local/sbin/qwen-benchmark-dispatch

Worker and state model

• SQLite WAL database with runs, instances, and append-only events
• systemd-managed worker with heartbeat and restart recovery
• state path:

QUEUED → PREPARING → RUNNING_AGENT → GRADING → UPLOADING
                                               ├─ SUCCEEDED
                                               └─ FAILED

A run is successful only when every instance frozen in its manifest has exactly one validated terminal result. Action completion, worker process liveness, or a partially written Harbor result is not treated as benchmark completion.

Harbor execution

The publishable POC suite runs the SWE-bench Verified task sympy__sympy-20590 through the open-source Harbor framework. Harbor starts the Docker testbed, installs @qwen-code/qwen-code@<release-version>, runs the Qwen Code agent, and invokes the verifier. Result parsing confirms that Harbor actually recorded the Qwen Code version derived from the requested Release tag.

The current POC executes one case at a time with Harbor --n-concurrent 1. It is intended to prove the control and data path, not to represent the resource model or runtime of the full 500-case SWE-bench Verified suite.

Artifact and publication contract

Private run artifacts contain:

• frozen request and suite manifest
• npm version metadata
• Harbor config, result, trial log, agent output, trajectory, and verifier report
• aggregate summary.json
• status.json
• checksums.sha256
• publisher-error.json only when GitHub publication fails

Only suites with publish: true may update GitHub. A successful public table contains:

• dataset and pinned revision
• suite and evaluation method
• completed, resolved, unresolved, and infrastructure-error counts
• score
• exact Qwen Code version and immutable commit
• ECS run ID

A failed terminal run publishes Failed — not scored and status/count information, but no numerical score. Raw trajectories, patches, Docker logs, secrets, internal paths, and model/provider configuration are never copied into the public Release.

Release content is bounded by paired markers:

<!-- qwen-code-benchmark:start -->
...
<!-- qwen-code-benchmark:end -->

A rerun replaces only this block and preserves content before and after it.

Retry and failure semantics

Automatic retry is limited to classified infrastructure failures, for example:

• transient model/provider 5xx or network failure
• Docker/Harbor startup failure before a valid trial exists
• interrupted worker attempt recoverable after restart

The POC allows one infrastructure retry (max_attempts=2). These benchmark outcomes are terminal and are not retried:

• valid but unresolved agent answer
• repository tests failing after the attempted patch
• normal task timeout
• malformed or semantically invalid agent result

A publication API failure does not rewrite the benchmark result. It is recorded separately as publisher-error.json; a durable publication outbox is a production follow-up.

Version and publication safety

• the workflow records both the human-readable release tag and immutable commit SHA
• the publisher fetches the target Release and rejects a mismatched tag_name
• manual dispatch can publish only to an existing prerelease
• gold and direct-Qwen diagnostic suites are non-publishable; only the Harbor smoke suite currently has publish: true
• no FastAPI service, public HTTP listener, Nginx route, webhook secret, release poller, or commit Check Run is introduced
• the GitHub runner does not receive the model key; model and publisher credentials remain in root-controlled ECS environment files

End-to-end validation

The exact PR head 1a1b0c233be182f101f3e90156721aadc5078ed8 was tested against an existing nightly prerelease. The test did not create a Release or tag and did not modify a stable Release.

Evidence	Result
GitHub Action	29990312519, success
ECS run	qwen-bench-fea3c395c9f04249, deduplicated=false
Test Release	v0.20.0-nightly.20260722.b98306b7e, prerelease=true
Evaluated Qwen commit	77115af615fca031a57505dee07deeaf702a0937
Dataset	swe-bench/swe-bench-verified@2
Case	sympy__sympy-20590
Terminal path	RUNNING_AGENT → GRADING → UPLOADING → SUCCEEDED
Runtime	4m51.7s from run creation to terminal state
Attempts	1; no retry or recovery
Result	1/1 completed, 1 resolved, 0 unresolved, 0 infrastructure errors
Score / Harbor reward	100.00% / 1.0
Agent result	exact nightly npm version, has_exception=false
Integrity	all 22 SHA-256 entries passed
Publication	Release GET 200, PATCH 200; existing notes preserved
Negative checks	no error.json, no publisher-error.json; worker remained active

The detailed evidence is also recorded in a separate PR comment so that future description edits do not erase the test trail.

Test coverage

• 16/16 unit tests passed on macOS
• 16/16 unit tests passed on the ECS Linux host
• YAML and TOML parsing passed
• shellcheck -e SC1090 benchmark-service/deploy/qwen-benchmark-dispatch passed
• staged-diff secret scan passed
• two consecutive clean self-audit passes completed after the final implementation change

The tests cover submit/idempotency, Harbor result parsing and version validation, worker state/failure handling, publishable-suite gating, Release tag/prerelease binding, paired-marker preservation, invalid Harbor tag handling, and checksum inclusion of publication errors.

Reviewer guide

Recommended review order:

1. .github/workflows/benchmark-dispatch.yml — trigger, immutable tag resolution, permissions, and dispatch boundary.
2. benchmark-service/qwen_benchmark/suites.json — allowlisted suites, pinned inputs, timeouts, and publication flags.
3. benchmark-service/qwen_benchmark/store.py — SQLite schema, idempotency, claiming, heartbeat, and state transitions.
4. benchmark-service/qwen_benchmark/worker.py — completion, retry, failure, checksums, and publication ordering.
5. benchmark-service/qwen_benchmark/harbor_runner.py — exact npm release installation and Harbor/grader parsing.
6. benchmark-service/qwen_benchmark/publisher.py — public schema, Release binding, and marker replacement.
7. benchmark-service/deploy/ — systemd, environment templates, Docker configuration, and restricted sudo command.
8. benchmark-service/tests/ and docs/design/qwen-code-benchmark-release-pipeline.md.

To reproduce the code-level checks:

python -m pytest benchmark-service/tests -q
shellcheck -e SC1090 benchmark-service/deploy/qwen-benchmark-dispatch

Deployment and rollback

The ECS deployment uses /srv/qwen-benchmark for source, state, artifacts, Harbor jobs, workspaces, and cache. The worker runs as qwen-benchmark-worker.service; the self-hosted runner account can invoke only the dispatcher through the provided sudoers rule.

For rollout:

1. install the package in the ECS virtual environment
2. install the dispatcher, sudoers rule, and systemd unit
3. configure root-controlled model and GitHub publication credentials
4. run the 16 tests on the host
5. restart the worker and verify active
6. use a prerelease smoke before enabling stable publication

Rollback is to stop the worker, restore the timestamped ECS worker backup and previous unit/config, restart the service, and revert this PR. SQLite state and historical artifacts are retained for diagnosis.

Known limitations and production follow-ups

• one ECS host, one worker, SQLite, and serial Harbor execution are POC constraints
• full-suite concurrency, sharding, resource classes, and multi-node scheduling are not implemented here
• multi-case policy such as “how many infrastructure failures make the whole 500-case run failed” remains intentionally undefined
• Release publication is best-effort and does not yet have a durable outbox
• raw artifacts currently remain on the ECS/private storage path; OSS lifecycle and public aggregate export are follow-ups
• Harbor's Docker backend may expand selected environment variables into child process arguments; the POC assumes a restricted single-tenant host, while production should use safer credential injection
• the POC-only stale.yml dispatch trampoline should be removed once the production workflow is registered on the default branch

---

中文摘要

本 PR 把之前分散的触发、ECS worker 和结果展示 POC 合并为一条可复现链路：GitHub Release → self-hosted ECS runner → SQLite → systemd worker → Harbor/Qwen Code Agent → grader → 校验和产物 → 同一个 GitHub Release 的公开汇总。

GitHub Action 只负责解析不可变版本并可靠入队，不会占用 runner 等待数小时；ECS worker 才负责 heartbeat、执行、retry、grader、完整性检查和最终回写。只有 manifest 中所有 case 都有唯一终态并通过校验后，才会发布成功分数。人工触发只能回写 tag 完全一致的 prerelease，正式 release.published 才进入稳定版本自动触发路径。

本次真实测试是 SWE-bench Verified 的单 case smoke，用于证明完整控制链路，不代表已经执行完整 500-case 数据集。现有 nightly prerelease 上的测试结果为 1/1 resolved、reward 1.0、100.00%，Action、ECS、Harbor、grader、22 项 checksum 和 Release 回写全部通过。

[DennisYu07]

End-to-end test report

The exact PR head 1a1b0c233be182f101f3e90156721aadc5078ed8 was tested against an existing nightly prerelease. No stable Release or new tag was created.

• GitHub Actions: run 29990312519, success
• Dispatch result: deduplicated=false, ECS run qwen-bench-fea3c395c9f04249
• Release: v0.20.0-nightly.20260722.b98306b7e, prerelease=true
• Evaluated Qwen Code commit: 77115af615fca031a57505dee07deeaf702a0937
• Suite/dataset: swebench_verified_harbor_smoke, swe-bench/swe-bench-verified@2
• Case: sympy__sympy-20590
• State path: RUNNING_AGENT → GRADING → UPLOADING → SUCCEEDED
• Runtime: 4m51.7s from run creation to terminal state
• Attempts: 1; no retry or recovery
• Result: 1/1 completed, 1 resolved, 0 unresolved, 0 infrastructure errors, score 100.00%
• Harbor trial: reward 1.0, has_exception=false, exact Qwen npm version matched the Release tag
• Artifact integrity: all 22 checksum entries passed
• Publication: Release GET 200, PATCH 200; paired benchmark markers present; existing release notes preserved
• Negative checks: no error.json, no publisher-error.json, worker remained active
• Test suites: 16/16 passed both locally and on ECS

This validates the requested path: Release/manual prerelease signal → GitHub Action → ECS self-hosted runner → SQLite worker → Harbor Qwen Code agent → grader → checksummed artifacts → public score in the same GitHub Release.

[gwinthis]

Architecture Review: COMMENT (C=0)

Summary

Large feature (+3330/-0) implementing an ECS-based release benchmark pipeline: GitHub Release trigger → SQLite queue → systemd worker → Harbor Docker execution → result published back to GitHub Release.

Architecture Assessment

Strengths:

1. Clear separation of concerns: GitHub Actions (trigger/validation), SQLite (durable queue), systemd worker (execution), Harbor (isolation). Each layer owns one responsibility.
2. Durable enqueue pattern: GitHub job ends after SQLite enqueue — doesn't hold a runner during multi-hour benchmark. "Successful Action" means "validated + accepted", not "finished".
3. Immutable version chain: annotated-tag dereferencing to 40-char commit SHA ensures the benchmarked version is immutable.
4. Idempotent run creation: SQLite WAL mode with run/instance/event records, heartbeat, attempts, terminal state.
5. Release tag verification: publisher verifies release tag == benchmarked qwen_ref; requires prerelease for manual publication.
6. Minimal POC scope: one runner, one suite, SQLite as local truth, Harbor as Docker framework — deliberately small.

Scope observations (non-blocking):

• Single self-hosted runner is a bottleneck for concurrent benchmarks — acceptable for POC
• SQLite is local to the runner — no multi-runner coordination yet
• The stale.yml POC test trampoline is a creative workaround for GitHub's workflow_dispatch registration limitation

Pattern

Durable enqueue for long-running CI jobs: When a CI trigger starts a multi-minute or multi-hour job, don't hold the runner. Validate + enqueue durably (SQLite), return success meaning "accepted", and let a separate worker own execution, heartbeat, retry, and completion. This separates trigger reliability from execution reliability.

中文说明

架构评审：COMMENT (C=0)

概要

大型功能（+3330/-0）：ECS 发布评测流水线。GitHub Release 触发 → SQLite 队列 → systemd worker → Harbor Docker 执行 → 结果发布回 GitHub Release。

架构评估

• 关注点分离： GitHub Actions（触发/验证）、SQLite（持久队列）、systemd worker（执行）、Harbor（隔离）
• 持久入队模式： GitHub job 在 SQLite 入队后结束——不占用 runner
• 不可变版本链： annotated-tag 解引用到 40 字符 commit SHA

模式

长时间 CI 作业的持久入队： 多分钟/多小时的 CI 作业不占用 runner。验证 + 持久入队（SQLite），返回"已接受"，让独立 worker 负责执行、心跳、重试和完成。

— qwen3.7-max via Qwen Code /review

[wenshao]

Review: release evaluation pipeline

Reviewed at head 1a1b0c233be182f101f3e90156721aadc5078ed8. I reproduced the test suite (16/16 pass on Python 3.12) and ran the repo's exact lint gates and two behavioural repros against the PR head.

This is a genuinely well-structured POC — the failure taxonomy (AgentError vs InfrastructureError), the "every manifest instance needs one validated terminal result" completion rule, WAL + BEGIN IMMEDIATE for the claim, atomic os.replace artifact writes, start_new_session + killpg for timeout cleanup, and publish-gating by suite are all sound. The workflows also get injection hygiene right (values go through env: and are referenced as "${VAR}", never interpolated into the run block), and actionlint is clean.

The blocking problems are in the trigger boundary and what reaches the public Release, plus three CI gates that were verified with different flags than CI uses.

---

Blocker 1 — the release trigger fires on every stable tag family, and writes to their release notes

benchmark-dispatch.yml filters release.published only on prerelease == false. This repo publishes far more than Qwen Code versions. Stable (non-prerelease) releases in the last 30 days:

13 x v0.19.3 ... v0.21.0        <- Qwen Code
     cua-driver-rs-v0.6.8
     desktop-v0.0.5
     sdk-typescript-v0.1.8
     pr6633-screenshots          <- ad-hoc asset-hosting tag

These events really do fire: release.yml:488 deliberately creates releases with CI_BOT_PAT, with the comment "CI_BOT_PAT required: GITHUB_TOKEN events cannot trigger downstream release-event workflows."

For a foreign tag the chain runs all the way to a public write. qwen_version_from_ref rejects the tag, so the run terminates as FAILED; the default suite (inputs.suite is empty on a release event) is the one with publish: true; and the publisher's only identity guard is release.tag_name == run.qwen_ref, which of course matches. I ran this for all four tags — every one issues a PATCH. Actual body written to cua-driver-rs-v0.6.8:

## What's Changed
* upstream notes here

<!-- qwen-code-benchmark:start -->
## Qwen Code benchmark

| Field | Result |
| --- | --- |
| Status | **Failed — not scored** |
| Dataset | `swe-bench/swe-bench-verified` at `2` |
| Suite | `swebench_verified_harbor_smoke` |
| Evaluation | Qwen Code agent (Harbor) |
| Cases | 0 / 1 completed |
| Score | Not published |
| Qwen Code | `cua-driver-rs-v0.6.8` · [`bbbbbbb`](.../commit/bbbb...) |
| Run | `qwen-bench-c10eb53c1ac54d50` |
<!-- qwen-code-benchmark:end -->

Suggested fixes, both cheap:

• Filter the trigger to the Qwen Code tag shape. The repo precedent is cd-cua-driver.yml: startsWith(github.ref, 'refs/tags/cua-driver-rs-v'). A guard like github.event.release.tag_name matching ^v[0-9] plus a github.repository == 'QwenLM/qwen-code' check (as in sync-release-to-oss.yml) covers it.
• Independently, make the publisher refuse to write when qwen_version_from_ref can't parse the ref. Tag equality is not the same as "this is a Qwen Code version", and defence in depth matters when the failure mode is writing to someone else's public release notes.

Blocker 2 — a single-case smoke publishes as "Score 100.00%" on the official release

Same default-suite path, but on a real v* release. This is what actually gets appended:

| Status | **Completed** |
| Dataset | `swe-bench/swe-bench-verified` at `2` |
| Suite | `swebench_verified_harbor_smoke` |
| Evaluation | Qwen Code agent (Harbor) |
| Cases | 1 / 1 completed |
| Results | 1 resolved · 0 unresolved · 0 infrastructure errors |
| Score | **100.00%** |

The PR description is careful that the smoke "is not intended to represent the full 500-case suite", but the published artifact carries none of that framing. A reader sees "Qwen Code benchmark", dataset swe-bench/swe-bench-verified, "Score 100.00%" on an official release. The only hint is the word smoke inside the suite slug. That will be quoted as a SWE-bench Verified result.

I'd set publish: false on swebench_verified_harbor_smoke until the real suite exists, and keep prerelease workflow_dispatch for validation (which already works, per your nightly test). If you want to keep publishing, _release_section needs an explicit caveat row and the score should read 1 / 500 rather than a bare percentage.

Blocker 3 — three CI gates fail; two were verified with non-CI flags

yamllint (this is the current red check, 17 errors). .yamllint.yml sets quoted-strings: {quote-type: single, required: true}:

benchmark-dispatch.yml:5    types: [published]
benchmark-dispatch.yml:30   runs-on: [self-hosted, Linux, X64, qwen-benchmark]   (4 errors)
stale.yml:14,19,24,32       type: boolean / string / choice / string
stale.yml:76                runs-on: [self-hosted, ...]                          (4 errors)
stale.yml:79                contents: read
stale.yml:83                id: revision
stale.yml:84                shell: bash
stale.yml:115               TRIGGER: workflow_dispatch

Every value needs single quotes (types: ['published'], shell: 'bash', …). Dropping the stale.yml trampoline removes 12 of the 17.

prettier — scripts/lint.js runs prettier --write . over the whole repo and .prettierignore does not exclude benchmark-service/. benchmark-service/README.md fails --check today; prettier wants the CJK config tables column-aligned. npx prettier --write benchmark-service/README.md fixes it.

shellcheck — the PR verified with shellcheck -e SC1090, but CI runs --check-sourced --enable=all --exclude=SC2002,SC2129,SC2310. SC1090 is not excluded. The file is extensionless but file --mime-type reports text/x-shellscript, so it is in scope:

qwen-benchmark-dispatch:7:8  SC1090  can't follow non-constant source
qwen-benchmark-dispatch:7:9  SC2250  prefer braces around variable references
qwen-benchmark-dispatch:15:10 SC2312 consider invoking separately to avoid masking return value

A # shellcheck source=/dev/null directive plus ${CONFIG} handles the first two.

---

Medium

_release_section crashes on a NULL qwen_commit, so terminal failures silently never publish. publisher.py:104 does commit[:7]; when run.qwen_commit is NULL that raises TypeError: 'NoneType' object is not subscriptable, _publish swallows it, and the "Failed — not scored" notice never reaches the Release. Worse, publisher-error.json then records the TypeError instead of the real cause.

Your own test_invalid_harbor_release_ref_becomes_a_terminal_failure walks this exact path and passes only because it sets github_token=None, which makes publish_release return SKIPPED before formatting anything. Changing that single line to github_token="token" — i.e. production — gives:

run status      : FAILED
real error      : Harbor runs require a Qwen Code release tag such as v0.19.7
qwen_commit     : None
Release PATCHed : False
publisher-error : 'NoneType' object is not subscriptable

--qwen-commit is optional on the CLI, and worker.run_once has an explicit request_commit or runner.resolve_qwen_commit(...) fallback precisely for that case, so NULL is a supported state. Guard the commit cell and add the token to that test.

The worker hands its entire environment, including BENCHMARK_GITHUB_TOKEN, to the process that runs untrusted agent code. The systemd unit loads benchmark.secret.env (the token with Contents read/write on this repo) into the worker environment, and both harbor_runner._command and runner._command do env = os.environ.copy(). The README already warns that "Harbor's Docker backend may expand selected environment variables into child process arguments" — that warning applies to the GitHub token as much as to the model key, and the token is the more damaging of the two. Build an explicit allowlist env for the Harbor subprocess and keep the GitHub token out of it; the publisher can read it directly from Settings.

The 16 tests never run in CI. benchmark-service appears nowhere in .github/, package.json, or scripts/, and there is no ruff/black/mypy config. This is the repo's first substantial Python tree, and right now nothing stops it from silently rotting. A small workflow filtered on benchmark-service/** would cover it.

---

Smaller items

• Drop the stale.yml trampoline before merge (you flag this yourself). Beyond being 65 duplicated lines grafted onto the stale-bot workflow, it has already drifted: it defaults suite to swebench_verified_gold_smoke while benchmark-dispatch.yml defaults to the harbor suite.
• README install order creates a root-code-execution path. sudo chown -R ecs-user:ecs-user /srv/qwen-benchmark (line 302) makes config/ writable by ecs-user, and benchmark.env is installed into it afterwards. Directory write permission allows unlink-and-recreate regardless of the file's root:root 0644, and the dispatcher sources that file as root under sudo. Chown only the subdirectories the worker actually writes (state, artifacts, workspaces, harbor, cache) and leave config/ root-owned.
• The legacy-marker branch discards release notes. existing_body.split(LEGACY_MARKER, 1)[0] drops everything after the marker, unlike the paired-marker branch which preserves the suffix. Destructive on a public release body — either give the legacy marker the same preserve-suffix treatment or delete the branch.
• A post-success bookkeeping error downgrades a real result. After the SUCCEEDED transition, _write_status can raise OSError; the generic except Exception then transitions the run to FAILED and publishes "Failed — not scored" over a genuine score. Narrow that handler, or bail out once the run is already terminal.
• Settings disagrees with itself on the npm registry: the dataclass default is registry.npmjs.org, from_env's default is registry.npmmirror.com. Tests exercise one and production the other.
• _wait_for_npm_release uses the total budget as its poll interval: time.sleep(min(30, max(npm_wait_seconds, 1))). With the default 600 that happens to yield 30s, but the poll interval should be its own constant.
• store.cancel() and store.retry() are unreachable — no CLI or caller invokes them.
• raise AgentError(f"Harbor trial timed out: {command[5]}") indexes the command list positionally; reordering an argument silently mislabels the error. Pass harbor_task_name in directly.
• load_suites() returns unvalidated JSON. The Suite TypedDict isn't enforced, so a suite added without publish raises KeyError at publish time — the worst place to find out. pydantic is already a dependency.
• write_checksums reads every artifact fully into memory (path.read_bytes()) after copy_tree has copied the whole Harbor job tree. Fine at one case; at 500 it will need chunked hashing.
• docker-daemon.ecs.json routes testbed images through third-party mirrors (docker.m.daocloud.io, docker.1ms.run). Worth a second look given these images produce numbers published as official scores.
• QWEN_REF is interpolated into a GitHub API URL without encoding. A tag containing ../ could redirect the ref lookup. Dispatch requires write access so this is low, but the tag-shape check belongs before the curl rather than in RunRequest afterwards.

---

Overlap with #7656

#7656 (same author, also draft) is another release.published-triggered SWE-bench pipeline with the same objective, and it has the same unfiltered-trigger and yamllint problems. Landing both would give the repo two independent benchmark pipelines racing on every release. Worth deciding which one is the path forward before either merges.

Verdict

Request changes. Blockers 1 and 2 both end in a public write that misrepresents things — one to unrelated products' release notes, one as an unqualified 100% SWE-bench score — and I'd want those closed before this leaves draft. Blocker 3 is mechanical. The underlying worker, state machine, and artifact design look solid and I don't think they need rework.

中文摘要

在 head 1a1b0c2 上复核：16/16 测试本地通过（Python 3.12），actionlint 干净，workflow 的注入防护（env: + "${VAR}"）写法正确，worker 状态机、完成判定、产物原子写入和超时清理都做得不错。

三个阻塞项：

1. release 触发器会命中所有 stable tag 家族，并写入它们的 release notes。 只按 prerelease == false 过滤。近 30 天 17 个 stable release 中有 4 个不是 Qwen Code：cua-driver-rs-v0.6.8、desktop-v0.0.5、sdk-typescript-v0.1.8、pr6633-screenshots。release.yml:488 明确用 CI_BOT_PAT 就是为了让 release 事件能触发下游 workflow，所以确实会触发。这四个 tag 我都复现了：全部会 PATCH 一张 "Qwen Code benchmark / Failed — not scored" 表格到对方的公开 release notes。建议按 tag 形状过滤（参考 cd-cua-driver.yml），并让 publisher 在 qwen_version_from_ref 解析失败时拒绝回写。
2. 单 case smoke 会以 "Score 100.00%" 发布到正式 release。 表格里 Dataset 写的是 swe-bench/swe-bench-verified，Score 写 100.00%，唯一提示只有 suite 名里的 smoke。建议先把该 suite 设为 publish: false，正式 suite 就绪前只用 prerelease workflow_dispatch 验证。
3. 三个 CI gate 红。 yamllint 17 个 quoted-strings 错误（当前 CI 失败原因，行号见上）；prettier 对 benchmark-service/README.md 不通过（.prettierignore 没有排除该目录）；shellcheck 用仓库配置（--enable=all，未排除 SC1090）会报 SC1090/SC2250/SC2312——PR 自测用的 -e SC1090 与 CI 配置不一致。

中等问题： publisher.py:104 在 qwen_commit 为 NULL 时抛 TypeError，导致终态失败永远发不出去，且 publisher-error.json 记录的是 TypeError 而非真实原因——PR 自带的 test_invalid_harbor_release_ref_becomes_a_terminal_failure 正好走这条路径，只因为 github_token=None 提前 SKIPPED 才通过；worker 通过 os.environ.copy() 把含 BENCHMARK_GITHUB_TOKEN（本仓库 Contents 读写）的完整环境传给运行不可信 agent 代码的 Harbor 子进程；16 个测试完全没有接入 CI。

其他： stale.yml 跳板应在合并前删除（且已与正式 workflow 的默认 suite 产生分歧）；README 的 chown -R ecs-user 让 config/ 可被 ecs-user 替换，而 dispatcher 以 root source 该文件；legacy marker 分支会丢弃 marker 之后的全部 release notes；成功后的 _write_status 报错会把 SUCCEEDED 降级成 FAILED 并覆盖真实分数；Settings 的 npm registry 默认值自相矛盾；cancel()/retry() 无人调用；command[5] 位置索引脆弱；load_suites() 未校验；write_checksums 全量读入内存；Docker 第三方镜像源；QWEN_REF 未编码即拼进 API URL。

另外 #7656 与本 PR 目标重合且同样是 release.published 触发，建议先确定保留哪一条链路。

结论： Request changes。阻塞项 1、2 都会造成对外的错误公开写入，建议在退出 draft 前解决；阻塞项 3 是机械修复。底层 worker / 状态机 / 产物设计不需要返工。