feat(benchmark): add release evaluation pipeline

[DennisYu07]

Summary

• add a stable-release GitHub Actions workflow that dispatches benchmark runs to the dedicated ECS self-hosted runner
• add the single-node worker, SQLite state machine, Harbor adapter, artifact/checksum handling, retry policy, and Release-only result publisher
• bind manual publication to the selected prerelease tag and preserve all non-benchmark Release content with paired markers
• keep the POC surface private: no FastAPI, public listener, polling service, or commit Check Run
• document deployment, suites, state transitions, failure semantics, security boundaries, and operator workflow

Why

Qwen Code needs a reproducible path from a published version to a public benchmark result. This PR consolidates the earlier trigger and display prototypes into one reviewable change based on current main. A stable release.published event dispatches the configured suite automatically; workflow_dispatch is limited to prerelease testing.

Reviewer test plan

1. Review .github/workflows/benchmark-dispatch.yml and the temporary manual POC entry in stale.yml.
2. Review suite publication flags and Release/tag binding in benchmark-service/qwen_benchmark/publisher.py.
3. Run python -m pytest benchmark-service/tests -q.
4. Run shellcheck -e SC1090 benchmark-service/deploy/qwen-benchmark-dispatch.
5. On the ECS host, deploy the worker, restart qwen-benchmark-worker, and dispatch swebench_verified_harbor_smoke against an existing nightly prerelease.
6. Confirm Action success, a terminal SQLite run, Harbor reward/artifacts, checksum verification, and paired benchmark markers in that same prerelease body.

Validation

• macOS: 16 unit tests passed; YAML/TOML parsing, ShellCheck, secret scan, and two clean self-audit passes completed
• Linux ECS: 16 unit tests passed; qwen-benchmark-worker is active after restart
• full E2E Action: 29990312519, success, head SHA 1a1b0c233be182f101f3e90156721aadc5078ed8
• ECS run: qwen-bench-fea3c395c9f04249, one attempt, SUCCEEDED, no recovery or error artifact
• nightly prerelease: v0.20.0-nightly.20260722.b98306b7e; no Release was created by this test
• exact evaluated Qwen commit: 77115af615fca031a57505dee07deeaf702a0937
• SWE-bench Verified smoke result: 1/1 completed and resolved, reward 1.0, score 100.00%, 0 infrastructure errors
• all 22 checksum entries passed; Release API GET 200 and PATCH 200; no publisher error
• Windows: not applicable to this Linux ECS/Docker worker

Risks and rollback

• stable releases will enqueue the configured suite after merge; suite publication is controlled in suites.json
• publication requires the Release tag to exactly match the requested Qwen version; manual dispatch additionally requires a prerelease
• retry is restricted to classified infrastructure failures; agent/test failures are terminal
• rollback is the prior ECS worker backup plus reverting this commit

---

中文说明

本 PR 将之前分散的触发与结果展示 POC 合并成一条基于最新 main 的完整链路：GitHub Release 发布后由自托管 ECS runner 派发，worker 通过 Harbor 执行指定版本 Qwen Code 的 SWE-bench 示例，SQLite 管理状态，最终仅将审核后的汇总结果写回同一个 GitHub Release。

人工测试只能写回 tag 完全一致的 prerelease；正式 release.published 才走稳定版自动触发。服务不开放 FastAPI 或公网端口，也不创建每次运行一个 PR。本次已用现有 nightly prerelease 完成真实端到端验证：Action、ECS/Harbor、grader、校验和以及 Release 回写全部通过。

[DennisYu07]

End-to-end test report

The consolidated PR branch was tested against an existing nightly prerelease. No stable Release or new tag was created.

• PR head used by Actions: 1a1b0c233be182f101f3e90156721aadc5078ed8
• GitHub Actions: run 29990312519, success
• Dispatch result: deduplicated=false, ECS run qwen-bench-fea3c395c9f04249
• Release: v0.20.0-nightly.20260722.b98306b7e, prerelease=true
• Evaluated Qwen Code commit: 77115af615fca031a57505dee07deeaf702a0937
• Suite/dataset: swebench_verified_harbor_smoke, swe-bench/swe-bench-verified@2
• Case: sympy__sympy-20590
• State path: RUNNING_AGENT → GRADING → UPLOADING → SUCCEEDED
• Runtime: 4m51.7s from run creation to terminal state
• Attempts: 1; no retry or recovery
• Result: 1/1 completed, 1 resolved, 0 unresolved, 0 infrastructure errors, score 100.00%
• Harbor trial: reward 1.0, has_exception=false, exact Qwen npm version matched the Release tag
• Artifact integrity: all 22 checksum entries passed
• Publication: Release GET 200, PATCH 200; paired benchmark markers present; existing release notes preserved
• Negative checks: no error.json, no publisher-error.json, worker remained active
• Test suites: 16/16 passed both locally and on ECS

This validates the requested path: Release/manual prerelease signal → GitHub Action → ECS self-hosted runner → SQLite worker → Harbor Qwen Code agent → grader → checksummed artifacts → public score in the same GitHub Release.

[DennisYu07]

Closing this draft in favor of #7584. GitHub kept this PR's head metadata at the pre-rebase SHA even though the organization branch and E2E Action had advanced. #7584 points directly to the tested, current-main-based SHA 1a1b0c233be182f101f3e90156721aadc5078ed8.