ci: add isolated DSW SWE-bench release pipeline#7656
Conversation
|
Full 500-case DSW release-chain run completed and was written back to the test prerelease.
Public JSON validation passed: 500 records, 500 unique instance IDs, no duplicate IDs, and state totals match the Release summary. DSW SSH is currently unreachable, so final PostgreSQL-to-OSS alignment and unified root-cause analysis of the 56 execution errors / 5 infrastructure failures remain pending host access restoration. This was a non-production prerelease validation; do not merge based on the quarantined score. |
Review:
|
What this PR does
This PR adds an isolated, asynchronous release benchmark pipeline:
GitHub Release -> short DSW dispatch job -> PostgreSQL queue -> persistent Coordinator + 10 Harbor executors -> SWE grader -> OSS -> persistent Publisher -> originating ReleaseFor every
release.publishedevent, the self-hosted DSW runner resolves the immutable Qwen Code tag/commit, freezes all 500 SWE-bench Verified instances, inserts the run plus Release callback metadata into PostgreSQL, records the poolrun_id, and exits. The GitHub Actions job does not remain active for the benchmark duration.The persistent DSW services then own execution and publication:
Harbor live trial directories use local NVMe because OSS FUSE does not support the POSIX
chmodoperations Harbor performs during trial initialization. Each completed attempt is copied to the dedicated OSS prefix. A score is published only when all 500 frozen instances have a unique terminal result and the run isSUCCEEDED; a quarantined run reports status/counts without a score.This implementation is isolated from #7584 through a new workflow, runner label
qwen-benchmark-dsw, suitedsw_release_swe_verified_v1, PostgreSQL databaseqwen_benchmark_dsw_release_v1, runtime root/mnt/workspace/qwen-benchmark-dsw-release-v1, OSS prefix/mnt/data/qwen-benchmark/dsw-release-v1, and dedicated Release markers.Why it's needed
Qwen Code releases need a public, reproducible score and a reliable chain from release publication to execution and presentation. Keeping Actions alive for a 12–16 hour benchmark couples orchestration to a temporary CI process and temporary
github.token. The asynchronous design lets GitHub perform only authenticated dispatch while the DSW pool owns long-running scheduling, recovery, validation, and final writeback.Reviewer Test Plan
How to verify
git diff --check, and a secret-pattern scan.self-hosted,Linux,X64, andqwen-benchmark-dsw.release.publishedautomatically starts the branch workflow without manual DSW dispatch.executor_count=10.SUCCEEDEDrun publishes version/commit, dataset revision, 500/500 completion, resolved/unresolved/error counts, score, run ID, and a JSON asset. Confirm aQUARANTINEDrun publishes no score.Current observed validation:
sympy__sympy-20590completedSUCCEEDED/RESOLVED, reward1.0, 40 trajectory steps, expected nightly Qwen Code version, and complete verifier output.release.publishedworkflow automatically.chmodincompatibility before container start. The run was stopped and quarantined; live Harbor work has been moved to local NVMe with per-attempt OSS archival.Evidence (Before & After)
N/A — CI and benchmark infrastructure only; no TUI behavior changes.
Tested on
Environment (optional)
Alibaba Cloud PAI DSW, 16 vCPU / 64 GiB, local NVMe Docker storage, OSS mounted at
/mnt/data, PostgreSQL 16, Harbor Docker testbeds, Qwen Codev0.20.0-nightly.20260722.b98306b7e, modelqwen3.7-max.Risk & Scope
Linked Issues
Related to #7584, with intentionally separate code, state, data, and experiments.
中文说明
本 PR 做了什么
本 PR 新增一条相互隔离的异步 Release 跑分链路:
GitHub Release -> DSW 短派发任务 -> PostgreSQL 队列 -> 常驻 Coordinator + 10 个 Harbor Executor -> SWE grader -> OSS -> 常驻 Publisher -> 原 Releaserelease.published触发后,DSW self-hosted runner 只负责解析不可变的 Qwen Code tag/commit、冻结 500 条 SWE-bench Verified manifest、把 run 和 Release 回调信息写入 PostgreSQL,并记录run_id后结束。GitHub Actions 不等待 12–16 小时的完整跑分。后续由 DSW 常驻服务负责:
Harbor 实时工作目录使用本地 NVMe,因为 OSS FUSE 不支持 Harbor 初始化 trial 时执行的 POSIX
chmod。每个 attempt 完成后逐文件归档到独立 OSS 前缀。只有 500 个冻结实例全部具有唯一终态且 run 为SUCCEEDED才公布分数;QUARANTINED只公布状态和数量,不公布分数。本实现通过新的 workflow、
qwen-benchmark-dswrunner 标签、suitedsw_release_swe_verified_v1、数据库qwen_benchmark_dsw_release_v1、runtime/mnt/workspace/qwen-benchmark-dsw-release-v1、OSS/mnt/data/qwen-benchmark/dsw-release-v1和独立 Release 标记,与 #7584 完全隔离。为什么需要
Release 跑分需要从发布、调度、执行、校验到公开展示的完整链路。让 GitHub Actions 持续等待长任务,会把运行生命周期绑定到临时 CI 进程和临时
github.token。异步方案让 GitHub 只做鉴权派发,DSW 资源池独立负责长任务调度、恢复、发布门禁和最终主动回写。Reviewer 测试计划
如何验证
qwen-benchmark-dsw标签。release.published自动触发,不人工 dispatch DSW。当前验证:
sympy__sympy-20590为SUCCEEDED/RESOLVED,reward1.0,40 个 trajectory step,Qwen Code nightly 版本正确,verifier 产物完整。release.publishedworkflow。chmod不兼容,发生在容器启动前;该运行已停止并隔离。Harbor 实时目录现已迁移到本地 NVMe,并在 attempt 结束后归档 OSS。风险与范围