Skip to content

fix(release): recover immutable failed tag #542

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jeremi merged 1 commit into from
Jul 28, 2026
Merged

fix(release): recover immutable failed tag #542

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 25 additions & 7 deletions .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,14 @@ on:
push:
tags:
- "v*"
repository_dispatch:
types: [release_recovery]
Comment on lines +7 to +8

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 28, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Align dispatch-generated evidence with its branch identity

A repository_dispatch run receives the default-branch ref, and checking out the requested tag does not change the OIDC identity used by cosign or the source ref recorded by the SLSA generator. The workflow nevertheless verifies release.yml@refs/tags/${tag} and --source-tag "${tag}" at lines 1152–1171, matching the public contract in release/VERIFY.md; after the images and GitHub Release have been written, reconciliation will reject the branch-identity certificates/provenance and leave a failed, partially completed promotion.

Useful? React with 👍 / 👎.


permissions:
contents: read

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
group: ${{ github.workflow }}-${{ github.event.client_payload.tag || github.ref }}

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 28, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use one concurrency key for both release entry points

For the same tag, a tag-push run uses a key ending in refs/tags/vX.Y.Z, while a recovery dispatch uses the bare payload vX.Y.Z, so GitHub does not serialize them despite cancel-in-progress: false. If recovery is dispatched while the original tag-push run or its rerun is still active, both workflows can pass the prewrite absence checks and enter the package and release write stages concurrently; normalize both event paths to the same tag-name key.

Useful? React with 👍 / 👎.

cancel-in-progress: false

env:
Expand Down Expand Up @@ -44,16 +46,19 @@ jobs:
- name: Checkout annotated release tag
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v4.2.2
with:
ref: ${{ github.event.client_payload.tag || github.ref }}
fetch-depth: 0
submodules: false
persist-credentials: false

- name: Resolve exact release identity
id: release
env:
REQUESTED_TAG: ${{ github.event.client_payload.tag || '' }}
shell: bash
run: |
set -euo pipefail
tag="${GITHUB_REF_NAME}"
tag="${REQUESTED_TAG:-${GITHUB_REF_NAME}}"
if [[ ! "${tag}" =~ ^v(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)$ ]]; then
echo "Release tag must be canonical semantic version text" >&2
exit 1
Expand Down Expand Up @@ -108,9 +113,10 @@ jobs:
from pathlib import Path
import release_candidate

binding = release_candidate.parse_tag_binding(
Path(sys.argv[1]).read_text(encoding="utf-8")
)
message = Path(sys.argv[1]).read_text(encoding="utf-8")
if message.endswith("\n\n"):
message = message[:-1]
binding = release_candidate.parse_tag_binding(message)
Comment on lines +116 to +119

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 28, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Normalize the message passed to every tag verifier

In the repository_dispatch recovery path, the later verify-candidate job checks out the immutable tag target, so it executes that tag's old release_candidate.py rather than the parser changed in this commit. This normalization fixes only the initial inline parse; lines 373–376 recreate the raw two-newline tag message and pass it to the old \n? parser, reproducing the exact failure before publication. The normalized message must be carried into downstream verification, or the fixed verifier must come from trusted default-branch code.

Useful? React with 👍 / 👎.

with Path(sys.argv[2]).open("a", encoding="utf-8") as handle:
for field in ("run_id", "run_attempt", "receipt_sha256"):
handle.write(f"{field}={binding[field]}\n")
Expand All @@ -132,10 +138,12 @@ jobs:
set -euo pipefail
git fetch --force origin refs/heads/main:refs/remotes/origin/main
test "$(git rev-parse HEAD)" = "${{ steps.release.outputs.tag_target }}"
test "$(git rev-parse refs/remotes/origin/main)" = \
"${{ steps.release.outputs.tag_target }}"
git merge-base --is-ancestor \
"${{ steps.release.outputs.tag_target }}" refs/remotes/origin/main
if [[ "${GITHUB_EVENT_NAME}" != "repository_dispatch" ]]; then
test "$(git rev-parse refs/remotes/origin/main)" = \
"${{ steps.release.outputs.tag_target }}"
fi
Comment on lines +143 to +146

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 28, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Require proof of failure before relaxing main equality

For every repository_dispatch, this skips the exact-current-main check and retains only the ancestor check. An operator can therefore push a new annotated tag at any still-valid older candidate commit and dispatch recovery: the normal tag-push run rejects that stale target, but this path accepts and publishes it. Because the payload is not constrained to the known failed tag and no prior failed prewrite run is verified, the exception is broader than recovery of an already-immutable failed tag.

Useful? React with 👍 / 👎.

release/scripts/registry-release validate \
"${{ steps.release.outputs.manifest }}"
release/scripts/registry-release validate-docsets
Expand Down Expand Up @@ -280,6 +288,16 @@ jobs:
gh api \
"/repos/${GITHUB_REPOSITORY}/actions/runs/${EXPECTED_RUN_ID}/artifacts?per_page=100" \
> promotion/control/artifact-metadata.json
if [[ "${GITHUB_EVENT_NAME}" == "repository_dispatch" ]]; then
jq '
.artifacts |= map(
select((.name | startswith("registry-stack-candidate-cli-")) | not)
)
' promotion/control/artifact-metadata.json \
> promotion/control/artifact-metadata.filtered.json
mv promotion/control/artifact-metadata.filtered.json \
promotion/control/artifact-metadata.json
fi
python3 release/scripts/release_candidate.py verify-attempt-artifacts \
--artifact-metadata promotion/control/artifact-metadata.json \
--run-id "${EXPECTED_RUN_ID}" \
Expand Down
2 changes: 1 addition & 1 deletion release/scripts/release_candidate.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1204,7 +1204,7 @@ def parse_tag_binding(message: str) -> dict[str, Any]:
re.escape(TAG_BINDING_HEADER)
+ r"\nrun_id: ([1-9][0-9]*)"
+ r"\nrun_attempt: ([1-9][0-9]*)"
+ r"\nreceipt_sha256: ([0-9a-f]{64})\n?",
+ r"\nreceipt_sha256: ([0-9a-f]{64})\n{0,2}",
message,
)
if match is None:
Expand Down
5 changes: 3 additions & 2 deletions release/scripts/test_registry_release.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1435,8 +1435,9 @@ def test_candidate_promotion_has_closed_no_rebuild_publish_gates(self) -> None:
self.assertNotIn("write", verification["permissions"].values())
self.assertIn("--release-id", verification_script)
self.assertIn(
'test "$(git rev-parse refs/remotes/origin/main)" = \\\n'
' "${{ steps.release.outputs.tag_target }}"',
'if [[ "${GITHUB_EVENT_NAME}" != "repository_dispatch" ]]; then\n'
' test "$(git rev-parse refs/remotes/origin/main)" = \\\n'
' "${{ steps.release.outputs.tag_target }}"',
source_verification,
)
self.assertEqual("read", publish_images["permissions"]["contents"])
Expand Down
5 changes: 5 additions & 0 deletions release/scripts/test_release_candidate.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -769,7 +769,12 @@ def test_tag_binding_is_closed_and_binds_exact_receipt(self) -> None:
},
self.module.parse_tag_binding(message),
)
self.assertEqual(
self.module.parse_tag_binding(message),
self.module.parse_tag_binding(message + "\n"),
)
for tampered in (
message + "\n\n",
message + "extra: accepted\n",
message.replace("run_attempt: 2", "run_attempt: 3"),
message.replace(
Expand Down
Loading