-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Add weekly TestFlight public-beta promotion #25712
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+812
−0
Merged
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| #!/bin/bash -eu | ||
|
|
||
| # Lists the nightly builds, opens the "choose a build" block step, and posts the | ||
| # candidate list to Slack. No build — just gems + secrets. | ||
|
|
||
| echo "--- :rubygems: Setting up Gems" | ||
| install_gems | ||
|
|
||
| echo "--- :closed_lock_with_key: Installing Secrets" | ||
| bundle exec fastlane run configure_apply | ||
|
|
||
| echo "--- :testflight: Gathering candidates and opening the block step" | ||
| bundle exec fastlane gather_testflight_candidates |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| #!/bin/bash -eu | ||
|
|
||
| # Promotes the build chosen in the preceding block step to public beta. No build — just gems + secrets. | ||
|
|
||
| # `build_to_promote` must stay in sync with PROMOTION_META_DATA_KEY in fastlane/lanes/promote.rb, | ||
| # which is the key the gather lane writes the block-step select field under. | ||
| BUILD_CODE="$(buildkite-agent meta-data get "build_to_promote" --default "")" | ||
|
|
||
| if [[ -z "${BUILD_CODE}" ]]; then | ||
| echo "+++ :x: No build was selected to promote." | ||
| exit 1 | ||
| fi | ||
|
|
||
| echo "--- :rubygems: Setting up Gems" | ||
| install_gems | ||
|
|
||
| echo "--- :closed_lock_with_key: Installing Secrets" | ||
| bundle exec fastlane run configure_apply | ||
|
|
||
| echo "--- :rocket: Promoting ${BUILD_CODE} to public beta" | ||
| bundle exec fastlane promote_build build_code:"${BUILD_CODE}" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| #!/bin/bash -eu | ||
|
|
||
| # Promotes the last build of the day to the nightly group. No build — just gems + secrets. | ||
|
|
||
| echo "--- :rubygems: Setting up Gems" | ||
| install_gems | ||
|
|
||
| echo "--- :closed_lock_with_key: Installing Secrets" | ||
| bundle exec fastlane run configure_apply | ||
|
|
||
| echo "--- :new_moon: Promoting last build of the day to nightly beta" | ||
| # The lane refuses to run anywhere but trunk. | ||
| bundle exec fastlane promote_nightly_build |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| # yaml-language-server: $schema=https://raw.githubusercontent.com/buildkite/pipeline-schema/main/schema.json | ||
| --- | ||
|
|
||
| # Daily nightly promotion — run by a Buildkite schedule (see docs/testflight-promotion.md). | ||
|
|
||
| agents: | ||
| queue: mac | ||
| env: | ||
| IMAGE_ID: $IMAGE_ID | ||
|
|
||
| steps: | ||
| - label: ":new_moon: Promote last build of the day to nightly" | ||
| command: ".buildkite/commands/promote-nightly.sh" | ||
| plugins: [$CI_TOOLKIT_PLUGIN] | ||
|
|
||
| # Backstop: the lane posts its own failures to Slack, but a setup failure (gems/secrets) | ||
| # under `set -e` aborts before fastlane runs and bypasses that rescue. A build-level failure | ||
| # notification ensures a broken unattended nightly run still reaches the channel. | ||
| notify: | ||
| - slack: "#build-and-ship" | ||
| if: build.state == "failed" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| # yaml-language-server: $schema=https://raw.githubusercontent.com/buildkite/pipeline-schema/main/schema.json | ||
| --- | ||
|
|
||
| # Weekly public-beta promotion — run by a Buildkite schedule (see docs/testflight-promotion.md). | ||
|
|
||
| agents: | ||
| queue: mac | ||
| env: | ||
| IMAGE_ID: $IMAGE_ID | ||
|
|
||
| steps: | ||
| - label: ":testflight: Gather builds to promote" | ||
| command: ".buildkite/commands/gather-testflight-candidates.sh" | ||
| plugins: [$CI_TOOLKIT_PLUGIN] | ||
|
|
||
| # Backstop: the gather lane posts its own failures to Slack, but a setup failure (gems/ | ||
| # secrets) under `set -e` aborts before fastlane runs and bypasses that rescue. A build-level | ||
| # failure notification ensures a broken weekly run still reaches the channel. | ||
| notify: | ||
| - slack: "#build-and-ship" | ||
| if: build.state == "failed" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,63 @@ | ||
| # TestFlight Promotion | ||
|
|
||
| Every trunk push uploads WordPress and Jetpack to the **internal** TestFlight group (`build_and_upload_app_for_testflight` in `fastlane/lanes/build.rb`). This covers promoting those builds onward through two more beta tiers: | ||
|
|
||
| ``` | ||
| per-commit → internal TestFlight | ||
| daily 00:00 → last build of the day → Nightly Beta Testers (automatic) | ||
| weekly → a developer picks a nightly → public beta (human-chosen) | ||
| ``` | ||
|
|
||
| Each promotion is a metadata-only App Store Connect call that adds an existing build to an external group — no rebuild. | ||
|
|
||
| ## Daily: last build of the day → nightly | ||
|
|
||
| `promote_nightly_build` (via `.buildkite/promote-nightly.yml`) finds the newest processed build uploaded in the last `NIGHTLY_LOOKBACK_HOURS` and distributes it to the **Nightly Beta Testers** group for both apps. Fully automatic; a quiet day (no new builds) is a no-op. | ||
|
|
||
| ## Weekly: a nightly → public beta | ||
|
|
||
| `.buildkite/promote-testflight.yml` runs three steps: | ||
|
|
||
| 1. **Gather** (`gather_testflight_candidates`) lists the `VALID` builds in the **Nightly Beta Testers** group, scoped to the highest marketing version and the last `PROMOTION_MAX_AGE_DAYS` days, newest first. It uploads a **block step** with those builds as a dropdown and posts the list to `#build-and-ship`. | ||
| 2. **Choose** — the Slack message links to the block step's unblock dialog, where a developer picks a build. | ||
| 3. **Promote** (`promote_build`) distributes the chosen build to the public beta group for both apps. | ||
|
|
||
| Both apps share a build code, so one selection promotes both. | ||
|
|
||
| ## Safety | ||
|
|
||
| Every lane refuses to run anywhere but **`trunk`** — including the weekly `gather_testflight_candidates`, which bails before it reads App Store Connect or opens a picker. Run from any other branch, or locally, they fail immediately — an off-trunk promotion is a mistake. The schedules only run on `trunk` regardless, so the guard is a backstop against a stray manual run. | ||
|
|
||
| ## Buildkite setup | ||
|
|
||
| The pipeline files live in the repo; the schedules that run them are managed in Terraform as `buildkite_pipeline_schedule` resources on the existing `wordpress-ios` pipeline. The trampoline uploads `.buildkite/${PIPELINE:-pipeline.yml}`, so a schedule selects its file with `env = { PIPELINE = "promote-nightly.yml" }`. | ||
|
|
||
| - **Nightly** — cron `0 0 * * *`, branch `trunk`. | ||
| - **Public** — cron `0 14 * * 5` (Fridays 14:00 UTC), branch `trunk`. The cron is the cadence knob. | ||
|
|
||
| No new secrets or queues — the steps reuse the existing App Store Connect API key, Slack webhook, and Buildkite token, and run on the `mac` queue. | ||
|
|
||
| ## Configuration | ||
|
|
||
| All constants are in `fastlane/lanes/promote.rb`: | ||
|
|
||
| | What | Where | | ||
| | --- | --- | | ||
| | Cadence | the Buildkite schedules (cron) | | ||
| | Candidates shown | `PROMOTION_CANDIDATE_LIMIT` | | ||
| | Recency window (days) | `PROMOTION_MAX_AGE_DAYS` | | ||
| | Nightly lookback (hours) | `NIGHTLY_LOOKBACK_HOURS` | | ||
| | Beta group names | `NIGHTLY_BETA_GROUP_NAME`, `WORDPRESS_PUBLIC_BETA_GROUPS`, `JETPACK_PUBLIC_BETA_GROUPS` | | ||
|
|
||
| ## Manual promotion | ||
|
|
||
| These lanes only run on `trunk`, so a manual promotion means triggering the relevant Buildkite pipeline on `trunk` (set `PIPELINE=promote-nightly.yml` or `PIPELINE=promote-testflight.yml` — see [Buildkite setup](#buildkite-setup)), not running fastlane locally. A local invocation refuses to run, by design. | ||
|
|
||
| ## Caveats | ||
|
|
||
| - The weekly list reads the **Nightly Beta Testers** group, so it's empty until the daily job has populated it on `trunk`. Both apps must have that group in App Store Connect. | ||
| - **Reader** isn't promoted — excluded from the internal build matrix while its App Store archive is broken (#25321). | ||
| - An un-actioned weekly promotion stays blocked in Buildkite until cancelled; the next week's build doesn't clear it. | ||
| - The nightly job promotes the newest **`VALID`** build in the lookback window, so a build still *processing* when the job runs is skipped — and won't be picked up later either (the next run only looks back `NIGHTLY_LOOKBACK_HOURS`). The `0 0 * * *` schedule makes this rare (the last build is usually hours old), but a commit landing minutes before the run can miss that night's promotion. | ||
| - The nightly job picks that build by upload time alone — it doesn't filter by branch or marketing version. While a release is in flight, `release/*` builds (a different `VERSION_SHORT`, distributed straight to public beta) land in the same App Store Connect app, so a release build that happens to be the most recent upload when the job runs could be promoted to nightly instead of the latest `trunk` build. Left unguarded by choice: the job only runs on `trunk`, a per-commit `trunk` build is normally the newest upload, and a release branch's version is out of step with `trunk` only briefly (it's merged back soon after the version bump) — so the window is short. The weekly public promotion is human-picked, so a stray nightly build wouldn't silently reach public beta. | ||
| - A promotion runs **per app** — WordPress and Jetpack are distributed independently (`distribute_build_to_apps`). If one succeeds and the other fails (a renamed beta group, a transient App Store Connect error), the lane reports the per-app result to Slack and exits non-zero, but the app that already succeeded **stays distributed** — there's no rollback. The two can sit a cycle apart until the next run reconciles them: the nightly job re-promotes the newest build automatically; the weekly promotion is re-triggered by hand. Re-promotion is idempotent — an already-distributed build isn't re-submitted for review or re-notified — so re-running after a partial failure is safe. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've been thinking about this on the Android end and I wonder if we should use one of our team's channels for it.
#build-and-shipis very noisy, so it'd be more likely for these failures to go unnoticed. Also, any issues would be addressed by our team and not infra folks - well, most of the time anyway - so, I was thinking either to use a team specific channel, or keep using#build-and-ship, but in the failure message ping our team. The second option would allow keeping that channel muted and still get a heads up. What do you think?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I figure we can adjust this later as it's an operational concern – right now we put most of the release chatter in
#build-and-shipso I figured we'd mirror that to start and can figure out something else later if needed 🤷