Repository navigation
Assemble a release from four runners instead of four machines - #21
Merged
Merged
Conversation
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
ShawnChen-Sirius
force-pushed
the
fix/issue-15-release-workflow
branch
from
September 9, 2026 07:44
f372fba to
c846c9c
Compare
ShawnChen-Sirius
changed the base branch from
v1/release-engineering
to
main
September 9, 2026 07:51
A release needs all four platform packages staged in one working directory, and no machine can produce them: build-native.sh refuses to cross-build on purpose, and macos-x86_64 needs an Intel Mac. `build` already builds all four on their own runners and throws the staged trees away at the end of every run. release.yml keeps them. Three things the issue asked to decide, decided: The version lives in git. The workflow never runs versions:set; a release is a commit carrying the release version plus a tag on that commit, and preflight refuses to run if the two disagree. versions:set in CI publishes bytes that correspond to no commit, so `git show <tag>` cannot say what went out and a re-run of the same tag can produce different POMs. The same workflow publishes snapshots, and that is the default for a manual run. It is the only way to move issue #10, and exercising the release path on every snapshot is what makes it already-tested when a release depends on it. Reproducibility was measured rather than left open. Two clean builds of the macos-aarch64 shim from the same source on the same machine differ in 116 bytes of a same-length file: 16 bytes of LC_UUID, four bytes holding the per-translation-unit N_OSO debug-map timestamps (the .o mtimes, three seconds apart), and ~96 bytes of ad-hoc code signature covering those. __text, __cstring, __const and __data are byte-identical. So the shim's content reproduces and only build metadata does not, and since the shim is ~130 KB of a ~326 MB package whose engine half is pinned by SHA-256, chasing the rest buys nothing a consumer can act on. The workflow records what it shipped instead: every staged library's SHA-256 in the job summary. Also replaces the release profile's <waitUntilValidated> with <waitUntil>VALIDATED</waitUntil>. central-publishing-maven-plugin 0.11.0 has no waitUntilValidated parameter -- the mojo declares waitUntil, waitMaxTime, waitPollingInterval and waitForPublishCompletion -- and Maven drops configuration elements a mojo does not declare rather than failing on them, so it had been doing nothing. VALIDATED is also the default, so the behaviour was right by accident. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every test to date builds the driver in the same reactor, where the POMs come from target/ and the native libraries are read from a directory beside them. Issue #10's gate is that nothing has ever depended on this as a published artifact, and the value of the gate is the list of things that only fail when consumed. That list did not exist. scripts/verify-consumer.sh produces it without publishing anything: it deploys to a throwaway file:// repository, builds a consumer project outside the checkout with its own empty local repository, imports the BOM, declares exactly one platform package with no version, and runs a query. What it found on the first run: The BOM and the transitive dependency were already fine, and are now asserted rather than assumed. A snapshot deploy does not write the filenames a reactor build uses. It writes chdb-jdbc-26.7.0.1-20260909.053707-1.jar and a maven-metadata.xml mapping -SNAPSHOT onto that timestamp; without the metadata a consumer gets a 404 for a version that is demonstrably there. Nothing exercised that indirection before. "Declared the wrong platform package" was indistinguishable from "declared none". Both produced the same message: three locations searched, none found, here is the coordinate to add. The advice is right either way, but somebody looking at chdb-native-linux-x86_64-gnu in their own POM while being told to add a platform package has no way to work out what they did wrong. The loader now names the chdb-native-* packages that are on the classpath when they are for another machine. NativePackage.find only ever looks for the current platform's resources, so a foreign JAR was indistinguishable from no JAR; foreignPackages() is how the two are told apart, and it is diagnostics only -- nothing there makes a foreign package loadable. Wired into `build` on every platform, because a verification nobody runs is a verification that rots. The one question it cannot answer is whether a ~100 MB artifact survives a real network transfer: a file:// deploy is a copy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`main` already has docs/release-readiness.md, from #6. This branch was cut from v1/release-engineering, where it did not exist, so it grew a second copy. Folded rather than replaced: 129 of main's 253 lines are kept verbatim, including the Track 1/2/3 structure, the licence ownership table, the untested-gap table and "What is already covered". What is added is the eight-step running order and what the consumption gate actually found. Also corrects four places where main's copy had come to disagree with docs/publishing.md, which is the reference for all of them. Two copies of a number is how this kind of drift survives, so this file now keeps none of them and points there instead: - §1.1 listed sources jars, javadoc jars, GPG signatures, <scm> and <developers> as absent. All five shipped in #13, which is in main. - §1.3 offered io.github.chdb-io as a namespace fallback. Sonatype does not register io.github.<organisation> automatically, so there is no fallback and DNS access to chdb.org is a hard prerequisite. - §1.4 said the publishing limits are compared as a rolling three-month average and concluded that a 2-3 month cadence divides the release size by two or three. Sonatype is explicit that each metric resets monthly and is not averaged; a slower cadence does not make a release month cheaper, it makes the other months empty. - §2.2 counted 156 contrib submodules by scanning licence files and found seven LGPL and eight GPL. Querying system.licenses in the binary we actually ship finds 968 components and twelve copyleft ones in two families. A filename scan calls every dual-licensed component GPL and describes a checkout rather than an artifact. Two review findings on the earlier copy, both about the soak test, are addressed here because they are the same paragraph: the soak has to be concurrent, since the milestone it discharges is that cancel, timeout, early close and cascading close "leak nothing and never deadlock" and a deadlock cannot occur in one thread; and it blocks a release, so it is no longer in a table the surrounding prose calls parallel and non-blocking. Track 3 is now split into §3.1 blocking and §3.2 not. The test count is measured rather than carried over: 257, from 150 unit and 107 integration on macos-aarch64. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…elease calling itself a dry run
Two review findings on the release workflow, one of which had a back door in it.
The tag requirement was enforced on one path only. `preflight` compared the tag name
to the POM version out of `github.ref_name`, which is a tag on a push and a *branch*
on a workflow_dispatch. So `channel=release` with `dry_run=false`, run by hand,
accepted any non-SNAPSHOT POM and went straight to `mvn -Prelease deploy` with no tag
anywhere -- the same "bytes on Central that correspond to no commit" outcome the design
claims to prevent, reached by the other door. docs/publishing.md promised the guarantee
without the qualifier, so the documentation was wrong rather than the workflow being
merely incomplete.
Fixed in the workflow rather than by downgrading the doc to describe only the push
path. The rule is worth having because `git show <tag>` should be able to say what was
published; a rule with a manual bypass does not give anyone that, and the bypass is the
path a person reaches for under time pressure. A new preflight step, gated on
`channel == 'release'`, looks up `v<POM version>` through the API and refuses unless it
exists and points at the commit being built.
It dereferences annotated tags, which is not incidental: `refs/tags/<x>` for a
`git tag -a` names a tag object, not a commit, so without the extra hop the gate would
have rejected every release cut the way our own runbook recommends. Resolved through the
API rather than git because actions/checkout fetches the triggering ref, not the tag
namespace.
The other finding: the deploy summary's heading used ${DRY_RUN:+, dry run: $DRY_RUN},
which tests for non-empty rather than for true, so a real release printed "dry run:
false" in its own heading. The reader of that summary is deciding whether to confirm an
irreversible upload, so it now says which of the two things happened and nothing else.
Both gates exercised locally against the same shell text: the version/channel and tag
steps over seven cases including "manual release, no tag" (now rejected) and both tag
shapes, and the heading over both values of dry_run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… before staging
Four review findings, one of which reopened the hole the previous commit closed.
THE TAG WINDOW. `preflight` checked that v<version> names the commit being built, and
then the run went on to four-platform staging -- ten minutes and more -- and the
`maven-central` environment's approval gate, which can hold it for as long as it takes
somebody to look. A tag can be force-moved or deleted inside that window, and nothing
would have noticed: the run would publish one commit under a version whose tag now
names another, permanently, because a Central release cannot be unpublished. Checking
once at the start describes the start, not the upload.
The check is now scripts/check-release-tag.sh, called twice: in `preflight`, where a
mistake costs nothing, and in `deploy` as the last step before Maven runs. Extracted to
a script rather than duplicated inline so that the two call sites cannot drift and so it
can be exercised locally against a stubbed `gh`.
Two checks shrink the window to the seconds between the second one and the upload; they
do not close it. Only an immutable tag does, so docs/release-readiness.md step 8 now
carries the tag-protection ruleset as the organisational half of the same guarantee,
with a note that the workflow cannot verify it exists.
Writing the harness found a second defect in the new step: `script | tee -a
$GITHUB_STEP_SUMMARY` returns tee's status, so without `set -o pipefail` the gate could
never fail. Measured both ways -- with pipefail exit 1, without it exit 0.
FINDING BY COORDINATE, NOT BY FILE NAME. The classpath assertion matched
"chdb-jdbc-${VERSION}.jar". The review argued that a snapshot resolves under a
timestamped name so the script would exit before reaching the two error paths
docs/native-loading.md says it reproduces. That is not what happens -- Maven's local
repository holds *both* names, and build-classpath emits the base-version one, so the
script does reach them, verified end to end on 26.7.0.1-SNAPSHOT and again on
26.7.2-rc.2.1-SNAPSHOT. But depending on which of two names a resolver happens to emit
is a bad thing for this assertion to depend on, and the failure mode is the one the
review described: it would not fail, it would skip both error paths and still report
that all consumer checks passed. Now located by org/chdb/chdb-jdbc/, which cannot
drift, with "exactly one" and "non-empty" assertions and the classpath printed on
failure.
LOGS SURVIVE FAILURE. The EXIT trap deleted $WORK, including the deploy.log the die
message had just told the reader to go and look at. Kept on failure with the paths
printed, removed on success. Verified by breaking the build on purpose: the retained log
had the whole 81-line Maven failure.
THE PUBLISHING-LIMIT STEP was complete once Sonatype "replied". A refusal is a reply,
and so is a request for more detail; either leaves the release liable to be rejected at
upload. Now complete only when the numbers are inside the thresholds or an exception
covers the numbers we actually publish.
Test count in that file re-measured on this baseline: 266, from 156 unit and 110
integration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ShawnChen-Sirius
force-pushed
the
fix/issue-15-release-workflow
branch
from
September 9, 2026 08:22
c846c9c to
bce4034
Compare
A <repository> in a POM is added to the Super POM's `central`, not substituted for it, so
the consumer project had two places an org.chdb artifact could come from. That defeats the
whole script. Once we publish, a broken artifact just deployed to the throwaway repository
could be answered for by a good one on Central, and the check would report that consuming
the driver from a repository works -- which is the single thing it exists to find out.
Simulated locally rather than argued about: a throwaway repository with chdb-jdbc removed,
a second repository under the id `central` that has it, standing in for Central holding a
previously published release.
central live BUILD SUCCESS, provenance chdb-jdbc-...jar>central=
central disabled BUILD FAILURE, "Could not find artifact org.chdb:chdb-jdbc:jar:
26.7.2-rc.2.1-SNAPSHOT in chdb-verify (file:///tmp/mask/broken)"
So `central` is now declared with both releases and snapshots disabled, and a URL that
names nothing, so re-enabling it by accident fails rather than reaching the network.
Under its own id rather than by renaming our block to `central`, which is the other way to
do it. Reusing the id would make the file repository indistinguishable from the real one in
_remote.repositories, and it collides with the plugin repositories, which still carry the
id `central` pointing at the real Central -- that bookkeeping is keyed by id, not by URL, so
an artifact arriving through the plugin path would afterwards look available from `central`
to project-scope resolution too.
Keeping the id distinct buys a second, independent check, because the configuration should
not be trusted just for being written down: every downloaded org.chdb file's recorded
provenance is now read back out of _remote.repositories and has to say `chdb-verify`.
Exercised both ways -- it rejects the masked repository above ("served by a repository other
than the throwaway one ... >central=") and accepts a correctly isolated one.
That assertion is also what covers the case configuration cannot: a <mirrorOf>*</mirrorOf>
in the caller's settings.xml intercepts even a file repository. The script does not suppress
settings.xml, because in the environments that have such a mirror it is also the only route
to Central for Maven's own plugins; instead the check fails there, loudly, with the -s
workaround in the message. Failing beats reporting a pass it cannot justify.
pluginRepositories are left inherited on purpose and the header now says why: plugin
repositories are consulted for plugins and their dependencies, org.chdb is neither, and
vendoring surefire and the dependency plugin into the throwaway repository to close a hole
nothing can come through would only make the script unrunnable. The header now states the
whole isolation model -- what is isolated, what is not, and which check enforces which --
since this finding was the second in two rounds to turn on exactly that question.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #15. Refs #10 — moves it as far as it can move without publishing anything.
Based on
v1/release-engineering(#13), notmain, because the release profile and theunstaged-native refusal it builds on live there.
The workflow
.github/workflows/release.yml, triggered bypush: tags: ["v*"]orworkflow_dispatch(inputs:
channel= snapshot|release, default snapshot;dry_run, default true).preflight— reads the version from Maven, resolves the channel, refuses a SNAPSHOT POM onthe release channel and a non-SNAPSHOT POM on the snapshot channel, refuses a tag that
disagrees with the POMs, and checks via
gh apithat thebuildworkflow concludedsuccessfor this exact SHA. Hard for a release, advisory for a snapshot, because
buildfilters onbranches and a tag push never runs it.
stage— matrix of four (ubuntu-22.04, ubuntu-22.04-arm, macos-15, macos-15-intel).Reuses
build's engine cache key, stages withbuild-native.sh/build-native-in-container.sh, runs the integration suite against the freshly staged bytes onthe floor JDK, uploads
target/native/asnative-<platform>withif-no-files-found: error,and debug symbols for 90 days.
deploy—environment: maven-centralas the first human gate.setup-javawrites thecentralserver entry and imports the GPG key; downloads all four artifacts, restores theexecutable bit, asserts each platform is complete and not
engine.source=local, re-verifiessha256sums.txtafter the artifact round trip, printsdf -h, thenmvn -Prelease deploy(orverifyfor a dry run).autoPublish=falseis unchanged, so the bundle waits in the portal —the second gate.
The three decisions the issue asked for
versions:set; it verifiestag ≡ POM.
versions:setin CI publishes bytes corresponding to no commit, sogit show <tag>cannot say what shipped and a re-run of the same tag can produce different POMs.
way to move Release gate: nothing has consumed the driver as a published artifact #10, and running the release path for every snapshot is what makes it already-tested
when a real release needs it. Confirmed from the plugin descriptor that
central-publishing-maven-plugin0.11.0 has acentralSnapshotsUrlparameter, so SNAPSHOTdeploys route to the snapshot repository instead of building a portal bundle.
byte-identical, but the gap is fully characterised: same length, 116 differing bytes — 16 of
LC_UUID, 4 that are the fourN_OSOdebug-map timestamps (one per .cpp;.omtimes1788931405 vs 1788931408), and ~96 of ad-hoc code signature covering those.
__text,__cstring,__constand__dataare byte-identical; the nlist entries were decoded toconfirm rather than assumed. Content reproduces, metadata does not. The shim is ~130 KB of a
~326 MB package whose engine half is SHA-256-pinned, so the workflow records hashes instead of
pursuing determinism.
Side finding, reported and not fixed: those
N_OSOstrings embed the builder's absolutesource and object paths in the shipped macOS dylib —
LC_ALL=C grep '/Users/…' libchdb_java_jni.dylibhits four times. Linux strips its debug info via
objcopy --strip-debug; macOS does not, andbuild-native.sh's existing absolute-path check only inspects load commands. Fixing it meansstrip -Splus a deterministic UUID and re-testing dylib loading — real risk, low payoff givenreleases come from CI where the path is
/Users/runner/work/…. It does mean a locally stagedpackage carries the builder's username.
scripts/verify-consumer.sh— #10 as an executable checkmvn deploy -DaltDeploymentRepository=…file://into a temp dir, then a consumer project outsidethe checkout with its own empty local repository, importing the BOM and declaring exactly one
platform package with no version. Wired into
build.yml'snativejob on all four platforms.What it found:
chdb-jdbcalready worked. Now asserted rather than assumed.chdb-jdbc-26.7.0.1-20260909.053707-1.jarplusmaven-metadata.xml, not-SNAPSHOT.jar. Without that metadata a consumer 404s on a versionthat is demonstrably present. No reactor test touches that indirection.
NativePackage.findonly looks for the current platform's resources, so a foreign JAR lookedlike no JAR — someone with
chdb-native-linux-x86_64-gnuin their POM was told to add aplatform package. The loader now names the
chdb-native-*packages on the classpath when theyare for another machine. Reproduced without staging a second platform by overriding
-Dos.arch, which is the branchPlatform.detect()actually takes.<waitUntilValidated>is not a parameter ofcentral-publishing-maven-plugin0.11.0. The mojo declares
waitUntil/waitMaxTime/waitPollingInterval/waitForPublishCompletion. Verified empirically that Maven drops unknown configurationelements rather than failing, so it was a no-op. Replaced with
<waitUntil>VALIDATED</waitUntil>— also the default, so the behaviour was right by accident.
file://is a copy.Verified locally vs CI-only
Ran for real:
actionlinton both workflows (only the four intentional$MAVEN_ARGSSC2086s,matching
build.yml's existing eight);shellcheckclean on the new script; the macos-aarch64stage twice; the release profile's unstaged refusal, with the right message, and its staged pass;
the deploy job's completeness and
sha256sum -cloop body; the preflight version/tag gate acrossall six branches; 150 unit tests; 92 integration tests;
verify-consumer.shend to end.CI-only: the matrix itself (especially
macos-15-intel), artifact upload/download of ~1.3 GB,executable-bit restoration,
setup-java's GPG import, thegh apibuild-conclusion gate,environment: maven-centralapproval, and any actual deploy.Nothing was deployed to any remote.
docs/release-readiness.mdcarries the remaining eight stepsin order, including what is still genuinely blocked on the
org.chdbnamespace claim.🤖 Generated with Claude Code
Note
Assemble release from four platform runners via
release.ymlworkflow.github/workflows/release.ymlwith preflight, four-platform staging matrix, and a deploy job that downloads staged artifacts, reconstructs native directories, and runs the release Maven profile to sign and publishscripts/check-release-tag.sh, and requires thebuildworkflow to have succeeded for releasesscripts/verify-consumer.shinvocation to.github/workflows/build.ymlso each platform verifies repository consumption of driver, BOM, POMs, and native librariesNativeLibraryLoaderfailure diagnostics to list foreign-platformchdb-native-*packages found on the classpath, using newNativePackage.foreignPackagesandPlatform.resourcePrefixForhelperspom.xmlto usewaitUntilset toVALIDATEDfor the central publishing plugin, and rewrites release docs to reflect the workflow-based release pathrelease.ymlreconstructs native module directories from uploaded artifacts and sets executable permissions; a missing or corrupted platform artifact fails the completeness gate in.github/workflows/release.ymldeploy jobMacroscope summarized af890b8.