ponyup Code: Multiple Download Sources (Cloudsmith and GitHub)

[SeanTAllen]

The artifacts are in their new home, and then some. Releases now live as GitHub release assets on every tool's repo, each archive with a raw-hex .sha512 sibling, and the historical release artifacts that had matching GitHub releases have been migrated over. Nightlies are dual-publishing to GHCR alongside Cloudsmith. That's workstreams 1 and 2 from #405, plus the GHCR nightly work from #412.

This discussion plans the ponyup side for the whole picture, both channels. And the picture has changed since #405: we are not migrating off Cloudsmith and we are not doing a hard cut-over. Cloudsmith stays. What we're building is choice. ponyup can pull from Cloudsmith or from GitHub, for releases and for nightlies, and the user picks. Cloudsmith stays the default, so nobody's install breaks the day this ships; GitHub is there for anyone who wants it, starting with everyone who's been getting throttled by Cloudsmith's bandwidth cap.

Two sources, two channels

There are two sources. Cloudsmith is the one we have, and it serves both channels through its API: you ask for a package, you get back a version, a checksum, and a CDN URL. GitHub is the new one, and it isn't a single thing. For the release channel it means GitHub release assets. For the nightly channel it means GHCR, the container registry, which is a different host and a different protocol from release assets. So "get it from GitHub" resolves to two different mechanisms depending on the channel, and the code has to treat them as such even though we present them to the user as one choice.

Selection is a stored preference plus a per-command override. ponyup stores a default source, set initially to Cloudsmith, the way it already stores a default platform. A global flag overrides it for a single invocation. No automatic fallback: if you ask for a source and it fails, you hear about it, you don't silently get the other one. Keeping Cloudsmith as the default means the change is opt-in and the blast radius on day one is zero.

The bootstrap installers, which run before any stored preference exists, write the source they used as that preference as their first act. A freshly installed ponyup always knows where it came from instead of leaning on an implicit default.

Order of work

This plans the entire thing, but it ships in two phases. GitHub release support for the release channel comes first: that's where the source seam, the selection mechanism, and the bulk of the new HTTP work get built. Nightlies from GHCR come second, because pulling from a container registry needs a Pony OCI client we don't have yet, which is the work in #401. Phase one stands on its own and delivers the option most people are asking for; phase two slots the nightly source in behind the same seam once #401 lands.

One ordering dependency sits in front of phase one: the redirect change to courier described under GitHub releases below. Add the option, cut a courier release, bump corral.json, then the download path can be built on it.

What gets built

The source seam

Two sources that the commands talk to through one shape. Each source implements three operations for a given tool, channel, and platform: resolve a request to a concrete version, hand back a verified download, and list what's available. The callers no longer depend on whether the bytes came from a CDN, a release asset, or a registry blob. Verification lives inside the source, because the three don't verify the same way: Cloudsmith and GitHub releases check a SHA-512, GHCR checks an OCI digest. The caller asks for a verified download and trusts the source to define what that means.

Selection

A stored default source living next to the platform and lockfile state, defaulting to Cloudsmith, set by a ponyup command and overridden by a global flag per run. The flag is the cheap part. The stored preference is a small new bit of persisted config and a small new bit of CLI. The exact shape of both is deferred; see Deferred below.

A neutral result shape

FindPackages and ShowPackages pass raw Cloudsmith JSON around today and reach into it for fields. With more than one source producing results, that has to become a small neutral record the callers read without depending on which source filled it in.

The error model

One query either returns rows or returns a single opaque error today, and any error retries. Every source forces the same distinction: not-found is permanent and shouldn't retry, transient failures should. Cloudsmith's empty result, GitHub's 404, GHCR's missing manifest all mean the same "it isn't there." A timeout or a 5xx or a dropped connection from any of them means "try again." The HTTP layer throws the response status away right now; it has to start carrying it so sync can distinguish the two.

Cloudsmith, refactored

The existing path doesn't change behavior, but it moves behind the seam and starts producing the neutral record and the shared error taxonomy instead of raw JSON. This is mostly relocation, not new logic, but it's where the seam gets proven before a second source leans on it.

GitHub releases (phase one)

The download URL is deterministic once you know the repo, the tag, and the asset filename, built with the same platform-to-filename transform Cloudsmith already needs. A pinned version is its own tag. latest needs a releases-API lookup to turn into a concrete tag, because ponyup needs the real version string to name the install directory and write the lockfile. The checksum comes from fetching the .sha512 sibling on its own. find is one releases-API call, walking each release's assets, filtering to the platform, skipping the siblings; its --count keeps a default of 10 and gains a single hard max of 100. show is one latest-release lookup per tool. The Application trait grows a method saying which GitHub repo each tool lives in.

GitHub's release-asset URLs answer with a redirect to a signed CDN URL, and courier doesn't follow redirects. We add redirect-following to courier as an opt-in option rather than reimplementing it inside ponyup. Re-dispatching to a new host, a fresh TLS handshake on the CDN hop, a hop cap, and Location parsing are HTTP-client work that has no business living in an installer. The option defaults off, caps the number of hops, and strips sensitive headers on a cross-origin hop so a future GITHUB_TOKEN never leaks to the signed URL. This is the courier release that has to land before the phase-one download path.

GHCR (phase two)

Nightlies from a container registry, built on the OCI client from #401. Listing versions is listing tags on the OCI repo. A download is pulling the manifest, then pulling the blob layer that holds the archive, then verifying by the digest the registry provides. GHCR also wants a token even for anonymous pulls of public packages, a small handshake unlike anything the Cloudsmith or GitHub-releases paths do. All of this rides behind the same source seam, so the commands don't change when it arrives; the work is the client and the source implementation.

Auth and rate limits

Anonymous GitHub API access is 60 requests an hour, and the calls that count are the version resolves and the find/show lookups, not the asset downloads, which come from a different host. We honor GITHUB_TOKEN when it's set so heavy users and CI get the higher ceiling. That's new plumbing: nothing reads env in the HTTP layer today. GHCR's anonymous token handshake in phase two is separate from this.

Bootstrap scripts

The shell and PowerShell installers download ponyup itself before ponyup exists on the machine, so they can't read the stored preference; they take a default and an opt-in, and write whichever source they used as the preference. The no-flag default is Cloudsmith, matching the tool. The GitHub opt-in both pulls ponyup from GitHub and records github, so the install doesn't end up split-brain, fetching the bootstrap from one place and everything after from another.

The GitHub bootstrap path can sidestep the API entirely: hit releases/latest/download/..., which is a redirect with no API call and no rate-limit exposure, and derive the version by running the downloaded binary, which is what the PowerShell script already does. The shell installer queries the Cloudsmith API and verifies SHA-256 today; the PowerShell one pulls a zip straight from Cloudsmith and verifies nothing. Both gain a GitHub path and SHA-512, and the PowerShell one finally gets a checksum check it's never had.

CI workflows

Most of the Cloudsmith reads in our workflows pull nightlies and stay put for now. The release-channel reads are the ones to repoint, and there's one of consequence: the Windows breakage workflow pulls a corral release zip from Cloudsmith. Our own publishing to Cloudsmith stays; we're keeping Cloudsmith alive as a source, which means we keep feeding it.

Tests

The integration tests make live calls to Cloudsmith. They grow GitHub equivalents, following the existing pattern of driving ponyup's internal APIs directly and asserting on side effects rather than shelling out. The suite fans out, so it should use the token Actions hands every job to stay under the anonymous limit.

Divergences from #405

#405 settled several things that this reframe reverses. Calling them out rather than quietly contradicting them:

• Cloudsmith to GitHub Releases Migration #405 called for a hard cut-over on the release channel with no Cloudsmith fallback. We're not cutting over. Cloudsmith stays a selectable source.
• Cloudsmith to GitHub Releases Migration #405 called for stopping release publishing to Cloudsmith. We keep publishing there, because a source you can't get fresh artifacts from isn't much of a source.
• Cloudsmith to GitHub Releases Migration #405 called the transition a flag-day with no dual-publish. Dual availability is now the entire point.
• Cloudsmith to GitHub Releases Migration #405 scoped nightlies out, leaving them on Cloudsmith. Nightlies are in scope now, gaining GHCR as a second source.

Decisions

Settled while drafting:

• Selection is one axis. The source is cloudsmith or github; github resolves per channel to release assets or GHCR. Per-tool source selection isn't planned, and it's easy to add later if anyone ever wants it.
• Bootstrap sets the backend explicitly. Whichever source the installer uses, it writes that as the stored preference first thing, so there's no split-brain install. No-flag default is Cloudsmith; the GitHub opt-in pulls ponyup from GitHub and records github.
• Redirects belong in courier. An opt-in option, default off, header-stripping on cross-origin hops, with a hop cap. This adds a courier release ahead of phase one's download work.
• One find cap. --count keeps its default of 10 and gains a single hard max of 100 across every source. No per-source caps. The help text that says 500 today changes with it.

Deferred

• The shape of the stored preference. Whether default generalizes to something like default <subject> <value> or a new command appears, and whether the saved settings fold into one config file or a new dotfile, is an implementation-time call. Not worth nailing down now.