Persistent Hetzner volume as the build store — the cache outlives the box - #84
Merged
Conversation
…e box The store moves from R2-hydrate-and-push to a long-lived volume (seascape-store, created once by the workflow, attached per build, never deleted). Content- addressed names are self-certifying, so whatever ANY prior run finished — completed or interrupted — is reused by the next run's freshness checks with zero machinery. Evidence from runs #5–#7: the hydrate/push/resume protocol cost ~10 min of line-rate download per boot, idled a box 26 min on prefix listings, lost a crashed run's vector forks mid-push-cycle, and made every failed attempt restart from zero (no pointer until the first complete build). Deleted, not ported: the manifest-driven store hydrate + its bootstrap branch, the 20-min push_agg loop (durability is now a property of the storage), the terrain-pmtiles store push, and the store manifest + pointer publish. R2 keeps the sources mirror (synced to the volume incrementally) and the PRODUCTS: the published mosaic (pointer-last, unchanged) and build/<sha>/ bundles. Aggregation scratch (covering plan + per-tile reproject/merge tmp, TBs of churn per planet run) stays on the box's local NVMe via a nested bind — regenerated every run, and it would saturate a Ceph-backed volume that shrugs at the durable artifacts' write rate. bbox smokes keep the NVMe store and attach nothing. Follow-up before the next GC run: re-root gc.yml onto mosaic indexes + build manifests (the store manifests it rooted on are no longer written).
- timeout-minutes: self-hosted escapes the 6h HOSTED platform cap, but the per-job DEFAULT timeout is 360 min on every runner type — omitting the key guillotined run 29371657768 at exactly 6h with 61% of the planet aggregated and memory textbook-healthy. 48h backstop. - One-time marker-gated seed of mosaic/tiles+contour+soundings+depare from R2 onto the blank volume: runs #6-#8's push loops banked ~1900+ tiles' artifacts that the manifest hydrate could never see; same code = same keys, so the first volume build skips that work instead of recomputing it a fifth time.
…ytes from R2 sources.yml now boots the same on-demand box pattern as build.yml, attaches the persistent store volume, and writes prepared sources, volatile mirrors (CUDEM/ S-102 objects), masks, and footprints DIRECTLY to the volume — the planet build's source of truth. R2 is demoted to the publish mirror that laptops (just preview) and bbox smokes stream from; every step's aws-cli calls become rclone (the box needs one pinned binary, not two). - create-runner/delete-runner copied from build.yml (ccx33 — sources prep is network-bound); the per-source matrix serializes onto the ONE non-ephemeral runner, keeping per-source GH-UI granularity and fail-fast:false isolation - staleness checks now require the recipe hash to match BOTH the volume catalog and the R2 catalog, so a crash between the two publishes self-heals - the seed-previous-bounds step is deleted: bounds.csv lives on the volume - volatile mirror lands upstream→volume, then volume→R2 (both incremental) - build.yml planet branch: deletes the bounds/catalog/polygon hydrates and the entire selective dirty-set source hydrate (sources-manifest machinery) — a local land.fgb existence check remains as the fail-fast; bbox unchanged - one-time seed extended to source/ polygon/ landmask/ (spares sources.yml a from-upstream re-prep of the whole catalog); volume 500→750 GB (sources ~250 GiB now live there)
There was a problem hiding this comment.
Pull request overview
Moves the build “store” and prepared source state from an R2-backed hydrate/push protocol to a long-lived Hetzner volume (seascape-store), so interrupted planet builds can resume from durable filesystem state and planet builds avoid build-time reads from data.openwaters.io.
Changes:
- Updates
build.ymlto attach and mount a persistent Hetzner volume for planet builds, use NVMe for aggregation scratch via a nested bind, extendtimeout-minutes, and remove the R2 store hydrate/push-loop/manifest-pointer machinery (replacing it with a one-time seed-from-R2 gate). - Updates
sources.ymlto run source prep on an on-demand Hetzner runner with the same persistent volume attached, write sources/masks/footprints to the volume first, and mirror that state to R2 using rclone env-var remotes (removing aws-cli usage). - Adds runner lifecycle management in
sources.yml(create + attach volume, then always delete runner/server) aligned with the existing build runner pattern and sharedr2-storeconcurrency.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| .github/workflows/sources.yml | Runs source prep/mirroring on a single non-ephemeral Hetzner runner with the persistent store volume attached; writes to volume and mirrors to R2 via rclone. |
| .github/workflows/build.yml | Attaches/mounts the persistent store volume for planet builds, keeps aggregation scratch on NVMe, removes R2 store hydrate/publish loops, and increases job timeout. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
327
to
330
| if grep -q source_polygonize "sources/${{ matrix.source }}/Justfile"; then | ||
| aws s3 ls "s3://$DATA_BUCKET/bathymetry/polygon/${{ matrix.source }}.gpkg" >/dev/null || poly_ok=false | ||
| [ -f "$STORE/polygon/${{ matrix.source }}.gpkg" ] || poly_ok=false | ||
| rclone lsf "r2:$DATA_BUCKET/bathymetry/polygon/${{ matrix.source }}.gpkg" 2>/dev/null | grep -q . || poly_ok=false | ||
| fi |
Comment on lines
+188
to
+192
| export DEBIAN_FRONTEND=noninteractive | ||
| command -v unzip >/dev/null || { apt-get update -qq; apt-get install -y -qq unzip; } | ||
| command -v docker >/dev/null || curl -fsSL https://get.docker.com | sh | ||
| if ! command -v rclone >/dev/null; then | ||
| curl -fsSL -o /tmp/rclone.zip "https://downloads.rclone.org/v1.74.4/rclone-v1.74.4-linux-amd64.zip" |
Comment on lines
303
to
305
| # 64 GB swap above is a safety net for a tile that peaks past its estimate, NOT budget | ||
| # headroom (swapping the hot merge arrays would thrash). ccx63 (48/192): pool 48, budget | ||
| # ~150 GB. ccx33 (8/32): reserve shrinks with RAM so a small box still gets a real budget. |
…setup) Dispatching sources.yml first on a blank volume re-prepped every source FROM UPSTREAM (the volume catalog is the staleness anchor, and it didn't exist) — caught live on the first branch dispatch, cancelled at 5m. The marker-gated seed now also lives in the sources box-setup: whichever workflow touches a blank volume first pays a line-rate R2 copy (source+polygon+landmask) instead; build.yml's fuller seed keeps its own marker and either order converges.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The store — and now the sources — move from R2-hydrate-and-push to a long-lived Hetzner volume (
seascape-store, created once by whichever workflow touches it first, attached per run, never deleted). Planet builds read zero bytes from R2/data.openwaters.io.Why
Content-addressed artifacts are self-certifying (the key is in the filename; local writes are atomic), so any prior run's finished tiles are safe to reuse — but the R2-as-store protocol only trusted the pointer published by a completed build, and no build had ever completed. The week's ledger:
timeout-minutes(360 applies to self-hosted too), 61% done, memory textbook — all 1928 tiles' compute lost againWith the volume, "what survived" and "what the next run sees" are the same filesystem. A re-dispatch redoes only unfinished tiles.
What changes
build.yml
seascape-store(create-if-missing, pinned hcloud; planet only — bbox attaches nothing)timeout-minutes: 2880(the default 360 killed run Milestone 4: soundings, feet/fathom contours, shallow-contour fix #8; self-hosted only escapes the hosted platform cap, not the per-job default)build/<sha>/bundles, bbox streaming pathsources.yml
fail-fast: falseisolation preserved)just preview) and bbox smokesCosts & risks
SERVER_LOCATION(fsn1) — volume attach is same-location-only; ccx63 stock there becomes a build dependency[res]sampler shows it; escape hatch = copy hot sources to NVMe at bootr2-storeconcurrency group serializes build/sources/GC, so the volume attach never racesFollow-ups (before/after merge)
Validation
run:script passesbash -n; actionlint via CI--ephemeral— matrix jobs queue onto the one runner)