Skip to content

Add AusSeabed per-survey bathymetry (Australia, 0.5 m–440 m, z13) #55

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bkeepers wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Add AusSeabed per-survey bathymetry (Australia, 0.5 m–440 m, z13) #55

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ff4c149
Work-sized build fan-outs + single vector tile-join
bkeepers Jul 5, 2026
e0ce807
Shard soundings + drying with the contours
bkeepers Jul 6, 2026
275aefa
Add AusSeabed per-survey bathymetry (Australia, 0.5 m–440 m, z13)
bkeepers Jul 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 54 additions & 38 deletions .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ on:
required: false
default: ""
shards:
description: "Aggregate matrix shards (<= 256)"
description: "Fan-out ceiling (<= 256): aggregate shard count; the other phases self-size to their work below it"
required: false
default: "24"
force:
Expand Down Expand Up @@ -366,10 +366,13 @@ jobs:
aws s3 sync store/aggregation "s3://$DATA_BUCKET/bathymetry/aggregation" --exclude '*' --include '*.done'

# Overview pyramid, sharded. The deep levels partition into read-closed subtrees
# (one per ancestor at the covering's seed zoom) so they fan out with no
# coordination, exactly like aggregate; the coarse global tail (a few cheap levels
# whose tiles span subtrees) is finished on the single bundle runner below. The
# level barrier stays inside each runner. A clean rebuild still spins one no-op shard.
# (one per ancestor at the covering's seed zoom), bin-packed by work into the shards
# (a shard ≈ the heaviest single subtree — sizing to ancestor *count* left one
# 77-minute straggler blocking bundle-plan next to hundreds of spin-up-only shards),
# so they fan out with no coordination, exactly like aggregate; the coarse global
# tail (a few cheap levels whose tiles span subtrees) is finished on the single
# bundle runner below. The level barrier stays inside each runner. A clean rebuild
# still spins one no-op shard.
downsample:
needs: [image, aggregate, plan]
if: ${{ !cancelled() && needs.aggregate.result == 'success' }}
Expand Down Expand Up @@ -419,9 +422,11 @@ jobs:
# whose footprint-sized archives outgrew a runner as sources accumulated. bundle-plan
# finishes the coarse pyramid tail (it spans ancestors, so it can't be a deep shard),
# verifies the pyramid is whole, and emits the matrix: each entry is a comma-joined
# CHUNK of group names (the partition rides in the matrix, so jobs can't drift from the
# plan's store view). A matrix job loops its chunk pull→bundle→push→clean one group at
# a time; bundle-merge stitches the fragments. Contours are built separately (below).
# CHUNK of group names, bin-packed by pmtiles bytes so a chunk ≈ the biggest single
# group (one-group chunks spent more runner time on setup than bundling). The
# partition rides in the matrix, so jobs can't drift from the plan's store view. A
# matrix job loops its chunk pull→bundle→push→clean one group at a time;
# bundle-merge stitches the fragments. Contours are built separately (below).
bundle-plan:
needs: downsample
if: ${{ !cancelled() && needs.downsample.result == 'success' }}
Expand Down Expand Up @@ -537,8 +542,10 @@ jobs:
aws s3 cp store/bundle/manifest.json "s3://$DATA_BUCKET/bathymetry/build/${{ github.sha }}/manifest.json" \
--content-type application/json

# ─── Contours (sharded — one global tippecanoe blows the 6 h cap at planet scale) ─
# Size the fan-out to the FGB count (no download — just an R2 listing).
# ── Vector layers (sharded — one global tippecanoe blows the 6 h cap at planet scale) ─
# Size the fan-out to the contour work — the dominant layer; soundings + drying
# slices ride the same shards for ~20% more per-shard time (no download — just an
# R2 listing).
contour-plan:
needs: aggregate
if: ${{ !cancelled() && needs.aggregate.result == 'success' }}
Expand All @@ -549,11 +556,18 @@ jobs:
- id: m
run: |
count=$(aws s3 ls "s3://$DATA_BUCKET/bathymetry/contour/" --recursive | grep -c '\.fgb$' || true)
shards=$(python3 -c "import json,sys; c=int(sys.argv[1]); n=min(${AGG_SHARDS}, max(c,1)); print(json.dumps([{'i':i,'n':n} for i in range(n)]))" "$count")
# ~200 FGBs ≈ 10 min of tippecanoe per shard (observed ~3 s/FGB planet-wide);
# sizing to the raw count spun one runner per ~1 min of work. Retune the divisor
# if shard runtimes drift past ~30 min. AGG_SHARDS stays the ceiling.
shards=$(python3 -c "import json,sys; c=int(sys.argv[1]); n=min(${AGG_SHARDS}, max((c + 199) // 200, 1)); print(json.dumps([{'i':i,'n':n} for i in range(n)]))" "$count")
echo "shards=$shards" >> "$GITHUB_OUTPUT"
echo "contour FGBs: $count → matrix $shards"

# tippecanoe one strided FGB slice per runner → a per-shard contours pmtiles.
# tippecanoe one strided slice of every vector layer per runner → per-shard
# contours/soundings/drying pmtiles. Soundings + drying ride the contour shards
# (separately they held contour-bundle's tile-join back ~26 min while the rest of
# the build sat finished); the slices stride each layer's own sorted list — no
# geographic alignment needed, the merge unions everything per tile anyway.
contours:
needs: [aggregate, contour-plan]
if: ${{ !cancelled() && needs.contour-plan.result == 'success' }}
Expand All @@ -566,32 +580,41 @@ jobs:
steps:
- uses: actions/checkout@v6
- uses: ./.github/actions/ghcr-login
# Pull ONLY this shard's FGB slice (not the whole set). List the FGB keys
# (sorted = the order Python globs), derive the global maxz (so every shard
# tiles to the same depth and tile-joins cleanly), and parallel-copy the
# strided slice this shard owns.
- name: Pull this shard's FGB slice from R2
run: |
mkdir -p store/contour store/bundle
aws s3 ls "s3://$DATA_BUCKET/bathymetry/contour/" --recursive | awk '{print $NF}' | grep '\.fgb$' | sort > /tmp/all.txt
sed 's#.*/##; s/\.fgb$//' /tmp/all.txt | awk -F- '{print $4}' | sort -n | tail -1 > store/contour-maxz.txt
# Pull ONLY this shard's slices (not the whole sets). List each layer's keys
# (sorted = the order Python globs), derive the global maxz from the contour
# list (so every shard and layer tiles to the same depth and tile-joins
# cleanly), and parallel-copy the strided slices this shard owns.
- name: Pull this shard's layer slices from R2
run: |
mkdir -p store/contour store/soundings store/drying store/bundle
aws s3 ls "s3://$DATA_BUCKET/bathymetry/contour/" --recursive | awk '{print $NF}' | grep '\.fgb$' | sort > /tmp/all-contour.txt
sed 's#.*/##; s/\.fgb$//' /tmp/all-contour.txt | awk -F- '{print $4}' | sort -n | tail -1 > store/contour-maxz.txt
aws s3 ls "s3://$DATA_BUCKET/bathymetry/soundings/" --recursive | awk '{print $NF}' | grep '\.geojson$' | sort > /tmp/all-soundings.txt
aws s3 ls "s3://$DATA_BUCKET/bathymetry/drying/" --recursive | awk '{print $NF}' | grep '\.fgb$' | sort > /tmp/all-drying.txt
# Bounded outer retry + visible errors — see the downsample pull above.
awk -v i="${{ matrix.shard.i }}" -v n="${{ matrix.shard.n }}" 'NR % n == (i + 1) % n' /tmp/all.txt \
| xargs -P 8 -I{} sh -c 'for a in 1 2 3 4 5; do aws s3 cp "s3://$DATA_BUCKET/$1" store/contour/ --only-show-errors && exit 0; sleep "$a"; done; echo "giving up on $1 after 5 attempts" >&2; exit 1' _ {}
echo "shard ${{ matrix.shard.i }}/${{ matrix.shard.n }}: $(ls store/contour/*.fgb 2>/dev/null | wc -l) FGBs, global maxz $(cat store/contour-maxz.txt)"
- name: Contour shard ${{ matrix.shard.i }}
for spec in "/tmp/all-contour.txt store/contour" "/tmp/all-soundings.txt store/soundings" "/tmp/all-drying.txt store/drying"; do
set -- $spec
awk -v i="${{ matrix.shard.i }}" -v n="${{ matrix.shard.n }}" 'NR % n == (i + 1) % n' "$1" \
| xargs -P 8 -I{} sh -c 'for a in 1 2 3 4 5; do aws s3 cp "s3://$DATA_BUCKET/$1" "$2/" --only-show-errors && exit 0; sleep "$a"; done; echo "giving up on $1 after 5 attempts" >&2; exit 1' _ {} "$2"
done
echo "shard ${{ matrix.shard.i }}/${{ matrix.shard.n }}: $(ls store/contour/*.fgb 2>/dev/null | wc -l) contour FGBs, $(ls store/soundings/*.geojson 2>/dev/null | wc -l) soundings, $(ls store/drying/*.fgb 2>/dev/null | wc -l) drying, global maxz $(cat store/contour-maxz.txt)"
- name: Vector shard ${{ matrix.shard.i }}
run: |
docker run --rm -v "$PWD/store:/app/pipelines/store" \
"$IMAGE:${{ github.sha }}" just contour-shard ${{ matrix.shard.i }}
"$IMAGE:${{ github.sha }}" just vector-shard ${{ matrix.shard.i }}
- name: Push shard pmtiles to R2
run: |
aws s3 cp store/bundle "s3://$DATA_BUCKET/bathymetry/build/${{ github.sha }}/contour-shards/" \
--recursive --exclude '*' --include 'contours-shard-*.pmtiles' --content-type application/octet-stream
--recursive --exclude '*' --include '*-shard-*.pmtiles' --content-type application/octet-stream
# Global contour maxz (identical from every shard) — contour-bundle's merge needs
# it to tile the coverage layer to the same depth as the contours.
aws s3 cp store/contour-maxz.txt "s3://$DATA_BUCKET/bathymetry/build/${{ github.sha }}/contour-maxz.txt"

# tile-join the per-shard pmtiles into one vector.pmtiles.
# tile-join the per-shard pmtiles (contours + soundings + drying slices) + coverage
# into one vector.pmtiles, in ONE join: tile-join rewrites every tile of the whole
# archive, so the old merge-then-fold-per-layer shape re-paid the planet-wide join
# per layer (~90 min each, serial, as the only job left running — the last 2 h 40
# of the build).
contour-bundle:
needs: contours
if: ${{ !cancelled() && needs.contours.result == 'success' }}
Expand All @@ -600,20 +623,13 @@ jobs:
steps:
- uses: actions/checkout@v6
- uses: ./.github/actions/ghcr-login
- name: Pull shard pmtiles + soundings + drying from R2
- name: Pull shard pmtiles from R2
run: |
mkdir -p store/bundle store/soundings store/drying
mkdir -p store/bundle
aws s3 sync "s3://$DATA_BUCKET/bathymetry/build/${{ github.sha }}/contour-shards" store/bundle
aws s3 cp "s3://$DATA_BUCKET/bathymetry/build/${{ github.sha }}/contour-maxz.txt" store/contour-maxz.txt
aws s3 sync "s3://$DATA_BUCKET/bathymetry/soundings" store/soundings
aws s3 sync "s3://$DATA_BUCKET/bathymetry/drying" store/drying
- name: Merge contours
- name: Merge shards + fold coverage
run: docker run --rm -v "$PWD/store:/app/pipelines/store" "$IMAGE:${{ github.sha }}" just contour-merge
# Soundings + drying aren't sharded (sparse) — bundle them here and fold into vector.pmtiles.
- name: Fold in soundings
run: docker run --rm -v "$PWD/store:/app/pipelines/store" "$IMAGE:${{ github.sha }}" just soundings
- name: Fold in drying
run: docker run --rm -v "$PWD/store:/app/pipelines/store" "$IMAGE:${{ github.sha }}" just drying
- name: Push vector.pmtiles to R2 build
run: |
aws s3 cp store/bundle/vector.pmtiles "s3://$DATA_BUCKET/bathymetry/build/${{ github.sha }}/vector.pmtiles" \
Expand Down
34 changes: 23 additions & 11 deletions Justfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,16 @@ sources:
just ../sources/"$id"/
done

# Planet build: cover -> aggregate -> downsample -> bundle -> contours (BBOX="W,S,E,N" for a region).
# Planet build: cover -> aggregate -> downsample -> bundle -> vector layers (BBOX="W,S,E,N"
# for a region). Soundings + drying bundle BEFORE contours: the contours tile-join folds
# their pmtiles into vector.pmtiles in the same single pass.
planet:
just cover
uv run python aggregation_run.py
just combine
just contours
just soundings
just drying
just contours

# Plan the covering: slice the planet into aggregation tiles (BBOX="W,S,E,N" for a region).
cover:
Expand Down Expand Up @@ -94,26 +96,33 @@ bundle-group name:
bundle-merge:
uv run python bundle.py merge

# Contours, whole set (local/regional). CI shards these across runners — see below.
# Contours, whole set (local/regional); the final tile-join also folds in any
# soundings/drying pmtiles already bundled. CI shards these across runners — see below.
contours:
uv run python contour_run.py bundle

# Soundings: bundle the per-tile points, then fold them into vector.pmtiles (one vector source).
# Soundings: bundle the per-tile points into soundings.pmtiles. Run BEFORE the contours
# bundle/merge — its single tile-join folds the layer into vector.pmtiles (a separate
# fold re-joined the whole planet archive per layer).
soundings:
uv run python soundings_run.py bundle
uv run python soundings_run.py fold

# Drying areas (green foreshore): bundle the per-tile polygons, then fold into vector.pmtiles.
# Drying areas (green foreshore): bundle the per-tile polygons into drying.pmtiles
# (folded into vector.pmtiles by the contours tile-join, same as soundings).
drying:
uv run python drying_run.py bundle
uv run python drying_run.py fold

# tippecanoe this shard's local FGBs -> contours-shard-{i}.pmtiles (CI pulls only the
# shard's slice + writes store/contour-maxz.txt; merged by contour-merge).
contour-shard i:
# tippecanoe this shard's local slice of every vector layer -> {contours,soundings,
# drying}-shard-{i}.pmtiles (CI pulls only the shard's slices + writes
# store/contour-maxz.txt so all layers tile to one depth; merged by contour-merge).
# Three invocations, not one -L run: the layers need different tippecanoe flags
# (soundings -r1, drying --drop-densest-as-needed, contours' per-zoom filter).
vector-shard i:
uv run python contour_run.py bundle-shard {{i}}
uv run python soundings_run.py bundle-shard {{i}}
uv run python drying_run.py bundle-shard {{i}}

# tile-join the per-shard contour pmtiles into vector.pmtiles.
# tile-join the per-shard pmtiles (all layers) + coverage into vector.pmtiles.
contour-merge:
uv run python contour_run.py bundle-merge

Expand Down Expand Up @@ -167,6 +176,9 @@ preview-local bbox="-74.30,40.40,-73.75,40.80": (preview bbox "local")
test-sources:
uv run python test_source_stage.py
uv run python source_register_remote_geopkg.py --check
uv run python source_unzip.py --check
uv run python source_download.py --check
uv run python source_polygonize.py --check
test-engine:
uv run python test_engine.py
uv run python aggregation_reproject.py --check
19 changes: 11 additions & 8 deletions pipelines/bundle.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -226,16 +226,19 @@ def _manifest_from_fragments(frags):


def groups_matrix(maxn):
"""Verify the pyramid is whole, then print the CI bundle matrix: <= maxn chunks,
each a comma-joined strided slice of the group names. The partition rides IN the
matrix (not re-derived per job from a live R2 listing), so every job bundles the
exact set this full-store runner saw — the same freeze-the-plan reasoning as the
aggregate/downsample shards."""
"""Verify the pyramid is whole, then print the CI bundle matrix: <= maxn chunks
of comma-joined group names, bin-packed by each group's local pmtiles bytes so
every chunk carries about the biggest single group (one-group chunks meant 235
runners each spending longer on spin-up than on bundling). The partition rides
IN the matrix (not re-derived per job from a live R2 listing), so every job
bundles the exact set this full-store runner saw — the same freeze-the-plan
reasoning as the aggregate/downsample shards."""
aggregation_id = utils.get_aggregation_ids()[-1]
verify_complete(aggregation_id)
names = sorted(group_filepaths(aggregation_id))
n = min(maxn, max(len(names), 1))
print(json.dumps([{"cells": ",".join(names[i::n])} for i in range(n)]))
groups = group_filepaths(aggregation_id)
weights = {name: sum(os.path.getsize(fp) for fp in fps) for name, fps in groups.items()}
n = min(maxn, math.ceil(sum(weights.values()) / max(max(weights.values()), 1))) if weights else 1
print(json.dumps([{"cells": ",".join(sorted(chunk))} for chunk in utils.lpt_bins(weights, n)]))


def group_keys(name):
Expand Down
29 changes: 17 additions & 12 deletions pipelines/contour_run.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -300,14 +300,19 @@ def _coverage_pmtiles(maxz):
return out


def _finalize_contours(contour_pmtiles, maxz):
"""tile-join the contour pmtiles (one local build or the CI shards) + the coverage
layer into store/bundle/vector.pmtiles. -pk keeps every feature of both layers;
coverage is dropped from the join when no footprints are present locally."""
def _finalize_contours(archives, maxz):
"""tile-join the layer archives (a local build's contour pmtiles, or the CI shards —
which carry contours, soundings, AND drying slices) + the coverage layer + the
whole-set soundings/drying pmtiles (local path, when their bundles ran first) into
store/bundle/vector.pmtiles. ONE join: tile-join rewrites every tile of the whole
archive, so folding each sparse layer in afterwards re-paid the planet-wide join
per layer (~90 min each in CI). -pk keeps every feature of every layer; a layer
whose pmtiles isn't present locally is simply not joined."""
cov = _coverage_pmtiles(maxz)
inputs = list(contour_pmtiles) + ([cov] if cov else [])
subprocess.run(["tile-join", "-o", "store/bundle/vector.pmtiles", "-f", "-pk", *inputs],
check=True)
layers = [p for p in [cov, "store/bundle/soundings.pmtiles", "store/bundle/drying.pmtiles"]
if p and os.path.isfile(p)]
subprocess.run(["tile-join", "-o", "store/bundle/vector.pmtiles", "-f", "-pk",
*archives, *layers], check=True)
return cov is not None


Expand Down Expand Up @@ -362,18 +367,18 @@ def bundle(shard=None):


def bundle_merge():
"""tile-join the per-shard contour pmtiles + the coverage layer into one
vector.pmtiles (-pk keeps every feature; the shards are disjoint FGB slices
unioned per tile)."""
shards = sorted(glob("store/bundle/contours-shard-*.pmtiles"))
"""tile-join the per-shard pmtiles — contours, soundings, drying (each shard job
bundles its slice of all three) — + the coverage layer into one vector.pmtiles
(-pk keeps every feature; the shards are disjoint file slices unioned per tile)."""
shards = sorted(glob("store/bundle/*-shard-*.pmtiles"))
if not shards:
print("contour merge: no shard pmtiles")
return
maxzfile = "store/contour-maxz.txt" # the CI shard path always writes this (shared global maxz)
if not os.path.isfile(maxzfile):
raise SystemExit("contour merge: store/contour-maxz.txt missing (the shard jobs write it)")
cov = _finalize_contours(shards, int(open(maxzfile).read().strip()))
print(f"contour merge: store/bundle/vector.pmtiles ({len(shards)} shards"
print(f"contour merge: store/bundle/vector.pmtiles ({len(shards)} shard archives"
f"{', + coverage layer' if cov else ''})")


Expand Down
Loading
Loading