Allen Coral Atlas SDB (Bahamas/N. Caribbean); multi-source pipeline improvements - #5
Allen Coral Atlas SDB (Bahamas/N. Caribbean); multi-source pipeline improvements#5bkeepers wants to merge 1 commit into
Conversation
…eline improvements General pipeline (independently useful): - tile_raster.py: split an oversized source raster into memory-safe per-tile rasters (windowed gdal_translate; drops all-nodata tiles) - source_smooth.py: gaussian denoise over water for speckly sources (+ optional --max-depth cutoff) - source_datum.py: --scale for unit conversion (e.g. cm->m), applied before negate - source_polygonize.py: generalize large/speckly coverage masks before polygonizing (pixel-exact polygons of scattered SDB reefs overflowed the SQLite blob limit) - aggregation_run.py: remove SKIP_CONTOURS/SKIP_SMOOTH; stages run independently via a `step` CLI - source priority quality tiers: noaa_s102=2, cudem/cudem_third=1 sources/aca_ncaribbean (ACA 10 m SDB, CC-BY 4.0): recipe complete end-to-end and the raw GeoTIFF is mirrored to R2, but the dataset is TABLED -- SDB is too noisy for a nautical chart (see PR for the write-up). Not wired into a shippable build; branch kept for revisit.
Revisit idea: mask SDB to ACA's own Reef Extent / Geomorphic footprintThe dominant terrain problem was ACA returning false-shallow depths over km-deep water Clipping the Note: the depth GeoTIFF is a single Int16 band (cm) — it contains only depth, none of the |
…e box The store moves from R2-hydrate-and-push to a long-lived volume (seascape-store, created once by the workflow, attached per build, never deleted). Content- addressed names are self-certifying, so whatever ANY prior run finished — completed or interrupted — is reused by the next run's freshness checks with zero machinery. Evidence from runs #5–#7: the hydrate/push/resume protocol cost ~10 min of line-rate download per boot, idled a box 26 min on prefix listings, lost a crashed run's vector forks mid-push-cycle, and made every failed attempt restart from zero (no pointer until the first complete build). Deleted, not ported: the manifest-driven store hydrate + its bootstrap branch, the 20-min push_agg loop (durability is now a property of the storage), the terrain-pmtiles store push, and the store manifest + pointer publish. R2 keeps the sources mirror (synced to the volume incrementally) and the PRODUCTS: the published mosaic (pointer-last, unchanged) and build/<sha>/ bundles. Aggregation scratch (covering plan + per-tile reproject/merge tmp, TBs of churn per planet run) stays on the box's local NVMe via a nested bind — regenerated every run, and it would saturate a Ceph-backed volume that shrugs at the durable artifacts' write rate. bbox smokes keep the NVMe store and attach nothing. Follow-up before the next GC run: re-root gc.yml onto mosaic indexes + build manifests (the store manifests it rooted on are no longer written).
…e box The store moves from R2-hydrate-and-push to a long-lived volume (seascape-store, created once by the workflow, attached per build, never deleted). Content- addressed names are self-certifying, so whatever ANY prior run finished — completed or interrupted — is reused by the next run's freshness checks with zero machinery. Evidence from runs #5–#7: the hydrate/push/resume protocol cost ~10 min of line-rate download per boot, idled a box 26 min on prefix listings, lost a crashed run's vector forks mid-push-cycle, and made every failed attempt restart from zero (no pointer until the first complete build). Deleted, not ported: the manifest-driven store hydrate + its bootstrap branch, the 20-min push_agg loop (durability is now a property of the storage), the terrain-pmtiles store push, and the store manifest + pointer publish. R2 keeps the sources mirror (synced to the volume incrementally) and the PRODUCTS: the published mosaic (pointer-last, unchanged) and build/<sha>/ bundles. Aggregation scratch (covering plan + per-tile reproject/merge tmp, TBs of churn per planet run) stays on the box's local NVMe via a nested bind — regenerated every run, and it would saturate a Ceph-backed volume that shrugs at the durable artifacts' write rate. bbox smokes keep the NVMe store and attach nothing. Follow-up before the next GC run: re-root gc.yml onto mosaic indexes + build manifests (the store manifests it rooted on are no longer written).
…e box The store moves from R2-hydrate-and-push to a long-lived volume (seascape-store, created once by the workflow, attached per build, never deleted). Content- addressed names are self-certifying, so whatever ANY prior run finished — completed or interrupted — is reused by the next run's freshness checks with zero machinery. Evidence from runs #5–#7: the hydrate/push/resume protocol cost ~10 min of line-rate download per boot, idled a box 26 min on prefix listings, lost a crashed run's vector forks mid-push-cycle, and made every failed attempt restart from zero (no pointer until the first complete build). Deleted, not ported: the manifest-driven store hydrate + its bootstrap branch, the 20-min push_agg loop (durability is now a property of the storage), the terrain-pmtiles store push, and the store manifest + pointer publish. R2 keeps the sources mirror (synced to the volume incrementally) and the PRODUCTS: the published mosaic (pointer-last, unchanged) and build/<sha>/ bundles. Aggregation scratch (covering plan + per-tile reproject/merge tmp, TBs of churn per planet run) stays on the box's local NVMe via a nested bind — regenerated every run, and it would saturate a Ceph-backed volume that shrugs at the durable artifacts' write rate. bbox smokes keep the NVMe store and attach nothing. Follow-up before the next GC run: re-root gc.yml onto mosaic indexes + build manifests (the store manifests it rooted on are no longer written).
…e box The store moves from R2-hydrate-and-push to a long-lived volume (seascape-store, created once by the workflow, attached per build, never deleted). Content- addressed names are self-certifying, so whatever ANY prior run finished — completed or interrupted — is reused by the next run's freshness checks with zero machinery. Evidence from runs #5–#7: the hydrate/push/resume protocol cost ~10 min of line-rate download per boot, idled a box 26 min on prefix listings, lost a crashed run's vector forks mid-push-cycle, and made every failed attempt restart from zero (no pointer until the first complete build). Deleted, not ported: the manifest-driven store hydrate + its bootstrap branch, the 20-min push_agg loop (durability is now a property of the storage), the terrain-pmtiles store push, and the store manifest + pointer publish. R2 keeps the sources mirror (synced to the volume incrementally) and the PRODUCTS: the published mosaic (pointer-last, unchanged) and build/<sha>/ bundles. Aggregation scratch (covering plan + per-tile reproject/merge tmp, TBs of churn per planet run) stays on the box's local NVMe via a nested bind — regenerated every run, and it would saturate a Ceph-backed volume that shrugs at the durable artifacts' write rate. bbox smokes keep the NVMe store and attach nothing. Follow-up before the next GC run: re-root gc.yml onto mosaic indexes + build manifests (the store manifests it rooted on are no longer written).
Summary
Ingests Allen Coral Atlas 10 m satellite-derived bathymetry (SDB) for the Northern
Caribbean / Bahamas as
sources/aca_ncaribbean(CC-BY 4.0; raw mirrored to R2), plus severalgeneral multi-source pipeline improvements. The ACA dataset is being tabled — the recipe
runs end-to-end, but SDB isn't good enough for a nautical chart. This PR preserves the work and
documents the challenges for a later revisit. Not intended to merge as-is (keeping it on a
branch leaves
main's build untouched).Pipeline improvements (keep — independently useful)
tile_raster.py— split an oversized source raster into memory-safe per-tile rasters(windowed
gdal_translate; drops all-nodata tiles). Also the tool for the deferred GSC Pacific ingest.source_smooth.py— gaussian denoise over water for speckly sources; optional--max-depthcutoff.source_datum.py—--scale(unit conversion applied before negate; e.g. ACA cm→m).source_polygonize.py— generalize large/speckly coverage masks before polygonizing(pixel-exact polygons of scattered SDB reefs overflowed the SQLite blob limit).
aggregation_run.py— removedSKIP_CONTOURS/SKIP_SMOOTH; aggregation stages(reproject/merge/smooth/tile/contour) now run independently via a
step <name>...CLI insteadof being skipped by env flags.
noaa_s102=2,cudem/cudem_third=1(real surveys outranklower-confidence sources in overlap).
Why ACA is tabled — the challenges
Research confirmed ACA is the only open, dense, redistributable bathymetry for the Bahamas banks
(the Andros "Median DEM" is paywalled; ICESat-2 ATL24 is open but sparse along-track; TCarta/EOMAP
are proprietary). But passive-optical SDB is fundamentally hard for a safety chart:
usable — i.e. its honest resolution is ~z11 (~69 m), not the nominal 10 m.
1,500–4,800 m deep. A self-referential depth cutoff can't catch this (the noise reads shallow).
A GEBCO-depth-gated feather (
w=1where GEBCO ≥ −12 m, ramp → 0 by −25 m;out = w·ACA + (1−w)·GEBCO)does fix the terrain (deep→GEBCO, shallow→ACA) and is a reusable building block.
pipeline's fine 1 m intervals yields ~1 M features per macrotile; tippecanoe takes 45 min+
to tile that (a global build would be untenable). The only lever that cuts the count is coarsening the
shallow interval — but fine-grained shallow contours are essential for navigation, so that's not
acceptable. Neither σ nor the GEBCO gate reduces the feature count.
Revisit path
and/or to bias-correct ACA.
See the
ROADMAP.mdCaribbean table for the tabled entry.