Skip to content

Allen Coral Atlas SDB (Bahamas/N. Caribbean); multi-source pipeline improvements - #5

Draft
bkeepers wants to merge 1 commit into
mainfrom
aca-ncaribbean-sdb
Draft

Allen Coral Atlas SDB (Bahamas/N. Caribbean); multi-source pipeline improvements#5
bkeepers wants to merge 1 commit into
mainfrom
aca-ncaribbean-sdb

Conversation

@bkeepers

@bkeepers bkeepers commented Jul 1, 2026

Copy link
Copy Markdown
Member

Summary

Ingests Allen Coral Atlas 10 m satellite-derived bathymetry (SDB) for the Northern
Caribbean / Bahamas as sources/aca_ncaribbean (CC-BY 4.0; raw mirrored to R2), plus several
general multi-source pipeline improvements. The ACA dataset is being tabled — the recipe
runs end-to-end, but SDB isn't good enough for a nautical chart. This PR preserves the work and
documents the challenges for a later revisit. Not intended to merge as-is (keeping it on a
branch leaves main's build untouched).

Pipeline improvements (keep — independently useful)

  • tile_raster.py — split an oversized source raster into memory-safe per-tile rasters
    (windowed gdal_translate; drops all-nodata tiles). Also the tool for the deferred GSC Pacific ingest.
  • source_smooth.py — gaussian denoise over water for speckly sources; optional --max-depth cutoff.
  • source_datum.py--scale (unit conversion applied before negate; e.g. ACA cm→m).
  • source_polygonize.py — generalize large/speckly coverage masks before polygonizing
    (pixel-exact polygons of scattered SDB reefs overflowed the SQLite blob limit).
  • aggregation_run.py — removed SKIP_CONTOURS / SKIP_SMOOTH; aggregation stages
    (reproject/merge/smooth/tile/contour) now run independently via a step <name>... CLI instead
    of being skipped by env flags.
  • Priority quality tiersnoaa_s102=2, cudem/cudem_third=1 (real surveys outrank
    lower-confidence sources in overlap).

Why ACA is tabled — the challenges

Research confirmed ACA is the only open, dense, redistributable bathymetry for the Bahamas banks
(the Andros "Median DEM" is paywalled; ICESat-2 ATL24 is open but sparse along-track; TCarta/EOMAP
are proprietary). But passive-optical SDB is fundamentally hard for a safety chart:

  1. Vertical noise. ±1–2 m with per-pixel speckle. It needed ~σ8 (≈80 m) gaussian smoothing to be
    usable — i.e. its honest resolution is ~z11 (~69 m), not the nominal 10 m.
  2. False-shallow over deep water. ACA returns shallow (<18 m) values over water GEBCO shows is
    1,500–4,800 m deep. A self-referential depth cutoff can't catch this (the noise reads shallow).
    A GEBCO-depth-gated feather (w=1 where GEBCO ≥ −12 m, ramp → 0 by −25 m; out = w·ACA + (1−w)·GEBCO)
    does fix the terrain (deep→GEBCO, shallow→ACA) and is a reusable building block.
  3. Contour explosion (the blocker). Even after the gate, contouring the vast shallow banks at the
    pipeline's fine 1 m intervals yields ~1 M features per macrotile; tippecanoe takes 45 min+
    to tile that (a global build would be untenable). The only lever that cuts the count is coarsening the
    shallow interval — but fine-grained shallow contours are essential for navigation, so that's not
    acceptable. Neither σ nor the GEBCO gate reduces the feature count.

Revisit path

  • Ingest ICESat-2 ATL24 (open, NASA public domain, RMSE ~0.3 m) as sparse depth control / QA,
    and/or to bias-correct ACA.
  • Or re-derive SDB with C-SHELPh (open) on ICESat-2 + Sentinel-2 for a cleaner, de-speckled owned grid.
  • The GEBCO-gated feather (fixes the terrain) is the reusable piece either way.

See the ROADMAP.md Caribbean table for the tabled entry.

…eline improvements

General pipeline (independently useful):
- tile_raster.py: split an oversized source raster into memory-safe per-tile rasters
  (windowed gdal_translate; drops all-nodata tiles)
- source_smooth.py: gaussian denoise over water for speckly sources (+ optional --max-depth cutoff)
- source_datum.py: --scale for unit conversion (e.g. cm->m), applied before negate
- source_polygonize.py: generalize large/speckly coverage masks before polygonizing
  (pixel-exact polygons of scattered SDB reefs overflowed the SQLite blob limit)
- aggregation_run.py: remove SKIP_CONTOURS/SKIP_SMOOTH; stages run independently via a `step` CLI
- source priority quality tiers: noaa_s102=2, cudem/cudem_third=1

sources/aca_ncaribbean (ACA 10 m SDB, CC-BY 4.0): recipe complete end-to-end and the raw
GeoTIFF is mirrored to R2, but the dataset is TABLED -- SDB is too noisy for a nautical
chart (see PR for the write-up). Not wired into a shippable build; branch kept for revisit.
@bkeepers
bkeepers marked this pull request as draft July 1, 2026 17:36
@bkeepers bkeepers changed the title Table Allen Coral Atlas SDB (Bahamas/N. Caribbean); multi-source pipeline improvements Allen Coral Atlas SDB (Bahamas/N. Caribbean); multi-source pipeline improvements Jul 4, 2026
@bkeepers

Copy link
Copy Markdown
Member Author

Revisit idea: mask SDB to ACA's own Reef Extent / Geomorphic footprint

The dominant terrain problem was ACA returning false-shallow depths over km-deep water
(e.g. <18 m reads where GEBCO shows −1,500 to −4,800 m). The GEBCO-depth-gated feather handles
that with an external depth threshold — but ACA ships its own boundary of "where a real
reef/bank bottom was actually mapped": the Reef Extent and Geomorphic Map layers (GeoJSON
polygons — Reef Slope, Lagoon, Patch Reefs, etc.).

Clipping the Bathymetry – composite depth raster to that footprint would drop the deep
false-shallow returns at the source, using ACA's confidence boundary instead of a depth guess —
likely cleaner than the GEBCO gate (and it's ACA's own definition of "valid"). It does not fix
the contour-count blocker (the mapped shallow banks are still speckly/dense at 1 m intervals), but
it makes the shallow mask honest.

Note: the depth GeoTIFF is a single Int16 band (cm) — it contains only depth, none of the
geomorphic classes. Reef Extent / Geomorphic Map are separate GeoJSON downloads from the same
custom-AOI "Download data" panel (draw/upload an AOI → save to My Areas → Download data → tick the
layers). "Bathymetry – composite depth" is only offered for a custom AOI, not for a pre-built
Mapped Area (whose archive is benthic/geomorphic vectors only, no bathymetry).

bkeepers added a commit that referenced this pull request Jul 14, 2026
…e box

The store moves from R2-hydrate-and-push to a long-lived volume (seascape-store,
created once by the workflow, attached per build, never deleted). Content-
addressed names are self-certifying, so whatever ANY prior run finished —
completed or interrupted — is reused by the next run's freshness checks with
zero machinery. Evidence from runs #5#7: the hydrate/push/resume protocol cost
~10 min of line-rate download per boot, idled a box 26 min on prefix listings,
lost a crashed run's vector forks mid-push-cycle, and made every failed attempt
restart from zero (no pointer until the first complete build).

Deleted, not ported: the manifest-driven store hydrate + its bootstrap branch,
the 20-min push_agg loop (durability is now a property of the storage), the
terrain-pmtiles store push, and the store manifest + pointer publish. R2 keeps
the sources mirror (synced to the volume incrementally) and the PRODUCTS: the
published mosaic (pointer-last, unchanged) and build/<sha>/ bundles.

Aggregation scratch (covering plan + per-tile reproject/merge tmp, TBs of churn
per planet run) stays on the box's local NVMe via a nested bind — regenerated
every run, and it would saturate a Ceph-backed volume that shrugs at the durable
artifacts' write rate. bbox smokes keep the NVMe store and attach nothing.

Follow-up before the next GC run: re-root gc.yml onto mosaic indexes + build
manifests (the store manifests it rooted on are no longer written).
bkeepers added a commit that referenced this pull request Jul 14, 2026
…e box

The store moves from R2-hydrate-and-push to a long-lived volume (seascape-store,
created once by the workflow, attached per build, never deleted). Content-
addressed names are self-certifying, so whatever ANY prior run finished —
completed or interrupted — is reused by the next run's freshness checks with
zero machinery. Evidence from runs #5#7: the hydrate/push/resume protocol cost
~10 min of line-rate download per boot, idled a box 26 min on prefix listings,
lost a crashed run's vector forks mid-push-cycle, and made every failed attempt
restart from zero (no pointer until the first complete build).

Deleted, not ported: the manifest-driven store hydrate + its bootstrap branch,
the 20-min push_agg loop (durability is now a property of the storage), the
terrain-pmtiles store push, and the store manifest + pointer publish. R2 keeps
the sources mirror (synced to the volume incrementally) and the PRODUCTS: the
published mosaic (pointer-last, unchanged) and build/<sha>/ bundles.

Aggregation scratch (covering plan + per-tile reproject/merge tmp, TBs of churn
per planet run) stays on the box's local NVMe via a nested bind — regenerated
every run, and it would saturate a Ceph-backed volume that shrugs at the durable
artifacts' write rate. bbox smokes keep the NVMe store and attach nothing.

Follow-up before the next GC run: re-root gc.yml onto mosaic indexes + build
manifests (the store manifests it rooted on are no longer written).
bkeepers added a commit that referenced this pull request Jul 14, 2026
…e box

The store moves from R2-hydrate-and-push to a long-lived volume (seascape-store,
created once by the workflow, attached per build, never deleted). Content-
addressed names are self-certifying, so whatever ANY prior run finished —
completed or interrupted — is reused by the next run's freshness checks with
zero machinery. Evidence from runs #5#7: the hydrate/push/resume protocol cost
~10 min of line-rate download per boot, idled a box 26 min on prefix listings,
lost a crashed run's vector forks mid-push-cycle, and made every failed attempt
restart from zero (no pointer until the first complete build).

Deleted, not ported: the manifest-driven store hydrate + its bootstrap branch,
the 20-min push_agg loop (durability is now a property of the storage), the
terrain-pmtiles store push, and the store manifest + pointer publish. R2 keeps
the sources mirror (synced to the volume incrementally) and the PRODUCTS: the
published mosaic (pointer-last, unchanged) and build/<sha>/ bundles.

Aggregation scratch (covering plan + per-tile reproject/merge tmp, TBs of churn
per planet run) stays on the box's local NVMe via a nested bind — regenerated
every run, and it would saturate a Ceph-backed volume that shrugs at the durable
artifacts' write rate. bbox smokes keep the NVMe store and attach nothing.

Follow-up before the next GC run: re-root gc.yml onto mosaic indexes + build
manifests (the store manifests it rooted on are no longer written).
bkeepers added a commit that referenced this pull request Jul 15, 2026
…e box

The store moves from R2-hydrate-and-push to a long-lived volume (seascape-store,
created once by the workflow, attached per build, never deleted). Content-
addressed names are self-certifying, so whatever ANY prior run finished —
completed or interrupted — is reused by the next run's freshness checks with
zero machinery. Evidence from runs #5#7: the hydrate/push/resume protocol cost
~10 min of line-rate download per boot, idled a box 26 min on prefix listings,
lost a crashed run's vector forks mid-push-cycle, and made every failed attempt
restart from zero (no pointer until the first complete build).

Deleted, not ported: the manifest-driven store hydrate + its bootstrap branch,
the 20-min push_agg loop (durability is now a property of the storage), the
terrain-pmtiles store push, and the store manifest + pointer publish. R2 keeps
the sources mirror (synced to the volume incrementally) and the PRODUCTS: the
published mosaic (pointer-last, unchanged) and build/<sha>/ bundles.

Aggregation scratch (covering plan + per-tile reproject/merge tmp, TBs of churn
per planet run) stays on the box's local NVMe via a nested bind — regenerated
every run, and it would saturate a Ceph-backed volume that shrugs at the durable
artifacts' write rate. bbox smokes keep the NVMe store and attach nothing.

Follow-up before the next GC run: re-root gc.yml onto mosaic indexes + build
manifests (the store manifests it rooted on are no longer written).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant