Add AusSeabed per-survey bathymetry (Australia, 0.5 m–440 m, z13) - #55
Open
bkeepers wants to merge 3 commits into
Open
Add AusSeabed per-survey bathymetry (Australia, 0.5 m–440 m, z13)#55bkeepers wants to merge 3 commits into
bkeepers wants to merge 3 commits into
Conversation
The last full planet build spent its final 2h40 running contour-bundle alone: tile-join rewrites every tile of the whole archive, and the merge-then-fold-per-layer shape re-paid that planet-wide join once per sparse layer (~90 min each for soundings and drying). Bundle both layers first and fold them into vector.pmtiles in the contour merge's ONE tile-join; fold() is gone from soundings_run/drying_run. The fan-outs also sized themselves to item counts, not work, saturating the shard ceiling (256) for every phase: - downsample: striding by ancestor left one 77-minute subtree straggling behind ~250 spin-up-only shards. Ancestors now bin-pack by parent-webp count (utils.lpt_bins, heaviest-first) and the matrix self-sizes to ceil(total/heaviest) — a shard ~= the heaviest subtree, the wall-clock floor anyway since a read-closed subtree can't split. - bundle: 235 one-group chunks each paid more runner setup than bundling. Groups now bin-pack by pmtiles bytes, same self-sizing. - contours: ~200 FGBs (~10 min of tippecanoe) per shard instead of one runner per ~1 min of work. Aggregate keeps max sharding on purpose: its shard count is the 6h job-cap headroom, and the phase is concurrency-bound regardless. The shards dispatch input is now documented as the ceiling the other phases self-size under. Tile content is unchanged, so no force rebuild is needed. Expected: ~9h40 -> ~7h wall, release ~2.5h earlier. New check_weighted_shards in test_engine.py covers the partition (complete + disjoint, self-sized n, LPT bound) and the pure packer.
Bundled whole-set inside contour-bundle, they held its tile-join back ~26 min while the rest of the build sat finished (run 28751489111: merge started 19:45, everything else done by 19:47, job ran solo to 21:05). Each contour shard now also tippecanoes its slice of soundings + drying (three invocations — the layers need different flags), so the bundling rides the existing fan-out and the merge starts as soon as the shards land. Slices stride each layer's own sorted list; no geographic alignment needed since the join unions everything per tile. Shard archives tile to the shared global maxz (store/contour-maxz.txt) like the contours always have: a slice whose own max child_z undershoots it would otherwise vanish from deeper tiles after the join.
137 survey zips (~30 GB) of gridded multibeam/lidar L3 products from Geoscience Australia, selected from the Marine Data Register WFS by sources/ausseabed/harvest.py: PUBLISHED + CC-BY 4.0 + no embargo + bathymetry-only. Backscatter products are excluded — some re-bundle their sibling survey's bathy tif and would collide at unzip — as is the compilations index (SDB; gbr30/AusBathyTopo already cover the useful ones). Register DATA_URLs pointing at the non-anonymous producthouse S3 bucket map by basename to files.ausseabed.gov.au; harvest range-reads every zip's central directory to verify fetchability and reject tif basename collisions. One source, no resolution split: merge order is per-file (priority, native maxzoom), so a 1 m harbor survey outranks gbr30 while a 100 m ocean transit yields to it; max_zoom=13 only caps the sub-metre surveys. The zips ship byproducts, so source_unzip grows generic member filters: --exclude drops the *_hs.tif hillshades, --prefer _cog.tif keeps only the COG where 21 zips include a raw twin of the same grid, and members extract with a lowercase .tif extension — every downstream step globs *.tif, so a .tiff member silently vanished from the source. Fetching ~30 GB over flaky links motivated two download fixes: http_download now streams to dest.part and resumes with a Range header across retries and re-runs (curl -C style), renaming into place only on completion, and source_download skips already-complete files — so a crashed fetch resumes instead of restarting. The filelist step's own .part handling folds into that. Ragged swath grids resurrect the NONNA polygonize pathology: pixel-exact footprint masks of two transit-corridor surveys (Wallaby-Zenith: 1,305 footprint parts) took 6:51 and unioned into a 23.5 MB polygon. The mask now caps at 1024 px on its long edge via gdalwarp -r max, which dilates: coverage is over-approximated, never lost — matching the coarse per-tile footprints streamed sources already get. Same two surveys: 11 s, 1 MB. New self-checks (unzip filters, download resume against a local Range-aware server, polygonize dilation) wired into just test-sources.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #29.
137 survey zips (~30 GB) of gridded multibeam/lidar L3 products from Geoscience Australia, selected from the Marine Data Register WFS by
sources/ausseabed/harvest.py:PUBLISHED+ CC-BY 4.0 + no embargo + bathymetry-only. One source, no resolution split — merge order is per-file(priority, native maxzoom), so a 1 m harbor survey outranks gbr30 while a 100 m ocean transit yields to it;max_zoom: 13only caps the sub-metre surveys.What the register data forced
DATA_TYPES+ URL basename; harvest range-reads every zip's central directory and hard-fails on any remaining tif basename collision._cogtwin) and most bundle*_hs.tif(f)hillshades →source_unzipgrew generic--exclude/--prefermember filters..tiffmembers are invisible to every downstream*.tifglob — silently dropped from the source. Members now extract with a lowercase.tifextension.http_downloadnow resumes via.part+ Range header (curl-Cstyle) andsource_downloadskips complete files, so the 30 GB fetch survives resets without restarting.This is our first source dominated (by area) by sparse survey swaths rather than continuous pre-gridded bathymetry — the profile that hurt us with NONNA. 88% of the footprint area is 21 deep-ocean transit/ridge compilations (Lord Howe Rise alone is 973k km²), some with hundreds-to-thousands of disjoint swath patches (Wallaby-Zenith: 1,305 footprint parts). What's mitigated vs. what still needs eyes:
Mitigated in this PR (measured): pixel-exact footprint masks reproduced the NONNA pathology — 6 min 51 s and a 23.5 MB union polygon for just the two worst surveys.
source_polygonizenow caps the mask at 1024 px (long edge) viagdalwarp -r max, which dilates: coverage is over-approximated, never lost. Same two surveys: 11 s, 1.0 MB. Footprint fuzz is ~extent/1024 (hundreds of m on an ocean corridor, a few tile-pixels on a harbor survey), consistent with the coarse per-tile rectangles streamed sources (CUDEM/S-102) already get.Verify manually on the first build / preview:
harvest.py.150.55,-35.20,150.90,-34.95, a 2 m full-bay survey), Port Phillip Bay, and Apollo Bay (143.40,-38.90,143.75,-38.60, validated locally). Datum is ~MSL by filename token, but 234 older grids are untagged and 2 are LAT — spot-check charted depths in a harbor against known values.coveragelayer — footprints tile into the provenance layer, so corridor footprints will read slightly fatter than the actual swaths at high zoom. Confirm that's acceptable there..tiffrename regressed.Validation done locally
.tiff-named grid, 7-tile 1999 EPSG:4326 product, and the two worst ragged-swath surveys. Values verified negative-down, embedded CRS, NoData 3.4e38 preserved through normalize.just test-sourcesgreen, including three new self-checks (unzip member filters, download resume against a local Range-aware server, polygonize dilation keeps sparse specks).