Add WSV DGM-W (German federal waterway bathymetry) source - #76
Draft
bkeepers wants to merge 13 commits into
Draft
Conversation
bkeepers
commented
Jul 13, 2026
There was a problem hiding this comment.
Pull request overview
Adds the WSV DGM‑W dataset as a prepared bathymetry/topobathy source (scoped to tidal/estuary tiles) and extends the pipeline to support spatially varying vertical datum subtraction via an offset surface (SKN-in-NHN).
Changes:
- Add a new prepared source definition for WSV DGM‑W (docs, metadata, recipe, curated tile URL list with inland tiles deferred).
- Add
source_reference_dgmw.pyto build an SKN (chart datum) reference surface raster for tidal Elbe + German Bight coverage. - Extend
source_datum.pywith--offset-surfaceto subtract a per-pixel reference raster (including a synthetic self-check).
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| sources/README.md | Adds DGM‑W to the public source index table. |
| sources/dgm_w/README.md | Source-specific documentation, scope rationale, and planned/implemented datum handling. |
| sources/dgm_w/metadata.json | Registers DGM‑W source metadata (producer, license, zoom cap, mixed CRS, datum description). |
| sources/dgm_w/Justfile | Defines the prepared-source pipeline steps, including reference build + datum surface subtraction. |
| sources/dgm_w/file_list.txt | Lists DGM‑W tile download URLs (tidal active; inland commented out). |
| pipelines/source_reference_dgmw.py | Builds the SKN reference raster used by the datum step. |
| pipelines/source_datum.py | Adds optional spatial reference subtraction (--offset-surface) and records it in datum.json. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
bkeepers
commented
Jul 13, 2026
93 zipped GeoTIFF tiles (2 m, terrain+riverbed) from the WSV INSPIRE "Höhe" Atom download service. mixed_crs (per-tile Gauss-Krüger / UTM-ETRS89 zones), NHN elevation up-positive so no negate/offset, nodata -32768. Contour (Höhenlinien) zips excluded. License: GeoNutzV, attribute the WSV. Closes #53. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add sources/dgm_w/README.md: source docs, datum problem, vintage, and a concrete "Datum normalization (planned)" plan for a low-water step in the prep (tidal: BSH SKN-Fläche CC-BY subtraction, ready; inland: WSV GlW/ Stauziel profile assembly, deferred). - file_list.txt: keep the 16 tidal/estuary tiles active (Nordsee, Außenelbe below Elbe-km 586, Jade, Unter-/Außenweser); comment out the 77 inland tiles (bed above MSL -> render as land until referenced). - metadata.json / Justfile / sources README row: reflect tidal-only scope and the planned SKN datum step. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Generalizes the scalar --offset to a reference-surface subtraction: bed - reference(x,y), reprojected onto each tile (bilinear, cross-CRS). Pixels the reference doesn't cover become nodata (can't be referenced). This is the low-water datum mechanism for DGM-W tidal reaches (subtract the BSH SKN-Fläche NHN grid -> depth below chart datum) and reusable for any spatially-varying separation. Self-check covers the partial-coverage case. Also address review: cache the reference under store/source/dgm_w. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire the datum step into the prep so the synced COG is depth below chart datum (SKN ~= LAT), not raw NHN: - source_reference_dgmw.py builds one SKN-in-NHN reference: the BSH SKN-Fläche Nordsee 2026 grid (CC-BY 4.0) for the outer estuaries, extended over the inner tidal Elbe from the GDWS per-gauge SKN values interpolated along the gauge polyline (no grid reaches Hamburg). Cached under store/source/dgm_w/reference/ so the pipeline globs skip it. - Justfile: unzip -> build reference -> source_datum --offset-surface reference/skn_reference.tif --clamp-positive -> normalize. - README/metadata: datum section now describes the implemented step; clamp caveat notes an OSM land-water reconciliation follow-up for drying areas. Validated end-to-end on outer Elbe km710-728 (grid) and inner Elbe km620-639 (assembled fill, 0% -> covered): both yield water-only depths below chart datum. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…alues - Move the reference builder to sources/dgm_w/build_reference.py (bespoke per-source script lives with the source), run via `uv run --with pdfplumber`. - Stop hard-coding the tidal-Elbe gauge SKN values: parse them from the GDWS "Seekartennull an den Tidepegeln" PDF and take gauge positions/km from PEGELONLINE, both at build time. Only the source URLs + gauge-name list remain. - Extract the BSH grid from the zip in memory and keep no temp archive, so the R2 sync of store/source/dgm_w never publishes the upstream zip. - Fix doc contradictions: dgm_w README pipeline note and the sources index row now say the SKN chart-datum step is applied, not planned; README references the new script path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per review, PDF parsing at build time was overkill and fragile. Move the tidal-Elbe gauge SKN values (and positions) into sources/dgm_w/tideelbe_skn.csv, with a header comment linking the GDWS SKN table + PEGELONLINE. build_reference.py reads the CSV, so the build only fetches the BSH grid — no pdfplumber, no live PDF/PEGELONLINE dependency. Output is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Below the Iffezheim barrage (km 336) the Rhein runs free, so its low-water datum is GlW (gleichwertiger Wasserstand). PEGELONLINE exposes each gauge's gaugeZero (PNP in NHN) and GlW characteristic value, so harvest_rhein_glw.py reads the 17 free-flowing gauges (Maxau km362 -> Emmerich km852) straight from the API into a committed rhein_glw.csv -- no PDF tables. build_reference.py now factors the inner-Elbe corridor-fill into a shared helper and adds the Rhein GlW reach as its own grid; the two reaches stitch into reference.vrt, which source_datum --offset-surface subtracts. Activates the 3 free-flowing Rhein tiles; the upper impounded reach (Stauziel) stays deferred.
Replace harvest_rhein_glw.py with a config-driven harvest_gauges.py covering every inland reach (Rhein GlW + Main Stauziel ZS_I so far), each a checked-in CSV with a shared datum_nhn_m column. Adds main_zs.csv (6 impounded-pool gauges, km 12-122). No build_reference wiring for the Main yet — the impounded step surface is the next step.
The canalised Main (km 0-102) is a staircase of pools, each held at a fixed Stauziel by its barrage. Three data files, each closing a gap: - main_stau.csv: per-barrage Stauziel (Wikipedia Liste der Mainstaustufen) — complete where the PEGELONLINE ZS_I gauges are too sparse (Eddersheim/Offenbach pools have none); the gauges become a cross-check build_reference asserts. - OSM man_made=weir line per barrage (endpoints in main_stau.csv): the exact pool divider, so each step lands on its dam, not an interpolated point. Muehlheim's weir sits by its lock way (river-km 53.2, source=WSV); it tags as lock, not weir. - main_centerline.wkt: the real OSM Main centerline, so the ~1.5 km corridor hugs every meander instead of a coarse gauge chord the river escapes at bends. build_main assigns each pixel the Stauziel of the barrage whose weir is immediately downstream (side-test count). Activates the 6 Main tiles (km 0-102).
Activate the 16 upper-Rhein tiles (km 164-334, Basel->Iffezheim). Ten impounded pools, each referenced to its barrage's normal retention level (rhein_stau.csv). PEGELONLINE publishes no ZS_I here (French EDF barrages), so retention levels are transcribed from the Grand Canal d'Alsace level profile, anchored on Iffezheim's measured NHN Stauziel. Divider is latitude, not weir lines: the reach flows due north, so pool boundaries fall on lines of latitude (barrage lats strictly increasing). Corridor follows the OSM navigation line (rhein_centerline.wkt) to stay on the impounded channel and off the low Restrhein. Verified on a km173-190 tile: water-only depths, step lands on the Kembs dam.
bkeepers
force-pushed
the
add-source-wsv-dgmw
branch
from
July 14, 2026 14:26
7ee7968 to
56f1b7a
Compare
bkeepers
changed the base branch from
main
to
plan-source-download-normalize-split
July 14, 2026 14:26
…the river Two related changes, both moving reaches to an OSM-river fill corridor. Fix (reported): the free-flowing Rhein filled around the gauge polyline, but 17 gauges over ~530 km make a chord that misses the wide outer channel at the Düsseldorf/Kaiserswerth bends, so the Rhine there fell outside the mask and didn't render. Corridor is now the OSM river union (rhein_river.wkt); GlW is still the gauge ramp (projected + interpolated by arc-length — only the fill needed the river). New: three impounded canalised rivers via build_impounded, the general Stauziel-staircase builder. Pools are assigned by arc-length along a proper centerline (shortest path through the OSM river graph — follows big meanders like the Cochem loop), with the river union as the fill corridor. Retention levels transcribed per barrage (Mosel: Wikipedia Moselstaustufen; Saar: Wikipedia + PEGELONLINE headwater gauges; Lahn: WSV Steckbriefe), each cross-checked against the reach's ZS_I gauges — the assert caught a Cochem-loop misprojection that forced the centerline approach. Validated end-to-end on the Mosel tile: water-only depths, correct pool steps. 12+8+11 pools. Refactor: shared _paint_corridor helper (build_rhein_upper now uses it too).
…Overture Elbe (25 tiles, km 0-586, Czech border -> Geesthacht weir): the only major un-canalised German river, so low water is a smooth downstream ramp, not a Stauziel staircase. The WSV references it to GlW, but GlW is not published per-gauge on the open web; MNW (mittleres Niedrigwasser, harvestable from PEGELONLINE) sits a few cm-dm below GlW (the conservative direction) and is a sound reproducible proxy. Generalized build_rhein -> build_freeflowing (a low-water ramp interpolated along the gauges, filled over the river corridor); Rhein (GlW) and Elbe (MNW) both use it. Validated on the km350-380 tile: water-only, mean -1.2 m. Geometry now comes from Overture Maps (base/water, the same anon S3 parquet landmask.py reads) via a new build_geometry.py, replacing ad-hoc Overpass queries: name-filterable, bbox pushdown, no public-instance timeouts. It regenerates every <river>_river.wkt / <river>_centerline.wkt; the river linestrings ARE centerlines, and the step rivers' single ordered centerline is still stitched by shortest-path through the river graph. Overture's schema has no weirs or CEMT locks, so the Main's weir lines and the upper Rhein's lock coords stay as committed OSM snapshots in their *_stau.csv. Regenerated rhein/mosel/saar/lahn from Overture; all ZS_I cross-checks still pass. harvest_gauges.py + build_geometry.py both take an optional reach/river filter.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #53.
Adds the German federal waterway bathymetry (WSV DGM-W) as a prepared source, low-water (chart-datum) normalized so the tiles read as depth, not raw NHN elevation.
What it is
DGM-W (Digitales Geländemodell des Wasserlaufs) is the WSV's 2 m terrain-plus-riverbed model for the federal waterways, published through the WSV INSPIRE "Höhe" Atom download service as 93 zipped GeoTIFF tiles (2 m, Float32, nodata −32768, per-region CRS →
mixed_crs). The feed'sHöhenliniencontour zips are excluded; Seascape builds its own contours.The datum problem
DGM-W stores orthometric NHN elevation, not depth. NHN ≈ MSL, so a bed value used as-is reads a metre or few deeper than a chart, which is the dangerous, optimistic direction. To render as depth it has to be referenced to a low-water datum, which differs by regime:
The prep subtracts a per-pixel reference surface (
source_datum --offset-surface), turning each bed into depth below its local low-water datum, then clamps above-datum cells (land/drying) to nodata.Done: tidal reaches (16 tiles)
build_reference.pybuilds the SKN-in-NHN surface — the BSH SKN-Fläche Nordsee 2026 grid (CC-BY 4.0) for the outer estuaries, extended over the inner tidal Elbe (Hamburg reach, east of the grid's 9.5° E edge) using the GDWS per-gauge SKN table intideelbe_skn.csv, interpolated along the gauge polyline. Validated end to end on outer Elbe km710–728 and inner Elbe km620–639 (0 % to covered); both yield water-only depths below chart datum.Inland: availability per waterway
PEGELONLINE carries
gaugeZero(PNP in NHN) plus characteristic values per gauge, so a low-water level in NHN isgaugeZero + value/100. Coverage varies by river:ZS_I, step-per-poolPlan
source_datum --offset-surface: spatially-varying reference subtractionZS_I, step-per-pool)build_impounded— arc-length pools along an OSM shortest-path centerline; ZS_I cross-checks assert each pool)build_freeflowing) — GlW un-published per-gauge, MNW is the conservative open proxy; validated on the km350–380 tilebuild_geometry.py) instead of Overpass — name-filterable, no timeouts; regenerated rhein/mosel/saar/lahn, cross-checks still passbuild_geometryclips corridors to bbox (Weser was bleeding into the tidal Unterweser)>0clamp against the OSM land–water mask so drying flats surviveNot for navigation. Reference licences: DEM GeoNutzV (attribute WSV), SKN-Fläche CC-BY 4.0 (attribute BSH), PEGELONLINE characteristic values (WSV).