Describe the bug
The tag whitelist in osm_buildings_filter.sql contradicts the one in
osm_buildings_filter_residential.sql and misses several OSM tag variants. As a
result, clearly inhabited buildings never reach
openstreetmap.osm_buildings_filtered, and they disappear from everything
downstream without a warning.
Three inconsistencies:
semidetached_house is whitelisted, the variants semidetached and
semi_detached are not.
bungalow and farmhouse are on the residential whitelist in
osm_buildings_filter_residential.sql, but not in osm_buildings_filter.sql.
- Further unambiguously residential tags are absent altogether:
dwelling_house, detached_house, houseboat.
Measured on a full Germany run: 134,725 buildings (0.71 %) that are
classified as residential do not appear in osm_buildings_filtered. The largest
groups carrying an unambiguously residential tag are bungalow (37,402),
cabin (6,591) and dwelling_house (4,849). The remainder is dominated by
outbuilding tags such as garage and shed — a separate matter, not this bug.
osm_buildings_filtered is the basis for load area aggregation, CTS building
selection, the MVGD building mapping and the rooftop PV potentials. A building
missing from it silently loses three things:
- Household load.
loadarea/loadareas_add_demand_hh.sql joins peak loads
against osm_buildings_filtered; with no geometry match the load drops out
of demand.egon_loadarea. No warning, just less demand.
- Grid assignment.
electricity_demand_timeseries/mapping.py writes no row
to boundaries.egon_map_zensus_mvgd_buildings, so the building gets no
MVGD/bus.
- Rooftop PV.
power_plants/pv_rooftop_buildings.py intersects against the
same set, so the building gets no rooftop potential.
To Reproduce
- Run the
OsmBuildingsStreets dataset.
- Run:
SELECT b.building,
count(*) AS in_osm_buildings,
count(f.id) AS in_osm_buildings_filtered
FROM openstreetmap.osm_buildings b
LEFT JOIN openstreetmap.osm_buildings_filtered f USING (id)
WHERE b.building IN ('bungalow', 'farmhouse', 'semidetached', 'semi_detached',
'dwelling_house', 'detached_house', 'houseboat')
GROUP BY b.building
ORDER BY in_osm_buildings DESC;
in_osm_buildings_filtered is 0 for every row.
Expected behavior
Buildings carrying these tags end up in osm_buildings_filtered, the way
semidetached_house, terraced_house, static_caravan and plain
building=yes already do.
Screenshots
n/a
Environment:
- OS: not relevant — the defect is in SQL, not in the runtime
- Python version: not relevant
- Scenario: independent of scenario
- Dataset Boundary: Everything (the figures above); the gap is visible on
Schleswig-Holstein as well
Proposed solution (optional)
Extend the whitelist in osm_buildings_filter.sql with the missing residential
tags, cross-checked against osm_buildings_filter_residential.sql so the two
whitelists stop drifting apart.
Two parts should be separated when doing this:
- The three inconsistencies above are a straightforward correction.
- Whether debatable tags such as
cabin, stilt_house or villa belong on the
list is a modelling decision, not a defect, and can be decided independently.
Additional context
The fix is not free. osm_buildings_filtered is also the CTS candidate set: in
census cells with CTS demand but no amenities, cts_buildings.py spreads the
demand over all filtered buildings in the cell, so adding buildings lowers
the demand per building. The effect is expected to be small, but it should be
measured rather than assumed — which is why this belongs in its own change
rather than riding along with #1310.
Part of #1311. Found while working on #1310, where residential buildings are
determined by intersecting with an external building dataset instead of by OSM
tag — which is how the gap became visible.
Possibly related: #1234 reports missing buildings in the residential layer. That
is a different table, but some of the reported gaps may share this cause. Not
verified.
Describe the bug
The tag whitelist in
osm_buildings_filter.sqlcontradicts the one inosm_buildings_filter_residential.sqland misses several OSM tag variants. As aresult, clearly inhabited buildings never reach
openstreetmap.osm_buildings_filtered, and they disappear from everythingdownstream without a warning.
Three inconsistencies:
semidetached_houseis whitelisted, the variantssemidetachedandsemi_detachedare not.bungalowandfarmhouseare on the residential whitelist inosm_buildings_filter_residential.sql, but not inosm_buildings_filter.sql.dwelling_house,detached_house,houseboat.Measured on a full Germany run: 134,725 buildings (0.71 %) that are
classified as residential do not appear in
osm_buildings_filtered. The largestgroups carrying an unambiguously residential tag are
bungalow(37,402),cabin(6,591) anddwelling_house(4,849). The remainder is dominated byoutbuilding tags such as
garageandshed— a separate matter, not this bug.osm_buildings_filteredis the basis for load area aggregation, CTS buildingselection, the MVGD building mapping and the rooftop PV potentials. A building
missing from it silently loses three things:
loadarea/loadareas_add_demand_hh.sqljoins peak loadsagainst
osm_buildings_filtered; with no geometry match the load drops outof
demand.egon_loadarea. No warning, just less demand.electricity_demand_timeseries/mapping.pywrites no rowto
boundaries.egon_map_zensus_mvgd_buildings, so the building gets noMVGD/bus.
power_plants/pv_rooftop_buildings.pyintersects against thesame set, so the building gets no rooftop potential.
To Reproduce
OsmBuildingsStreetsdataset.in_osm_buildings_filteredis 0 for every row.Expected behavior
Buildings carrying these tags end up in
osm_buildings_filtered, the waysemidetached_house,terraced_house,static_caravanand plainbuilding=yesalready do.Screenshots
n/a
Environment:
Schleswig-Holstein as well
Proposed solution (optional)
Extend the whitelist in
osm_buildings_filter.sqlwith the missing residentialtags, cross-checked against
osm_buildings_filter_residential.sqlso the twowhitelists stop drifting apart.
Two parts should be separated when doing this:
cabin,stilt_houseorvillabelong on thelist is a modelling decision, not a defect, and can be decided independently.
Additional context
The fix is not free.
osm_buildings_filteredis also the CTS candidate set: incensus cells with CTS demand but no amenities,
cts_buildings.pyspreads thedemand over all
filteredbuildings in the cell, so adding buildings lowersthe demand per building. The effect is expected to be small, but it should be
measured rather than assumed — which is why this belongs in its own change
rather than riding along with #1310.
Part of #1311. Found while working on #1310, where residential buildings are
determined by intersecting with an external building dataset instead of by OSM
tag — which is how the gap became visible.
Possibly related: #1234 reports missing buildings in the residential layer. That
is a different table, but some of the reported gaps may share this cause. Not
verified.