[BUG] Small unexplained residual in heat-pump capacity conservation after accounting for known causes (nationwide run) #1512

Summary

On a nationwide (--dataset-boundary: Everything) status2024/reGon2037 run, total distributed hp_capacity falls slightly short of the scenario's capacity target in both scenarios. Most of the gap is explained by already-known causes, but a small residual remains unexplained after ruling those out.

Numbers

Truncation figure computed with the diagnostic from a prior SH-run writeup:

SELECT scenario,
       sum(hp_capacity::double precision) - sum(hp_capacity)::double precision AS trunc_err
FROM demand.egon_hp_capacity_buildings GROUP BY 1;

This matches closely (status2024: 0.141 MW, reGon2037: 5.351 MW) and is expected/benign — hp_capacity is real (float32) while the target in supply.egon_scenario_capacities-style tables is double precision; desaggregate_hp_capacity() computes exactly in float64 but each per-building value truncates on write.

What's ruled out

• Not the scaling step. desaggregate_hp_capacity() (min * fac + min with fac = (target - sum(min)) / sum(min)) is algebraically exact whenever it runs — it scales the selected buildings to sum to exactly the grid's remaining budget, by construction. There is no quantization/rounding gap left over from this step to explain a residual of this size.
• Not PV-rooftop duplicates (a previously-fixed cause, 699bc1f7e) as the direct mechanism — the code-level guards (.building_id.drop_duplicates() in the PV-weighting query, drop_duplicates() on buildings_with_hp) are present in this checkout and should structurally neutralize duplication before any capacity is allocated, regardless of scale. (Separately: status2024's egon_power_plants_pv_roof_building has 3,928,270 duplicate building_id rows on this run, vs. 72 on the SH run that originally motivated the fix — the upstream duplication was apparently regenerated. Not confirmed to be causing the residual, but flagging since it's a large jump in scale for a mechanism that was tricky to diagnose once already.)
• Not the /16 double-boundary-scaling bug or the proxy-scenario-mapping bug — both already-fixed in this checkout and confirmed not applicable (--dataset-boundary: Everything makes the /16 bug moot anyway; the datasets.yml proxy mappings resolve to real, populated scenarios).

What's a known, separate confounder (already reported) but doesn't fully explain this

A genuine Airflow DAG-dependency gap allows some reGon2037 heat-pump-capacity bulk tasks to run before all status2024 bulk tasks commit, causing ~15 MV grids to lose their inherited floor (reported separately). This measurably removes some capacity from those specific grids. However, quantifying its exact contribution to the national residual requires a correct building→MV-grid mapping, which is where this investigation stalled (see below).

Open question / where I got stuck

I tried to localize the residual to specific MV grids by joining demand.egon_hp_capacity_buildings → boundaries.egon_map_zensus_buildings_filtered_all → boundaries.egon_map_zensus_grid_districts and comparing each grid's distributed sum against its target in supply.egon_individual_heating. This produced an implausible result (1,046 of 4,079 grids apparently >1% under target, 353.6 MW "missing", zero grids over target) that turned out to be an artifact of that join silently dropping ~16,600 buildings (0.85%) — accounting for ~484 MW of apparent shortfall on its own, i.e. more than the real residual. So this mapping is not the (or not a complete) building→MV-grid association the pipeline itself uses for get_total_heat_pump_capacity_of_mv_grid/determine_hp_cap_buildings_pvbased_per_mvgd, and I didn't find the correct one.

Question for whoever picks this up: what is the authoritative building→MV-grid mapping used inside determine_hp_cap_peak_load_mvgd_ts_status_quo/_2035 (the mvgd_ids loop), and does redoing the per-grid target-vs-distributed comparison with that mapping localize the remaining ~0.28–0.73 MW residual to specific grids or a specific code path? Given the scaling step is exact by construction, the residual most likely comes from grids that never fully entered the top-up branch at all (e.g. remaining_cap_grid <= 0.0 or min_hp_cap_remaining.empty short-circuiting before scaling), rather than from an error within the scaling itself — but this needs the correct grid mapping to confirm.

Environment

• eGon-data commit 8dc7727b1482cef0caf4f0850224f102be82a263 (branch server-run-24-08)
• Nationwide (--dataset-boundary: Everything) run, status2024/reGon2037 scenarios