1475 feature rework status2024 demand data rerun generation independently instead of proxying egon2035 and verify hp capacity placeholder - #1490
Draft
Draft
Conversation
added 6 commits
August 7, 2026 10:56
status2024 has no demand or PV data of its own yet, so route its scenario-dependent reads (peta_heat, CTS heat demand share/profile, DH-exclusion mapping) through eGon2035, and PV rooftop weighting through the legacy status_quo rows, via a central mapping in datasets.yml rather than scattered hardcoded strings. Also adds a placeholder national HP capacity (2M x 5kW) and reuses the status2023 battery-storage figure, both marked TODO pending verification against a primary source.
egon_cts_heat_demand_building_share and egon_etrago_heat_cts already contain real, independently-computed status2024 rows, so the earlier demand_source_scenario redirect in calc_cts_building_profiles was silently substituting eGon2035 data instead of using what's already there. Also restores the original sourced comment on rural_heat_capacity in scenario_capacities.py, which a prior commit had overwritten with a stale "needs verification" note even though the figure was already cited to NEP 2035/2021 and NEP 2037/2045/2025.
insert_nep_list_powerplants() loads the NEP2021 list for eGon2035 and the NEP2025 list for reGon2037/reGon2045. When only one of those scenario groups is configured, the other DataFrame stays empty, so the concatenated result is missing that list's capacity columns. Two places assumed all columns are present, so a run with --scenarios [status2024, eGon2035] and a federal-state boundary failed twice in a row: KeyError: 'c2037_capacity' KeyError: "['c2037_capacity', 'c2045_capacity'] not in index" Restrict the testmode scaling loop to columns that were actually loaded, and use reindex() in aggr_nep_capacities() so missing columns become NaN. No fabricated values enter the results: the reGon aggregation filters on scenario == 'reGon', which yields no rows when NEP2025 was not loaded, and dropna(subset=["nuts"]) clears the empty frame. Both conditions must hold to trigger this: a partial NEP scenario set and --dataset-boundary != "Everything". Full-Germany runs skip the scaling block entirely.
status2024 originally had no demand or PV data of its own, so demand_source_scenario() and pv_source_scenario() pointed it at other scenarios. Both now resolve to status2024 itself: real data is present in egon_peta_heat, egon_map_zensus_district_heating_areas and egon_power_plants_pv_roof_building. The demand proxy mattered: status2024 residential heat demand is ~58% higher than eGon2035's (22.0 vs 13.9 TWh), so proxying sized heat pumps from the wrong profile. The PV mapping pointed at "status_quo", which does not exist in egon_power_plants_pv_roof_building (it holds only eGon2035 and status2024). The lookup matched zero rows, so every building fell back to equal weight and the PV weighting had no effect at all -- silently, with no error. The comment now warns against pointing a mapping at an absent scenario. Refs #1475
…ection
Two defects made the status2024 heat pump distribution wrong.
1. Double testmode scaling.
The 'rural_heat_pump' target is already reduced to the dataset boundary by
population_share() when insert_capacities_status_quo() writes it (3.49% for
Schleswig-Holstein). cascade_per_technology()'s national branch then divided
by 16 again as a crude "1 of 16 federal states" proxy, so the distributed
capacity was ~16x too low: 21.84 MW instead of 349.42 MW.
eGon2035 never hit this because it takes the federal_states branch, which
reads a per-state row and has no such division. The national branch is used
only by status scenarios, so the /16 predates them and assumed an unscaled
target.
2. Duplicate building ids in the PV-weighted selection.
egon_power_plants_pv_roof_building contains every status2024 row twice
(233,700 rows for 116,850 buildings -- tracked separately as a PowerPlants
bug). Those duplicates entered the weights index, so np.random.choice could
draw the same building twice. Both copies were allocated capacity, and the
second was then silently dropped by drop_duplicates("building_id") in the
bulk export -- removing that capacity from the grid's budget with no
rescaling. Measured at 72 rows / 1.68 MW of 349.42 MW in one run.
Deduplicate and sort the PV query result, and guard the returned selection.
The sort also makes the seeded draw reproducible: the query has no ORDER BY,
so the weights index was built in arbitrary order and --random-seed did not
pin the outcome. Repeated runs previously returned 87/90/92 buildings for
the same grid.
Versions bumped since both change dataset output.
Refs #1475
Sanity checks only cover eGon2035 and eGon100RE. With neither configured -- e.g. --scenarios [status2024, eGon2035] once eGon2035 is dropped, or a status-only run -- the tasks tuple was empty and Dataset raised, breaking DAG construction. Fall back to a no-op task that reports which scenarios were configured and that nothing was checked, so the pipeline builds and the gap is visible in the logs rather than silent. Status-quo scenario sanity checks are still to be written; this only stops their absence from breaking the DAG. Refs #1475
MoritzSchloesser
pushed a commit
that referenced
this pull request
Aug 24, 2026
MoritzSchloesser
pushed a commit
that referenced
this pull request
Aug 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Replaces the
status2024demand/PV proxying with direct reads, and fixes three defects foundwhile verifying the resulting heat pump distribution.
Closes #1475 (partially — see Still missing below, which is why this is a draft).
Changes
1. Read
status2024data directly instead of proxying eGon2035demand_source_scenario()andpv_source_scenario()now resolvestatus2024to itself. Realdata is present in
egon_peta_heat,egon_map_zensus_district_heating_areasandegon_power_plants_pv_roof_building.The demand proxy was not cosmetic:
status2024residential heat demand is ~58 % higher thaneGon2035's (22.0 vs 13.9 TWh), so heat pumps were being sized from the wrong profile.The PV mapping pointed at
"status_quo", a scenario that does not exist inegon_power_plants_pv_roof_building(it holds onlyeGon2035andstatus2024). The lookupmatched zero rows, so every building fell back to equal weight and the PV weighting had no effect
at all — silently, with no error.
2. Fix double boundary scaling of the status-quo HP target
insert_capacities_status_quo()already reduces the nationalrural_heat_pumptarget to thedataset boundary via
population_share()(3.49 % for Schleswig-Holstein).cascade_per_technology()'snationalbranch then divided by 16 again as a crude"1 of 16 federal states" proxy.
population_share()eGon2035never hit this: it takes thefederal_statesbranch, which reads a per-state row andhas no such division. The
nationalbranch is used only by status scenarios, so the/16predates them and assumed an unscaled target. Full-Germany runs were unaffected.
3. Fix duplicate building ids in the PV-weighted selection
egon_power_plants_pv_roof_buildingcontains everystatus2024row twice (233,700 rows for116,850 buildings). Those duplicates entered the
weightsindex indetermine_buildings_with_hp_in_mv_grid(), sonp.random.choicecould draw the same buildingtwice. Both copies were allocated capacity, and the second was then silently dropped by
drop_duplicates("building_id")in the bulk export — removing that capacity from the grid'sbudget with no rescaling.
Measured via temporary instrumentation:
pre_dedup_sum= 349.417503 MW (exactly the budget),then 72 rows / 1.68 MW dropped before the write.
Deduplicating and sorting the PV query result also makes the seeded draw reproducible. The
query has no
ORDER BY, so the weights index was built in arbitrary order and--random-seeddid not pin the outcome — repeated runs returned 87 / 90 / 92 buildings for the same grid.
4. Fix
KeyErrorwhen only part of the NEP scenarios are configuredinsert_nep_list_powerplants()loads NEP2021 foreGon2035and NEP2025 forreGon2037/reGon2045. With only one group configured the other frame stays empty, so theconcatenated result lacks that list's capacity columns. Two places assumed all columns exist, so
a run with
[status2024, eGon2035]on a federal-state boundary failed twice in a row:Pre-existing on
dev; needs both a partial NEP scenario set and--dataset-boundary != "Everything"to trigger, which is why CI has not caught it.5. Let
SanityChecksbuild with no implemented scenario checksSanity checks only cover
eGon2035andeGon100RE. With neither configured the tasks tuple wasempty and
Datasetraised, breaking DAG construction. Falls back to a no-op that logs whichscenarios ran unchecked.
Verification
status2024vseGon2035in a Schleswig-Holstein run:status2024eGon2035Checks that hold for both scenarios:
sum(hp_capacity)equals theHeatSupplybudget: 349.41800 vs349.41750 MW for
status2024(0.0001 % float rounding), 1654.36000 vs 1654.36833 MW foreGon2035. Every MV grid reaches its budget.peak_load × (24/18) / 1.7; no undersized buildingshp_capacityin MW,peak_load_in_win W, eTraGo series length 8760A review notebook covering these is available on request (not committed — it is analysis, not
pipeline code).
Still missing (why this is a draft)
DAG; the checks themselves are unwritten.
docs/adr/. Either they werenever committed or they live elsewhere — worth resolving before merge so the references are not
dangling.
Out of scope
status2024's per-building assignments as a fixed floor ([FEATURE] Make future-scenario HP distribution treat status2024's assignments as a fixed floor #1477)power_plants/,chp/,demandregio/,motorized_individual_travel/pv_rooftop_to_buildings()duplication itself (filed separately)