Skip to content

1475 feature rework status2024 demand data rerun generation independently instead of proxying egon2035 and verify hp capacity placeholder - #1490

Draft
MoritzSchloesser wants to merge 6 commits into
devfrom
1475-feature-rework-status2024-demand-data-rerun-generation-independently-instead-of-proxying-egon2035-and-verify-hp-capacity-placeholder
Draft

MoritzSchloesser wants to merge 6 commits into
devfrom
1475-feature-rework-status2024-demand-data-rerun-generation-independently-instead-of-proxying-egon2035-and-verify-hp-capacity-placeholder

Conversation

@MoritzSchloesser

Copy link
Copy Markdown
Contributor

What this does

Replaces the status2024 demand/PV proxying with direct reads, and fixes three defects found
while verifying the resulting heat pump distribution.

Closes #1475 (partially — see Still missing below, which is why this is a draft).

Changes

1. Read status2024 data directly instead of proxying eGon2035

demand_source_scenario() and pv_source_scenario() now resolve status2024 to itself. Real
data is present in egon_peta_heat, egon_map_zensus_district_heating_areas and
egon_power_plants_pv_roof_building.

The demand proxy was not cosmetic: status2024 residential heat demand is ~58 % higher than
eGon2035's (22.0 vs 13.9 TWh), so heat pumps were being sized from the wrong profile.

The PV mapping pointed at "status_quo", a scenario that does not exist in
egon_power_plants_pv_roof_building (it holds only eGon2035 and status2024). The lookup
matched zero rows, so every building fell back to equal weight and the PV weighting had no effect
at all — silently, with no error.

2. Fix double boundary scaling of the status-quo HP target

insert_capacities_status_quo() already reduces the national rural_heat_pump target to the
dataset boundary via population_share() (3.49 % for Schleswig-Holstein).
cascade_per_technology()'s national branch then divided by 16 again as a crude
"1 of 16 federal states" proxy.

before after
national target 10,000 MW 10,000 MW
× population_share() 349.42 MW 349.42 MW
÷ 16 21.84 MW (removed)
distributed to MV grids 21.84 MW 349.42 MW

eGon2035 never hit this: it takes the federal_states branch, which reads a per-state row and
has no such division. The national branch is used only by status scenarios, so the /16
predates them and assumed an unscaled target. Full-Germany runs were unaffected.

3. Fix duplicate building ids in the PV-weighted selection

egon_power_plants_pv_roof_building contains every status2024 row twice (233,700 rows for
116,850 buildings). Those duplicates entered the weights index in
determine_buildings_with_hp_in_mv_grid(), so np.random.choice could draw the same building
twice. Both copies were allocated capacity, and the second was then silently dropped by
drop_duplicates("building_id") in the bulk export — removing that capacity from the grid's
budget with no rescaling.

Measured via temporary instrumentation: pre_dedup_sum = 349.417503 MW (exactly the budget),
then 72 rows / 1.68 MW dropped before the write.

Deduplicating and sorting the PV query result also makes the seeded draw reproducible. The
query has no ORDER BY, so the weights index was built in arbitrary order and --random-seed
did not pin the outcome — repeated runs returned 87 / 90 / 92 buildings for the same grid.

The duplicated rows are a bug in pv_rooftop_to_buildings() (the accumulator is seeded with the
status-quo rows before a loop that appends them again). That is filed separately, since it also
doubles status-quo PV capacity for home_batteries and sanity_checks. The change here is
defensive so the HP path is correct regardless.

4. Fix KeyError when only part of the NEP scenarios are configured

insert_nep_list_powerplants() loads NEP2021 for eGon2035 and NEP2025 for
reGon2037/reGon2045. With only one group configured the other frame stays empty, so the
concatenated result lacks that list's capacity columns. Two places assumed all columns exist, so
a run with [status2024, eGon2035] on a federal-state boundary failed twice in a row:

KeyError: 'c2037_capacity'
KeyError: "['c2037_capacity', 'c2045_capacity'] not in index"

Pre-existing on dev; needs both a partial NEP scenario set and
--dataset-boundary != "Everything" to trigger, which is why CI has not caught it.

5. Let SanityChecks build with no implemented scenario checks

Sanity checks only cover eGon2035 and eGon100RE. With neither configured the tasks tuple was
empty and Dataset raised, breaking DAG construction. Falls back to a no-op that logs which
scenarios ran unchecked.

Verification

status2024 vs eGon2035 in a Schleswig-Holstein run:

status2024 eGon2035
heat pumps 14,934 103,022
capacity 349.417 MW 1,654.370 MW
mean HP size 23.40 kW 16.06 kW
conservation 100.0000 % 99.9998 %

Checks that hold for both scenarios:

  • Capacity conservationsum(hp_capacity) equals the HeatSupply budget: 349.41800 vs
    349.41750 MW for status2024 (0.0001 % float rounding), 1654.36000 vs 1654.36833 MW for
    eGon2035. Every MV grid reaches its budget.
  • Sizing rule — every building gets at least peak_load × (24/18) / 1.7; no undersized buildings
  • Unitshp_capacity in MW, peak_load_in_w in W, eTraGo series length 8760
  • PV weighting active — PV-owning buildings are over-represented among HP buildings

A review notebook covering these is available on request (not committed — it is analysis, not
pipeline code).

Still missing (why this is a draft)

Out of scope

Moritz.Schloesser added 6 commits August 7, 2026 10:56
status2024 has no demand or PV data of its own yet, so route its
scenario-dependent reads (peta_heat, CTS heat demand share/profile,
DH-exclusion mapping) through eGon2035, and PV rooftop weighting
through the legacy status_quo rows, via a central mapping in
datasets.yml rather than scattered hardcoded strings. Also adds a
placeholder national HP capacity (2M x 5kW) and reuses the status2023
battery-storage figure, both marked TODO pending verification against
a primary source.
egon_cts_heat_demand_building_share and egon_etrago_heat_cts already
contain real, independently-computed status2024 rows, so the earlier
demand_source_scenario redirect in calc_cts_building_profiles was
silently substituting eGon2035 data instead of using what's already
there. Also restores the original sourced comment on
rural_heat_capacity in scenario_capacities.py, which a prior commit
had overwritten with a stale "needs verification" note even though
the figure was already cited to NEP 2035/2021 and NEP 2037/2045/2025.
insert_nep_list_powerplants() loads the NEP2021 list for eGon2035 and the
NEP2025 list for reGon2037/reGon2045. When only one of those scenario
groups is configured, the other DataFrame stays empty, so the concatenated
result is missing that list's capacity columns.

Two places assumed all columns are present, so a run with
--scenarios [status2024, eGon2035] and a federal-state boundary failed
twice in a row:

  KeyError: 'c2037_capacity'
  KeyError: "['c2037_capacity', 'c2045_capacity'] not in index"

Restrict the testmode scaling loop to columns that were actually loaded,
and use reindex() in aggr_nep_capacities() so missing columns become NaN.
No fabricated values enter the results: the reGon aggregation filters on
scenario == 'reGon', which yields no rows when NEP2025 was not loaded,
and dropna(subset=["nuts"]) clears the empty frame.

Both conditions must hold to trigger this: a partial NEP scenario set and
--dataset-boundary != "Everything". Full-Germany runs skip the scaling
block entirely.
status2024 originally had no demand or PV data of its own, so
demand_source_scenario() and pv_source_scenario() pointed it at other
scenarios. Both now resolve to status2024 itself: real data is present in
egon_peta_heat, egon_map_zensus_district_heating_areas and
egon_power_plants_pv_roof_building.

The demand proxy mattered: status2024 residential heat demand is ~58%
higher than eGon2035's (22.0 vs 13.9 TWh), so proxying sized heat pumps
from the wrong profile.

The PV mapping pointed at "status_quo", which does not exist in
egon_power_plants_pv_roof_building (it holds only eGon2035 and status2024).
The lookup matched zero rows, so every building fell back to equal weight
and the PV weighting had no effect at all -- silently, with no error. The
comment now warns against pointing a mapping at an absent scenario.

Refs #1475
…ection

Two defects made the status2024 heat pump distribution wrong.

1. Double testmode scaling.

The 'rural_heat_pump' target is already reduced to the dataset boundary by
population_share() when insert_capacities_status_quo() writes it (3.49% for
Schleswig-Holstein). cascade_per_technology()'s national branch then divided
by 16 again as a crude "1 of 16 federal states" proxy, so the distributed
capacity was ~16x too low: 21.84 MW instead of 349.42 MW.

eGon2035 never hit this because it takes the federal_states branch, which
reads a per-state row and has no such division. The national branch is used
only by status scenarios, so the /16 predates them and assumed an unscaled
target.

2. Duplicate building ids in the PV-weighted selection.

egon_power_plants_pv_roof_building contains every status2024 row twice
(233,700 rows for 116,850 buildings -- tracked separately as a PowerPlants
bug). Those duplicates entered the weights index, so np.random.choice could
draw the same building twice. Both copies were allocated capacity, and the
second was then silently dropped by drop_duplicates("building_id") in the
bulk export -- removing that capacity from the grid's budget with no
rescaling. Measured at 72 rows / 1.68 MW of 349.42 MW in one run.

Deduplicate and sort the PV query result, and guard the returned selection.
The sort also makes the seeded draw reproducible: the query has no ORDER BY,
so the weights index was built in arbitrary order and --random-seed did not
pin the outcome. Repeated runs previously returned 87/90/92 buildings for
the same grid.

Versions bumped since both change dataset output.

Refs #1475
Sanity checks only cover eGon2035 and eGon100RE. With neither configured --
e.g. --scenarios [status2024, eGon2035] once eGon2035 is dropped, or a
status-only run -- the tasks tuple was empty and Dataset raised, breaking
DAG construction.

Fall back to a no-op task that reports which scenarios were configured and
that nothing was checked, so the pipeline builds and the gap is visible in
the logs rather than silent.

Status-quo scenario sanity checks are still to be written; this only stops
their absence from breaking the DAG.

Refs #1475
MoritzSchloesser pushed a commit that referenced this pull request Aug 24, 2026
MoritzSchloesser pushed a commit that referenced this pull request Aug 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] Rework status2024 demand data: rerun generation independently instead of proxying eGon2035, and verify HP capacity placeholder

1 participant