Features/#1460 integrate new mit lgv dataset

[nesnoj]

Fixes #1460 .

Replaces the eMobility dataset for motorized individual travel with a new one that covers vehicle class M1 (passenger cars, as before) and N1 (light commercial vehicles < 3.5 t), and that is generated together with the charging infrastructure, so events, vehicles and charging points are mutually consistent.

What changes

Input data instead of derivation. The vehicle pool, the events, the vehicle counts per municipality, the charging locations and the allocation of vehicles to municipalities are delivered by the data providers as parquet, published on Zenodo as one zip archive per scenario and downloaded by the pipeline itself. The KBA/RegioStaR7 chain — registration-district level, population-based disaggregation — is no longer used for the new scenarios. Vehicle profiles are still generated with SimBEV; the charging sites are now placed with GeoLIS from the same run.

Two methodologies side by side. status2024, reGon2037 and reGon2045 use the new methodology; eGon2035 keeps the simBEV trip tarball from the data bundle and the KBA-derived allocation, as a long-term stable scenario. Every task splits the configured scenario list and dispatches on is_legacy_scenario(), so the DAG shape does not depend on --scenarios. Both write the same tables, discriminated by scenario.

Nine vehicle types instead of six: the six private M1 types, plus bev_commercial and phev_commercial (commercially registered passenger cars, still M1) and bev_light_duty_vehicle (the only N1 type; N1 is BEV-only). They are merged into the existing eTraGo components — one Li_ion bus, one BEV_charger link, one battery_storage store and one land_transport_EV load per MV grid district. The eTraGo model shape is unchanged.

Flexible charging is keyed on the charging use case (depot, home_detached, home_apartment, work) rather than on the (location, use_case) pair. street, retail, urban_fast and highway_fast are inflexible.

New: flexibility diagnostics. The pipeline now writes what it already computed and discarded — see Additional Notes below.

Table renames (breaking for external consumers)

All MIT tables move to the egon_ev_mit_lgv_ prefix, for both methodologies, so there is exactly one naming family:

before	after
demand.egon_ev_pool	demand.egon_ev_mit_lgv_pool
demand.egon_ev_trip	demand.egon_ev_mit_lgv_trip
demand.egon_ev_count_registration_district	demand.egon_ev_mit_lgv_count_registration_district
demand.egon_ev_count_municipality	demand.egon_ev_mit_lgv_count_municipality
demand.egon_ev_count_mv_grid_district	demand.egon_ev_mit_lgv_count_mv_grid_district
demand.egon_ev_mv_grid_district	demand.egon_ev_mit_lgv_mv_grid_district
demand.egon_ev_metadata	demand.egon_ev_mit_lgv_metadata

The FK column egon_ev_pool_ev_id becomes ev_id, and the ORM classes follow the tables (EgonEvPool → EgonEvMitLgvPool, …) so the code stays greppable from the table name. grid.egon_emob_charging_infrastructure is not renamed — it is TracBEV output and stays with the legacy path.

Five new tables, all in demand: egon_ev_mit_lgv_mapping_ev_municipality, egon_ev_mit_lgv_charging_location, egon_ev_mit_lgv_flex_timeseries, egon_ev_mit_lgv_energy_balance, egon_ev_mit_lgv_charging_profile_use_case.

egon_ev_mit_lgv_metadata changes shape entirely: the per-key columns (eta_cp, stepsize, start_date, …) are replaced by two JSONB documents holding the simBEV and GeoLIS run configurations verbatim. The delivered key set is not stable — between the old bundle and delivery v1.4 it lost two keys and gained a dozen — and the old code raised KeyError on the current delivery. eDisGo's simbev_config_from_oedb() reads this table and has to unpack the JSON instead.

Other taxonomy changes that break silently and are worth flagging to downstream users: egon_ev_mit_lgv_trip.use_case moves from public/home/work/empty to the eight new values, and .location carries two parallel vocabularies (English for private vehicles, German for commercial ones) selected by the vehicle group.

📋 Pull Request Guidelines

Please read the Pull Request Guidelines carefully before creating your PR.

---

🧑‍💻 Contributor Checklist

Before requesting a review, make sure you've completed all of the following:

• All tests pass locally or via CI
  (for more information on local test, check tox in the Contributing section)
  (CI tests are automatically executed when creating a PR, you can see the results of the checks below)
• Workflow has run at least once in Test mode
  (optional if no dataset changes are involved)
  Not yet — blocked, see Open before merge below.
• Relevant documentation is updated (API, new features, etc.)
• Dataset-versions are updated when existing datasets are adjusted.
  MotorizedIndividualTravel 0.0.12 → 0.1.0, MITChargingInfrastructure 0.0.9 → 0.1.0
• Added a note to CHANGELOG.rst about the changes
• Added yourself to AUTHORS.rst

Optional:

• Changes have been tested in Everything mode
• Extend the checklist for reviewers: Which aspects should be reviewed in particular?

Please focus on:

1. `mit_import.allocate_ev_instances_to_grid_districts()` — the municipality → MV grid
   district split. It is the one genuinely new piece of modelling logic and it operates
   on up to ~40 million vehicle instances, so it is written in numpy rather than pandas
   and is correspondingly dense. The invariants it must hold are: municipal totals
   preserved exactly, no RNG, and every vehicle placed in a grid district that actually
   intersects its own municipality.
2. The dual code path. Every task has to no-op cleanly for the methodology it does not
   serve, in both directions — please sanity-check that a legacy-only and a new-only
   scenario list both produce a sensible DAG run.
3. Whether `egon_ev_mit_lgv_charging_location` belongs to the charging infrastructure
   dataset (as implemented) or to the MIT dataset — see design decision 1 below.
4. Metadata: eleven OEP metadata blocks are now generated from one table-spec loop
   rather than copy-pasted. Worth a look at whether the shared parts (sources, licences,
   contributors) are right per table.

---

🔍 Reviewer Checklist

During your review, please check the following:

• Is the code clean, readable, and efficient? Are there any oddities or obvious inefficiencies?
• Does the code work as expected? (should already be verified by contributor)
• Do all tests pass? (see CI results)
• Is the documentation complete and up to date?
• Is CHANGELOG.rst updated accordingly?
• Is all necessary metadata complete and correct?
  • If metadata is pending: Is there an appropriate issue filed?

---

📝 Additional Notes (optional)

Flexibility diagnostics (D23–D26)

The land_transport_EV load does not always mean the same quantity: it is the grid-side charging energy in dumb and lowflex scenarios but the battery-side driving energy in flexible ones, with no record of which. That was the core methodological defect of the previous dataset, and it is why the reported 37.3 vs. 41.4 TWh looked like an inconsistency when it is in fact the identity Σ driving_load = eta_cp · Σ charging_load_grid.

Three exports now make the reference points explicit, for the new methodology only, without changing the eTraGo model shape:

• egon_ev_mit_lgv_flex_timeseries — charging_load_grid, charging_load_grid_flex and driving_load per grid district and hour, so both sides are recorded for every scenario;
• egon_ev_mit_lgv_energy_balance — annual balance per grid district, use case and vehicle type, carrying the battery-side and the grid-side charging energy;
• egon_ev_mit_lgv_charging_profile_use_case — the dumb charging load per grid district, hour and use case.

All three are computed inside the existing event loop and the existing hourly resample — no extra pass. ~4.5 GB worst case for three scenarios, against the ~350 GB of a full run. docs/data/mobility_demand.rst documents the reference points and the post-processing recipe for Δ(t) and E_shift.

⚠️ charging_load_grid_flex and the flexible flag are a potential, not a realised flexibility — they are populated for status2024 too, which gets no bus, link or store at all. The documentation says so explicitly, in a warning box, because the data invites the misreading.

Design decisions worth a second opinion

1. egon_ev_mit_lgv_charging_location is created and filled by the charging infrastructure dataset, although it lives in demand next to the other M1+N1 tables. The plan claimed it for both datasets in different sections; since the two run in parallel and create_tables() drops before it creates, exactly one of them may own it.
2. The four import tasks run sequentially, not in parallel. A set nested inside a set is unhashable in TaskGraph, and serialising avoids four concurrent bulk loads competing for the same database anyway. The event import dominates the runtime regardless.
3. A cross-process download lock (_DownloadLock) guards the shared archive. D3 keeps the two datasets independent, so both fetch the same multi-GB zip; without mutual exclusion one would read a half-written file.
4. Imports and diagnostics are idempotent (delete-then-write, per scenario or per bus_id), so an Airflow retry recovers instead of hitting the primary key on rows the failed attempt already wrote.
5. add_metadata is now wired into the MIT DAG. It was imported but never used as a task, so MIT metadata was never actually written. Refactored from ~370 lines of copy-paste into a table-spec loop covering all eleven tables.
6. pyarrow added to pyproject.toml — the parquet import path needs it and it was not a dependency.

ev_mapping_event_location is deliberately not imported

It carries one row per (charging event × drawn vehicle): 499,677,447 rows measured for status2024, and an estimated ~6.9·10⁹ / ~8.2·10⁹ for reGon2037 / reGon2045 — roughly 370 GB and 440 GB in PostgreSQL before indexes, against a ~350 GB budget for the entire pipeline. Nothing in the database needs it, because charging locations carry their own use_case since delivery v1.4. It stays a parquet file in the extracted scenario directory and the documentation says where to find it.

Vehicles in unknown municipalities are dropped, loudly

Delivery v1.4 covers 10,615 municipalities; VG250 has 11,003 with an MV grid district. The split joins on ags, so a delivered ags that VG250 does not know contributes no population share. Behaviour: continue, drop, and log the number of affected municipalities, the vehicles lost per type and in total, and that total as a share of the delivered fleet — plus a pointed hint when it exceeds ~0.5 %. Aborting the pipeline on ordinary reference-data drift (municipalities merge and are renumbered) is a worse outcome than a documented, quantified loss. This decision is provisional and needs revisiting once the number is measured on a real full delivery of each scenario; if it turns out material, the fix is an AGS-vintage translation table, not a tolerance.

What was verified, and how

Since the Zenodo records do not exist yet and no database was available, verification was done against the local v1.4 delivery and in isolation:

• Allocation logic — unit-tested: municipal totals preserved exactly, deterministic across input permutations (remainder ties broken by bus_id, no RNG), and every placed vehicle lands in a grid district that intersects its own municipality.
• Test-mode filter — run against the real status2024 delivery, reproducing the plan's projections exactly: 1,074 municipalities, 92,047 vehicle instances, 23,840 distinct pool EVs (56.9 % of the pool). Note that the event table shrinks far less than the region share suggests, because the pool is national and each profile is instantiated ~65× across Germany. That is inherent, not a defect — plan test-mode runtime and disk accordingly.
• COPY encoding — checked on real events and geoparquet: use_case NULLs survive as unquoted empty fields (PostgreSQL CSV reads those as NULL, empty strings as ""), booleans as true/false, geometries as valid WKB hex.
• DDL — inspected for every new and modified table.
• pytest tests/ passes, the pipeline DAG imports, and black / isort / flake8 are clean on everything touched (only pre-existing F401s remain elsewhere).

Open before merge

These need the real data and are not done:

• grep -rn PLACEHOLDER src/ must come back empty. The Zenodo records do not exist yet, so the URLs are deliberate literal PLACEHOLDERs — an unreplaced one fails with an obvious 404 rather than fetching something plausible-looking.
• Switch ZENODO_ENVIRONMENT from zenodo_sandbox to zenodo and fill in the production record ids — these are separate deployments, so it is not a host substitution.
• Test-mode run from scratch for status2024 + reGon2037 + reGon2045, and one for eGon2035 proving the legacy path is intact.
• Revisit the AGS-drop decision on the measured vehicle loss (see above).
• omi-validate the thirteen metadata strings — eleven MIT tables plus TracBEV's and the charging locations (needs a populated database).

Not in scope

• sanity_checks.py is not adapted. It is written against the KBA-derived allocation and the old event semantics, is already excluded from the pipeline, and will be superseded by a new validation concept in a separate effort. Only its imports were updated so the module still imports. What this PR provides instead is one hard assertion at import (every pool type has tech_data) plus logged, non-fatal consistency information after each import and after the diagnostics.
• The orphaned duplicate src/egon/data/datasets/helpers.py (imported nowhere, carries a stale copy of read_simbev_metadata_file) is left alone — drop_old_methodology.md §6.2 owns its deletion.
• Removal of the legacy path itself is deferred until eGon2035 is retired.

⚠️ Structural constraint: create_tables() drops and recreates all MIT tables, so eGon2035 and the new scenarios must be built in the same run. Running eGon2035 afterwards destroys the new scenarios' rows.

[nesnoj]

Hey @CarlosEpia , SH was successful and I merged this into server-run-11-09.
The required input data will be downloaded from zenodo in the future. For testing, I copied the required files to your server run-de/emobility/input_data/, please do not remove prior to the run - thx.