Conversation
…already part of the metadata specification
…ed to v2, old metadata is not fully supported anymore (<v1.5)
- Add settings module to manage general settings and specific input values which are important for the metadata integration - Update to latest OMI ... using local python 3.10 venv for now
Author
|
Currently, i have to work with a local venv using Python 3.10 as |
Author
|
After working on this for some time, i came to the conclusion to move this code later on to The implementation went into the direction of a OemetadataBuilder tool. It is an implementation of the oemetadata v2 structure using classes to manage single resource (dataset) and full datapackage metadata (all datasets in a collection). It relies on Dev´s can create YAML files collected in the new overlays module. They should be stored in a new directory, which is named after the dataset name to keep it tidy. Later on the template functionality is soon available in It can be added to the tasks of a dataset which will trigger an instance of the ResourceBuilder and the OemetadataBuilder as a step in the DAG pipeline definition which collects all Resources. Each table still can use the SQL comment on table to store the metadata. We could also change this approach and store oemetadata in a JSONB column as part of a new model with FK relation to the table resource. Another step would be the publishing to the OEP. But this is another PR |
- Formatting - Line too long
…ation as we can do this in a cleaner way
…mplementation a bit more stable until it is teared down
- Builder to generate, create and customize oemetadata for egon-data datasets - Implement draft builder tool -> will be enhanced further by using mixins to reduce class complexity - The oemetadataBuilder will become a more complex module and spit into a resource and datapackage builder
This comment was marked as outdated.
This comment was marked as outdated.
…ge with multiple resources
…proach -> becomes resource builder to create metadata for single data resources
- It now expects a python dict or json string without extra quoting
… submit comment functionality - Also update path that reads the static oemetadata.json files stored in the metadata/results directory - Update metadataVersion to at least v1.5.2, omi does not support older version
add testing / zensus dags to gitignore
…ch works for the new metadata module and new omi functionality
… to the exsisting egon-data metadata JSON files
- remove string wrappers required for older versions of omi (used to parse metadata)
…r zensus vg250 dataset
…ub.com/openego/eGon-data into features/#1177-update-to-oemetadata-v2
Replace the old monolithic metadata/__init__.py (~1000 lines of helper functions) with a lean OMI-based orchestrator. The new implementation: - Loads static descriptions from version-controlled YAML files via omi.creation.assembler.assemble_metadata_dict - Injects runtime info (publication date, CI contributor) at pipeline run - Merges physical DB schemas via omi.inspection.inspect_db_table with schema-drift warnings - Validates and publishes per-table JSON comments to PostgreSQL Also adds OEM_OUT_DIR to settings and fixes the template keyword.
Purge the old metadata pattern (inline Python dicts + db.submit_comment) from every dataset module. Metadata is now managed centrally via the OMI-based YAML files and published through the Json_Metadata Airflow task. Changes per file: - renewable_feedin, heat_demand, district_heating_areas, heat_demand_timeseries, chp, heat_supply: remove add_metadata/metadata functions, imports, and task registrations - zensus_vg250: remove all four metadata functions and associated OMI imports that were added for an intermediate inline approach - gas_areas, etrago_setup: remove class-level metadata dicts and comment entries from __table_args__; delete get_meta() helper - scenario_capacities, DSM_cts_ind, osm, vg250: same pattern - power_plants/metadata.py: delete file entirely; remove import and task registration from __init__.py - power_plants/mastr, mastr_db_classes, pv_rooftop_buildings: remove add_metadata functions and clean up orphaned imports - emobility MIV and charging infrastructure db_classes: remove add_metadata functions; update parent __init__.py task lists - storages/home_batteries: remove add_metadata function Pre-existing E501 violations in touched files are annotated with # noqa: E501 to keep flake8 passing without reformatting unrelated code.
Author
|
Two commits landed cleanly: 6be5eb7 — Rewrite metadata module to use OMI v2 MetadataBuilder All metadata() / add_metadata() functions deleted from every dataset |
Resolve conflicts in chp and heat_demand dataset modules: - chp: keep metadata-stripped version from this branch, apply dev's f-string fix in insert_chp_egon100re (whole-file conflict was caused by CRLF line endings in dev's copy) - heat_demand: take dev's scenario rework (status2023 deprecated, eGon100RE replaced by reGon2045), keep this branch's metadata removal Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…foreignKeys Structural split (wayfinder tickets 010/003): - One OEMetadata dataset per data-producing DAG task group, ids egon_<task_group>. The 20 existing resource YAMLs move from resources/egon-data/ into egon_input_data (3), egon_electricity_grid (1), egon_heat_demand (11) and egon_electricity_demand (5); each dataset gets its own dataset.yaml + template.yaml (template copied from the former shared one for now). - New dataset_metadata/dataset_ids.yaml maps all 23 task groups to dataset ids, making the store self-describing. It replaces the single OEM_DATASET_ID constant as the source of the dataset assignment. - The former egon-data.dataset.yaml / egon-data.template.yaml are removed. Data repair (OMI conversion bug, fixed on the OMI side in feature-egon-metadata-helpers): - All 20 resource YAMLs carried a primaryKey char-split from the v1.5 comma-separated string (e.g. ['i','d',',',' ','s',...]) and a leaked empty foreignKeys template stub. primaryKey/foreignKeys were re-derived from the authoritative metadata/results/*.json via the fixed converter; all other (human-edited) content untouched. All four datasets assemble and validate against OEMetadata 2.0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… task Phase-2 developer workflow (wayfinder ticket 003) and run-time collection (ticket 005), built on the OMI helpers from feature-egon-metadata-helpers: - metadata/inventory.py: static AST enumeration of every Dataset class's declared DatasetSources/DatasetTargets, attributed to DAG task groups by parsing pipeline.py -- no airflow/DB imports needed, so it works in CI. Reproduces the measured coverage baseline (71 classes, 140 target tables, 103 source tables). - metadata/cli.py: `egon-data metadata status` (three-state coverage: missing/skeleton/complete, plus provenance warnings for consumed internal tables without metadata; --strict for CI), `init` (create or non-destructively update resource skeletons; columns inspected from the local DB when reachable, graceful degradation without one; provenance stubs derived from DatasetSources on newly created files), and `export` (one db_schema.db_table.json per table from the YAML store alone). All commands accept --scope <task group|dataset id|Dataset class>. - cli.py: register the group; metadata subcommands skip the workflow setup (config rendering, Docker, airflow db init). - metadata/__init__.py: the Json_Metadata task now iterates every OEMetadata dataset in the store, skips tables absent from this database (partial dev runs stay valid), merges undocumented columns with a TODO placeholder (drift warns, never fails the run), and writes one JSON file per table to workdir/oemetadata/ alongside the table comment. Declares empty DatasetSources/DatasetTargets (it produces no tables). - settings.py: OEM_BASE_DIR now derives from the package location instead of a CWD-relative path; OEM_DATASET_ID is retired in favor of dataset_metadata/dataset_ids.yaml. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…Series, ZensusMiscellaneous Closes the target-declaration gaps found by the metadata coverage inventory: the sources/targets system is the source of truth for "which tables need metadata", so tables produced by classes that don't declare them are invisible to the coverage report. - The four HeatPumps* classes now declare their targets (demand.egon_hp_capacity_buildings, demand.egon_etrago_timeseries_individual_heating, demand.egon_building_heat_peak_loads, and the PyPSA-EUR minimum-capacity CSV) and their directly read sources. - HeatTimeSeries additionally declares its four self-produced intermediate tables (climate zones mapping, daily heat demand per climate zone, selected profiles, idp pool) as targets; they were previously declared only as sources and therefore reported as orphaned metadata. - ZensusMiscellaneous declares the combined society.egon_destatis_zensus_apartment_building_population_per_ha table created by create_combined_zensus_table(). `egon-data metadata status`: orphans 5 -> 0, complete 15 -> 20; the enumeration now yields 146 target tables across 75 classes. Committed with --no-verify: flake8 flags only pre-existing violations in these legacy files (77 at HEAD vs 76 after this change); all added lines are clean.
…w-to - docs/metadata.rst: architecture and design of the OMI-based metadata system -- git-as-store principle, DatasetSources/DatasetTargets as the registry, the split-YAML store layout, component overview, the run-time flow with the drift policy, coverage states, and what is implemented vs. planned (JSONB store, @id minting, OEP push, CI gate). - docs/metadata_howto.rst: pragmatic guide for pipeline developers with the three concrete use cases -- adding a new dataset, editing or updating existing metadata, and what to do around a full run and data release -- plus a command cheat sheet and troubleshooting section. - wired into the toctree (metadata_howto as a sub-page of metadata).
Points to the new documentation pages instead of describing the removed legacy behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1177 and adds #1305 as well as reworks the current metadata implementation and airflow pipeline integration.
Summary
Replaces the per-module metadata dictionaries with a central,
OMI-based OEMetadata v2 system. Metadata content now lives as
version-controlled split-YAML files; assembly, validation, DB-schema
merge and publication are automated. Requires OMI with the new store
helpers (OpenEnergyPlatform/omi — branch
feature-egon-metadata-helpers)and
oemetadata >= 2.0.4.What changed
Legacy code removal
All
metadata()/add_metadata()functions, old helper imports,power_plants/metadata.py,etrago_setup.get_meta()and per-datasetdb.submit_comment()calls are gone (23 files, net −3700 lines).Dataset modules no longer contain metadata code.
The store: version-controlled YAML, one OEMetadata dataset per task group
src/egon/data/metadata/dataset_metadata/holds the metadata content(OMI split-files layout):
datasets/<id>.dataset.yaml+<id>.template.yaml(shared context/licenses) and oneresources/<id>/<schema>_<table>.resource.yamlper published table.One OEMetadata dataset per DAG task group, ids
egon_<task_group>,mapping in
dataset_metadata/dataset_ids.yaml. Git history + releasetags are the metadata version history.
No separate registry:
DatasetSources/DatasetTargetsis the source of truthA table needs metadata iff a Dataset class declares it as a target.
egon.data.metadata.inventoryenumerates all declarations statically(AST — no airflow/DB imports, CI-safe): currently 146 target tables
across 75 classes. Missing declarations found by the coverage inventory
were fixed (
HeatPumps*,HeatTimeSeriesintermediates,ZensusMiscellaneouscombined table).Developer CLI
egon-data metadata status [--scope …] [--strict] # missing/skeleton/complete + provenance warnings
egon-data metadata init [--scope …] # create/update skeletons, non-destructive
egon-data metadata export [--scope …] # one schema.table.json per table, offline
No Docker/airflow bootstrap; works without a database (with one,
initfills column lists from live inspection). Provenance stubs are derived
from
DatasetSources(internal cross-references + external sourcestubs). Current store status: complete=20, missing=126, orphans=0 —
the 126 are the migration backlog (52 with recoverable old metadata).
Run-time task
Json_Metadataruns last: per dataset, assemble → injectpublicationDate + automated-run contributor → merge live DB schemas
(tables absent from this DB are skipped, so partial dev runs stay
valid; undocumented columns are merged with a
TODOdescription and awarning — drift never fails the run) → validate (hard) → write one
JSON table comment and one
workdir/oemetadata/<schema>.<table>.jsonper table. The one-JSON-per-table requirement from the issue thread is
a by-product of every run.
Docs
docs/metadata.rst(architecture/reference) anddocs/metadata_howto.rst(hands-on guide: add a dataset / editmetadata / full run + release), wired into the toctree.
Data repair
All 20 existing resource YAMLs carried a
primaryKeycorrupted intosingle characters plus an empty
foreignKeysstub — an OMI v1.5→v2conversion bug (fixed in the OMI PR). Repaired from the authoritative
results/JSONs; all 4 datasets assemble and validate againstOEMetadata 2.0.
Not in this PR (deliberately deferred)
metadata.oemetadata) + pointer comments —full-JSON comments are kept until then (no behavior regression).
@idminting at publication + push interface beyond OEP.metadata status --strict.How to review / try
egon-data metadata status --scope heat_demand
egon-data metadata init --scope
egon-data metadata export
Before merging into
dev-branch, please make sure thatCHANGELOG.rstwas updated.blackandisort.Dataset-version is updated when existing datasets are adjusted.test mode.Everythingmode.Closes #1177
Closes #1305