Grid Matcher

Grid Matcher is a specialized tool for matching transmission line data between JAO (Joint Allocation Office) and PyPSA (Python for Power System Analysis) formats. It provides automated spatial matching with manual override options, enabling accurate conversion between these formats, side-by-side parameter comparisons, and export/visualization utilities.

Features

• Automated Line Matching – Spatially match transmission lines between JAO and PyPSA datasets.
• Transformer Matching (MV/HV Substations – Match JAO and PyPSA transformers/substations within a configurable distance and propagate identifiers/parameters.
• Parameter Transfer – Transfer electrical parameters between matched lines.
• Manual Matching – Define and apply manual matches for complex cases.
• DC Link Support – Optionally include DC links in matching and outputs.
• 110 kV Line Support – Optionally include 110 kV voltage level lines.
• Visualization – Generate interactive maps and parameter comparison visuals.
• Comprehensive Reports – Produce detailed CSV exports and summary tables.

Requirements

• Python: 3.8+
• Dependencies:
  • pandas
  • geopandas
  • shapely
  • matplotlib
  • folium (for visualization)
  • numpy
  • requests (for data downloading)
• Optional (for troubleshooting / extra visuals):
  • plotly

Installation

Clone the repository and install dependencies:

git clone https://github.com/yourusername/grid_matcher.git
cd grid_matcher
pip install -r requirements.txt

Data Acquisition

Before running the matcher, you must download the required dataset files from Zenodo. The project includes helper scripts to automate this process.

Downloading JAO Data

Run the provided script to download JAO transmission line data:

python scripts/materialize_jao.py

This script will:

• Download the JAO dataset from Zenodo record ID 13367535.
• Process and standardize the data format.
• Save the result to grid_matcher/data/jao_lines.csv.

Note: The CORE-TSO data has been georeferenced using the methodology described in https://zenodo.org/records/13367535 To use the most up-to-date TSO lines with proper geometry, please follow that methodology; the dataset provided here reflects the state of September 2022.

Downloading PyPSA Data

To download the PyPSA line data, use the following Python code:

from pathlib import Path
from grid_matcher.io.zenodo_fetch import download_prepare_pypsa_lines_from_zenodo

download_prepare_pypsa_lines_from_zenodo(
    url="https://zenodo.org/records/14144752/files/lines.csv?download=1",
    output_csv=Path("grid_matcher/data/pypsa_lines.csv"),
    verbose=True
)

Note: Both download functions have parameters to control Germany-specific clipping, caching, and other options. See the docstrings for details.

Data Preparation

After downloading the core datasets, you can optionally prepare additional input files:

• pypsa_lines_110kv.csv – PyPSA 110 kV line data (optional)
• pypsa_dc_links.csv – PyPSA DC link data (optional)
• manual_matches.json – Manual matches file (created automatically if it doesn't exist)

Additional notes:

• The download scripts ensure both input formats contain geometry information in WKT (Well-Known Text) format and appropriate electrical parameters.
• Both datasets are standardized to use a consistent CRS (EPSG:4326).

Transformers

Place transformer CSVs in grid_matcher/data/:

• jao_transformers.csv

• pypsa_transformers.csv

Both files must include either:

• a geometry column containing WKT, or

• lon/lat columns (WGS84).

During the run, cleaned versions (with valid geometry and optional de-duplication) are produced automatically.

Basic Usage

After downloading the required data, run the matcher with default settings:

python run_matcher.py

This will:

1. Load JAO and PyPSA transmission line data.
2. Apply any predefined manual matches.
3. Run the automated matching algorithm.
4. Generate output files in the output/matcher directory.
5. Create visualizations for parameter comparison.
6. (If transformer inputs exist and the flag is enabled) run transformer matching and produce a combined map.

Command-Line Options

usage: run_matcher.py [-h] [--include-dc-matching] [--include-110kv-matching]
                      [--no-dc-output] [--no-110kv-output]
                      [--no-viz] [--no-length-comparison]
                      [--grid-comparison] [--no-grid-comparison]
                      [--quiet]
                      [--manual] [--no-manual]
                      [--add-predefined] [--no-predefined]
                      [--import-new-lines]
                      [--include-transformers] [--no-transformers]
                      [--transformers-distance TRANSFORMERS_DISTANCE]
                      [--output OUTPUT]

options:
  -h, --help                       Show help and exit.
  --include-dc-matching            Include DC links in matching.
  --include-110kv-matching         Include 110 kV lines in matching.
  --no-dc-output                   Exclude DC links from outputs.
  --no-110kv-output                Exclude 110 kV lines from outputs.
  --no-viz                         Skip parameter visualization.
  --no-length-comparison           Skip line length comparison.
  --grid-comparison                Generate grid comparison visuals.
  --no-grid-comparison             Skip grid comparison visuals.
  --quiet                          Less verbose logging.
  --manual / --no-manual           Enable/disable manual line matching.
  --add-predefined / --no-predefined
                                   Add/skip predefined manual matches.
  --import-new-lines               Import *-new-lines.csv before matching.
  --include-transformers           **Run transformer matching** (default in code: enabled).
  --no-transformers                Skip transformer matching.
  --transformers-distance FLOAT    **Max match distance (km) for transformers** (default: 5.0).
  --output OUTPUT, -o OUTPUT       Output directory (default: output/matcher).

Example Commands

Match only high-voltage AC lines (exclude DC and 110 kV from output):

python run_matcher.py --no-dc-output --no-110kv-output

Include DC links in both matching and output:

python run_matcher.py --include-dc-matching

Disable manual matches entirely:

python run_matcher.py --no-manual

Import new lines before matching:

python run_matcher.py --import-new-lines

Run transformer matching with a 1 km radius:

python run_matcher.py --include-transformers --transformers-distance 1

Output Files

Generated in the specified output directory (default: output/matcher):

Lines:

• jao_pypsa_matches.csv – All matched lines and their parameters.
• pypsa_with_eic.csv – PyPSA lines with added JAO identifiers.
• pypsa_with_eic_enhanced.csv – Enhanced PyPSA set including DC/110 kV lines.
• jao_with_pypsa.csv – JAO lines with PyPSA electrical parameters.
• jao_pypsa_matches.html – Interactive map visualization of matches.
• parameter_comparison.html – Visualization of electrical parameter comparison.
• parameter_summary.html – Summary table of parameter statistics.

Transformers:

• jao_transformers_clean.csv – Cleaned JAO transformer geometries.

• pypsa_transformers_clean.csv – Cleaned PyPSA transformer geometries.

• transformer_matches.csv (filename created by the pipeline; may vary) – Pairings with distances and flags.

• pypsa_transformers_updated.csv – PyPSA transformers updated with JAO attributes (and EIC identifiers, if available).

• lines_plus_transformers_matched.html – Integrated interactive map (voltage-filtered lines plus matched/unmatched transformers).

Manual Matching

For complex cases where automated matching doesn't provide satisfactory results, you can define manual matches in grid_matcher/manual/manual_matching.py. Manual matches override automated results for the specified JAO IDs.

Example structure:

# Dictionary of predefined matches (JAO ID -> PyPSA IDs)
predefined_matches = {
    "jao_id": ["pypsa_id1", "pypsa_id2", ...],
    # Example:
    "2611": ["merged_relation/3916226-380-c+3", "merged_way/240543053-1-380-a+5"]
}

You can also keep a manual_matches.json file (created on first run if missing) that persists manual mappings between runs.

Importing New Lines

If you have new transmission line data to add:

1. Place new JAO lines in grid_matcher/data/jao-new-lines.csv.
2. Place new PyPSA lines in grid_matcher/data/pypsa-new-lines.csv.
3. Run with the import flag:

python run_matcher.py --import-new-lines

Available Results

The main results of the matching and parameter comparison are available as an interactive HTML view here.

Troubleshooting

Missing Geometry Data

• Ensure input files contain valid geometry in WKT format.
• Verify CRS consistency between datasets.

No Matches Found

• Adjust matching parameters in grid_matcher/matcher/original_matcher.py.
• Add manual matches for problematic lines.

Visualization Errors

Ensure visualization dependencies are installed:

pip install folium matplotlib plotly

Memory Errors with Large Datasets

Exclude 110 kV lines and DC links to reduce memory usage:

python run_matcher.py --no-dc-output --no-110kv-output

Contributing

Contributions are welcome! Please open an issue or submit a pull request with improvements, bug fixes, or feature proposals.

License

This project is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0) — see the LICENSE file for details.