Major improvement to OSM-based electricity grid (e.g. using relations, preserving substation locations) #1384

bobbyxng · 2024-10-22T15:34:03Z

Changes proposed in this Pull Request

Major improvement to the OSM-based electricity grid
build_osm_network completely rewritten:
- Use of OSM relations, where available and where a single linestring could be extracted to replace member ways
- substantial speed improvements
- Exact locations of original substations are preserved
- Lines with identical geometries and voltages are aggregated (num_parallel is increased)
- Lines overpassing "virtual nodes" (not actual substations as per OSM raw data) are merged, if their electric properties allow it (same voltage levels, same number of circuits)
- Geometries are preserved. Method to extend line/link to the substation has been improved significantly. No "mess" inside of a substation cluster anymore.
- Method to determine substation point improved, based on "Point of Inaccessibility" (PoI) method developed by Garcia-Castellanos & Lombardo 2007, doi.org/10.1080/14702540801897809 which ensures that an interior point in non-convex shapes is always found.
- Clustering resolution has been sharpened from 5000 meter radius to only 500 meters, while still ensuring topological connectivity.
- Single bus for each voltage level in each substation. Buses of different voltages are arranged in geometric shape (three buses form a triangle, see screenshot)
- Transformers now have a capacity s_nom based on the maximum of connected capacity connected to each bus
- Resolves many issues related around cross-border capacity, especially if the substation was located close to a border. @lindnemi @toniseibold @JulianGeis @nworbmot

Todo:

Validation runs still open

Checklist

I tested my contribution locally and it works as intended.
Code and workflow changes are sufficiently documented.
Changed dependencies are added to envs/environment.yaml.
Changes in configuration options are added in config/config.default.yaml.
Changes in configuration options are documented in doc/configtables/*.csv.
Sources of newly added data are documented in doc/data_sources.rst.
A release note doc/release_notes.rst is added.

…eating voltage-specific bus endings in stations.

for more information, see https://pre-commit.ci

… osm-relations

github-actions · 2024-10-22T15:59:25Z

Validator Report

I am the Validator. Download all artifacts here.
I'll be back and edit this comment for each new commit.

General

Plots comparison

Main branch	Feature branch

Files comparison

	Status	NRMSE	MAE (norm)
prices.csv	⚠️Changed	0.000	0.08
weighted_prices.csv	⚠️Changed	0.449	0.42
market_values.csv	⚠️Changed	0.000	0.10
price_statistics.csv	⚠️Changed	0.000	0.17
curtailment.csv	⚠️Changed	0.000	0.14
metrics.csv	⚠️Changed	0.000	0.06
nodal_supply_energy.csv	✅ Almost equal	0.000	0.00
nodal_cfs.csv	✅ Almost equal	0.065	0.02
nodal_costs.csv	✅ Almost equal	0.003	0.00
nodal_capacities.csv	✅ Almost equal	0.000	0.00
supply.csv	✅ Almost equal	0.000	0.01
capacities.csv	✅ Almost equal	0.000	0.02
costs.csv	✅ Almost equal	0.001	0.00
cfs.csv	✅ Almost equal	0.030	0.02
supply_energy.csv	✅ Almost equal	0.000	0.01
energy.csv	✅ Almost equal	0.000	0.01

NRMSE: Normalized (combined-min-max) Root Mean Square Error
MAE (norm): Mean Absolute Error on normalized data (min-max)
Status Threshold: MAE (norm) > 0.05 and NRMSE > 0.3

Model Metrics

Benchmarks

Comparing osm-relations (c7c453d) with master (ef20ee8).
Branch is 25 commits ahead and 0 commits behind.
Last updated on 2024-10-25 10:00:59 CEST.

for more information, see https://pre-commit.ci

…to osm-relations

for more information, see https://pre-commit.ci

fneum

I didn't check every line in detail, but the structure looks good and comparable to the previous version. I have a series of small comments that are easy to address:

fneum · 2024-11-04T10:21:03Z

scripts/base_network.py

+    if "s_nom" not in transformers:
+        transformers["s_nom"] = config.get("s_nom", 2000)


What happens if some transformers have s_nom, but not all of them. Will missing values be filled?

fneum · 2024-11-04T10:21:20Z

scripts/build_gdp_pop_non_nuts3.py

+        snakemake = mock_snakemake(
+            "build_gdp_pop_non_nuts3", configfiles=["config/config.osm-raw.yaml"]
+        )


fneum · 2024-11-04T10:21:53Z

config/config.default.yaml

@@ -84,7 +84,7 @@ co2_budget:

 # docs in https://pypsa-eur.readthedocs.io/en/latest/configuration.html#electricity
 electricity:
-  voltages: [200., 220., 300., 380., 500., 750.]
+  voltages: [220., 300., 330., 380., 400., 500., 750.]


Just a short note why 200 kV disappears for clarification would be good.

fneum · 2024-11-04T10:24:56Z

scripts/build_osm_network.py

+        # # list of all countries
+        # country = ';'.join([G.nodes[node].get('country', '') for node in subgraph.nodes()])
+        # country = ';'.join(sorted(set(country.split(';'))))


could remove unused code?

fneum · 2024-11-04T10:27:18Z

scripts/build_osm_network.py

-        temp_list = [split(l, p) for l in list_linestrings]
-        # nest all geometries
-        list_linestrings = [lstring for tval in temp_list for lstring in tval.geoms]
+    for g_name, g_value in buses_all.groupby("station_id"):


If this is slow, could be converted to an apply function applied to each group.

bobbyxng and others added 13 commits October 14, 2024 19:56

Implemented line merge over virtual bus functionality.

a1d076c

Implemented: Aggregating identical lines, bus merging to stations, cr…

0287872

…eating voltage-specific bus endings in stations.

Implemented: Mapping of lines to buses and extending lines to buses.

ce15373

Finished implementing converters and links.

77f77c8

Finished implementation of entirely new build_osm_network.py script.

24145f7

Merge branch 'master' into osm-interconnectors

9aca61a

Updated configtables.

fc5b36c

[pre-commit.ci] auto fixes from pre-commit.com hooks

43cf668

for more information, see https://pre-commit.ci

Temporarily disabled tqdm in retrieve. Line splitting disabled as well.

caf8eb9

[pre-commit.ci] auto fixes from pre-commit.com hooks

6b00178

for more information, see https://pre-commit.ci

Implemented relations. Running, converging workflow.

fb653fb

Added updated simplify_network.py

b2c21f7

Cleaned up code.

2aefa9b

bobbyxng requested review from fneum and p-glaum October 22, 2024 15:34

[pre-commit.ci] auto fixes from pre-commit.com hooks

5420fec

for more information, see https://pre-commit.ci

bobbyxng mentioned this pull request Oct 22, 2024

Major improvement to OSM-based electricity grid (e.g. preserves exact locations of substations and converters) #1375

Closed

7 tasks

bobbyxng added 2 commits October 22, 2024 17:35

Updated doc

9b24312

Merge remote-tracking branch 'refs/remotes/origin/osm-relations' into…

4ef062f

… osm-relations

bobbyxng and others added 9 commits October 22, 2024 19:13

Reactivated line splitting.

8aec857

Updated build_osm_network.

13391d6

Cleaned up code.

639eef7

[pre-commit.ci] auto fixes from pre-commit.com hooks

aca66fb

for more information, see https://pre-commit.ci

Added 400 kV voltage level to config.default

0cca562

Merge branch 'osm-relations' of https://github.com/pypsa/pypsa-eur in…

b3fec47

…to osm-relations

Merge branch 'master' into osm-relations

db2f593

Updated Zenodo link

e9f7798

[pre-commit.ci] auto fixes from pre-commit.com hooks

c7c453d

for more information, see https://pre-commit.ci

fneum reviewed Nov 4, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Major improvement to OSM-based electricity grid (e.g. using relations, preserving substation locations) #1384

Major improvement to OSM-based electricity grid (e.g. using relations, preserving substation locations) #1384

bobbyxng commented Oct 22, 2024

github-actions bot commented Oct 22, 2024 •

edited

Loading

fneum left a comment

fneum Nov 4, 2024

fneum Nov 4, 2024

fneum Nov 4, 2024

fneum Nov 4, 2024

fneum Nov 4, 2024

		if "s_nom" not in transformers:
		transformers["s_nom"] = config.get("s_nom", 2000)

Major improvement to OSM-based electricity grid (e.g. using relations, preserving substation locations) #1384

Are you sure you want to change the base?

Major improvement to OSM-based electricity grid (e.g. using relations, preserving substation locations) #1384

Conversation

bobbyxng commented Oct 22, 2024

Changes proposed in this Pull Request

Todo:

Checklist

github-actions bot commented Oct 22, 2024 • edited Loading

Validator Report

fneum left a comment

Choose a reason for hiding this comment

fneum Nov 4, 2024

Choose a reason for hiding this comment

fneum Nov 4, 2024

Choose a reason for hiding this comment

fneum Nov 4, 2024

Choose a reason for hiding this comment

fneum Nov 4, 2024

Choose a reason for hiding this comment

fneum Nov 4, 2024

Choose a reason for hiding this comment

github-actions bot commented Oct 22, 2024 •

edited

Loading