add total system cost to exporter #296

JulianGeis · 2024-12-04T10:40:23Z

Before asking for a review for this PR make sure to complete the following checklist:

Workflow with target rule ariadne_all completes without errors
The logic of export_ariadne_variables has been adapted to the changes
One or several figures that validate the changes in the PR have been posted as a comment
A brief description of the changes has been added to Changelog.md
The latest main has been merged into the PR
The config has a new prefix of the format YYYYMMDDdescriptive_title

for more information, see https://pre-commit.ci

github-actions · 2024-12-04T10:42:52Z

Validator Report

I am the Validator. Download all artifacts here.
I'll be back and edit this comment for each new commit.

⚠️ Config changes detected!

Results may differ due to these changes:

diff --git a/config/config.yaml b/config/config.yaml
index 6cc5bdd..713449c 100644
--- a/config/config.yaml
+++ b/config/config.yaml
@@ -4,7 +4,7 @@
 
 # docs in https://pypsa-eur.readthedocs.io/en/latest/configuration.html#run
 run:
-  prefix: 20241202-more-waste-CHP
+  prefix: 20241121-fix-missing-gas-capa
 
   name:
   # - CurrentPolicies
@@ -393,17 +393,14 @@ industry:
     2040: 0.12
     2045: 0.16
     2050: 0.20
-  # 15 Mt HVC production (from IDEES) -> 6 Mt Plastikabfälle,
-  # To substitute for other waste, assume all Plastikabfall is used energetically whereas in reality ~40% is recycled
-  # see https://github.com/PyPSA/pypsa-ariadne/pull/292
   HVC_environment_sequestration_fraction:
-    2020: 0.6
-    2025: 0.6
-    2030: 0.6
-    2035: 0.6
-    2040: 0.6
-    2045: 0.6
-    2050: 0.6
+    2020: 0.1
+    2025: 0.1
+    2030: 0.12
+    2035: 0.15
+    2040: 0.18
+    2045: 0.20
+    2050: 0.20
   waste_to_energy: true
   waste_to_energy_cc: true
 
@@ -483,14 +480,6 @@ solving:
             2035: 2.5
             2040: 2.5
             2045: 2.5
-        HVC to air:
-          DE:
-            2020: 0 # all HVC in Germany is either burned or recycled
-            2025: 0
-            2030: 0
-            2035: 0
-            2040: 0
-            2045: 0
     limits_capacity_min:
       Generator:
         onwind:
@@ -502,8 +491,8 @@ solving:
         offwind:
           DE:
             2030: 22.5   # 75% Wind-auf-See Law
-            2035: 35
-            2040: 42
+            2035: 30
+            2040: 40
             2045: 50
         solar:
           DE:
@@ -663,8 +652,8 @@ pypsa_eur:
 
 
 co2_price_add_on_fossils:
-  # 2020: 25
-  # 2025: 60
+  2020: 25
+  2025: 60
 
 must_run:
   2020:

Ariadne Variables

Comparison

	NRMSE	Pearson
Capacity\|Hydrogen\|Gas\|w/ CCS	0.816	-0.251
Capacity\|Hydrogen\|Gas	0.810	0.636
Capacity\|Electricity\|Gas\|OC	0.707	-0.316
Investment\|Energy Supply\|Electricity\|Biomass\|w/o CCS	0.693	0.510
Investment\|Energy Supply\|Electricity\|Biomass\|Solids	0.693	0.510
Investment\|Energy Supply\|Electricity\|Biomass	0.693	0.510
Emissions\|CO2\|Supply\|Non-Renewable Waste	0.686	0.000
Investment\|Energy Supply\|Heat\|Non-Renewable Waste	0.581	0.000
Capacity Additions\|Heat\|Non-Renewable Waste	0.581	-0.275
Capacity Additions\|Electricity\|Non-Renewable Waste	0.578	-0.276
Investment\|Energy Supply\|Heat\|Hydrogen	0.577	0.000
Investment\|Energy Supply\|Heat\|Biomass\|w/o CCS	0.577	0.000
Investment\|Energy Supply\|Heat\|Biomass	0.577	0.000
Investment\|Energy Supply\|Hydrogen\|Gas\|w/ CCS	0.576	0.000
Capacity\|Electricity\|Storage Converter	0.569	-0.155
Investment\|Energy Supply\|Heat\|Gas	0.554	-0.510
Investment\|Energy Supply\|Heat\|Processes	0.542	0.000
Final Energy\|Waste	0.527	0.000
Trade\|Secondary Energy\|Methanol\|Gross Import\|Volume	0.522	0.162
Price\|Carbon\|National Climate Target	0.510	0.604
Trade\|Secondary Energy\|Methanol\|Volume	0.484	0.487
Capacity Additions\|Electricity\|Biomass\|Solids	0.482	0.598
Capacity Additions\|Electricity\|Biomass	0.482	0.598
Capacity Additions\|Electricity\|Biomass\|w/o CCS	0.482	0.598
Capacity Additions\|Heat\|Biomass	0.480	0.591
Capacity Additions\|Heat\|Biomass\|w/o CCS	0.480	0.591
Capacity\|Methanol	0.476	0.893
Investment\|Energy Supply\|Hydrogen\|Transmission and Distribution\|Endogen\|New-build	0.471	0.560
Secondary Energy\|Electricity\|Hydro	0.464	-0.239
Investment\|Energy Supply\|Heat	0.445	0.170
Secondary Energy\|Heat\|Gas	0.443	-0.129
Investment\|Energy Supply\|Heat\|Heat pump	0.437	0.426
Investment\|Energy Supply\|Heat\|Heatpump	0.437	0.426
Capacity Additions\|Heat\|Heat pump	0.437	0.058
Primary Energy\|Waste\|Heat	0.433	0.064
Secondary Energy\|Heat\|Waste	0.432	0.063
Capacity\|Heat\|Non-Renewable Waste	0.432	-0.287
Primary Energy\|Waste	0.432	0.063
Secondary Energy Input\|Electricity\|Liquids	0.429	0.788
Secondary Energy\|Methanol	0.429	0.788
Price\|Secondary Energy\|Hydrogen\|Green	0.429	0.000
Investment\|Energy Supply\|Liquids	0.426	0.651
Primary Energy\|Waste\|Electricity	0.426	0.057
Secondary Energy\|Electricity\|Waste	0.425	0.055
Capacity\|Electricity\|Non-Renewable Waste	0.424	-0.291
Price\|Final Energy\|Residential and Commercial\|Liquids\|Oil	0.423	0.000
Price\|Final Energy\|Residential and Commercial\|Liquids	0.423	0.000
Secondary Energy Input\|Electricity\|Hydrogen	0.411	0.743
Capacity Additions\|Methanol	0.410	0.406
Capacity Additions\|Heat\|Gas	0.410	0.015
Secondary Energy\|Hydrogen\|Electricity	0.409	0.740
Final Energy\|Carbon Dioxide Removal	0.408	-0.581
Final Energy\|Carbon Dioxide Removal\|Electricity	0.408	-0.447
Carbon Sequestration\|DACCS	0.408	0.000
Capacity Additions\|Hydrogen\|Gas\|w/ CCS	0.408	0.632
Secondary Energy\|Heat\|Other	0.391	0.803
Price\|Final Energy\|Industry\|Liquids\|Efuel	0.391	0.731
Price\|Final Energy\|Industry\|Liquids\|Oil	0.391	0.731
Investment\|Energy Supply\|Liquids\|Biomass	0.387	1.000
Primary Energy\|Gas\|Heat	0.386	0.188
Price\|Final Energy\|Industry\|Liquids	0.385	0.735
Final Energy\|Industry excl Non-Energy Use\|Liquids\|Efuel	0.373	0.468
Final Energy\|Liquids\|Efuel	0.373	0.446
Final Energy\|Residential and Commercial\|Liquids\|Petroleum	0.372	0.662
Capacity Additions\|Hydrogen\|Gas	0.362	0.687
Capacity Additions\|Electricity\|Wind\|Onshore	0.361	0.392
Final Energy\|Transportation\|Liquids\|Efuel	0.360	0.407
Secondary Energy\|Hydrogen\|Gas	0.356	0.443
Investment\|Energy Supply\|Hydrogen\|Transmission and Distribution\|Endogen	0.352	0.750
Final Energy\|Residential and Commercial\|Liquids	0.352	0.690
Primary Energy\|Gas\|Hydrogen	0.349	0.468
Secondary Energy\|Liquids\|Hydrogen	0.348	0.849
Secondary Energy\|Electricity\|Gas	0.348	0.570
Investment\|Energy Supply\|Methanol	0.346	0.654
Capacity Additions\|Heat	0.344	0.350
Secondary Energy Input\|Hydrogen\|Liquids	0.342	0.852
Primary Energy\|Biomass\|Solids	0.341	-0.039
Secondary Energy\|Solids\|Biomass	0.341	-0.039
Capacity\|Liquids\|Hydrogen	0.338	0.949
Secondary Energy\|Heat\|Oil	0.332	0.667
Investment\|Energy Supply\|Hydrogen\|Fossil	0.326	0.753
Investment\|Energy Supply\|Hydrogen\|Gas	0.326	0.753
Primary Energy\|Gas\|Electricity	0.321	0.646
Capacity Additions\|Electricity\|Wind	0.320	0.487
Investment\|Energy Supply\|Electricity\|Non-Renewable Waste	0.316	0.506
Primary Energy\|Oil\|Heat	0.315	0.910
Emissions\|CO2\|Model\|CCU	0.315	0.829
Capacity\|Hydrogen\|Electricity	0.315	0.968
Capacity Additions\|Liquids	0.312	0.734
Emissions\|Gross Fossil CO2\|Energy\|Supply\|Heat	0.310	0.810
Emissions\|CO2\|Energy\|Supply\|Heat	0.310	0.810
Capacity\|Heat\|Processes	0.306	0.975
Final Energy\|Solids\|Biomass	0.304	0.243
Price\|Secondary Energy\|Gases\|Hydrogen	0.303	0.000
Emissions\|Gross Fossil CO2\|Energy\|Supply\|Hydrogen	0.302	0.785
Emissions\|CO2\|Energy\|Supply\|Hydrogen	0.302	0.785
Capacity\|Liquids	0.301	0.954
Secondary Energy\|Heat\|Electricity\|Heat Pumps	0.298	0.863
Capacity\|Hydrogen\|Gas\|w/o CCS	0.294	1.000
Secondary Energy\|Electricity\|Oil	0.293	0.662
Capacity Additions\|Liquids\|Hydrogen	0.290	0.755
Investment\|Energy Supply\|Hydrogen\|Electrolysis	0.284	0.961
Investment\|Energy Supply\|Hydrogen\|Electricity	0.284	0.961
Price\|Final Energy\|Industry\|Hydrogen	0.282	0.698
Final Energy\|Bunkers\|Navigation\|Liquids\|Biomass	0.280	0.787
Secondary Energy\|Electricity\|Curtailment	0.280	0.753
Price\|Primary Energy\|Coal	0.278	0.848
Price\|Carbon	0.276	0.866
Price\|Final Energy\|Industry\|Solids\|Coal	0.275	0.863
Capacity\|Liquids\|Biomass	0.274	1.000
Capacity Additions\|Liquids\|Biomass	0.274	1.000
Final Energy\|Industry excl Non-Energy Use\|Liquids\|Biomass	0.268	0.779
Investment\|Energy Supply\|Electricity\|Wind Onshore	0.268	0.760
Investment\|Energy Supply\|Electricity\|Wind\|Onshore	0.268	0.760
Capacity\|Heat\|Heat pump	0.265	0.849
Secondary Energy\|Heat\|Electricity	0.263	0.890
Price\|Final Energy\|Residential and Commercial\|Gases	0.261	0.851
Secondary Energy\|Electricity\|Coal\|Lignite	0.261	0.761
Emissions\|CO2\|Efuels\|Liquids	0.260	0.819
Primary Energy\|Coal\|Lignite	0.256	0.773
Investment\|Energy Supply\|Liquids\|Hydrogen	0.248	0.840
Investment\|Energy Supply\|Electricity\|Solar\|PV\|Open Field	0.247	0.890
Primary Energy\|Coal\|Electricity	0.247	0.797
Final Energy\|Residential and Commercial\|Solids	0.243	0.626
Final Energy\|Residential and Commercial\|Solids\|Biomass	0.243	0.626
Capacity Additions\|Hydrogen\|Transmission\|Endogenous	0.242	0.879
Price\|Secondary Energy\|Liquids	0.234	0.875
Price\|Primary Energy\|Gas	0.233	0.858
Capacity\|Hydrogen\|Transmission\|Endogenous	0.233	0.984
Final Energy\|Transportation\|Liquids\|Biomass	0.230	0.835
Primary Energy\|Biomass\|Liquids	0.230	0.846
Price\|Final Energy\|Transportation\|Electricity	0.229	0.873
Price\|Final Energy\|Transportation\|Passenger\|Liquids	0.229	0.873
Price\|Final Energy\|Transportation\|Freight\|Liquids	0.229	0.873
Investment\|Energy Supply\|Electricity\|Wind	0.229	0.819
Trade\|Secondary Energy\|Liquids\|Hydrogen\|Gross Import\|Volume	0.229	0.806
Capacity\|Heat\|Gas	0.226	0.909
Capacity Additions\|Heat\|Processes	0.225	0.927
Price\|Secondary Energy\|Gases	0.222	0.875
Secondary Energy Input\|Electricity\|Heat	0.215	0.912
Secondary Energy\|Hydrogen	0.210	0.973
Capacity\|Heat	0.208	0.949
Capacity Additions\|Electricity\|Solar\|PV\|Open Field	0.207	0.883
Capacity Additions\|Hydrogen\|Electricity	0.206	0.937
Trade\|Secondary Energy\|Liquids\|Biomass\|Gross Import\|Volume	0.197	0.896
Capacity\|Heat\|Resistive heater	0.196	0.970
Secondary Energy\|Electricity\|Coal	0.195	0.894
Trade\|Primary Energy\|Biomass\|Volume	0.194	0.988
Price\|Final Energy\|Industry\|Solids\|Biomass	0.191	0.849
Emissions\|Gross Fossil CO2\|Energy\|Supply\|Electricity	0.189	0.895
Emissions\|CO2\|Energy\|Supply\|Electricity	0.189	0.895
Price\|Primary Energy\|Oil	0.185	0.000
Price\|Final Energy\|Transportation\|Passenger\|Hydrogen	0.180	0.000
Price\|Final Energy\|Transportation\|Freight\|Hydrogen	0.180	0.000
Trade\|Secondary Energy\|Hydrogen\|Gross Import\|Volume	0.179	0.978
Investment\|Energy Supply\|Electricity\|Distribution	0.178	0.891
Final Energy\|Industry\|Liquids\|Efuel	0.178	0.897
Investment\|Energy Supply\|Heat\|Resistive heater	0.176	0.934
Secondary Energy\|Heat\|Electricity\|Resistive	0.176	0.947
Final Energy\|Liquids\|Biomass	0.175	0.891
Emissions\|CO2\|Efuels\|Gases	0.174	0.921
Capacity Additions\|Heat\|Resistive heater	0.170	0.931
Final Energy\|Non-Energy Use\|Liquids\|Efuel	0.164	0.912
Investment\|Energy Supply\|Hydrogen\|Transmission and Distribution\|Endogen\|Retrofitted	0.161	0.997
Price\|Final Energy\|Industry\|Solids	0.161	0.955
Investment\|Energy Supply\|Hydrogen	0.158	0.992
Primary Energy\|Coal	0.155	0.927
Final Energy\|Solids	0.153	0.922
Capacity\|Electricity\|Solar\|PV\|Open Field	0.153	0.912
Capacity\|Electricity\|Wind\|Onshore	0.153	0.941
Price\|Final Energy\|Residential and Commercial\|Heat	0.150	0.953
Secondary Energy\|Heat\|Coal	0.148	0.982
Capacity Additions\|Electricity	0.148	0.883
Secondary Energy\|Electricity\|Wind\|Onshore	0.147	0.948
Primary Energy\|Coal\|Heat	0.146	0.981
Investment\|Energy Supply\|Electricity	0.146	0.918
Price\|Primary Energy\|Biomass	0.145	0.958
Price\|Final Energy\|Residential and Commercial\|Electricity	0.145	0.939
Final Energy\|Residential and Commercial\|Electricity\|Heat Pumps	0.144	0.960
Secondary Energy\|Electricity\|Coal\|Hard Coal	0.143	0.948
Price\|Carbon\|EU-wide Regulation All Sectors	0.143	0.963
Primary Energy\|Coal\|Hard Coal	0.143	0.963
Final Energy\|Residential and Commercial\|Electricity	0.142	0.957
Trade\|Secondary Energy\|Liquids\|Hydrogen\|Volume	0.141	0.896
Final Energy\|Residential and Commercial\|Liquids\|Biomass	0.141	0.976
Secondary Energy\|Electricity\|Storage Losses	0.139	0.969
Price\|Final Energy\|Industry\|Gases	0.139	0.941
Final Energy\|Industry\|Liquids\|Biomass	0.138	0.938
Final Energy\|Bunkers\|Aviation\|Liquids\|Efuel	0.137	0.939
Price\|Final Energy\|Industry\|Heat	0.137	0.935
Trade\|Secondary Energy\|Liquids\|Biomass\|Volume	0.136	0.952
Primary Energy\|Hydro	0.136	0.971
Price\|Final Energy\|Residential and Commercial\|Solids\|Biomass	0.135	0.965
Price\|Final Energy\|Residential and Commercial\|Solids	0.135	0.965
Emissions\|CO2\|Energy\|Supply\|Electricity and Heat	0.134	0.956
Price\|Final Energy\|Transportation\|Passenger\|Electricity	0.134	0.970
Secondary Energy\|Gases\|Biomass	0.133	0.946
Primary Energy\|Biomass\|Gases	0.133	0.946
Price\|Secondary Energy\|Liquids\|Hydrogen	0.133	0.980
Price\|Final Energy\|Industry\|Electricity	0.133	0.974
Price\|Secondary Energy\|Electricity	0.131	0.967
Emissions\|Gross Fossil CO2\|Energy\|Supply	0.130	0.959
Emissions\|CO2\|Energy\|Demand\|Residential and Commercial	0.129	0.970
Investment\|Energy Supply\|Electricity\|Transmission and Distribution	0.127	0.940
Final Energy\|Non-Energy Use\|Liquids\|Biomass	0.124	0.950
Trade\|Secondary Energy\|Hydrogen\|Volume	0.123	0.973
Capacity\|Electricity\|Wind	0.123	0.961
Primary Energy\|Biomass	0.119	0.968
Trade\|Secondary Energy\|Electricity\|Volume	0.119	0.931
Price\|Secondary Energy\|Hydrogen	0.119	0.949
Trade\|Secondary Energy\|Electricity\|Gross Import\|Volume	0.116	0.981
Capacity Additions\|Electricity\|Wind\|Offshore	0.116	0.949
Capacity Additions\|Hydrogen	0.113	0.970
Primary Energy\|Oil\|Electricity	0.113	0.966
Price\|Secondary Energy\|Gases\|Biomass	0.113	0.000
Capacity\|Hydrogen	0.112	0.983
Emissions\|CO2\|Energy\|Supply\|Liquids	0.111	0.983
Primary Energy\|Biomass\|Heat	0.111	0.951
Secondary Energy\|Heat	0.110	0.983
Emissions\|CO2\|Energy\|Supply	0.109	0.969
Final Energy\|Bunkers\|Aviation\|Liquids\|Biomass	0.107	0.964
Secondary Energy\|Heat\|Biomass	0.104	0.956
Secondary Energy\|Electricity	0.104	0.981

NRMSE: Normalized (combined-min-max) Root Mean Square Error
Pearson: Pearson correlation coefficient
Threshold: NRMSE > 0.1
Only variables reaching the threshold are shown. Find the equivalent plot for all of them below.

Plots

Main branch	Feature branch

⚠️ Note: Only the first 20 variables are shown, but more are above the threshold. Find all of them in the artifacts.

General

Plots comparison

Main branch	Feature branch

Files comparison

	Status	NRMSE	MAE (norm)
csvs/prices.csv	⚠️Changed	0.041	0.08
csvs/market_values.csv	⚠️Changed	0.087	0.06
csvs/price_statistics.csv	⚠️Changed	0.091	0.22
csvs/cumulative_cost.csv	⚠️Changed	0.020	0.13
csvs/metrics.csv	⚠️Changed	0.031	0.06
csvs/nodal_cfs.csv	⚠️ NaN mismatch
csvs/nodal_supply_energy.csv	✅ Almost equal	0.001	0.00
csvs/supply.csv	✅ Almost equal	0.006	0.04
csvs/capacities.csv	✅ Almost equal	0.003	0.00
csvs/nodal_costs.csv	✅ Almost equal	0.003	0.00
csvs/nodal_capacities.csv	✅ Almost equal	0.001	0.00
csvs/curtailment.csv	✅ Almost equal	0.000	0.04
csvs/cfs.csv	✅ Almost equal	0.038	0.00
csvs/costs.csv	✅ Almost equal	0.011	0.01
csvs/supply_energy.csv	✅ Almost equal	0.005	0.03
csvs/energy.csv	✅ Almost equal	0.006	0.02
csvs/weighted_prices.csv	✅ Equal

NRMSE: Normalized (combined-min-max) Root Mean Square Error
MAE (norm): Mean Absolute Error on normalized data (min-max)
Status Threshold: MAE (norm) > 0.05 and NRMSE > 0.3

Model Metrics

Benchmarks

Comparing report-more-vars (2321470) with main (40ba80e).
Branch is 4 commits ahead and 8 commits behind.
Last updated on 2024-12-04 17:31:57 CET.

for more information, see https://pre-commit.ci

* add total system cost to exporter * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add capex and opex per sub sector * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

JulianGeis and others added 2 commits December 4, 2024 11:39

add total system cost to exporter

2a44a94

[pre-commit.ci] auto fixes from pre-commit.com hooks

6272658

for more information, see https://pre-commit.ci

JulianGeis and others added 2 commits December 4, 2024 14:59

add capex and opex per sub sector

7f4eca0

[pre-commit.ci] auto fixes from pre-commit.com hooks

2321470

for more information, see https://pre-commit.ci

JulianGeis marked this pull request as ready for review December 4, 2024 13:59

JulianGeis requested a review from lindnemi December 4, 2024 13:59

lindnemi merged commit 125a2c2 into main Dec 4, 2024
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add total system cost to exporter #296

add total system cost to exporter #296

JulianGeis commented Dec 4, 2024 •

edited

Loading

github-actions bot commented Dec 4, 2024 •

edited

Loading

add total system cost to exporter #296

add total system cost to exporter #296

Conversation

JulianGeis commented Dec 4, 2024 • edited Loading

github-actions bot commented Dec 4, 2024 • edited Loading

Validator Report

JulianGeis commented Dec 4, 2024 •

edited

Loading

github-actions bot commented Dec 4, 2024 •

edited

Loading