Sweep across multiple weather years

[fneum]

Closes #200 .

Changes proposed in this Pull Request

• Energy statistics from eurostat and IDEES now stored initially for full number of years.
• Synthetic electricity load time series for years not in OPSD (from https://zenodo.org/record/7070438/files/demand_hourly.csv).
• Heat demand for years not in eurostat or IDEES scaled based on HDD regression.
• Hydro generation for years not in EIA (outside 1980-2021) scaled based on runoff regression.
• Hydro generation data scaled by installed capacity to eliminate changes in generation due to newly built capacity.
• Option to drop leap day.
• Option to make hydro data independent of weather year.
• Option to make electric load data independent of weather year.
• Include time series of Swiss number of passenger vehicles.

How To Use

Run create_scenarios.py to build a scenarios.yaml with the following snippet for the template:

template = """
weather_year_{year}:

  snapshots:
    start: "{year}-01-01 00:00"
    end: "{year}-12-31 23:00"
    inclusive: "right"
  
  atlite:
    default_cutout: europe-era5-{year}
    cutouts:
      europe-era5-{year}:
        module: era5
        x: [-12., 42.]
        y: [33., 72]
        dx: 0.3
        dy: 0.3
        time: ['{year}', '{year}']
    
  renewable:
    onwind:
      cutout: europe-era5-{year}
    offwind-ac:
      cutout: europe-era5-{year}
    offwind-dc:
      cutout: europe-era5-{year}
    solar:
      cutout: europe-era5-{year}
    hydro:
      cutout: europe-era5-{year}
  
  solar_thermal:
    cutout: europe-era5-{year}
  
  sector:
    heat_demand_cutout: europe-era5-{year}
"""

config_values = dict(year=range(2010, 2020))

Future PRs

• add feature to run years from July to June
• add feature to run multiple consecutive years in one optimization
• add feature to run multiple non-consecutive years in one optimization (requires rule to concatenate cutouts)
• update https://zenodo.org/records/7070438 to year range 1941-2023

[fneum]

Leap years! Set as early as base_networks?

Option A: Ignore/cut leap day.

def cut_leap_days(n):
    if not n.snapshots.is_leap_year.any(): return
    snapshots = pd.DatetimeIndex([sn for sn in n.snapshots if (sn.month != 2) | (sn.day != 29)])
    n.set_snapshots(snapshots)
    n.snapshot_weightings[:] = 8760/len(n.snapshots)

Option B: Accept lower snapshot weightings (e.g. 2.99 instead of 3.0).

[EDIT: but for option B need to consider that we fall back to non-leap years for load and hydro total annual generation, also in leap years data on Feb 29 might be missing (e.g. load)]

[euronion]

• Needs update to new atlite version, see Atlite availability #224 .
• Needs disentanglement of all the different features changed into separate PRs.

[fneum]

I am quite happy with the removal of the {weather_year} wildcard. The PR is now in a state that works for different single years and reproduces results for the default year 2013 locally. There are some additional features one could add, but I think it would be best to do that step by step in separate PRs (wishlist at the top).

Thanks, @koen-vg, for the transition to scenario management. It was very fast and easy to implement, and it saved a lot of time!

Just a consideration: I think it would be great if you could refer to a cutout name from the configuration, and that there's then some magic in an input function somewhere to "translate" this to a cutout for the right year.

The create_scenarios.py snippet mentioned at the top of the PR (updated now) somewhat covers this.

On other note, I would love to be to summary plots across weather years using the plot_summary rule, which, if I understand it correctly, doesn't currently collect results from across scenarios.

Yes, that's the plan, but for a separate PR.

Finally, I think it would be great to eventually support optimization over multiple weather years at once, but this might actually somehow be easier with a scenario-based approach.

Yes, that's a feature I would also want, but I would propose this for a separate PR after this one (just to keep the changes somewhat digestible). It requires a rule to merge cutouts, but I think you already implemented this in https://github.com/aleks-g/intersecting-near-opt-spaces/, right? In the scripts, there shouldn't be too many hurdles as far as I can see. I know of one I built on purpose for now in pop_weighted_energy_totals, which should be straightforward to resolve.

[koen-vg]

Thanks, @koen-vg, for the transition to scenario management. It was very fast and easy to implement, and it saved a lot of time!

Thanks, glad to know that I could help a little!

Just a consideration: I think it would be great if you could refer to a cutout name from the configuration, and that there's then some magic in an input function somewhere to "translate" this to a cutout for the right year.

The create_scenarios.py snippet mentioned at the top of the PR (updated now) somewhat covers this.

Yes, this looks nice to me; if the scenario system is used anyway to select the weather year, then it seems very sensible to use it also to select the correct cutout for the solar thermal etc. cutouts instead of adding extra complexity in input functions. One could always think about streamlining these things but I 100% agree that these "nice-to-have"s should be left for future PRs.

Finally, I think it would be great to eventually support optimization over multiple weather years at once, but this might actually somehow be easier with a scenario-based approach.

Yes, that's a feature I would also want, but I would propose this for a separate PR after this one (just to keep the changes somewhat digestible). It requires a rule to merge cutouts, but I think you already implemented this in https://github.com/aleks-g/intersecting-near-opt-spaces/, right? In the scripts, there shouldn't be too many hurdles as far as I can see. I know of one I built on purpose for now in pop_weighted_energy_totals, which should be straightforward to resolve.

Yes, the merging of cutouts was quite simple, see https://github.com/koen-vg/pypsa-eur/blob/7ea264f2744a8bd165f9301f23dfc0e2c4b86c6a/scripts/build_renewable_profiles.py#L220-L224 (from the pypsa-eur commit included in the repo you linked). It's a bit quirky that we found we had to supply a filename to the atlite.Cutout constructor and used a named temporary file for this, but it should never be written to. Maybe there is a better way, but it did work fine.

I can see if I have any time going forward to open a PR for this. The approach I've used before is to write an input function for the relevant rules (mainly build_renewable_profiles and siblings) which determines which cutouts are needed (based in this case on the snapshot config section), and then merge them inside the build_renewable_profiles rule. But that doesn't mesh very well with the current approach of using the config to manage which cutout is for which year. I can see that one could write a rule to produce cutouts of the form europe-era5-2010-2020 and then put that in the config, but I will say that this would take quite a lot of disk space. Not sure if there's a good way around that. Just some loose thoughts, should maybe be discussed elsewhere anyway.

I'm looking forward to seeing this merged; I definitely agree it makes sense to get the basic functionality merged at first and worry about additional features in later PRs. I'll be using the multiyear functionality quite actively in the next couple of months and I'll be sure to nail down and fix any potential issues I find on the way. Thanks so much for the amazing work!

[FabianHofmann]

Wow! Great job @fneum and @koen-vg !

[fneum]

only after looming around for 3.5 years (Oct 20, 2020) ;)