Atlite availability

[FabianHofmann]

closes #162
closes #156
closes #132
closes #169
closes #147
closes #230

Changes proposed in this Pull Request

Switch to new atlite version v0.2.1. The version upgrade comes along with significant speed up for the rule build_renewable_profiles.py (~factor 2). A lot of the code which calculated the landuse availability is now outsourced and does not rely on glaes, geokit anymore. This facilitates the environment building and version compatibility of gdal, libgdal with other packages.

It requires an update of the cutout files at zenodo.

The new implementation is not able to exactly reproduce the old profiles, but very close. Reasons:

1. The differences result from a slightly different approach of the distance calculations from raster points: atlite uses the dilation function of scipy.ndimage, glaes made a conversion of the raster to geometries -> calculating a buffer -> backconverting to a raster. Non of the approach is objectively better, their results diverge just a very little.
2. The gdal resampling method average, is not purely consistent as their might be a small subpixel shift (Pixels shift when resampling using "average" or "Lanczos" OSGeo/gdal#2235).

Open question

• Do we still need the piezker script (build_country_flh.py)? I did not update it. If wanted please shout out. Otherwise we could delete the rule.

Checklist

• I tested my contribution locally and it seems to work fine.
• Code and workflow changes are sufficiently documented.
• Newly introduced dependencies are added to envs/environment.yaml and envs/environment.docs.yaml.
• Changes in configuration options are added in all of config.default.yaml, config.tutorial.yaml, and test/config.test1.yaml.
• Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/configuration.rst and doc/tutorial.rst.
• A note for the release notes doc/release_notes.rst is amended in the format of previous release notes.

[fneum]

Anything left to do?

[FabianHofmann]

I will just double check the non-tutorial workflow and I think then we're fine :)

[euronion]

I'll briefly check the tutorial workflow.

[euronion]

Tutorial:

• Atlite part runs without problems direct problems.
• The progressbar is glitchy, presumably due to multithreading (the % numbers appear and disappear in unordered manner). Not an issue for this PR, wort a issue for atlite @FabianHofmann ?
• Using snakemake --configfile config.tutorial.yaml -jall without having a config.yaml in the folder fails as it tries to create the electricity network from config.defaul.yaml and downloads the full europe-2013-era5. Cause is this convinience code line in the Snakefile which in turn triggers the build_hydro rule:

if not exists("config.yaml"):
    copyfile("config.default.yaml", "config.yaml")

But if one follows the documentation and does mv config.tutorial.yaml config.yaml everything works out.

[FabianHofmann]

yep, the jittering progressbar appears when having parallel processes that result in parallel progressbars which all want to be displayed at the same spot.
What we could actually do is to set atlite.show_progress in the config to false. That leaves out the progressbars of the compute_availability process and is actually almost twice as fast (this has to do with the synchronization of callbacks of parallel processes). Downside you don't have a direct feedback. Is that an option?

[euronion]

I prefer jiggly feedback on whether the process is alive or not over no feedback. (In favour of the current situation)

[fneum]

I see the positive aspects of both :) no preference.

Re config: I see, so the option --configfile only overwrites fields of the main config but does not substitute it fully? I think it's fine to leave as is for now.

[euronion]

Re config: I see, so the option --configfile only overwrites fields of the main config but does not substitute it fully? I think it's fine to leave as is for now.

Yes. That's intentional behaviour: Use configfile: <...> from Snakefile as a basis and overwrite (update) with --configfile <..>. We could think about removing the lazy copy-mechanism from the Snakefile and prefer to have the workflow fail (if no explicit creation of config.yaml happens) or encourage the use of --configfile <...> . But that's a separate issue. Worth raising?

[FabianHofmann]

Short update. Everything works fine, but (!) the memory consumption is not stable for solar profiles when executing with the latest dask version (2021.4.1). I try to find the latest version that works smoothly and then add a requirement to the envs.

Unfortunately, I have no clue where the memory issue comes from (my guess it has to do with xarray clip functions). But i'm pretty sure this will be solved in the future.

[fneum]

dask/dask#7583?

[FabianHofmann]

Yes, exactly! That costed me some hours...

[fneum]

restarting tests, got stuck on build_cutout csdapi request.

[FabianHofmann]

right, which reminds me that we can/should set the test config to retrieve_cutout: true

[FabianHofmann]

Alright, I think we are there. Thumbs up if I should merge.

[fneum]

🥳 🎈 🎉

[FabianHofmann]

Yes, finally :)

[euronion]

Indeed. Finally! (= 😄

[MichelJD]

Hi together,
I've just encountered an error, while running pypsa-eur-sec with the new atlite-version. In the build_population_layouts.py script, there is still the old style of loading an atlite cutout. I think this needs to be changed.
Best regards
Michél

[fneum]

Hi, yes, PyPSA-Eur-Sec is not yet updated for the new atlite version. We are working on this. Please use PyPSA-Eur release v0.3.0 or a commit pre #224 in combination with PyPSA-Eur-Sec for now.