Atlite availability (PyPSA#224)

* adjust buil_cutout.py and Snakefile * try adjusting build_renewable_profiles, currently crashing due to weird pyproj error * build_renewable_profiles: -remove printing gid * build_renewable_profiles: use dask for paralellization, use dense functions * build_renewable_profiles: - revise imports - add logging for long calculation - revise explaining comment - revise distance calculation * build profiles: adjust to cutout.grid * * fix area to square km * rename potmatrix -> capacity_potential * rename available to availibility * config.default update cutout params build_renewable_potentials: major refactoring and simplification hydro_profiles: update code * build profiles: fix weight output dimensions * build profiles: fix typo, fix selection of buses * build profiles: reinsert paths variable * follow up * build profiles: move to dask calculation only * CI: set build cutout to true (add CDSAPI) * build profiles: use pyproj, test with gleas and geokit upstream * environment.yaml fix atlite version * build profiles: use dask 'processes' for more than 25 regions * build profiles: specify dask scheduler according to number of regions * backpedal a bit, only allow scheduler='processes' * follow up, code style and fixup * build profiles: add logger info for underwater fraction calc * config adjust cutout parameters Snakefile fixup * config.default.yaml: adjust resolution * config: use one cutout in total build_cutout: automatic detetection of geographical boundaries * env: add python>=3.8 requirement build_cutout: fixup for region bound * config: allow base cutout * folllow up, fix up * follow up II * clean up * clean up II * build profiles: move back to multiprocessing due to performance issues * small code style corrections * move in pool context * swqitch to ratsterio * switch to rasterio for availibility calculation * tiny fixup * * build continental raster for offshore distance calculation * adjust Snakefile to new script build_raster * rename continental raster to onshore raster add projected_mask function (not yet tested) add docstrings, modularize * Snakefile: remove build_onhore_raster rule, build mask directly from geometry instead build_natura_raster: adjust code, add function for exporting build_profiles: * add buffer to shore distance to init_globals function * update docstrings * improve handling of nodata grid codes * add geometry mask if natura raster not activated (the 255 value is an 'eligible' value for the corine data base, do this for excluding data outside the shape) * build_profiles: adjust docstrings * update environment * build profiles: fixup reproject woth padding * follow up, small fixups * fix resampling method checkpoint: reproduces solar profile in tut data * reintegrate plot map code style * config: rename cutout into "base" * build profiles: adjust to new atlite code * natura raster: small fixup * build natura raster: compress tiff file * config: adjust cutout names * build profiles: cover case if no or partial overlap between natura raster and cutout * config-tutorial: adjust cutout params * buid-profifiles: fixup in gebco filter * follow up * update config files * build profiles: select layoutmatrix != 0 * build profiles: speed up average_distance and underwaterfraction * build profiles: fix typo * update release notes build_cutout: only build needed features * update envs * config: add temperature to sarah features * temporary fix for atlite v0.2.1 and new xarray version release * env: remove xarray specification * * remove rule build_country_flh * build profiles: remove sneaked in line * doc: update configuration.rst (section atlite) and corresponding csv table * release notes: fix quotes * build profiles: use 3035 for area calculation * Update envs/environment.docs.yaml * Update scripts/build_cutout.py * Update doc/release_notes.rst Co-authored-by: euronion <42553970+euronion@users.noreply.github.com> * Update doc/configuration.rst Co-authored-by: euronion <42553970+euronion@users.noreply.github.com> * Update scripts/build_cutout.py Co-authored-by: euronion <42553970+euronion@users.noreply.github.com> * update release notes * release notes: add deprecation of 'keep_all_available_areas' build profiles: remove warning for 'keep_all_available_areas' * build cutout: rearrage code, set buffer correctly * Rename tutorial cutout to remove name clash with real cutout. * Update release_notes.rst: Rename tutorial cutout. * retrieve: update cutouts and downloads (alternative) (#237) * retrieve: update cutouts and downloads * retrieve: remove unnecessary import * use snakemake remote file functionality * Snakefile: update zenodo link * update natura remote link (closes PyPSA#234) * env: update atlite version to 0.2.2 * env: fix dask version due to memory issues * test: retrieve cutout instead of build * test: use tutorial cutout for CI Co-authored-by: euronion <42553970+euronion@users.noreply.github.com> Co-authored-by: Fabian Neumann <fabian.neumann@outlook.de>
LukasFrankenQ · Apr 27, 2021 · b8a7685 · b8a7685
1 parent c29d59b
commit b8a7685
Show file tree

Hide file tree

Showing 20 changed files with 298 additions and 734 deletions.
diff --git a/.travis.yml b/.travis.yml
@@ -29,6 +29,9 @@ before_install:
   # list packages for easier debugging
   - conda list
 
+before_script:
+  - 'echo -ne "url: ${CDSAPI_URL}\nkey: ${CDSAPI_TOKEN}\n" > ~/.cdsapirc'
+
 script:
   - cp ./test/config.test1.yaml ./config.yaml
   - snakemake -j all solve_all_networks

diff --git a/Snakefile b/Snakefile
@@ -5,6 +5,9 @@
 from os.path import normpath, exists
 from shutil import copyfile
 
+from snakemake.remote.HTTP import RemoteProvider as HTTPRemoteProvider
+HTTP = HTTPRemoteProvider()
+
 if not exists("config.yaml"):
     copyfile("config.default.yaml", "config.yaml")
 
@@ -135,10 +138,12 @@ rule build_bus_regions:
     resources: mem=1000
     script: "scripts/build_bus_regions.py"
 
-
 if config['enable'].get('build_cutout', False):
     rule build_cutout:
-        output: directory("cutouts/{cutout}")
+        input: 
+            regions_onshore="resources/regions_onshore.geojson",
+            regions_offshore="resources/regions_offshore.geojson"
+        output: "cutouts/{cutout}.nc"
         log: "logs/build_cutout/{cutout}.log"
         benchmark: "benchmarks/build_cutout_{cutout}"
         threads: ATLITE_NPROCESSES
@@ -148,26 +153,26 @@ if config['enable'].get('build_cutout', False):
 
 if config['enable'].get('retrieve_cutout', True):
     rule retrieve_cutout:
-        output: directory(expand("cutouts/{cutouts}", **config['atlite'])),
-        log: "logs/retrieve_cutout.log"
-        script: 'scripts/retrieve_cutout.py'
+        input: HTTP.remote("zenodo.org/record/4709858/files/{cutout}.nc", keep_local=True)
+        output: "cutouts/{cutout}.nc"
+        shell: "mv {input} {output}"
 
 
 if config['enable'].get('build_natura_raster', False):
     rule build_natura_raster:
         input:
             natura="data/bundle/natura/Natura2000_end2015.shp",
-            cutouts=expand("cutouts/{cutouts}", **config['atlite'])
+            cutouts=expand("cutouts/{cutouts}.nc", **config['atlite'])
         output: "resources/natura.tiff"
         log: "logs/build_natura_raster.log"
         script: "scripts/build_natura_raster.py"
 
 
 if config['enable'].get('retrieve_natura_raster', True):
     rule retrieve_natura_raster:
+        input: HTTP.remote("zenodo.org/record/4706686/files/natura.tiff", keep_local=True)
         output: "resources/natura.tiff"
-        log: "logs/retrieve_natura_raster.log"
-        script: 'scripts/retrieve_natura_raster.py'
+        shell: "mv {input} {output}"
 
 
 rule build_renewable_profiles:
@@ -181,11 +186,10 @@ rule build_renewable_profiles:
         country_shapes='resources/country_shapes.geojson',
         offshore_shapes='resources/offshore_shapes.geojson',
         regions=lambda w: ("resources/regions_onshore.geojson"
-                           if w.technology in ('onwind', 'solar')
-                           else "resources/regions_offshore.geojson"),
-        cutout=lambda w: "cutouts/" + config["renewable"][w.technology]['cutout']
-    output:
-        profile="resources/profile_{technology}.nc",
+                                   if w.technology in ('onwind', 'solar')
+                                   else "resources/regions_offshore.geojson"),
+        cutout=lambda w: "cutouts/" + config["renewable"][w.technology]['cutout'] + ".nc"
+    output: profile="resources/profile_{technology}.nc",
     log: "logs/build_renewable_profile_{technology}.log"
     benchmark: "benchmarks/build_renewable_profiles_{technology}"
     threads: ATLITE_NPROCESSES
@@ -198,7 +202,7 @@ if 'hydro' in config['renewable'].keys():
         input:
             country_shapes='resources/country_shapes.geojson',
             eia_hydro_generation='data/bundle/EIA_hydro_generation_2000_2014.csv',
-            cutout="cutouts/" + config["renewable"]['hydro']['cutout']
+            cutout="cutouts/" + config["renewable"]['hydro']['cutout'] + ".nc"
         output: 'resources/profile_hydro.nc'
         log: "logs/build_hydro_profile.log"
         resources: mem=5000
@@ -388,29 +392,3 @@ rule plot_p_nom_max:
     log: "logs/plot_p_nom_max/elec_s{simpl}_{clusts}_{techs}_{country}_{ext}.log"
     script: "scripts/plot_p_nom_max.py"
 
-
-rule build_country_flh:
-    input:
-        base_network="networks/base.nc",
-        corine="data/bundle/corine/g250_clc06_V18_5.tif",
-        natura="resources/natura.tiff",
-        gebco=lambda w: ("data/bundle/GEBCO_2014_2D.nc"
-                         if "max_depth" in config["renewable"][w.technology].keys()
-                         else []),
-        country_shapes='resources/country_shapes.geojson',
-        offshore_shapes='resources/offshore_shapes.geojson',
-        pietzker="data/pietzker2014.xlsx",
-        regions=lambda w: ("resources/country_shapes.geojson"
-                           if w.technology in ('onwind', 'solar')
-                           else "resources/offshore_shapes.geojson"),
-        cutout=lambda w: "cutouts/" + config["renewable"][w.technology]['cutout']
-    output:
-        area="resources/country_flh_area_{technology}.csv",
-        aggregated="resources/country_flh_aggregated_{technology}.csv",
-        uncorrected="resources/country_flh_uncorrected_{technology}.csv",
-        plot="resources/country_flh_{technology}.pdf",
-        exclusion=directory("resources/country_exclusion_{technology}")
-    log: "logs/build_country_flh_{technology}.log"
-    resources: mem=10000
-    benchmark: "benchmarks/build_country_flh_{technology}"
-    script: "scripts/build_country_flh.py"
diff --git a/config.default.yaml b/config.default.yaml
@@ -62,18 +62,28 @@ electricity:
 atlite:
   nprocesses: 4
   cutouts:
+    # use 'base' to determine geographical bounds and time span from config
+    # base: 
+      # module: era5  
     europe-2013-era5:
-      module: era5
-      xs: [-12., 35.]
-      ys: [72., 33.]
-      years: [2013, 2013]
+      module: era5 # in priority order 
+      x: [-12., 35.]
+      y: [33., 72]
+      dx: 0.3
+      dy: 0.3
+      time: ['2013', '2013']
     europe-2013-sarah:
-      module: sarah
-      resolution: 0.2
-      xs: [-12., 42.]
-      ys: [65., 33.]
-      years: [2013, 2013]
+      module: [sarah, era5] # in priority order 
+      x: [-12., 45.]
+      y: [33., 65]
+      dx: 0.2
+      dy: 0.2
+      time: ['2013', '2013']
+      sarah_interpolate: false
+      sarah_dir: 
+      features: [influx, temperature]
 
+
 renewable:
   onwind:
     cutout: europe-2013-era5

diff --git a/config.tutorial.yaml b/config.tutorial.yaml
@@ -54,16 +54,15 @@ electricity:
 atlite:
   nprocesses: 4
   cutouts:
-    europe-2013-era5:
+    europe-2013-era5-tutorial:
       module: era5
-      xs: [4., 15.]
-      ys: [56., 46.]
-      months: [3, 3]
-      years: [2013, 2013]
+      x: [4., 15.]
+      y: [46., 56.]
+      time: ["2013-03", "2013-03"]
 
 renewable:
   onwind:
-    cutout: europe-2013-era5
+    cutout: europe-2013-era5-tutorial
     resource:
       method: wind
       turbine: Vestas_V112_3MW
@@ -80,7 +79,7 @@ renewable:
     potential: simple # or conservative
     clip_p_max_pu: 1.e-2
   offwind-ac:
-    cutout: europe-2013-era5
+    cutout: europe-2013-era5-tutorial
     resource:
       method: wind
       turbine: NREL_ReferenceTurbine_5MW_offshore
@@ -92,7 +91,7 @@ renewable:
     potential: simple # or conservative
     clip_p_max_pu: 1.e-2
   offwind-dc:
-    cutout: europe-2013-era5
+    cutout: europe-2013-era5-tutorial
     resource:
       method: wind
       turbine: NREL_ReferenceTurbine_5MW_offshore
@@ -105,7 +104,7 @@ renewable:
     potential: simple # or conservative
     clip_p_max_pu: 1.e-2
   solar:
-    cutout: europe-2013-era5
+    cutout: europe-2013-era5-tutorial
     resource:
       method: pv
       panel: CSi

diff --git a/doc/configtables/atlite.csv b/doc/configtables/atlite.csv
@@ -1,8 +1,9 @@
 ,Unit,Values,Description
 nprocesses,--,int,"Number of parallel processes in cutout preparation"
 cutouts,,,
--- {name},--,"Convention is to name cutouts like ``<region>-<year>-<source>`` (e.g. ``europe-2013-era5``).","Directory to write cutout data to. The user may specify multiple cutouts under configuration ``atlite: cutouts:``. Reference is used in configuration ``renewable: {technology}: cutout:``"
--- -- module,--,"One of {'era5','sarah'}","Source of the reanalysis weather dataset (e.g. `ERA5 <https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5>`_ or `SARAH-2 <https://wui.cmsaf.eu/safira/action/viewDoiDetails?acronym=SARAH_V002>`_)"
--- -- xs,°,"Float interval within [-180, 180]","Range of longitudes to download weather data for."
--- -- ys,°,"Float interval within [-90, 90]","Range of latitudes to download weather data for."
--- -- years,--,"Integer interval within [1979,2018]","Range of years to download weather data for."
+-- {name},--,"Convention is to name cutouts like ``<region>-<year>-<source>`` (e.g. ``europe-2013-era5``).","Name of the cutout netcdf file. The user may specify multiple cutouts under configuration ``atlite: cutouts:``. Reference is used in configuration ``renewable: {technology}: cutout:``. The cutout ``base`` may be used to automatically calculate temporal and spatial bounds of the network."
+-- -- module,--,"Subset of {'era5','sarah'}","Source of the reanalysis weather dataset (e.g. `ERA5 <https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5>`_ or `SARAH-2 <https://wui.cmsaf.eu/safira/action/viewDoiDetails?acronym=SARAH_V002>`_)"
+-- -- x,°,"Float interval within [-180, 180]","Range of longitudes to download weather data for. If not defined, it defaults to the spatial bounds of all bus shapes."
+-- -- y,°,"Float interval within [-90, 90]","Range of latitudes to download weather data for. If not defined, it defaults to the spatial bounds of all bus shapes."
+-- -- time,,"Time interval within ['1979', '2018'] (with valid pandas date time strings)","Time span to download weather data for. If not defined, it defaults to the time interval spanned by the snapshots."
+-- -- features,,"String or list of strings with valid cutout features ('inlfux', 'wind').","When freshly building a cutout, retrieve data only for those features. If not defined, it defaults to all available features."
diff --git a/doc/configuration.rst b/doc/configuration.rst
@@ -95,9 +95,12 @@ Specifies the temporal range to build an energy system model for as arguments to
 ``atlite``
 =============
 
+Define and specify the ``atlite.Cutout`` used for calculating renewable potentials and time-series. All options except for ``features`` are directly used as `cutout parameters <https://atlite.readthedocs.io/en/latest/ref_api.html#cutout>`_.
+
 .. literalinclude:: ../config.default.yaml
    :language: yaml
-   :lines: 62-75
+   :start-at: atlite:
+   :end-before: renewable:
 
 .. csv-table::
    :header-rows: 1

diff --git a/doc/plotting.rst b/doc/plotting.rst
@@ -9,50 +9,6 @@ Plotting and Summary
 
 .. warning:: The corresponding code is currently under revision and has only minimal documentation.
 
-.. _flh:
-
-Rule ``build_country_flh``
-=============================
-
-.. graphviz::
-    :align: center
-
-    digraph snakemake_dag {
-        graph [bgcolor=white,
-            margin=0,
-            size="8,5"
-        ];
-        node [fontname=sans,
-            fontsize=10,
-            penwidth=2,
-            shape=box,
-            style=rounded
-        ];
-        edge [color=grey,
-            penwidth=2
-        ];
-        0	 [color="0.31 0.6 0.85",
-            fillcolor=gray,
-            label=build_country_flh,
-            style=filled];
-        1	 [color="0.06 0.6 0.85",
-            label=base_network];
-        1 -> 0;
-        2	 [color="0.42 0.6 0.85",
-            label=build_natura_raster];
-        2 -> 0;
-        3	 [color="0.58 0.6 0.85",
-            label=build_shapes];
-        3 -> 0;
-        4	 [color="0.14 0.6 0.85",
-            label=build_cutout];
-        4 -> 0;
-    }
-
-|
-
-.. automodule:: build_country_flh
-
 .. _plot_potentials:
 
 Rule ``plot_p_nom_max``

diff --git a/doc/preparation/retrieve.rst b/doc/preparation/retrieve.rst
@@ -21,9 +21,59 @@ Rule ``retrieve_databundle``
 Rule ``retrieve_cutout``
 ------------------------
 
-.. automodule:: retrieve_cutout
+.. image:: https://zenodo.org/badge/DOI/10.5281/zenodo.3517949.svg
+   :target: https://doi.org/10.5281/zenodo.3517949
+
+Cutouts are spatio-temporal subsets of the European weather data from the `ECMWF ERA5 <https://software.ecmwf.int/wiki/display/CKB/ERA5+data+documentation>`_ reanalysis dataset and the `CMSAF SARAH-2 <https://wui.cmsaf.eu/safira/action/viewDoiDetails?acronym=SARAH_V002>`_ solar surface radiation dataset for the year 2013.
+They have been prepared by and are for use with the `atlite <https://github.com/PyPSA/atlite>`_ tool. You can either generate them yourself using the ``build_cutouts`` rule or retrieve them directly from `zenodo <https://doi.org/10.5281/zenodo.3517949>`_ through the rule ``retrieve_cutout``.
+The :ref:`tutorial` uses a smaller cutout than required for the full model (30 MB), which is also automatically downloaded.
+
+.. note::
+    To download cutouts yourself from the `ECMWF ERA5 <https://software.ecmwf.int/wiki/display/CKB/ERA5+data+documentation>`_ you need to `set up the CDS API <https://cds.climate.copernicus.eu/api-how-to>`_.
+
+
+**Relevant Settings**
+
+.. code:: yaml
+
+    tutorial:
+    enable:
+        build_cutout:
+
+.. seealso::
+    Documentation of the configuration file ``config.yaml`` at
+    :ref:`toplevel_cf`
+
+**Outputs**
+
+- ``cutouts/{cutout}``: weather data from either the `ERA5 <https://www.ecmwf.int/en/forecasts/datasets/reanalysis-datasets/era5>`_   reanalysis weather dataset or `SARAH-2 <https://wui.cmsaf.eu/safira/action/viewProduktSearch>`_ satellite-based historic weather data.
+
+.. seealso::
+    For details see :mod:`build_cutout` and read the `atlite documentation <https://atlite.readthedocs.io>`_.
+
 
 Rule ``retrieve_natura_raster``
 -------------------------------
 
-.. automodule:: retrieve_natura_raster
+.. image:: https://zenodo.org/badge/DOI/10.5281/zenodo.4706686.svg
+   :target: https://doi.org/10.5281/zenodo.4706686
+
+This rule, as a substitute for :mod:`build_natura_raster`, downloads an already rasterized version (`natura.tiff <https://zenodo.org/record/4706686/files/natura.tiff>`_) of `Natura 2000 <https://en.wikipedia.org/wiki/Natura_2000>`_ natural protection areas to reduce computation times. The file is placed into the ``resources`` sub-directory.
+
+**Relevant Settings**
+
+.. code:: yaml
+
+    enable:
+        build_natura_raster:
+
+.. seealso::
+    Documentation of the configuration file ``config.yaml`` at
+    :ref:`toplevel_cf`
+
+**Outputs**
+
+- ``resources/natura.tiff``: Rasterized version of `Natura 2000 <https://en.wikipedia.org/wiki/Natura_2000>`_ natural protection areas to reduce computation times.
+
+.. seealso::
+    For details see :mod:`build_natura_raster`.
diff --git a/doc/release_notes.rst b/doc/release_notes.rst
@@ -11,8 +11,12 @@ Release Notes
 Upcoming Release
 ================
 
+* Switch to new major release, ``>=v0.2.1`` of ``atlite``.  The version upgrade comes along with significant speed up for the rule ``build_renewable_profiles.py`` (~factor 2). A lot of the code which calculated the landuse availability is now outsourced and does not rely on ``glaes``, ``geokit`` anymore. This facilitates the environment building and version compatibility of ``gdal``, ``libgdal`` with other packages. 
+* The minimum python version was set to ``3.8``.
+* The rule and script ``build_country_flh`` are removed as they're no longer used or maintained.
+* The flag ``keep_all_available_areas`` in the configuration for renewable potentials (config.yaml -> renewable -> {technology}) was deprecated and now defaults to ``True``. 
+* The tutorial cutout was renamed from ``cutouts/europe-2013-era5.nc`` to ``cutouts/europe-2013-era5-tutorial.nc`` to accomodate tutorial and productive cutouts side-by-side.
 * Fix: Value for ``co2base`` in ``config.yaml`` adjusted to 1.487e9 t CO2-eq (from 3.1e9 t CO2-eq). The new value represents emissions related to the electricity sector for EU+UK. The old value was ~2x too high and used when the emissions wildcard in ``{opts}`` was used.
-
 * Add option to include marginal costs of links representing fuel cells, electrolysis, and battery inverters 
   [`#232 <https://github.com/PyPSA/pypsa-eur/pull/232>`_].
 

diff --git a/doc/wildcards.rst b/doc/wildcards.rst
@@ -130,8 +130,7 @@ It can take the values ``onwind``, ``offwind-ac``, ``offwind-dc``, and ``solar``
 
 The wildcard can moreover be used to create technology specific figures and summaries.
 For instance ``{technology}`` can be used to plot regionally disaggregated potentials
-with the rule :mod:`plot_p_nom_max` or to summarize a particular technology's
-full load hours in various countries with the rule :mod:`build_country_flh`.
+with the rule :mod:`plot_p_nom_max`.
 
 .. _attr: