Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Generalize retrieve databundle #260

Merged
merged 70 commits into from
Mar 9, 2022
Merged

Generalize retrieve databundle #260

merged 70 commits into from
Mar 9, 2022

Conversation

davide-f
Copy link
Member

@davide-f davide-f commented Feb 6, 2022

Closes # #259

Changes proposed in this Pull Request

This pull request aims at providing a general retrieve_databundle structure that enables a flexible addition of diversified bundles for different regions.

Checklist

  • I tested my contribution locally and it seems to work fine.
  • Code and workflow changes are sufficiently documented.
  • Newly introduced dependencies are added to envs/environment.yaml and envs/environment.docs.yaml.
  • Changes in configuration options are added in all of config.default.yaml, config.tutorial.yaml, and test/config.test1.yaml.
  • Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/configuration.rst and doc/tutorial.rst.
  • A note for the release notes doc/release_notes.rst is amended in the format of previous release notes.

@restyled-io restyled-io bot mentioned this pull request Feb 6, 2022
Closed
@davide-f
Copy link
Member Author

davide-f commented Feb 6, 2022 •
edited
Loading

The pull request is already in good shape, yet, I would like to improve how it works.
Currently, for every category [resources, data, cutouts, common], it downloads a single databundle, yet in general, since we may want a databundle per continent (if not more):

  • the procedure shall download more databundles per category.

Moreover, debugging is needed including the appropriate urls and we may need to re-organize the databundles for consistency. In particular:

  • need to merge common world-level data [e.g. GDP raster] to a single zip file
  • load to zenodo only zip files

davide-f and others added 23 commits February 15, 2022 17:13
@davide-f
Add bundle_config docstring and specify urls for selection of bundles
00dc1f4
@davide-f
Merge branch 'pypsa-meets-africa:main' into generalize-retrieve_datab…
0aebc7b 
…undle
@davide-f
Set bundle_data countries according to world_iso
333f31e
@davide-f
Re-enable comments in yaml format
8e8cf92
@davide-f
Improve bundle selection algorithm: select bundles till matching each…
92fd949 
… category
@davide-f
Add summary
b0f6da2
@davide-f
Add debug notebook and revise downloading data_bundle
f6958b5
@davide-f
Revise data_bundle.yaml
c06917d
@davide-f
Typo
db01c1c
@davide-f
Remove typo
6c8fd4b
@davide-f
Improve bundle_configs links and code; bug fixing
008e51f
@davide-f
Fixing tutorial data google drive
d3ae35e
@davide-f
Update bundle data with sandbox zenodo
e4adb40
@davide-f
Standardization bundle_config and improvement zips
2b4dbd1
@davide-f
Update tutorial google drive links
97c5cdb
@davide-f
Add test zenodo links for all bundles to bundle_configs
6ee6b96
@davide-f
Add docstring and add hot_run parameter when downloading files for te…
2ffd216 
…st purposes
@davide-f
Typo zenodo link
42592ad
@davide-f
Add and improve docstrings
f6d7149
@davide-f
Specify folder cutouts_africa
9642699
@restyled-commits
Restyled by autopep8
bc63019
@restyled-commits
Restyled by black
074156b
@restyled-commits
Restyled by isort
32a2a90
@restyled-io restyled-io bot mentioned this pull request Feb 17, 2022
Closed
@davide-f
Copy link
Member Author

davide-f commented Feb 17, 2022 •
edited
Loading

I've just noted that the resource files from gegis are just the raw files; they don't have the fix for the empty countries, thus the known issue arise.

  • Need to update the raw files in drive and zenodo

Moreover:

  • Once settled, and we agree on merging, we shall move the zenodo files from sandbox [the test repo I've used] to the official one

@davide-f davide-f mentioned this pull request Feb 17, 2022
Closed
@davide-f
Copy link
Member Author

I filled the databundles the "empty" Africa countries demand data for all years [2030, 2040, 2050 and 2100] with era5 data 2013, and updated all databundles [zenodo&googledrive] accordingly.

This branch may be merged as-is and it should work; anyway, we may need to use the links to the official PyPSA-Earth zenodo.
The easy way if all zip files are moved from one zenodo to the other one

@davide-f
Copy link
Member Author

davide-f commented Feb 19, 2022 •
edited
Loading

Tests performed:

  • Turkey test (with manual change of input file Africa.nc to Asia.nc as input file to add_electricity)
  • Africa test

Once both tests are complete (expected soon), I think this PR could be ready to merge, however, we shall update the official Zenodo; @pz-max I'm not sure if you can add permissions for pushing file to Zenodo, otherwise, you may just download and upload all zip files available (except resources_africa that is a duplication) at sandbox Zenodo.
As a little comment, resources_world contains demand data for the whole world, but the natura.tiff is calibrated only for africa.
I may update that file, but also the current version is file as natura.tiff can be created by enabling build_cutout.

@pz-max
Copy link
Member

pz-max commented Feb 20, 2022

Awesome! Regarding Zenodo, I made a fatal error. I uploaded everything from my Github linked account. Unfortunately, I cannot give anyone edit access or something like that. Suggestion. I created now a new gmail account called pypsameetsearth@gmail.com. We register a Zenodo account over it so that everyone can upload data?

Login details can be found in the trusted coordinator discord channel

@davide-f davide-f mentioned this pull request Feb 20, 2022
Closed
@davide-f
Copy link
Member Author

@pz-max I totally agree on creating the Zenodo account for the organization. Add some comments in discord

@pz-max
Copy link
Member

pz-max commented Mar 8, 2022

@davide-f can this be merged?

@davide-f
Copy link
Member Author

davide-f commented Mar 9, 2022

I think this PR can be merged; though there can be improvements but postponed into future PRs and commits.
In particular, I foresee:

  • automatically download the eez zones
  • automatically download the WPDA data
  • add license package to the documentation

@pz-max pz-max mentioned this pull request Mar 9, 2022
Closed
3 tasks
@pz-max pz-max merged commit 2d4cbb9 into pypsa-meets-earth:main Mar 9, 2022
@davide-f davide-f deleted the generalize-retrieve_databundle branch March 19, 2022 20:04
pz-max pushed a commit that referenced this pull request Sep 24, 2022
@davide-f @restyled-commits
Generalize retrieve databundle (#260)
11bf7d8 
* Draft bundle file configurations

* Revision bundle config

* Modify Snakemake and draft of retrieve_databundle

* First implementation of flexible bundles specified by config.yaml

* Update release_notes

* Add bundle_config docstring and specify urls for selection of bundles

* Set bundle_data countries according to world_iso

* Re-enable comments in yaml format

* Improve bundle selection algorithm: select bundles till matching each category

* Add summary

* Add debug notebook and revise downloading data_bundle

* Revise data_bundle.yaml

* Typo

* Remove typo

* Improve bundle_configs links and code; bug fixing

* Fixing tutorial data google drive

* Update bundle data with sandbox zenodo

* Standardization bundle_config and improvement zips

* Update tutorial google drive links

* Add test zenodo links for all bundles to bundle_configs

* Add docstring and add hot_run parameter when downloading files for test purposes

* Typo zenodo link

* Add and improve docstrings

* Specify folder cutouts_africa

* Add retrieve_bundle outputs and adapt bundles

* Fix hydrobasins dependencies for general case

* Use tqdm instead of progressbar for progress_retrieve

* Generalize hydrobasins and minor edits

* Round progress bar to the first 0.01

* Revise hydro profiles to avoid computational issues with large hydrobasins

* Revise hydro calculation

* Revise link common_data

* Update docstring and description retrieve_databundle

* Fix docstrings

* Test remove version limit to ipopt

* Attempt to fix CI-windows; ipopt<=3.11.1

* Update links databundles with complete data for Africa demand

* Restore ipopt version

* Improve Snakemake and bundle_config for world compatibility

* Merge https://github.com/davide-f/pypsa-africa into generalize-retrieve_databundle

* Add set_level to retrieve_databundle to enable logging info

Co-authored-by: Restyled.io <commits@restyled.io>
FabianHofmann pushed a commit that referenced this pull request Aug 16, 2024
@energyLS
Merge pull request #260 from pypsa-meets-earth/clean_data
347748c 
Clean data
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pz-max pz-max pz-max left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Generalize retrieve_databundle to the world
3 participants
@davide-f @pz-max @restyled-commits