Add load_dataset and save_dataset functions #392
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #392 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 31 32 +1
Lines 2230 2247 +17
=========================================
+ Hits 2230 2247 +17
Continue to review full report at Codecov.
|
jeromekelleher
approved these changes
Nov 19, 2020
eric-czech
reviewed
Nov 19, 2020
| """ | ||
| store = str(path) | ||
| for v in ds: | ||
| ds[v].encoding.pop("chunks", None) |
There was a problem hiding this comment.
Worth mentioning pydata/xarray#4380 in a comment?
eric-czech
approved these changes
Nov 19, 2020
tomwhite
force-pushed
the
load_and_save
branch
from
fcfa0c5 to
13559a9
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This fixes #298, and also incorporates some of the implementation from https://github.com/related-sciences/ukb-gwas-pipeline-nealelab/blob/4f862e31b8093d25fdaa8da7f841b9be8583cda4/scripts/gwas.py#L53-L71
Unlike #298 it does not rechunk the dataset before saving. I worry about having implicit rechunk methods that the user can't control since we have seen them perform poorly in some cases. For the moment I think it's preferable for the user to explicitly rechunk before saving. This is what I have been doing in the MalariaGEN notebooks, and Eric has too judging by this example.
I also haven't added
fsspecsupport even though the GWAS pipeline uses it. This is because I was getting a warning that files were not being closed when running on local files. That could be addressed separately.