Use pooch

[Zeitsperre]

Pull Request Checklist:

• This PR addresses an already opened issue (for bug fixes / features)
  • This PR relies on changes to be merged in Adapt to pooch and cleanup testing data xclim-testdata#29
• Tests for the changes have been added (for bug fixes / features)
  • (If applicable) Documentation has been added / updated (for bug fixes / features)
• CHANGELOG.rst has been updated (with summary of main changes)
  • Link to issue (:issue:number) and pull request (:pull:number) has been added

What kind of change does this PR introduce?

• Replaces the logic for file gathering and caching from the in-house developed version to instead use pooch.
  • In order to fetch testing data, one can now use the following:

  from xclim.testing.helpers import nimbus
  
  n = nimbus()
  # from a fork of xclim-testdata: nimbus(repo="https://github.com/Me/My_Repo", branch="my_test_branch")
  file = n.fetch("some_folder/some_data.nc")

• Removes the remote GitHub calls for every file request (which was performed by _get()).
• Exports most of the file request and cache handling to pooch, while maintaining a relatively unchanged API for users.
• (To be confirmed) Speeds up the delivery of test data to tests by reducing the amount of redundant calls to fixtures and relying on a single pooch instance of pooch to prevent multiple setup stages.

Does this PR introduce a breaking change?

Absolutely. get_file and open_dataset no longer fetch remote files from GitHub. Instead, a locally-stored registry.txt file contains all the checksums of all files needed to run the tests and returns the appropriate file from a locally-held cache. If the file checksum does not match the expected value, it will attempt to replace it from the remote storage.

Additionally, the md5 files that accompanied all testing data files are now obsolete thanks to the use of the registry. The testing data is now versioned according to the xclim-testdata version/tag.

All the prefetch logic baked into the pytest calls has been removed, making the setup code much easier to follow. There is no longer a need to run $ xclim prefetch_testing_data unless users are running on Windows (for the very first run of pytest only).

There are now three environment variables to help developers:

• XCLIM_TESTDATA_BRANCH
  • Controls the branch name of xclim-testdata.
• XCLIM_TESTDATA_CACHE_DIR
  • Controls the local folder to be used when fetching the test data.
• XCLIM_TESTDATA_REPO_URL
  • Controls the repository URL for xclim-testdata (for forks)

platformdirs is no longer a hard dependency. The default cache directory will only be determined if pooch is installed.

Other information:

There is still a lot of potential here to tighten this up; I'd like to land on a design that is clean and easily portable to other projects.

What is unchanged is that pytest will still do the following on every run:

1. Check that a locally stored copy of the test data exists in a platform-dependent default location and, if not found, will fetch a copy.
2. Each worker of pytest creates its own copy of the test data, which is delivered by its own pooch instance, written to a threadsafe temporary directory
3. The equivalent to the get_file() fixture is now nimbus.fetch(), providing the absolute paths to files, respective of platform and workers.

Many tests related to testing the file accessors have been removed (as these are now out of scope), but I'd like to test the implementation a bit before merging this (for both coverage and peace of mind).

[github-actions]

Note

It appears that this Pull Request modifies the main.yml workflow.

On inspection, the XCLIM_TESTDATA_BRANCH environment variable is set to the most recent tag (v2024.8.23).

No further action is required.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[coxipi]

THe only change on user side is that open_dataset will automatically use caching if pooch is installed correct?

I tried reviewing the code but there is a lot of things I don't really understand, my approval would not be much better than rubber stamp

[aulemahal]

This looks good! Thanks.
A suggestion and yet again a nitpicky comment on the changelog ;).