Improve reproducibility dd

[Tjalling-dejong]

Issue addressed

Fixes #356 #521 and partially #537

Explanation

This PR updates the deltares data catalog to include new features such as versions/variants of data sources. Also if a data source has been pre-processed the path to the pre-processing script or the notes describing the pre-processing is included.
In addition I added the temporal extent of the datasets with a temporal dimension.

Checklist

• Updated tests or added new tests
• Branch is up to date with main
• Tests & pre-commit hooks pass
• Updated documentation if needed
• Updated changelog.rst if needed

Additional Notes (optional)

Add any additional notes or information that may be helpful.

[savente93]

Nice work Tjalling. It would be nice if we can also add the spatial extent and also host the pre-processing scripts we now have locally (I assume on the P drive) somewhere but these can be separate issues I think, so we can merge this for now.

[DirkEilander]

Thanks for this work @Tjalling-dejong! Unfortunately this PR got merged too soon I think seeing some comments on Teams. This change directly affects all users and we need to think about how to properly test it. I've reverted the merge (see #703) and restored the branch. You can create a new PR from the branch to continue the review process.

[Tjalling-dejong]

I replicated the error with the soilgrids dataset that people have raised in the HydroMT chat. The error persists in earlier versions of the data catalog as well. What has changed with this PR is that I added versions to data catalog items. With this change the users are served the latest version of a dataset by default. Thus, the soilgrids dataset given to the users is the latest, namely soilgrids_v2.0. This version of the dataset also breaks in earlier versions of the data catalog.

Error can be replicated with datacatalog.get_rasterdataset("soilgrids_v2.0").

[Tjalling-dejong]

I have made an issue regarding the error with soilgrids_v2.0 dataset (#706).

[DirkEilander]

Thanks for checking this issue @Tjalling-dejong 👍

The issue was indeed not caused by this PR, but it did change the default dataset as you already mention. I think it would be good to:

1. have a small test that only runs locally to actually reads (a random slice of) all the datasets in the catalog
2. keep track and communicate for which datasets the default version changes when we publish. Perhaps we can also add a note in the meta data of the items like updated_in_version: xx
3. make it common practice to add the version of your dataset in your hydromt scripts / model builder yml files (for discussion in the next plugin meeting @alimeshgi)

Would it be possible to add point 1 to this PR? And let's discuss about point 2.