Test data in unit tests and tutorials

[VeckoTheGecko]

How will we be handling test data in v4 of Parcels? At the moment we download it via parcels/tools/exampledata_utils.py, but perhaps it would be better to:

• Generate the data (i.e., xarray datasets) on the fly in the idealised cases
• Download the data (from some hosting provider) for realistic cases. Perhaps using pooch rather than our current custom downloading mechanism.

This will also mean that we can remove parcels/data (and any other committed data files) which I think would be good.

Let's continue this discussion in an issue :)

Originally posted by @VeckoTheGecko in #1946 (comment)

Okay. Let's flesh this out...

Summary

• For idealised cases: Data can be generated
• For real world cases: Download the data
  • For small datasets (<100Mb per file): Host on GitHub via https://github.com/OceanParcels/parcels-data
  • For large datasets: We won't have any example datasets in this category. If we need them, can investigate using Zenodo

In all cases the return should be an xarray dataset object (feedback welcome on this point. Is this suitable for unstructured grids - would it be better to return a uxarray dataset? Would there be instances where we want a collection (e.g., list) of xr.Dataset objects?)

TODO

• Create Parcels data repo
• Migrate data downloading to use Pooch
• Update MovingEddies_data dataset to be generated
• Remove download_example_dataset in favor of get_example_dataset (which will return an xarray object)
  • This is best to do down the line once we have support for easily creating Fields etc from xarray objects, and would require updating the tutorials etc.
• Remove datasets from tests/test_data that don't need to be there (i.e., they can be generated with code)

[VeckoTheGecko]

• Download the data (from some hosting provider) for realistic cases. Perhaps using pooch rather than our current custom downloading mechanism.

Since the examples we're dealing with are quite small, I propose that we host them in a separate GitHub repo (parcels-data). That way we don't have to increase the size of this repo, and enables us to easily update the data if needed (via tags). We could also do this via Zenodo, but that seems like a bit of a headache. If we want alternatives I can go looking at discussions in PyOpenSci or Scientific Python communities - bound to be many alternatives there.

[VeckoTheGecko]

Here are the data files that we currently have in the tests/test_data folder. I want to move as many as possible to code only (i.e., idealised datasets that are created on the fly with xarray) for readability/maintainability.

Status	File
Remove (can be replaced with other pop datasets from functions later)	POPtestdata_time.nc
Keep for now (but reduce dependency of test suite on this data)	Uu_eastward_nemo_cross_180lon.nc
Keep for now (but reduce dependency of test suite on this data)	Vv_eastward_nemo_cross_180lon.nc
Convert to function	access-om2-01_u.nc
Convert to function	access-om2-01_v.nc
Convert to function	access-om2-01_wt.nc
Keep for now (but reduce dependency of test suite on this data)	mask_nemo_cross_180lon.nc
Remove (created by create_testfields.py)	perlinfieldsU.nc
Remove (created by create_testfields.py)	perlinfieldsV.nc
Remove (can be replaced with other pop datasets from functions later)	popcoordinates.nc
Remove (can be replaced with other pop datasets from functions later)	popdata.nc
Keep (validation with v3)	test_interpolation_data_random_cgrid_velocity.nc
Keep (validation with v3)	test_interpolation_data_random_freeslip.nc
Keep (validation with v3)	test_interpolation_data_random_linear.nc
Keep (validation with v3)	test_interpolation_data_random_nearest.nc
Keep (validation with v3)	test_interpolation_jit_cgrid_velocity.zarr
Keep (validation with v3)	test_interpolation_jit_freeslip.zarr
Keep (validation with v3)	test_interpolation_jit_linear.zarr
Keep (validation with v3)	test_interpolation_jit_nearest.zarr
Remove (created by create_testfields.py)	testfieldsP.nc
Remove (created by create_testfields.py)	testfieldsU.nc
Remove (created by create_testfields.py)	testfieldsV.nc

Still need to work out what to do with the files that don't have a status there. Not sure whether they were created by code or are from somewhere else (or where they are used in the codebase). cc @erikvansebille