Copernicusmarine bug

[j-atkins]

I've been getting some things ready for an application of VirtualShip in a course at the start of September, and found virtualship fetch is consistently failing with:

...

  File "/opt/anaconda3/envs/ship/lib/python3.10/site-packages/botocore/client.py", line 1058, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (429) when calling the HeadObject operation: Too Many Requests

This same bug has already been reported in the copernicusmarine issue tracker and appears related to zarr dependencies. There is a PR with a fix but it's not yet merged and I'm not sure what their timeline will be on that.

Hopefully the fix should be implemented in time for early September but it leaves me wondering how we should handle updating VirtualShip once it is... I suppose ideally copernicusmarine will do a release and we can update our dependencies. If not, are there options to patch the change into the VirtualShip build ourselves?

@VeckoTheGecko would you have any ideas here?

cc @ammedd @erikvansebille

[VeckoTheGecko]

Good investigation! Indeed looks to be related to Zarr v2. Unfortunately zarr v3 support in Parcels will only arrive in v4 of Parcels.

We can:

• upper-pin copernicus marine so that it gets a lower version (which I think would work better than the latest)
• Wait for the PR to be merged and for a new release to come
• pip install directly from the branch with the fix (i.e., pip install git+https://github.com/mercator-ocean/copernicus-marine-toolbox@adapting-zarr-v2)

Patching from the virtualship side will either not be possible or be more effort than its worth

[j-atkins]

Update: I've tried upper-pinning copernicusmarine but I'm either running into the same bug with versions > 2, whilst versions < 1 are incompatible with VirtualShip.

There is a fix + release in progress on the copernicusmarine side so I think waiting on that is the best option at this point 👍

[uriii3]

You can try the version 2.2.2, it should avoid the ClientErrors. Let us know if you have any problems related with it! @VeckoTheGecko @j-atkins

[j-atkins]

Many thanks @uriii3, I'll give this a go!