file: part3/chunking_introduction

[tinaok]

• Update obsolete commands
• Put visibilities of 'Zarr' in table of contents and/or title
• Fix warning on hdf.SingleHdf5ToZarr
• update example for exporting zarr,
• add intake
• put intake catalogue on s3? update file path name update from foss4g to ?

[tinaok]

The data we use to show chunking explanation could be better if we can use data from former 3 notebooks.

[tinaok]

The title of this section also include 'zarr'.

[tinaok]

on pangeo-eosc cluster
fsspec 2022.7.1
kerchunk '0.0.7'

on my Mac-m1 local enviroment
fsspec 2023.3.0
kerchunk 0.1.2
This works, but with my new local enviroment (same version as https://github.com/pangeo-data/pangeo-docker-images/blob/master/pangeo-notebook/packages.txt)
fsspec==2023.10.0
kerchunk==0.2.1

Kerchunk shows warning and does not translate all informations

UserWarning: The following excepion was caught and quashed while traversing HDF5
'SingleHdf5ToZarr' object has no attribute 'decoders'
Traceback (most recent call last):
File "/Users/todaka/micromamba/envs/bids23/lib/python3.11/site-packages/kerchunk/hdf.py", line 262, in _translator
filters = self._decode_filters(h5obj)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/todaka/micromamba/envs/bids23/lib/python3.11/site-packages/kerchunk/hdf.py", line 246, in _decode_filters
f" not supported by kerchunk., supported filters are {self.decoders.keys()}"
^^^^^^^^^^^^^
AttributeError: 'SingleHdf5ToZarr' object has no attribute 'decoders'
warnings.warn(msg)

[tinaok]

on pangeo-eosc cluster fsspec 2022.7.1 kerchunk '0.0.7'

on my Mac-m1 local enviroment fsspec 2023.3.0 kerchunk 0.1.2 This works, but with my new local enviroment (same version as https://github.com/pangeo-data/pangeo-docker-images/blob/master/pangeo-notebook/packages.txt) fsspec==2023.10.0 kerchunk==0.2.1

Kerchunk shows warning and does not translate all informations

UserWarning: The following excepion was caught and quashed while traversing HDF5
'SingleHdf5ToZarr' object has no attribute 'decoders'
Traceback (most recent call last):
File "/Users/todaka/micromamba/envs/bids23/lib/python3.11/site-packages/kerchunk/hdf.py", line 262, in _translator
filters = self._decode_filters(h5obj)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/todaka/micromamba/envs/bids23/lib/python3.11/site-packages/kerchunk/hdf.py", line 246, in _decode_filters
f" not supported by kerchunk., supported filters are {self.decoders.keys()}"
^^^^^^^^^^^^^
AttributeError: 'SingleHdf5ToZarr' object has no attribute 'decoders'
warnings.warn(msg)

Fixed with fsspec/kerchunk#382
kerchunk==0.2.2

[tinaok]

Update data set, and xarray kerchunk backend will be later

[tinaok]

#22

[tinaok]

#41
for updating zarr, and update with snow example.

[tinaok]

we will keep kerchunk, and to do so we will go back to original NVDI example.