ATL03 Cloud Optimized Samples

[betolink]

I have a few ATL03 files that were cloud optimized and put in a cryocloud-accessible S3 bucket. I wonder if they could be used to benchmark access to some of the SlideRule workflows or just to test that we can open them in a performant way. This could be h5coro or h5py, if h5py is used we'll have to adjust some parameters to let the library know we are opening a file with a different file layout, see reference gist

Size	URL
7760000000	s3://nasa-cryo-persistent/h5cloud/atl03/test/ATL03_20181120182818_08110112_006_02-page-only-8mb.h5
7720000000	s3://nasa-cryo-persistent/h5cloud/atl03/test/ATL03_20181120182818_08110112_006_02-rechunked-100k-page-8mb-repacked.h5
7728000000	s3://nasa-cryo-persistent/h5cloud/atl03/test/ATL03_20181120182818_08110112_006_02-rechunked-100k-page-8mb.h5
7754735138	s3://nasa-cryo-persistent/h5cloud/atl03/test/ATL03_20181120182818_08110112_006_02.h5
336000000	s3://nasa-cryo-persistent/h5cloud/atl03/test/ATL03_20191225111315_13680501_006_01-page-only-8mb.h5
336000000	s3://nasa-cryo-persistent/h5cloud/atl03/test/ATL03_20191225111315_13680501_006_01-rechunked-100k-page-8mb-repacked.h5
344000000	s3://nasa-cryo-persistent/h5cloud/atl03/test/ATL03_20191225111315_13680501_006_01-rechunked-100k-page-8mb.h5
326397686	s3://nasa-cryo-persistent/h5cloud/atl03/test/ATL03_20191225111315_13680501_006_01.h5
6936000000	s3://nasa-cryo-persistent/h5cloud/atl03/test/ATL03_20200217204710_08110612_006_01-page-only-8mb.h5
6888000000	s3://nasa-cryo-persistent/h5cloud/atl03/test/ATL03_20200217204710_08110612_006_01-rechunked-100k-page-8mb-repacked.h5
6896000000	s3://nasa-cryo-persistent/h5cloud/atl03/test/ATL03_20200217204710_08110612_006_01-rechunked-100k-page-8mb.h5
6925710500	s3://nasa-cryo-persistent/h5cloud/atl03/test/ATL03_20200217204710_08110612_006_01.h5
896000000	s3://nasa-cryo-persistent/h5cloud/atl03/test/ATL03_20230618223036_13681901_006_01-page-only-8mb.h5
888000000	s3://nasa-cryo-persistent/h5cloud/atl03/test/ATL03_20230618223036_13681901_006_01-rechunked-100k-page-8mb-repacked.h5
896000000	s3://nasa-cryo-persistent/h5cloud/atl03/test/ATL03_20230618223036_13681901_006_01-rechunked-100k-page-8mb.h5
885330898	s3://nasa-cryo-persistent/h5cloud/atl03/test/ATL03_20230618223036_13681901_006_01.h5

So we have 3 different sizes and each file has 3 different flavors.

• -page-only-8mb.h5 files were repacked with the paged aggregation strategy, this affects only the metadata by consolidating it and forcing the library to use a fixed page size for remote requests.
• rechunked-100k-page-8mb.h5: This flavor has the same page aggregation of the metadata and also repacked the heights group, this means that h_ph are now in chunks of 100k items instead of 10k, same level 6 compression (not a lot of compression).
• rechunked-100k-page-8mb-repacked.h5: same as above but I ran the h5repack command at the end to reduce wasted internal space. Mixed results on that, the h5stat still indicates that we have a lot of unaccounted space in the files after we rechunk them.

I'll run some benchmarks on these files again and will post them here, again if they could be tested with SlideRule that would be great.

[betolink]

I just noticed that the rechunked versions (most of them) are not readable out of the box with xarray(via h5netcdf), I get a long stack-trace exception and the last lines indicate that some references for the group are not there. Using h5py directly seems to be working fine, will investigate further.

File h5py/_objects.pyx:55, in h5py._objects.with_phil.wrapper()

File /srv/conda/envs/notebook/lib/python3.11/site-packages/h5py/_hl/group.py:353, in Group.__getitem__(self, name)
    350 """ Open an object in the file """
    352 if isinstance(name, h5r.Reference):
--> 353     oid = h5r.dereference(name, self.id)
    354     if oid is None:
    355         raise ValueError("Invalid HDF5 object reference")

File h5py/_objects.pyx:54, in h5py._objects.with_phil.wrapper()

File h5py/_objects.pyx:55, in h5py._objects.with_phil.wrapper()

File h5py/h5r.pyx:83, in h5py.h5r.dereference()

KeyError: 'Unable to open object by token (bad object header version number)'

[jefflee-icesat2]

Dimension scales are causing the problem. I failed to re-reference the dimension scales correctly. Let me see if I can figure out how to do that in a sensible way...

[betolink]

Thanks Jeff! I think we may have a workaround but will test it tomorrow. @jefflee-icesat2

[betolink]

Update: we have generated a new round of cloud optimized samples for ATL03: https://its-live-data.s3.amazonaws.com/index.html#test-space/cloud-experiments/h5cloud/atl03/ (we need to navigate to this location)

The files are now on a publicly accessible bucket:

s3://its-live-data/test-space/cloud-experiments/h5cloud/atl03/

There are 3 original files with different sizes (300MB, 800MB, 7GB) and each of them has 4 variants, 2 using only paged aggregation and 2 using paged aggregation and rechunked photons (100k per chunk). I also created the dmrpp and kerchunk references for future benchmarks. I'm going to create a new notebook to benchmark access using h5py, xarray, h5coro and perhaps the most recent native ROS3 driver if time allows.

cc. @jefflee-icesat2 @asteiker @jpswinski @ajelenak