What's wrong with my kerchunk json?

[cspencerjones]

Hi,

I am trying to use kerchunk to open some netcdfs stored on the Open Storage Network (all the data in this problem is public). We don't have local compute so we create both the files and the kerchunk json file elsewhere. Then I did a search-and-replace on the json file so that the paths should point to locations on OSN before uploading netcdfs and json. But I can't open the data with the json. Can you help me identify what is wrong please?

Here is the code I am using to open data via my kerchunk-generated json:

import s3fs
import xarray as xr

endpoint_url = 'https://mghp.osn.xsede.org'
fs = s3fs.S3FileSystem(anon=True,
    client_kwargs={'endpoint_url': endpoint_url}
)

mapper = fs.open('cnh-bucket-1/csjones/temp/replaced_fields.json', mode='rb')

ds = xr.open_dataset("reference://", engine="zarr",backend_kwargs={"storage_options": {"fo": mapper,
    "remote_options": {"client_kwargs":{'endpoint_url': endpoint_url}}},"consolidated": False})
ds

The code runs, but the data is just nans, which suggests that the json doesn't point to the correct place.

Here is some code to open one of the individual netcdf files

endpoint_url = 'https://mghp.osn.xsede.org'
fs = s3fs.S3FileSystem(
    anon=True,
    client_kwargs={'endpoint_url': endpoint_url}
)
file_url='cnh-bucket-1/csjones/temp/llc4320_Eta-U-V-W-Theta-Salt_f1_k0_chunka_iter_10512.nc'
mapper = fs.open(file_url, mode='rb')

# open the data 
ds= xr.open_dataset(mapper)

You can download the json directly from https://mghp.osn.xsede.org/cnh-bucket-1/csjones/temp/replaced_fields.json if you want to look at it directly.

[martindurant]

I wrote it out a little more verbosely:

refs = ujson.load(mapper)
ds = xr.open_dataset("reference://", engine="zarr",backend_kwargs={"storage_options": {"fo": refs,
    "remote_options": {"client_kwargs":{'endpoint_url': endpoint_url}, "anon": True}},"consolidated": False})

and this does have values

<xarray.Dataset>
Dimensions:  (time: 100, face: 1, j: 4320, i: 4320, k: 1, i_g: 4320, j_g: 4320,
              k_l: 1, k_p1: 2, k_u: 1)
Coordinates:
  * face     (face) float64 1.0
  * i        (i) float64 0.0 1.0 2.0 3.0 ... 4.317e+03 4.318e+03 4.319e+03
  * i_g      (i_g) float64 0.0 1.0 2.0 3.0 ... 4.317e+03 4.318e+03 4.319e+03
  * j        (j) float64 0.0 1.0 2.0 3.0 ... 4.317e+03 4.318e+03 4.319e+03
  * j_g      (j_g) float64 0.0 1.0 2.0 3.0 ... 4.317e+03 4.318e+03 4.319e+03
  * k        (k) float64 0.0
  * k_l      (k_l) float64 0.0
  * k_p1     (k_p1) float64 0.0 1.0
  * k_u      (k_u) float64 0.0
    niter    (time) float64 ...
  * time     (time) datetime64[ns] 2011-09-13T01:00:00 ... 2011-09-17T04:00:00
Data variables:
    Eta      (time, face, j, i) float32 ...
    Salt     (time, k, face, j, i) float32 ...
    Theta    (time, k, face, j, i) float32 ...
    U        (time, k, face, j, i_g) float32 ...
    V        (time, k, face, j_g, i) float32 ...
    W        (time, k_l, face, j, i) float32 ...

Perhaps "remote_protocol" was the missing thing.

[cspencerjones]

I am pleased that you are able to open my dataset, but interestingly, this didn't solve the problem for me! It sounds as though the json isn't the problem...perhaps I have a versioning problem. I have xarray version 2022.6.0, s3fs version 2023.1.0

Do you think that's the cause? Thanks for your help on this.

My data is still filled with nans when I run:

import s3fs
import xarray as xr
import ujson

endpoint_url = 'https://mghp.osn.xsede.org'
fs = s3fs.S3FileSystem(anon=True,
    client_kwargs={'endpoint_url': endpoint_url}
)

mapper = fs.open('cnh-bucket-1/csjones/temp/replaced_fields.json', mode='rb')


refs = ujson.load(mapper)
ds = xr.open_dataset("reference://", engine="zarr",backend_kwargs={"storage_options": {"fo": refs,
    "remote_options": {"client_kwargs":{'endpoint_url': endpoint_url}, "anon": True}},"consolidated": False})

<xarray.Dataset>
Dimensions:  (time: 100, face: 1, j: 4320, i: 4320, k: 1, i_g: 4320, j_g: 4320,
              k_l: 1, k_p1: 2, k_u: 1)
Coordinates:
  * face     (face) float64 1.0
  * i        (i) float64 nan nan nan nan nan nan nan ... nan nan nan nan nan nan
  * i_g      (i_g) float64 nan nan nan nan nan nan ... nan nan nan nan nan nan
  * j        (j) float64 nan nan nan nan nan nan nan ... nan nan nan nan nan nan
  * j_g      (j_g) float64 nan nan nan nan nan nan ... nan nan nan nan nan nan
  * k        (k) float64 0.0
  * k_l      (k_l) float64 0.0
  * k_p1     (k_p1) float64 0.0 1.0
  * k_u      (k_u) float64 0.0
    niter    (time) float64 1.051e+04 1.066e+04 1.08e+04 ... 2.462e+04 2.477e+04
  * time     (time) datetime64[ns] 2011-09-13T01:00:00 ... 2011-09-17T04:00:00
Data variables:
    Eta      (time, face, j, i) float32 ...
    Salt     (time, k, face, j, i) float32 ...
    Theta    (time, k, face, j, i) float32 ...
    U        (time, k, face, j, i_g) float32 ...
    V        (time, k, face, j_g, i) float32 ...
    W        (time, k_l, face, j, i) float32 ...

[martindurant]

I have s3fs/fsspec always at main/master on my local system - always a good idea for this experimental suff.

[cspencerjones]

Thanks for the advice. I installed these from the source on github and now I have s3fs.__version__ is 2023.3.0+1.g78e6c9a and fsspec.__version__ is 2023.3.0+13.g95eb5f9. But I am STILL getting nans in the output.

You mentioned remote_protocol in your original answer, but as far as I can tell the answer you posted doesn't provide the remote_protocol variable, though you did add "anon": True. Is that what you were talking about? I'm just wondering if there's a line missing from your answer, or something like that.

[martindurant]

the answer you posted doesn't provide the remote_protocol variable

I wonder where it went??
Should be "remote_protocol": "s3" inside storage_options

[cspencerjones]

It works! Thank you