netCDF + lazy backend: Error when sel is used with slice, reverse arrange

[jesieleo]

What happened?

import xarray as xr
data = xr.open_dataset('data_example.nc')

# wrong
a = data.sel(time='1979-1',isobaricInhPa=200).z[:,  ::10,::10][:, ::-1,:]

It seem that when use sel() , [: , ::-1 , :] and [: , ::10 , ::10] at the same time will cause the second coord will wrong like this.
Why just when use sel and[: , ::10 , ::10][: , ::-1 , :] ,it will goes wrong in axis=2( the data DO NOT correspond to the coordinates) ?

What did you expect to happen?

The data correspond to the coordinates on Latitude.

Minimal Complete Verifiable Example

problem_example.zip

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

INSTALLED VERSIONS

---

commit: None
python: 3.8.12 (default, Oct 12 2021, 03:01:40) [MSC v.1916 64 bit (AMD64)]
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: AMD64 Family 25 Mod
el 33 Stepping 0, AuthenticAMD
byteorder: little
LC_ALL: None
LANG: en
LOCALE: ('Chinese (Simplified)_China', '936')
libhdf5: 1.12.1
libnetcdf: 4.8.1
xarray: 2022.3.0
pandas: 1.4.2
numpy: 1.22.3
scipy: 1.7.3
netCDF4: 1.5.7
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.6.0
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.2.10
cfgrib: 0.9.10.1
iris: None
bottleneck: None
dask: None
distributed: None
matplotlib: 3.5.1
cartopy: 0.20.2
seaborn: None
numbagg: None
fsspec: None
cupy: None
pint: None
sparse: None
setuptools: 56.0.0
pip: 22.0.4
conda: None
pytest: None
IPython: 8.3.0
sphinx: 4.5.0

[max-sixty]

@jesieleo — I'm probably being unclear, and I'm sure you're well-intentioned, but this & the previous issues aren't constructive for either of us so far.

In the politest possible way: please make an issue that is complete and "copy & pastes into an IPython prompt or Binder notebook, returning the result.". You cannot reference data that is only on your computer. This will require work from your end to take the example you have there, and reduce it to the bare minimum.

[jesieleo]

Sorry, this is the first time I use Github to report a problem, I don't understand it very well, it seems that I need to learn more. I have added a zip dile with a .ipynb file , don't know if it is ok?

…

------------------ 原始邮件 ------------------ 发件人: "Maximilian ***@***.***>; 发送时间: 2022年5月2日(星期一) 下午4:32 收件人: ***@***.***>; 抄送: ***@***.***>; ***@***.***>; 主题: Re: [pydata/xarray] error when using .sel() and [:, ::10, ::10][:, ::-1, :] (Issue #6560) @jesieleo — I'm probably being unclear, and I'm sure you're well-intentioned, but this & the previous issues aren't constructive for either of us so far. In the politest possible way: please make an issue that is complete and "copy & pastes into an IPython prompt or Binder notebook, returning the result.". You cannot reference data that is only on your computer. This will require work from your end to take the example you have there, and reduce it to the bare minimum. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[max-sixty]

OK — I would like to make your experience GitHub a good one! It's a magical place.

I have added a zip dile with a .ipynb file , don't know if it is ok?

But no, this is not sufficient — can you show your problem on an array like xr.DataArray(np.arange(12).reshape(2,3,4)), only adding extra as necessary? Please have another read of the links above showing how to minimize the examples.

[jesieleo]

But no, this is not sufficient — can you show your problem on an array like xr.DataArray(np.arange(12).reshape(2,3,4)), only adding extra as necessary? Please have another read of the links above showing how to minimize the examples.

Ok ,thanks. I try to simplify the problem on an array
a = data3.sel(time='1979-1',isobaricInhPa=200).z[:, ::10,::10][:, ::-1,:] ,my question is that the coordinates DO NOT correspond to the data, there is 73 on coordinates latitude but only the there is 72 only on real data latitude.

[max-sixty]

What is data3? We need an example that I can paste into a blank notebook and will show the problem.

[jesieleo]

What is data3? We need an example that I can paste into a blank notebook and will show the problem.

I packed them in one .zip this time ,the second-dimensional coordinates of variable a do not correspond to the data

[max-sixty]

But no, this is not sufficient — can you show your problem on an array like xr.DataArray(np.arange(12).reshape(2,3,4)), only adding extra as necessary? Please have another read of the links above showing how to minimize the examples.

Don't bundle data — build a new example from a basic array and demonstrate the bug.

[jesieleo]

Don't bundle data — build a new example from a basic array and demonstrate the bug.

Since I don't know how the grib file is generated, I can't reproduce it. But the data I use is the ECWMF official website data converted by engine='cfgrib', this problem may be caused by the 'cfgrib' engine, although it is not very big wrong, but it's a hidden danger.
I'm not competent enough, sorry to bother you.

[dcherian]

OK turns out my attempts at a minimal example didn't work.

There's some bug here (on xarray main)

1. the size of latitude is different in .sizes, .shape, .data.shape and
2. latitude really should be 73 elements. I don't see why a subsetting happens when slicing with ::-

---

@jesieleo here's how I tried to create a "minimal example"

import xarray as xr
import numpy as np

ds = xr.Dataset(
    {
        "z": (
            ("time", "isoBaricInhPa", "latitude", "longitude"),
            np.ones((1, 5, 721, 1440)), # use numpy.ones to create array of same shape.
        )
    },
    coords={"latitude": np.linspace(-90, 90, 721)},
)
ds.isel(isoBaricInhPa=1).z[:, ::10, :][:, ::-1, :]

Though this doesn't replicate it!

[dcherian]

OK it has to do with the lazy backend. Calling data_example.load() fixes things. @jesieleo in the mean time, if your data is small use load_dataset or load to fix it.

Though I still can't make a minimal example. The example below works fine.

import numpy as np
import xarray as xr
ds = xr.Dataset(
    {
        "z": (
            ("time", "isoBaricInhPa", "latitude", "longitude"),
            np.ones((1, 5, 721, 1440)),
        )
    },
    coords={"latitude": np.linspace(-90, 90, 721)},
).to_netcdf("test.nc")
ds = xr.open_dataset('test.nc')
ds.isel(time=[0], isoBaricInhPa=1).z[:, ::10, :][:, ::-1, :]

[max-sixty]

@dcherian thanks a lot for looking into this. @jesieleo thanks for your patience in us finding the bug.

[jesieleo]

First of all, thank you for your help @dcherian @max-sixty . I finally reproduced this problem.
I put the ds.isel(time=[0], isoBaricInhPa=1).z[:, ::10, :][:, ::-1, :] to
ds.isel(time=[0], isoBaricInhPa=1).z[:, ::10, ::10][:, ::-1, :] can make a minimal example.

import numpy as np
import xarray as xr
ds = xr.Dataset(
    {
        "z": (
            ("time", "isoBaricInhPa", "latitude", "longitude"),
            np.ones((1, 5, 721, 1440)),
        )
    },
    coords={"latitude": np.linspace(-90, 90, 721)},
).to_netcdf("test.nc")
ds = xr.open_dataset('test.nc')
a = ds.isel(time=[0], isoBaricInhPa=1).z[:, ::10, ::10][:, ::-1, :]
print(a.sizes)
print(a.shape)
print(a)

Frozen({'time': 1, 'latitude': 73, 'longitude': 144})
(1, 73, 144)
<xarray.DataArray 'z' (time: 1, latitude: 72, longitude: 144)>
array([[[1., 1., ..., 1., 1.],
        [1., 1., ..., 1., 1.],
        ...,
        [1., 1., ..., 1., 1.],
        [1., 1., ..., 1., 1.]]])
Coordinates:
  * latitude  (latitude) float64 90.0 87.5 85.0 82.5 ... -82.5 -85.0 -87.5 -90.0
Dimensions without coordinates: time, longitude