🐛 NetCDF4 RuntimeWarning if xarray is imported before netCDF4 #7259
Comments
s-weigand
added
bug
needs triage
|
It is true that the import behavior had changed in the linked PR (it is now lazy importing netCDF4) but I don't see how this can cause such problems. Normally this warning indicates that something is wrong with your install, but since you can solve it by importing netcdf4 first, I have no idea what is going wrong ... |
|
We were/are also entirely baffled and have no clue what kind of black magic is going is going on with this 😅 |
|
I have run into this problem already several times and the problem was always a somehow broken numpy install. A reinstall usually fixed this for me. |
|
Yeah we first saw it on the CI, which moved it from
😅 |
|
Is it repeatable locally with the exact same package versions installed? |
|
It is also reproducible locally only |
|
It is also reproducible on binder: It seems that the binder uses conda-forge, which is why i'm commenting here. It is really strange in the sense that xarray doesn't compile anything. https://github.com/conda-forge/xarray-feedstock/blob/main/recipe/meta.yaml#L16 So it must be something that gets lazy loaded that triggers things. |
|
Can we find a minimum environment that can reproduce this results? I.e. only numpy, pandas, xarray and netcdf4? |
`mamba list` |
|
I think it is a numpy thing mamba create --name np numpy netcdf4 --channel conda-forge --override-channels
conda activate np
python -c "import numpy; import warnings; warnings.filterwarnings('error'); import netCDF4"Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/mark/mambaforge/envs/np/lib/python3.11/site-packages/netCDF4/__init__.py", line 3, in <module>
from ._netCDF4 import *
File "src/netCDF4/_netCDF4.pyx", line 1, in init netCDF4._netCDF4
RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility. Expected 16 from C header, got 96 from PyObject |
|
So it happens when you import numpy before netCDF? Then I would claim that it is a netCDF problem, haha. |
|
one or the other. |
|
If you are still motivated, you could try figuring out if it happens with older versions of numpy and netcdf as well... |
|
I'm really not sure. It seems to happen with a large swath of versions from my recent search. Also running from the python REPL, i don't see the warning. which makes me feel like numpy/cython/netcdf4 are trying to suppress the harmless warning. https://github.com/cython/cython/blob/0.29.x/Cython/Utility/ImportExport.c#L365 |
|
I would suggest to open an issue in netcdf4, since it seems to happen when you import numpy before netcdf4. |
headtr1ck
added
upstream issue
and removed
needs triage
|
weirdly enough, this: |
Xarray > 2022.10.0 was causing warnings on numpy.ndarray in the test suite and appears to be related to: pydata/xarray#7259 To eliminate the warnings I had to add an empty netCDF4 import to `data/tests/conftest.py`.
|
➕ 1 No errors when the order is |
problem si described in pydata/xarray#7259
* first pieces of code and data for cml path plotting * let mypy ignore missing import types, also unpinned xarray * first implementation of plotting of CML paths * had to add hack to solve problem with netcdf4 and numpy compatibility, problem is described in pydata/xarray#7259 * drop Python 3.12 in CI * drop 3.12 from poetry dependency, which hopefully also limits pypi builds to 3.10 and 3.11 * add info regarding codecov setup * add jupyterlab as dev devependency to be able to run notebooks from poetry env * exclude notebooks from codespell checks * add simple example notebook * add notebook to docs
similar issue to: pydata/xarray#7259
|
Is an MCVE available where the warning shows when xarray is installed, but not when it's not? That would isolate whether this in an xarray issue. |
max-sixty
added
needs mcve
|
Closing to limit the number of open issues without MCVEs, please feel free to reopen with an MCVE |
What happened?
Yesterday we got a dependabot update PR to upgrade
xarrayfrom2022.10.0to2022.11.0and a test where we check for our own deprecation warnings failed because there was an additional unexpected warning.After some debugging we found that the warning was caused by calling
xarray.Dataset.to_netcdffor the first time in our test suite, but did not trigger when calling it again.After a lot of head-scratching and confusion, we found that it is an import order problem that can be solved by importing netCDF4 before importing xarray (we didn't import netCDF4 at all in our code).
What did you expect to happen?
No RuntimeWarning from
netCDF4Minimal Complete Verifiable Example
MVCE confirmation
Relevant log output
Anything else we need to know?
The problem can be reproduced by running
python -c "import xarray;import warnings;warnings.filterwarnings('error');import netCDF4"which throws the error
When importing netCDF4 first all runs as expected
python -c "import netCDF4;import xarray;import warnings;warnings.filterwarnings('error');import netCDF4"A git bisect shows that the first commit with this problem was f32d354
Environment
INSTALLED VERSIONS
commit: None
python: 3.10.6 | packaged by conda-forge | (main, Oct 7 2022, 20:14:50) [MSC v.1916 64 bit (AMD64)]
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: AMD64 Family 23 Model 8 Stepping 2, AuthenticAMD
byteorder: little
LC_ALL: None
LANG: de_DE.UTF-8
LOCALE: ('de_DE', 'cp1252')
libhdf5: 1.12.1
libnetcdf: 4.8.1
xarray: 2022.11.0
pandas: 1.5.1
numpy: 1.23.4
scipy: 1.9.3
netCDF4: 1.6.1
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.6.2
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: None
distributed: None
matplotlib: 3.6.1
cartopy: None
seaborn: None
numbagg: None
fsspec: None
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 59.8.0
pip: 22.2.2
conda: None
pytest: 7.1.3
IPython: 8.5.0
sphinx: 5.2.3
None
The text was updated successfully, but these errors were encountered: