Numpy version incompatibility

[Arkangus]

Code Sample, a copy-pastable example if possible

Python 3.6.7 (v3.6.7:6ec5cf24b7, Oct 20 2018, 13:35:33) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\SoftInstall\Python\lib\site-packages\pandas\__init__.py", line 31, in <module>
    from pandas._libs import hashtable as _hashtable, lib as _lib, tslib as _tslib
  File "D:\SoftInstall\Python\lib\site-packages\pandas\_libs\__init__.py", line 3, in <module>
    from .tslibs import (
  File "D:\SoftInstall\Python\lib\site-packages\pandas\_libs\tslibs\__init__.py", line 3, in <module>
    from .conversion import localize_pydatetime, normalize_date
  File "__init__.pxd", line 918, in init pandas._libs.tslibs.conversion
ValueError: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216 from C header, got 192 from PyObject

Problem description

Although Pandas 1.0.1 has a minimum requirement for numpy of 1.13.3 (see release notes of Pandas 1.0.0), it fails with numpy 1.15.4 (Python 3.6.7).

Problem is solved by upgrading numpy (version 1.18.1 works).

[TomAugspurger]

You'll need to provide more information on how you installed NumPy and pandas.

[Arkangus]

python -m pip install pandas
python -m pip install numpy

I was previously using pandas 0.25.1 I believe, along with numpy 1.15.4

The issue appeared when upgrading to pandas with python -m pip install pandas --upgrade.
It dissapeared when upgrading numpy to the latest version with python -m pip install numpy --upgrade

[TomAugspurger]

Did you build from source or install wheels? Ideally you could provide the full inputs and outputs of a terminal session.

[jbrockmendel]

I'm pretty sure ive seen this on either the cython tracker or mailing list

[Arkangus]

Hi again,

I believe this is what you're asking for:

D:\>python
Python 3.6.7 (v3.6.7:6ec5cf24b7, Oct 20 2018, 13:35:33) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> pd.__version__
'1.0.1'
>>> import numpy as np
>>> np.__version__
'1.18.1'
>>> exit()
D:\>python -m pip install numpy==1.15.4
Collecting numpy==1.15.4
  Using cached numpy-1.15.4-cp36-none-win_amd64.whl (13.5 MB)
ERROR: pygem 1.1 has requirement Sphinx==1.4, but you'll have sphinx 2.2.1 which is incompatible.
Installing collected packages: numpy
  Attempting uninstall: numpy
    Found existing installation: numpy 1.18.1
    Uninstalling numpy-1.18.1:
      Successfully uninstalled numpy-1.18.1
Successfully installed numpy-1.15.4

D:\>python
Python 3.6.7 (v3.6.7:6ec5cf24b7, Oct 20 2018, 13:35:33) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\SoftInstall\Python\lib\site-packages\pandas\__init__.py", line 31, in <module>
    from pandas._libs import hashtable as _hashtable, lib as _lib, tslib as _tslib
  File "D:\SoftInstall\Python\lib\site-packages\pandas\_libs\__init__.py", line 3, in <module>
    from .tslibs import (
  File "D:\SoftInstall\Python\lib\site-packages\pandas\_libs\tslibs\__init__.py", line 3, in <module>
    from .conversion import localize_pydatetime, normalize_date
  File "__init__.pxd", line 918, in init pandas._libs.tslibs.conversion
ValueError: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216 from C header, got 192 from PyObject
>>>

Also, seems like pandas was installed from wheels.

D:\>python -m pip install pandas==0.25.3
Collecting pandas==0.25.3
  Using cached pandas-0.25.3-cp36-cp36m-win_amd64.whl (9.0 MB)
Requirement already satisfied: pytz>=2017.2 in d:\softinstall\python\lib\site-packages (from pandas==0.25.3) (2018.7)
Requirement already satisfied: python-dateutil>=2.6.1 in d:\softinstall\python\lib\site-packages (from pandas==0.25.3) (2.7.5)
Requirement already satisfied: numpy>=1.13.3 in d:\softinstall\python\lib\site-packages (from pandas==0.25.3) (1.18.1)
Requirement already satisfied: six>=1.5 in d:\softinstall\python\lib\site-packages (from python-dateutil>=2.6.1->pandas==0.25.3) (1.12.0)
Installing collected packages: pandas
  Attempting uninstall: pandas
    Found existing installation: pandas 1.0.1
    Uninstalling pandas-1.0.1:
      Successfully uninstalled pandas-1.0.1
Successfully installed pandas-0.25.3

D:\>python -m pip install pandas --upgrade
Collecting pandas
  Using cached pandas-1.0.1-cp36-cp36m-win_amd64.whl (8.8 MB)
Requirement already satisfied, skipping upgrade: pytz>=2017.2 in d:\softinstall\python\lib\site-packages (from pandas) (2018.7)
Requirement already satisfied, skipping upgrade: python-dateutil>=2.6.1 in d:\softinstall\python\lib\site-packages (from pandas) (2.7.5)
Requirement already satisfied, skipping upgrade: numpy>=1.13.3 in d:\softinstall\python\lib\site-packages (from pandas) (1.18.1)
Requirement already satisfied, skipping upgrade: six>=1.5 in d:\softinstall\python\lib\site-packages (from python-dateutil>=2.6.1->pandas) (1.12.0)
Installing collected packages: pandas
  Attempting uninstall: pandas
    Found existing installation: pandas 0.25.3
    Uninstalling pandas-0.25.3:
      Successfully uninstalled pandas-0.25.3
Successfully installed pandas-1.0.1

[TomAugspurger]

I might naively expect the following to work

pip install pandas numpy==1.18.1
pip install numpy==1.15.4
python -c import pandas

assuming that pandas was built against NumPy 1.13.3 as specified in https://github.com/pandas-dev/pandas/blob/master/pyproject.toml#L8-L9 (for Python 3.6)

@cgohlke do you know what version of NumPy pandas 1.0.1 was compiled with?

[cgohlke]

what version of NumPy pandas 1.0.1 was compiled with?

1.14.6

[cgohlke]

I can reproduce the error. Numpy >= 1.16 work.
Strange. If I rebuild pandas 1.0.1 today from source with the same build script, I can import pandas with numpy 1.14.6 as expected...

[jbrockmendel]

@seberg is this a resolved issue?

[seberg]

I think this is a sanity check in cython, which is being a bit overzealous, and probably is simply fixed in Cython (in the sense that it is now normally a warning, and NumPy actually filters the warning away).

I am not sure there is anything we can do about it, it is possible the size of the ufunc struct grew, but unless you subclass ufunc (which nobody really does), it shouldn't be a big issue. But, there may be more to it, I am not too familiar with these build issues.

[seberg]

Yeah, we now have the ctypedef class numpy.ufunc [object PyUFuncObject, check_size ignore]: "check_size ignore" flag in the .pyd, so the issue is fixed but would show up if you used a cython which did not include this when compiling, and then upgrade to a newer numpy version. Making sure you are using a newish cython will thus be sufficient.