BUG: Combining SciPy 1.14.1 and statsmodels 0.14.2 on OSX causes "symbol not found in flat namespace '_npy_cabs'" #9333
Comments
|
What happens if you downgrade to NumPy 2.0.x?
…On Thu, Aug 22, 2024, 19:18 Nick ODell ***@***.***> wrote:
Describe the bug
When importing statsmodels.api, the following error appears:
Traceback (most recent call last):
File "/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/test.py", line 1, in <module>
import statsmodels.api as sm
File "/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels/api.py", line 136, in <module>
from .regression.recursive_ls import RecursiveLS
File "/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels/regression/recursive_ls.py", line 14, in <module>
from statsmodels.tsa.statespace.mlemodel import (
File "/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels/tsa/statespace/mlemodel.py", line 33, in <module>
from .simulation_smoother import SimulationSmoother
File "/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels/tsa/statespace/simulation_smoother.py", line 11, in <module>
from .kalman_smoother import KalmanSmoother
File "/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels/tsa/statespace/kalman_smoother.py", line 11, in <module>
from statsmodels.tsa.statespace.representation import OptionWrapper
File "/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels/tsa/statespace/representation.py", line 10, in <module>
from .tools import (
File "/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels/tsa/statespace/tools.py", line 14, in <module>
from . import (_initialization, _representation, _kalman_filter,
File "statsmodels/tsa/statespace/_initialization.pyx", line 1, in init statsmodels.tsa.statespace._initialization
ImportError: dlopen(/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels/tsa/statespace/_representation.cpython-312-darwin.so, 0x0002): symbol not found in flat namespace '_npy_cabs'
This issue was originally reported on StackOverflow
<https://stackoverflow.com/questions/78900274/scipy-1-14-1-breaks-statsmodels-0-14-2>.
I have independently reproduced the failure on OSX, but failed to reproduce
it on Linux.
Curiously, this issue only shows up when SciPy 1.14.1 is installed. If
SciPy 1.14.0 is installed, this error does not show up. Similarly, if
statsmodels 0.14.1 is installed, the issue also goes away. Strangely,
downgrading from statsmodels 0.14.2 to 0.14.1 and upgrading back to 0.14.2
also fixes the issue.
I'm unsure if this is a bug in statsmodels or SciPy - it seems very
strange that a change in another package could cause this kind of error.
Code Sample, a copy-pastable example if possible
import statsmodels.api as sm
If you are using a released version, have you verified that the bug exists
in the main branch of this repository? It helps the limited resources if we
know problems exist in the current main branch so that they do not need to
check whether the code sample produces a bug in the next release.
I was unable to verify whether this issue has been fixed in main, because
building the package from source also makes the issue go away.
For example, if I install scipy and statsmodels like this:
pip install scipy==1.14.1 ***@***.***
the error goes away, despite being the same version that breaks when I use
pre-built binaries.
Expected Output
I expected that the latest version of SciPy and statsmodels could be used
together.
Output of import statsmodels.api as sm; sm.show_versions()
This is the output of import statsmodels.api as sm; sm.show_versions()
for the working version of SciPy. After installing SciPy 1.14.1, import
statsmodels.api as sm no longer works.
import statsmodels.api as sm; sm.show_versions()
INSTALLED VERSIONS
Python: 3.12.5.final.0
OS: Darwin 23.4.0 Darwin Kernel Version 23.4.0: Wed Feb 21 21:44:54 PST
2024; root:xnu-10063.101.15~2/RELEASE_ARM64_T6030 arm64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
statsmodels
Installed: 0.14.2
(/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels)
Required Dependencies
cython: Not installed
numpy: 2.1.0
(/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/numpy)
scipy: 1.14.0
(/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/scipy)
pandas: 2.2.2
(/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/pandas)
dateutil: 2.9.0.post0
(/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/dateutil)
patsy: 0.5.6
(/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/patsy)
Optional Dependencies
matplotlib: Not installed
cvxopt: Not installed
joblib: Not installed
Developer Tools
IPython: Not installed
jinja2: Not installed
sphinx: Not installed
pygments: Not installed
pytest: Not installed
virtualenv: Not installed
Additional details:
x-ref scipy/scipy#21434 <scipy/scipy#21434>
—
Reply to this email directly, view it on GitHub
<#9333>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABKTSRKGBYOFVWDEFQ3XME3ZSYTQHAVCNFSM6AAAAABM6VWXQKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGQ4DCNBUGI4TQNI>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
|
Should have nothing to do with SciPy, but perhaps SciPy 1.14 is bumping NumPy to 2.1.0, while SciPy 1.13 is happy with NumPy 2.0.x. |
Curiously, the configuration
According to Working: Failing: |
|
I've done a test of all combinations of SciPy 1.14.0/1.14.1, NumPy 2.0.1/2.1.0, and statsmodels 0.14.1/0.14.2. The result is that the only failing configuration is latest SciPy plus latest NumPy plus latest statsmodels.
Everything else either works, or pip refuses to install that combination of packages. |
|
Sounds like the both used to export the symbol but have both stopped.
…On Thu, Aug 22, 2024, 20:01 Nick ODell ***@***.***> wrote:
I've done a test of all combinations of SciPy 1.14.0/1.14.1, NumPy
2.0.1/2.1.0, and statsmodels 0.14.1/0.14.2.
The result is that the only failing configuration is latest SciPy plus
latest NumPy plus latest statsmodels.
scipy Version statsmodels Version numpy Version Compatibility
1.14.0 0.14.1 2.0.1 incompatible (statsmodels requires numpy < 2)
1.14.0 0.14.1 2.1.0 incompatible (statsmodels requires numpy < 2)
1.14.0 0.14.1 1.26.4 works
1.14.0 0.14.2 2.0.1 works
1.14.0 0.14.2 2.1.0 works
1.14.1 0.14.1 2.0.1 incompatible (statsmodels requires numpy < 2)
1.14.1 0.14.1 2.1.0 incompatible (statsmodels requires numpy < 2)
1.14.1 0.14.1 1.26.4 works
1.14.1 0.14.2 2.0.1 works
1.14.1 0.14.2 2.1.0 fails
Everything else either works, or pip refuses to install that combination
of packages.
—
Reply to this email directly, view it on GitHub
<#9333 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABKTSRMFUBBZUVROO5YV5SLZSYYPTAVCNFSM6AAAAABM6VWXQKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBVGQZTQNRQGU>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
You're right. It took me a bit of googling to work out how to view the symbol tables, but NumPy 2.1.0 no longer exports this symbol, and SciPy 1.14.1 no longer exports it either. SciPy used to export the symbol via |
|
We can't reproduce on CI: https://dev.azure.com/statsmodels/statsmodels-testing/_build/results?buildId=5869&view=logs&j=ff02c0bf-a6da-574b-cd48-e6a8a9d1f05d I wonder if it matters that the CI one is built against scipy 1.14.1 - not 100% but could imagine that since SciPy doesn't have the symbol anymore, it is finding it from NumPy and so is happy. The release 0.14.2 was built against SciPy 1.13 and probably expects to find the symbol there. At least this is my working hypothesis. |
I just tried running with a Output: MAC:
|
|
We have a similar issue in our CI, see this run for example. Numpy 2.1.1, SciPy 1.14.1 and statsmodels 0.14.2. Also happening only on macOS. |
|
Also on macOS. this run. Numpy 2.1.0, SciPy 1.14.1 and statsmodels 0.14.2. |
|
I am having the same problem while using any combinations of python==3.10.15, scipy==1.14.0/1.14.1/1.14.2, numpy==2.0.1/2.1.1 and statsmodels==0.14.2 |
|
There seems to be a decent bug here that isn't easy to fix. It appears in the 0.14.2 code even if built against SciPy 1.14.1. |
Simplify setup since NumPy wikk always be installed when using pip install. Standardize library usage xref statsmodels#9333
|
The source of the issue has been found. It is due to some changes in the arch used by the GitHub MaxOS runners, which were previously x86_64 and have moved to arm64. This means that the x86_64 wheels were not correctly linked. The fix is in place, although there are a number of bugs due to changes in other libraries (probably pandas) that need to be addressed before we can release 0.14.3. https://github.com/MacPython/statsmodels-wheels/actions/runs/10844514094 |
|
Appears to be fixed now. Will need a release note and then likely to release early next week. https://github.com/MacPython/statsmodels-wheels/actions/runs/10848683287 |
|
Wheels are on PyPi (excluding a couple that are slow building). Closing. Please let us know if there continues to be a problem. |
|
I can confirm statsmodels 0.14.3 resolves this bug for us, and our CI now passes again! Thanks a lot for the time and effort to crack this complex bug! |
|
Hi, original reporter here, I can confirm the issue is fixed on my computer with statmodels 0.14.3. |
Describe the bug
When importing
statsmodels.api, the following error appears:This issue was originally reported on StackOverflow. I have independently reproduced the failure on OSX, but failed to reproduce it on Linux.
Curiously, this issue only shows up when SciPy 1.14.1 is installed. If SciPy 1.14.0 is installed, this error does not show up. Similarly, if statsmodels 0.14.1 is installed, the issue also goes away. Strangely, downgrading from statsmodels 0.14.2 to 0.14.1 and upgrading back to 0.14.2 also fixes the issue.
I'm unsure if this is a bug in statsmodels or SciPy - it seems very strange that a change in another package could cause this kind of error.
Code Sample, a copy-pastable example if possible
I was unable to verify whether this issue has been fixed in main, because building the package from source also makes the issue go away.
For example, if I install scipy and statsmodels like this:
the error goes away, despite being the same version that breaks when I use pre-built binaries.
Expected Output
I expected that the latest version of SciPy and statsmodels could be used together.
Output of
import statsmodels.api as sm; sm.show_versions()This is the output of
import statsmodels.api as sm; sm.show_versions()for the working version of SciPy. After installing SciPy 1.14.1,import statsmodels.api as smno longer works.INSTALLED VERSIONS
Python: 3.12.5.final.0
OS: Darwin 23.4.0 Darwin Kernel Version 23.4.0: Wed Feb 21 21:44:54 PST 2024; root:xnu-10063.101.15~2/RELEASE_ARM64_T6030 arm64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
statsmodels
Installed: 0.14.2 (/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/statsmodels)
Required Dependencies
cython: Not installed
numpy: 2.1.0 (/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/numpy)
scipy: 1.14.0 (/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/scipy)
pandas: 2.2.2 (/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/pandas)
dateutil: 2.9.0.post0 (/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/dateutil)
patsy: 0.5.6 (/private/var/folders/7v/pg90myg55bs218ff8dpkrfrc0000gn/T/tmp.8CjfzJEYnh/venv/lib/python3.12/site-packages/patsy)
Optional Dependencies
matplotlib: Not installed
cvxopt: Not installed
joblib: Not installed
Developer Tools
IPython: Not installed
jinja2: Not installed
sphinx: Not installed
pygments: Not installed
pytest: Not installed
virtualenv: Not installed
Additional details:
x-ref scipy/scipy#21434
Original report from StackOverflow
The text was updated successfully, but these errors were encountered: