Installations from PyPI tarballs fail #308

ulupo · 2020-02-20T09:07:13Z

Description

I believe that all installs from tarballs on PyPI currently fail. The reason seems to be that we do not package the CMakeLists.txt file needed by CMake.

Steps/Code to Reproduce

Create a new environment in an OS not supported by current wheels. E.g., for giotto-tda-nightly, wheels only support macOS 10.14 or above following updates in the Azure VM images https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops, so one can test this on macOS 10.13.
Run pip install giotto-tda-nightly to download the package from PyPI.
pip will skip the wheels as no supported wheels will be found, and will attempt to install from tarball.

Expected Results

Package is installed as if it were being installed from source (pip install . from the root of a cloned repo).

Actual Results

On my macOS 10.13 machine and running pip install giotto-tda-nightly: https://gist.github.com/ulupo/8bdf6f97a55cf86688a62e9728348409

The text was updated successfully, but these errors were encountered:

rth · 2020-02-20T11:16:14Z

There seem to be another issue with nightly wheels: I get a segfault when running tests in continuumio/miniconda3 container,


# needed to allow running GDB inside docker https://stackoverflow.com/a/46676907/1791279
$  docker run --rm -it --cap-add=SYS_PTRACE --security-opt seccomp=unconfined continuumio/miniconda3 /bin/bash

apt update
apt install gdb

conda create -n gtda-env python=3.7
conda activate gtda-env 
pip install giotto-tda-nightly

pip install hypothesis pytest

For instance,

$ python -c "import gtda.externals.python" 
Illegal instruction (core dumped)

and with a full backtrace,

$ gdb run --args python -c "import gtda.externals.python"
[...]
Reading symbols from python...done.                                                                                                    
(gdb) r                                                                                                                                
Starting program: /opt/conda/envs/gtda-env/bin/python -c import\ gtda.externals.python                                                 
[Thread debugging using libthread_db enabled]                                                                                          
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".                                                             
[New Thread 0x7ffff5178700 (LWP 596)]                                                                                                  
[New Thread 0x7ffff4977700 (LWP 597)]                                                                                                  
...                                                                                               
                                                                                                                                       
Thread 1 "python" received signal SIGILL, Illegal instruction. 
0x00007fffad9188b0 in ?? () from /opt/conda/envs/gtda-env/lib/python3.7/site-packages/giotto_ripser.cpython-37m-x86_64-linux-gnu.so    
(gdb) bt
#0  0x00007fffad9188b0 in ?? ()
   from /opt/conda/envs/gtda-env/lib/python3.7/site-packages/giotto_ripser.cpython-37m-x86_64-linux-gnu.so
#1  0x00007fffad92bac0 in ?? ()
   from /opt/conda/envs/gtda-env/lib/python3.7/site-packages/giotto_ripser.cpython-37m-x86_64-linux-gnu.so
#2  0x00007fffad92ce7e in PyInit_giotto_ripser ()
   from /opt/conda/envs/gtda-env/lib/python3.7/site-packages/giotto_ripser.cpython-37m-x86_64-linux-gnu.so
#3  0x0000555555773695 in _PyImport_LoadDynamicModuleWithSpec (spec=0x7fffae9a7e10, fp=<optimized out>)
    at /tmp/build/80754af9/python_1578510683607/work/Python/importdl.c:159
#4  0x00005555557738a9 in _imp_create_dynamic_impl.isra.15 (file=0x0, spec=0x7fffae9a7e10)
    at /tmp/build/80754af9/python_1578510683607/work/Python/import.c:2170
#5  _imp_create_dynamic (module=<optimized out>, args=<optimized out>, nargs=<optimized out>)
    at /tmp/build/80754af9/python_1578510683607/work/Python/clinic/import.c.h:289
#6  0x000055555568cab2 in _PyMethodDef_RawFastCallDict (method=0x55555586f800 <imp_methods+320>, self=0x7ffff7a32cb0,
    args=0x7ffff78c31a8, nargs=1, kwargs=<optimized out>) at /tmp/build/80754af9/python_1578510683607/work/Objects/call.c:530
#7  0x000055555568cbd1 in _PyCFunction_FastCallDict (func=0x7ffff7a4b7d0, args=<optimized out>, nargs=<optimized out>,
    kwargs=<optimized out>) at /tmp/build/80754af9/python_1578510683607/work/Objects/call.c:586
#8  0x000055555571a839 in do_call_core (kwdict=0x7fffb8e58c80, callargs=0x7ffff78c3190, func=0x7ffff7a4b7d0)
    at /tmp/build/80754af9/python_1578510683607/work/Python/ceval.c:4641
#9  _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /tmp/build/80754af9/python_1578510683607/work/Python/ceval.c:3191
#10 0x000055555566e6f9 in _PyEval_EvalCodeWithName (_co=0x7ffff7a3a150, globals=<optimized out>, locals=<optimized out>,
    args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x7ffff7850ea0, kwcount=0, kwstep=1, defs=0x0, defcount=0,
    kwdefs=0x0, closure=0x0, name=0x7ffff7a38300, qualname=0x7ffff7a38300)
    at /tmp/build/80754af9/python_1578510683607/work/Python/ceval.c:3930

which is a Ubuntu based system,

$ uname -a
Linux d614d763e5fb 5.0.0-38-generic #41-Ubuntu SMP Tue Dec 3 00:27:35 UTC 2019 x86_64 GNU/Linux

Not much ideas what could be wrong so far.

ulupo · 2020-02-20T12:51:43Z

@rth, thanks for the report!

rth · 2020-02-20T12:56:57Z

For the error with .tar.gz I can confirm I get the same error on Linux with,

pip install https://files.pythonhosted.org/packages/be/b3/cbb2d3bf866f8f1cf820dbe32bfc69cf5bdb0265c26e875670f2bd20be3f/giotto-tda-nightly-20200214.9.tar.gz

which yields,

  fatal: not a git repository (or any of the parent directories): .git                                                       [185/1905]
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-req-build-3_74n4r_/setup.py", line 178, in <module>
      cmdclass=dict(build_ext=CMakeBuild))
    File "/opt/conda/envs/gtda-env/lib/python3.7/site-packages/setuptools/__init__.py", line 144, in setup
      return distutils.core.setup(**attrs)
    File "/opt/conda/envs/gtda-env/lib/python3.7/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/opt/conda/envs/gtda-env/lib/python3.7/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/opt/conda/envs/gtda-env/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/conda/envs/gtda-env/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 223, in run
      self.run_command('build')
    File "/opt/conda/envs/gtda-env/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/conda/envs/gtda-env/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/conda/envs/gtda-env/lib/python3.7/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/opt/conda/envs/gtda-env/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/conda/envs/gtda-env/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/tmp/pip-req-build-3_74n4r_/setup.py", line 94, in run
      self.install_dependencies()
    File "/tmp/pip-req-build-3_74n4r_/setup.py", line 129, in install_dependencies
      '--init', '--recursive'])
    File "/opt/conda/envs/gtda-env/lib/python3.7/subprocess.py", line 363, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['git', 'submodule', 'update', '--init', '--recursive']' returned non-zero exit status 128.

The issue is in this line

giotto-tda/setup.py

Line 119 in 6b0411b

subprocess.check_call(['git', 'submodule', 'update',

which assumes that gtda is a git repository, which is not the case in a .tar.gz.

ulupo · 2020-02-20T13:00:42Z

@rth thanks for diagnosing this and for opening #314! It seems that in the current approach we should simply not publish the tarballs at all as they are guaranteed to fail.

madarax64 · 2020-03-12T18:19:43Z

Hello,
So is there a workaround for this issue at the moment? For me, install via pip(3) fails with the same error regarding git, and when I try to install it directly as a wheel by saying
pip3 install https://github.com/giotto-ai/giotto-tda/releases/download/v0.1.4/giotto_tda-0.1.4-cp36-cp36m-manylinux2010_x86_64.whl
I get an error saying that xxxxx is not a supported wheel on this platform. My Python version is 3.6.8 and my pip3 version is 9.0.1

ulupo · 2020-03-12T18:24:58Z

Thanks for the report @madarax64! The message seems to suggest you are running on a potentially exotic (for giotto-tda at least) Linux version. Could you tell us what that is? Notice that the meaning of manylinux2010 is explained in https://www.python.org/dev/peps/pep-0571/.

Finally, what you mean by "fails with the same error regarding git"? Which error exactly?

madarax64 · 2020-03-12T18:31:04Z

Hey @ulupo
Weird - I'm on Ubuntu 18.04.

This one:
subprocess.CalledProcessError: Command '['git', 'submodule', 'update', '--init', '--recursive']' returned non-zero exit status 128.

ulupo · 2020-03-12T18:33:52Z

Ubuntu 18.04 should definitely be supported by the manylinux wheels. So just to be clear, what does a run of

pip install giotto-tda

on a fresh environment, not in the repository root, yield?

madarax64 · 2020-03-12T18:35:16Z

So I upgraded my pip3, and then simply did:
pip3 install giotto-tda
And that worked off the bat. But when I actually run code, I get an error reading:
Illegal Instruction (core dumped)

Running with gdb like above gives the following error:
Thread 1 "python3" received signal SIGILL, Illegal instruction. 0x00007fffd3b24870 in ?? () from /usr/local/lib/python3.6/dist-packages/giotto_ripser.cpython-36m-x86_64-linux-gnu.so

ulupo · 2020-03-12T18:39:17Z

But when I actually run code, I get an error reading:
Illegal Instruction (core dumped)

Could you give us the code you ran, to reproduce?

ulupo · 2020-03-12T18:40:33Z

Running with gdb like above gives the following error:
Thread 1 "python3" received signal SIGILL, Illegal instruction. 0x00007fffd3b24870 in ?? () from /usr/local/lib/python3.6/dist-packages/giotto_ripser.cpython-36m-x86_64-linux-gnu.so

So this is installing from a clone of the repository? By wich I mean

git clone https://github.com/giotto-ai/giotto-tda
pip install -e .

The error looks like an esoteric C++-related error, which @MonkeyBreaker may be in a better position to diagnose.

madarax64 · 2020-03-12T18:55:55Z

I'm not working in Anaconda or a Virtualenv, so this is directly on my machine. Running pip3 install giotto-tda in say, an empty folder works perfectly without any errors, after I upgraded my pip3 to version 20.

To reproduce the error, you can run the following:
python3 -c "import gtda.homology as hl"

No, I never installed from the clone of the repository. This came from the version I got from pip after I upgraded pip.

MonkeyBreaker · 2020-03-13T07:03:46Z

Hi @madarax64,

Sorry that you encounter Illegal Instruction (core dumped), this was fixed yesterday in PR #360.
This was caused by illegal instructions running on the CPU, in this case some AVX instructions that your CPU does not support.

I don't know when will the wheels integrate this change, but I think that giotto-tda-nightly should now integrate them, or am I wrong @ulupo ?

ulupo · 2020-03-13T07:11:24Z

@MonkeyBreaker @madarax64, I don't think a nightly with these changes has been pushed yet. But the new stable version 0.2.0 which will include these fixes will be out today or Monday.

madarax64 · 2020-03-13T08:21:12Z

Hi @ulupo @MonkeyBreaker
Thanks, I look forward to it. I really appreciate it.

ulupo added the bug Something isn't working label Feb 20, 2020

ulupo assigned rth Feb 20, 2020

ulupo added this to the 0.2.0 milestone Feb 20, 2020

rth mentioned this issue Feb 20, 2020

Run auditwheel repair for manylinux2010 wheels #312

Merged

rth mentioned this issue Feb 20, 2020

pybind11 and cmake as build dependencies in pyproject.toml #314

Open

ulupo mentioned this issue Feb 20, 2020

Remove packaging of source distribution #317

Merged

ulupo mentioned this issue Sep 27, 2021

Issue installing giotto-tda (MacOS Mojave V.10.14.6) #612

Closed

anmolguppta mentioned this issue Jan 24, 2022

Installing on Apple Silicon #617

Closed

ulupo mentioned this issue Feb 27, 2022

Pip installation from PyPI fails on Python 3.10 #621

Closed

ulupo mentioned this issue Apr 8, 2022

[BUG] Installation error on Windows 10 (user) #629

Closed

ulupo mentioned this issue Aug 15, 2022

Cannot install Giotto with pip on MAC OS #641

Closed

jGaboardi mentioned this issue Jan 4, 2024

Failing to install v 0.6.0 with pypi tarball pysal/spopt#435

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Installations from PyPI tarballs fail #308

Installations from PyPI tarballs fail #308

ulupo commented Feb 20, 2020 •

edited

Loading

rth commented Feb 20, 2020

ulupo commented Feb 20, 2020

rth commented Feb 20, 2020

ulupo commented Feb 20, 2020 •

edited

Loading

madarax64 commented Mar 12, 2020

ulupo commented Mar 12, 2020 •

edited

Loading

madarax64 commented Mar 12, 2020

ulupo commented Mar 12, 2020

madarax64 commented Mar 12, 2020 •

edited

Loading

ulupo commented Mar 12, 2020

ulupo commented Mar 12, 2020 •

edited

Loading

madarax64 commented Mar 12, 2020

MonkeyBreaker commented Mar 13, 2020 •

edited

Loading

ulupo commented Mar 13, 2020 •

edited

Loading

madarax64 commented Mar 13, 2020

Installations from PyPI tarballs fail #308

Installations from PyPI tarballs fail #308

Comments

ulupo commented Feb 20, 2020 • edited Loading

Description

Steps/Code to Reproduce

Expected Results

Actual Results

rth commented Feb 20, 2020

ulupo commented Feb 20, 2020

rth commented Feb 20, 2020

ulupo commented Feb 20, 2020 • edited Loading

madarax64 commented Mar 12, 2020

ulupo commented Mar 12, 2020 • edited Loading

madarax64 commented Mar 12, 2020

ulupo commented Mar 12, 2020

madarax64 commented Mar 12, 2020 • edited Loading

ulupo commented Mar 12, 2020

ulupo commented Mar 12, 2020 • edited Loading

madarax64 commented Mar 12, 2020

MonkeyBreaker commented Mar 13, 2020 • edited Loading

ulupo commented Mar 13, 2020 • edited Loading

madarax64 commented Mar 13, 2020

ulupo commented Feb 20, 2020 •

edited

Loading

ulupo commented Feb 20, 2020 •

edited

Loading

ulupo commented Mar 12, 2020 •

edited

Loading

madarax64 commented Mar 12, 2020 •

edited

Loading

ulupo commented Mar 12, 2020 •

edited

Loading

MonkeyBreaker commented Mar 13, 2020 •

edited

Loading

ulupo commented Mar 13, 2020 •

edited

Loading