Unable to compile a statically linked wheel #1033

mrossinek · 2020-11-11T13:21:38Z

Please apologize the length of this bug report in advance, but I am trying to provide as much information as possible.

Informations

Qiskit Aer version: master
Python version: 3.6.9
Operating system: uname -a provides:

Linux qt00 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

To provide some context: I am trying to compile Aer from source on a compute server to allow multiple users to access the same master-branch installation from a centrally managed virtualenv. While doing so I ran into the problem of not being able to produce a correctly statically linked wheel.
Specifically this means that I log in onto a login node, create a virtualenv there and expect all computations on the respective compute nodes to work normally.

I am using the following Python script for testing:

import sys
print(sys.version)
from qiskit.providers.aer import Aer
print(Aer)

and the following bash script for testing the execution on the compute nodes:

#!/bin/bash
locate libopenblas.so.0
source ~/path/to/venv/bin/activate
which python
which pip
pip list | grep qiskit
python test.py  # this is the above script

What is the current behavior?

In order to explain the problem at hand I will step through a number of different scenarios and explain the results which I have achieved with each:

A stable installation using `pip install qiskit-terra qiskit-aer`

Starting from a clean virtualenv and running the above pip installation command everything works fine.
On the login node I obtain:

3.6.9 (default, Oct  8 2020, 12:12:24)
[GCC 8.4.0]
/home/a/oss/software/qiskit-stable/venv/lib/python3.6/site-packages/qiskit/__init__.py:75: RuntimeWarning: Could not import the IBMQ provider from the qiskit-ibmq-provider package. Install qiskit-ibmq-provider or check your installation.
  RuntimeWarning)
AerProvider

And on the compute nodes:

/path/to/stable/venv/bin/python
/path/to/stable/venv/bin/pip
qiskit-aer         0.7.0
qiskit-terra       0.16.0
/home/a/oss/software/qiskit-stable/venv/lib/python3.6/site-packages/qiskit/__init__.py:75: RuntimeWarning: Could not import the IBMQ provider from the qiskit-ibmq-provider package. Install qiskit-ibmq-provider or check your installation.
  RuntimeWarning)
3.6.9 (default, Oct  8 2020, 12:12:24)
[GCC 8.4.0]
AerProvider

A master installation using `python setup.py bdist_wheel`

When trying to compile Aer manually from master everything works as expected on the login node (i.e. same as above) but on the compute node I obtain the following error:

Traceback (most recent call last):
  File "test.py", line 3, in <module>
    from qiskit.providers.aer import Aer
  File "/home/a/oss/software/qiskit-master/venv/lib/python3.6/site-packages/qiskit/providers/aer/__init__.py", line 64, in <module>
    from .aerprovider import AerProvider
  File "/home/a/oss/software/qiskit-master/venv/lib/python3.6/site-packages/qiskit/providers/aer/aerprovider.py", line 20, in <module>
    from .backends.qasm_simulator import QasmSimulator
  File "/home/a/oss/software/qiskit-master/venv/lib/python3.6/site-packages/qiskit/providers/aer/backends/__init__.py", line 17, in <module>
    from .qasm_simulator import QasmSimulator
  File "/home/a/oss/software/qiskit-master/venv/lib/python3.6/site-packages/qiskit/providers/aer/backends/qasm_simulator.py", line 25, in <module>
    from .controller_wrappers import qasm_controller_execute
ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory

This was to be expected as the first line of the bash script (locate libopenblas.so.0) does not return anything which shows that OpenBLAS cannot be found on the compute nodes (while it is available normally on the login node).
Having talked to the system administrator, we agreed that it would be best if, thus, we can compile Aer with static linking (as this appears to work with the stable installation using pip install which ships a Python wheel bundled with libopenblas).

Static linking: `python setup.py bdist_wheel -- -DSTATIC_LINKING=True`

This attempt causes even the installation on the login node to break with the following error:

Traceback (most recent call last):
  File "test.py", line 3, in <module>
    from qiskit.providers.aer import Aer
  File "/home/a/oss/software/qiskit-master/venv/lib/python3.6/site-packages/qiskit/providers/aer/__init__.py", line 64, in <module>
    from .aerprovider import AerProvider
  File "/home/a/oss/software/qiskit-master/venv/lib/python3.6/site-packages/qiskit/providers/aer/aerprovider.py", line 20, in <module>
    from .backends.qasm_simulator import QasmSimulator
  File "/home/a/oss/software/qiskit-master/venv/lib/python3.6/site-packages/qiskit/providers/aer/backends/__init__.py", line 17, in <module>
    from .qasm_simulator import QasmSimulator
  File "/home/a/oss/software/qiskit-master/venv/lib/python3.6/site-packages/qiskit/providers/aer/backends/qasm_simulator.py", line 25, in <module>
    from .controller_wrappers import qasm_controller_execute
ImportError: /home/a/oss/software/qiskit-master/venv/lib/python3.6/site-packages/qiskit/providers/aer/backends/controller_wrappers.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _gfortran_concat_string

Static linking with `FindBLAS.cmake.fix-static-linking`

I then found out about this patch: https://github.com/Qiskit/qiskit-aer/blob/master/cmake/FindBLAS.cmake.fix-static-linking
But after applying that and trying the same -DSTATIC_LINKING=True compilation as above, I still obtain the same error.

Steps to reproduce the problem

The following summarizes the steps necessary to reproduce this issue. However, your mileage may vary as I have tried to reproduce this on my Fedora laptop where I am unable to do so. Both of the last two steps do not result in the problem but I am not sure how to verify that this is because the static linking was successful or whether it falls back to dynamic linking.
The Fedora laptop specs are:

$ uname -a
Linux pulsano 5.8.15-201.fc32.x86_64 #1 SMP Thu Oct 15 15:56:44 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ python --version
Python 3.6.12  # the same works with Python 3.8.6
$ gcc --version
gcc (GCC) 9.3.0
$ cmake --version
cmake version 3.18.2  # same as on the server

mkdir tmp
cd tmp
python3 -m virtualenv venv
source venv/bin/activate
git clone https://github.com/Qiskit/qiskit-terra.git
cd qiskit-terra
pip install -c constraints.txt -r requirements.txt -r requirements-dev.txt
pip install -e .
cd ..
git clone https://github.com/Qiskit/qiskit-aer.git
cd qiskit-aer
pip install -c constraints.txt -r requirements-dev.txt
python setup.py bdist_wheel -- -DSTATIC_LINKING=True
cd ..
python
% from qiskit.providers.aer import Aer

What is the expected behavior?

I would expect to be able to compile a statically linked Python wheel of Aer

Suggested solutions

I am at a loss to be honest, which is why I came here 🙂

The text was updated successfully, but these errors were encountered:

mtreinish · 2020-11-11T15:25:51Z

We build statically linked wheels for release on pypi. We use the cibuildwheel tool to automate this in CI. But the basic workflow it uses is:

pip wheel . --no-deps
auditwheel repair qiskit_aer-*.whl

The auditwheel repair command there vendors the shared libs into the binary. Although it may only work if the system has a version of system libs (like glibc) that is compatible with a manylinux python packaging spec (which is the other thing cibuildwheel does for us, it runs everything in the manylinux docker images).

mrossinek · 2020-11-12T07:43:54Z

Thanks a lot for the information! I went ahead and tried it out and as slightly anticipated by your comment about auditwheel it complains about the versions of the system libs:

auditwheel: error: cannot repair "qiskit_aer-0.8.0-cp36-cp36m-linux_x86_64.whl" to "manylinux1_x86_64" ABI because of the presence of too-recent versioned symbols. You'll need to compile the wheel on an older toolchain.

From reading a little bit through auditwheels Readme it doesn't seem like there is anything I can do about this unless I have access to older versions of these libraries, correct?

vvilpas · 2020-11-12T10:11:52Z

Under the hood we use manylinux2010 to build the linux wheels. Not sure if you can in your system, but maybe you can follow the workflow @mtreinish described, using the manylinux2010 docker image.

mrossinek · 2020-11-12T10:49:01Z

On this compute server where I would like to compile Aer I do not have access to docker. And I would like to avoid having to compile this on a different system just to push the wheel onto the server every time I want to update it.
I checked the PEPs for manylinux1, manylinux2010 and manylinux2014 for the maximally allowed system library versions. But it appears as though none of them will support what is available on the server. The output of auditwheel --verbose repair *.whl confirms this as it shows incompatibility of all three manylinux* variants.

Is there no way of statically linking the libraries into this specific wheel? It doesn't need to support any platform but just the compute nodes which should be identical to the login node (but apparantly missing the libraries because otherwise the dynamic linking should work...)

vvilpas · 2020-11-12T11:59:39Z

Can you compile with static_linking enabled (but forget about FindBLAS.cmake.fix-static-linking) and then check for dependency on shared libraries (ldd qiskit/providers/aer/backends/controller_wrappers*.so under _skbuild/**YOUR_OS**/cmake-install/)?

mrossinek · 2020-11-12T12:12:59Z

Here is the ouptut of what you described:

(venv) oss@qt00:~/software/qiskit-master/src/qiskit-aer/_skbuild/linux-x86_64-3.6/cmake-install$ ldd qiskit/providers/aer/backends/controller_wrappers.cpython-36m-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007ffd47d76000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5905a27000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5905689000)
        libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f590545a000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5905069000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f5907912000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5904e65000)

mrossinek · 2020-11-12T13:31:45Z

So I took a closer look at auditwheel as it appeared odd to me that it shouldn't be possible to restrict the platform to something else than manylinux*. Turns out I must have overlooked the --plat argument to the repair command earlier.
Anyways, I believe that this should work. However, now I need to get patchelf to work on the server. I will update once that is done and hopefully be able to close this issue then 👍

vvilpas · 2020-11-12T13:47:21Z

ok, let us know. In any case, not sure if the problem comes from openblas as it seems to be correctly linked statically. However, it seems there is a missing gfortran dependency. Maybe it works adding -static-libgfortran.

mrossinek · 2020-11-12T14:43:10Z

Actually scratch the above. I did not notice that auditwheel stored the new wheel in a different path. Actually installing the wheel under wheelhouse/... still results in a problem however:

Inconsistency detected by ld.so: dl-version.c: 205: _dl_check_map_versions: Assertion `needed != NULL' failed!

mrossinek · 2020-11-12T14:56:08Z

When I compile with python setup.py bdist_wheel -- -DSTATIC_LINKING=True -DCMAKE_CXX_FLAGS="-static-libgfortran" -- -j16 I still obtain the same inconsistency as above but I can finally verify that some of the libraries are correctly included in the wheel:

qiskit_aer.libs/libc-2-f0ad9639.27.so
qiskit_aer.libs/libdl-2-a017a02c.27.so
qiskit_aer.libs/libpthread-2-3dba1c1f.27.so
qiskit_aer.libs/libgomp-52f2fd74.so.1.0.0
qiskit_aer.libs/libm-2-b631b1f5.27.so

However, I don't understand why libopenblas does not show up there...

mrossinek · 2020-11-12T15:10:08Z

And now for a final test with python setup.py bdist_wheel -- -- -j16 and subsequent auditwheel repair --plat linux_x86_64 dist/qiskit_aer*.whl I again obtain the inconsistency error but this time all libraries appear to show up correctly inside the wheel:

qiskit_aer.libs/libc-2-f0ad9639.27.so
qiskit_aer.libs/libgfortran-d87b8024.so.4.0.0
qiskit_aer.libs/libdl-2-a017a02c.27.so
qiskit_aer.libs/libstdc++-a7ce5e3a.so.6.0.25
qiskit_aer.libs/libquadmath-9dd08b42.so.0.0.0
qiskit_aer.libs/libpthread-2-3dba1c1f.27.so
qiskit_aer.libs/libgomp-52f2fd74.so.1.0.0
qiskit_aer.libs/libm-2-b631b1f5.27.so
qiskit_aer.libs/libopenblasp-r0-164709ac.2.20.so
qiskit_aer.libs/libgcc_s-2f0e4866.so.1

Come to think of it, now that I see this, I probably understand that in the previous case those libraries were not included since the static linking directly incorporated these libraries into the wrapper .so...

mrossinek · 2020-11-12T15:15:00Z

Okay I think I found the solution. The same happened to another user of auditwheel here: pypa/auditwheel#103
As he outlines in the first comment this patch from patchelf (NixOS/patchelf#85) fixes this but the Ubuntu version is still fixed at 0.9 (prior to that PR).

I will compile patchelf from source and try again. Should this finally work I will close this issue without further comment.
Please excuse the excessive commenting/spamming in the past hour...

mtreinish · 2020-11-12T16:43:28Z

I'm actually going to reopen this. I think this is a potentially common use case and we should have something in our CONTRIBUTING.md about building a statically linked wheel from master.

As discussed in Qiskit#1033 we should document how to compile a static wheel of Qiskit Aer. This commit adds such documentation to `CONTRIBUTING.md`. I have attempted to make the explanation easy to follow but also easily searchable for potential pitfalls (thus, the inclusion of possible error messages). Hopefully, this will help with the visibility of this section and guide users through the process more easily. Closes Qiskit#1033

mrossinek · 2020-11-16T07:54:59Z

@mtreinish I opened a PR with to update CONTRIBUTING.md and I only just noticed that you were not automatically requested as a reviewer (thus, the ping 😉)

As discussed in Qiskit#1033 we should document how to compile a static wheel of Qiskit Aer. This commit adds such documentation to `CONTRIBUTING.md`. I have attempted to make the explanation easy to follow but also easily searchable for potential pitfalls (thus, the inclusion of possible error messages). Hopefully, this will help with the visibility of this section and guide users through the process more easily. Closes Qiskit#1033

As discussed in #1033 we should document how to compile a static wheel of Qiskit Aer. This commit adds such documentation to `CONTRIBUTING.md`. I have attempted to make the explanation easy to follow but also easily searchable for potential pitfalls (thus, the inclusion of possible error messages). Hopefully, this will help with the visibility of this section and guide users through the process more easily.

mrossinek added the bug Something isn't working label Nov 11, 2020

This comment has been minimized.

Sign in to view

mrossinek closed this as completed Nov 12, 2020

mtreinish reopened this Nov 12, 2020

mrossinek mentioned this issue Nov 13, 2020

Add explanation for static wheel compilation #1038

Merged

vvilpas closed this as completed in #1038 Jan 14, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to compile a statically linked wheel #1033

Unable to compile a statically linked wheel #1033

mrossinek commented Nov 11, 2020

mtreinish commented Nov 11, 2020

mrossinek commented Nov 12, 2020

vvilpas commented Nov 12, 2020

mrossinek commented Nov 12, 2020

vvilpas commented Nov 12, 2020

mrossinek commented Nov 12, 2020

mrossinek commented Nov 12, 2020

vvilpas commented Nov 12, 2020

This comment has been minimized.

mrossinek commented Nov 12, 2020

mrossinek commented Nov 12, 2020

mrossinek commented Nov 12, 2020

mrossinek commented Nov 12, 2020

mtreinish commented Nov 12, 2020

mrossinek commented Nov 16, 2020

Unable to compile a statically linked wheel #1033

Unable to compile a statically linked wheel #1033

Comments

mrossinek commented Nov 11, 2020

Informations

What is the current behavior?

A stable installation using pip install qiskit-terra qiskit-aer

A master installation using python setup.py bdist_wheel

Static linking: python setup.py bdist_wheel -- -DSTATIC_LINKING=True

Static linking with FindBLAS.cmake.fix-static-linking

Steps to reproduce the problem

What is the expected behavior?

Suggested solutions

mtreinish commented Nov 11, 2020

mrossinek commented Nov 12, 2020

vvilpas commented Nov 12, 2020

mrossinek commented Nov 12, 2020

vvilpas commented Nov 12, 2020

mrossinek commented Nov 12, 2020

mrossinek commented Nov 12, 2020

vvilpas commented Nov 12, 2020

This comment has been minimized.

mrossinek commented Nov 12, 2020

mrossinek commented Nov 12, 2020

mrossinek commented Nov 12, 2020

mrossinek commented Nov 12, 2020

mtreinish commented Nov 12, 2020

mrossinek commented Nov 16, 2020

A stable installation using `pip install qiskit-terra qiskit-aer`

A master installation using `python setup.py bdist_wheel`

Static linking: `python setup.py bdist_wheel -- -DSTATIC_LINKING=True`

Static linking with `FindBLAS.cmake.fix-static-linking`