binary for linux is failing to execute properly #373

k9ert · 2020-09-09T16:27:43Z

We're currently reworking the creation of binaries. Therefore the Linux-binary is created automatically inside a docker-container like this:

https://gitlab.com/cryptoadvance/specter-desktop/-/jobs/727455155

39 INFO: PyInstaller: 4.0
39 INFO: Python: 3.8.5
43 INFO: Platform: Linux-5.4.0-47-generic-x86_64-with-glibc2.2.5
45 INFO: UPX is not available.
48 INFO: Extending PYTHONPATH with paths
['/mnt/pyinstaller']
54 INFO: checking Analysis
54 INFO: Building Analysis because Analysis-00.toc is non existent
54 INFO: Initializing module dependency graph...
55 INFO: Caching module graph hooks...
63 INFO: Analyzing base_library.zip ...
2871 INFO: Processing pre-find module path hook distutils from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks/pre_find_module_path/hook-distutils.py'.
2885 INFO: distutils: retargeting to non-venv dir '/usr/local/lib/python3.8'
4549 INFO: Caching module dependency graph...
4666 INFO: running Analysis Analysis-00.toc
4680 INFO: Analyzing specter_desktop.py
4938 INFO: Analyzing hidden import 'pkg_resources.py2_warn'
5056 ERROR: Hidden import 'pkg_resources.py2_warn' not found
5056 INFO: Processing module hooks...
5057 INFO: Loading module hook 'hook-PyQt5.QtGui.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
5121 INFO: Loading module hook 'hook-PyQt5.QtWebEngineWidgets.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
5343 INFO: Loading module hook 'hook-encodings.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
5382 INFO: Loading module hook 'hook-PyQt5.QtQuick.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
5465 INFO: Loading module hook 'hook-PyQt5.QtCore.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
5481 INFO: Loading module hook 'hook-PyQt5.QtWidgets.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
5512 INFO: Loading module hook 'hook-pkg_resources.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
6152 INFO: Processing pre-safe import module hook win32com from '/usr/local/lib/python3.8/site-packages/_pyinstaller_hooks_contrib/hooks/pre_safe_import_module/hook-win32com.py'.
6154 WARNING: Hidden import "pkg_resources.py2_warn" not found!
6155 WARNING: Hidden import "pkg_resources.markers" not found!
6155 INFO: Excluding import '__main__'
6156 INFO:   Removing import of __main__ from module pkg_resources
6157 INFO: Loading module hook 'hook-PyQt5.py' from '/usr/local/lib/python3.8/site-packages/PyInstaller/hooks'...
6182 WARNING: Hidden import "sip" not found!
...

Note how it contains the error but imports the hook some lines later.

The working-build looks like this:

30 INFO: PyInstaller: 4.0
31 INFO: Python: 3.8.2
78 INFO: Platform: Linux-5.4.0-47-generic-x86_64-with-glibc2.27
80 INFO: UPX is not available.
84 INFO: Extending PYTHONPATH with paths
['/home/kim/src/specter-desktop/pyinstaller']
90 INFO: checking Analysis
90 INFO: Building Analysis because Analysis-00.toc is non existent
90 INFO: Initializing module dependency graph...
91 INFO: Caching module graph hooks...
96 INFO: Analyzing base_library.zip ...
1960 INFO: Processing pre-find module path hook distutils from '/home/kim/src/specter-desktop/.env/lib/python3.8/site-packages/PyInstaller/hooks/pre_find_module_path/hook-distutils.py'.
1963 INFO: distutils: retargeting to non-venv dir '/home/kim/.pyenv/versions/3.8.2/lib/python3.8'
3758 INFO: Caching module dependency graph...
3845 INFO: running Analysis Analysis-00.toc
3852 INFO: Analyzing specter_desktop.py
4068 INFO: Analyzing hidden import 'pkg_resources.py2_warn'
4153 INFO: Processing module hooks...
4154 INFO: Loading module hook 'hook-PyQt5.QtGui.py' from '/home/kim/src/specter-desktop/.env/lib/python3.8/site-packages/PyInstaller/hooks'...
4240 INFO: Loading module hook 'hook-PyQt5.QtWebEngineWidgets.py' from '/home/kim/src/specter-desktop/.env/lib/python3.8/site-packages/PyInstaller/hooks'...
4583 INFO: Loading module hook 'hook-encodings.py' from '/home/kim/src/specter-desktop/.env/lib/python3.8/site-packages/PyInstaller/hooks'...
4628 INFO: Loading module hook 'hook-PyQt5.QtQuick.py' from '/home/kim/src/specter-desktop/.env/lib/python3.8/site-packages/PyInstaller/hooks'...
4693 INFO: Loading module hook 'hook-PyQt5.QtCore.py' from '/home/kim/src/specter-desktop/.env/lib/python3.8/site-packages/PyInstaller/hooks'...
4710 INFO: Loading module hook 'hook-PyQt5.QtWidgets.py' from '/home/kim/src/specter-desktop/.env/lib/python3.8/site-packages/PyInstaller/hooks'...
4748 INFO: Loading module hook 'hook-_tkinter.py' from '/home/kim/src/specter-desktop/.env/lib/python3.8/site-packages/PyInstaller/hooks'...
4796 INFO: checking Tree
4796 INFO: Building Tree because Tree-00.toc is non existent
4796 INFO: Building Tree Tree-00.toc
4801 INFO: checking Tree
4801 INFO: Building Tree because Tree-01.toc is non existent
4801 INFO: Building Tree Tree-01.toc
4803 INFO: Loading module hook 'hook-pkg_resources.py' from '/home/kim/src/specter-desktop/.env/lib/python3.8/site-packages/PyInstaller/hooks'...
5115 INFO: Processing pre-safe import module hook win32com from '/home/kim/src/specter-desktop/.env/lib/python3.8/site-packages/_pyinstaller_hooks_contrib/hooks/pre_safe_import_module/hook-win32com.py'.
5285 INFO: Processing pre-safe import module hook six.moves from '/home/kim/src/specter-desktop/.env/lib/python3.8/site-packages/PyInstaller/hooks/pre_safe_import_module/hook-six.moves.py'.
5323 WARNING: Hidden import "pkg_resources.markers" not found!
5324 INFO: Excluding import '__main__'
5325 INFO:   Removing import of __main__ from module pkg_resources
5325 INFO: Loading module hook 'hook-PyQt5.py' from '/home/kim/src/specter-desktop/.env/lib/python3.8/site-packages/PyInstaller/hooks'...
5339 WARNING: Hidden import "sip" not found!
...

So this doesn't contain the error but also doesn't process the hook.

This created this binary:
https://github.com/cryptoadvance/specter-desktop/releases/download/v0.7.3-pre1/specter_desktop-v0.7.3-pre1-x86_64-linux-gnu.tar.gz

Unfortunately, it's failing to execute properly like this:

➜  specter_desktop-v0.7.3-pre1-x86_64-linux-gnu ./Specter                                                  
/usr/lib/x86_64-linux-gnu/gio/modules/libgvfsdbus.so: undefined symbol: g_date_time_format_iso8601
Failed to load module: /usr/lib/x86_64-linux-gnu/gio/modules/libgvfsdbus.so
libGL error: MESA-LOADER: failed to open iris (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri)
libGL error: failed to load driver: iris
libGL error: MESA-LOADER: failed to open iris (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri)
libGL error: failed to load driver: iris
libGL error: MESA-LOADER: failed to open swrast (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri)
libGL error: failed to load driver: swrast
QGLXContext: Failed to create dummy context
WebEngineContext used before QtWebEngine::initialize() or OpenGL context creation failed.
[2020-09-09 18:22:24,471] INFO in specterd: Logging configured
[2020-09-09 18:22:24,472] INFO in server: pyinstaller based instance running in /tmp/_MEI1JDnf0/specterd
[2020-09-09 18:22:24,473] INFO in server: Initializing LoginManager
[2020-09-09 18:22:24,473] INFO in server: Initializing Specter
[2020-09-09 18:22:24,479] INFO in server: Login disabled
[2020-09-09 18:22:24,479] INFO in server: Initializing Controller ...
* Started Specter daemon...
[2020-09-09 18:22:24,507] INFO in _internal:  * Running on http://127.0.0.1:25441/ (Press CTRL+C to quit)
Failed to create OpenGL context for format QSurfaceFormat(version 2.0, options QFlags<QSurfaceFormat::FormatOption>(), depthBufferSize 24, redBufferSize -1, greenBufferSize -1, blueBufferSize -1, alphaBufferSize -1, stencilBufferSize 8, samples 0, swapBehavior QSurfaceFormat::DefaultSwapBehavior, swapInterval 1, colorSpace QSurfaceFormat::DefaultColorSpace, profile  QSurfaceFormat::NoProfile) 
[1]    283287 abort (core dumped)  ./Specter
➜  specter_desktop-v0.7.3-pre1-x86_64-linux-gnu

The text was updated successfully, but these errors were encountered:

k9ert · 2020-09-11T14:42:43Z

I've created a post on the pyinstaller-list:
https://www.mail-archive.com/pyinstaller@googlegroups.com/msg08327.html
Hope to get some answers from there.

k9ert · 2020-09-15T21:31:57Z

I've tried to import a lot of dependencies explained here:
https://groups.google.com/g/pyinstaller/c/zqf8SPt5iQM/m/lqHcP0BqBQAJ

... but that didn't work. So i got this hint:

You can use psutil to work out which binaries your code needs:

# Put me at the end of your code.
import pustil
print(psutil.Process().memory_maps())

Then compare the output with what PyInstaller includes in your bundle.

I tryto use this to add the additional psutils-dependency in:
https://stackoverflow.com/questions/31261528/any-pyinstaller-detailed-example-about-hidden-import-for-psutil

So i created the file in the hook-directory but still get this:

However, i still get this:

(.env) ➜  specter_desktop--x86_64-linux-gnu git:(master) ✗ ./Specter       
Initializing HWI...
Traceback (most recent call last):
  File "specter_desktop.py", line 473, in <module>
  File "specter_desktop.py", line 387, in init_desktop_app
ModuleNotFoundError: No module named 'psutils'
[246072] Failed to execute script specter_desktop
(.env) ➜  specter_desktop--x86_64-linux-gnu git:(master) ✗

@stepansnigirev any hint here?

stepansnigirev · 2020-09-16T09:17:00Z

Maybe you can just include psutil in the hidden imports of the spec file?
https://github.com/cryptoadvance/specter-desktop/blob/master/pyinstaller/specter_desktop.spec#L30-L33

k9ert · 2020-09-17T14:57:31Z

The psutil-issue was solved with a hook-file like this:

from PyInstaller.utils.hooks import (collect_data_files, collect_submodules)
datas = [('../.env/lib/python3.8/site-packages/psutil/_psutil_linux.cpython-38-x86_64-linux-gnu.so', 'psutil'),
         ('../.env/lib/python3.8/site-packages/psutil/_psutil_posix.cpython-38-x86_64-linux-gnu.so', 'psutil')]
hiddenimports = collect_submodules('psutil')

Useful snippets which i'm dumping here if something like this happens again:

cat psutil_works.txt| tr ',' '\n' | grep pmmap | cut -d "'" -f2 | sed -e 's/\/tmp\/_MEI[^\/]*//' | sort > psutil_works_cleaned.txt
diff psutil_works_cleaned.txt psutil_nworks_cleaned.txt > psutil_diff.txt
cat psutil_diff.txt | grep "^<" > missing_stuff.txt

Anyway, i finally figured out the root cause of it. It seems that a binary built on debian does not work on ubuntu which really surprised me as we're not playing 3d-shooters which need hardware-acceleration but just representing a webpage in a local app.

Details here:
https://groups.google.com/g/pyinstaller/c/zqf8SPt5iQM/m/Wwf3VU4oBQAJ

I also tried to build via an ubuntu-image and this suceeded. I'll do a PR.

rokm · 2020-09-17T20:17:45Z

For what is worth, the issue could be that the libraries bundled with the built program conflict with the system libraries, preventing the DRI driver from properly loading.

The culprit could be either standard c/c++ libraries (libgcc_s.so.1, libstdc++.so.6) or maybe the X11 libraries (libX11.so.6, libXau.so.6, libXdmcp.so.6, libXext.so.6, `libXrender.so.1˙). Perhaps more likely former than the latter.

For example, if libstdc++.so.6 on the build system is older than the one used by the target system, then the non-bundled libraries will fail to load due to missing symbols (which are present in the newer, system version of the library, but not in the bundled one). This is actually quite a common issue with binary-only software on linux, especially on more bleeding edge distributions. In those cases, removing the bundled version of the offending library may help.

(You have a similar issue with system libgvfsdbus.so, which is missing a symbol that is not available in the bundled libglib-2.0.so.0, which is probably older than the glib library available on the system).

k9ert · 2020-09-18T20:24:02Z

Thanks for your thoughts. I hope i have addressed the issue by using a bit of an older build-environment (bionic 18.04 LTS) combined with referring not the exact version but the softlink version of the binary (libgcc_s.so.1 instead of libgcc_s.so.1.x.y).
https://github.com/cryptoadvance/specter-desktop/blob/master/pyinstaller/specter_desktop.spec#L15-L30

I chose the older build-env-version (bionic) intentionally because of the maybe slightly different rules for glibc which caused #356.
I think this could potentially hit us again if the above list of binaries bundled is not complete as you have described.

On the other hand, this is just a simple web-served flask-application which hopefully doesn't need so complex system-level dependencies and the above list is indeed complete.

We will see. If you're interested, please feel free to test the pre-release:
https://github.com/cryptoadvance/specter-desktop/releases/tag/v0.7.3-pre3

k9ert · 2023-02-01T09:10:13Z

For the reference (thank chatGPT):

Distribution	Version	Release Date	End of Support	glibc Version
Debian 8 (Jessie)	8	April 25, 2015	2020-06-30	2.19
Debian 9 (Stretch)	9	June 17, 2017	2020-06-30	2.24
Ubuntu 18.04 LTS (Bionic Beaver)	18.04 LTS	April 26, 2018	2023-04-26	2.27
Debian 10 (Buster)	10	July 6, 2019	2022-06-30	2.28
Ubuntu 20.04 LTS (Focal Fossa)	20.04 LTS	April 23, 2020	2025-04-23	2.31
Ubuntu 20.10 (Groovy Gorilla)	20.10	October 22, 2020	2021-07-22	2.31
Debian 11 (Bullseye)	11	August 14, 2021	2026-06-30	2.31
Ubuntu 22.04 LTS (Jammy Jellyfish)	22.04 LTS	April 21, 2022	2027-04-21	2.35

k9ert mentioned this issue Sep 11, 2020

Error In Ubuntu #356

Closed

k9ert added the bug Something isn't working label Sep 11, 2020

k9ert self-assigned this Sep 11, 2020

k9ert added the chore testing, automation, releasing and the like label Sep 11, 2020

k9ert pushed a commit to k9ert/specter-desktop that referenced this issue Sep 17, 2020

fixes cryptoadvance#373 hopefully - still needs finalisation

8985482

k9ert mentioned this issue Sep 17, 2020

Fixing the linux-binary-build #396

Merged

k9ert closed this as completed in #396 Sep 18, 2020

k9ert mentioned this issue Sep 26, 2020

Linux Binary fails on Linux-Mint 20 (Virtualbox) #441

Closed

k9ert mentioned this issue Oct 12, 2020

Doesn't run on systems with GCC version < 2.25 #495

Closed

k9ert mentioned this issue Sep 11, 2022

Chore: Upgrade to python 3.10 #1688

Merged

14 tasks

k9ert mentioned this issue Feb 1, 2023

Clarifying glibc support for version2 #2118

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

binary for linux is failing to execute properly #373

binary for linux is failing to execute properly #373

k9ert commented Sep 9, 2020 •

edited

Loading

k9ert commented Sep 11, 2020

k9ert commented Sep 15, 2020 •

edited

Loading

stepansnigirev commented Sep 16, 2020

k9ert commented Sep 17, 2020 •

edited

Loading

rokm commented Sep 17, 2020

k9ert commented Sep 18, 2020

k9ert commented Feb 1, 2023

binary for linux is failing to execute properly #373

binary for linux is failing to execute properly #373

Comments

k9ert commented Sep 9, 2020 • edited Loading

k9ert commented Sep 11, 2020

k9ert commented Sep 15, 2020 • edited Loading

stepansnigirev commented Sep 16, 2020

k9ert commented Sep 17, 2020 • edited Loading

rokm commented Sep 17, 2020

k9ert commented Sep 18, 2020

k9ert commented Feb 1, 2023

k9ert commented Sep 9, 2020 •

edited

Loading

k9ert commented Sep 15, 2020 •

edited

Loading

k9ert commented Sep 17, 2020 •

edited

Loading