Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

InternalError with Python 3.10: libname absent from _added_libs but library is present #223

Closed
jmsmkn opened this issue May 18, 2022 · 6 comments · Fixed by #225
Closed

InternalError with Python 3.10: libname absent from _added_libs but library is present #223

jmsmkn opened this issue May 18, 2022 · 6 comments · Fixed by #225
Labels
awaiting feedback bug

Comments

@jmsmkn
Copy link

jmsmkn commented May 18, 2022

I have a PyInstalled app that is successfully bundled with staticx 0.13.6 on Python 3.8.10, but with Python 3.10.4 I get error:

staticx: libname ld-linux-x86-64.so.2 absent from _added_libs but library ld-linux-x86-64.so.2 present

From:

staticx/staticx/api.py

Lines 210 to 211 in 86298f6

raise InternalError("libname {} absent from _added_libs but"
" library {} present".format(libname, arcname))

Unsure how to handle this, do you have any pointers? Thank you for the great tool!
@JonathonReinhart
Copy link
Owner

Please run staticx with --debug and provide the full output.

If you're comfortable running staticx from source, it may also be helpful to add these two lines:

from pprint import pformat
        if arcname in self._added_libs:
            logging.info("_added_libs is: {}".format(pformat(self._added_libs)))    # <<<<<<<<
            raise InternalError("libname {} absent from _added_libs but"
                    " library {} present".format(libname, arcname))

I don't have time to investigate very much right now, but it appears to be some discrepancy between the library's name and its actual resolved (through symlinks) file base name.

@jmsmkn
Copy link
Author

jmsmkn commented May 19, 2022

Super, thank you for the pointers, they helped a lot. Problem was on my end.

@jmsmkn jmsmkn closed this as completed May 19, 2022
@JonathonReinhart
Copy link
Owner

Do you mind sharing the details? InternalError was something that was never supposed to happen. So if you found an external way to make it happen, I'd like to consider updating the error message to be more helpful, or adding a wiki page, etc.

@liujin1993
Copy link

liujin1993 commented Jul 28, 2022 •
edited
Loading

@JonathonReinhart, hi, I reproduced this error in ubuntu:22.04

docker run -it ubuntu:22.04 /bin/bash
apt update & apt install -y python3-pip patchelf
pip install pyinstaller staticx
echo "print(1)" > a.py
pyinstaller -F a.py
cd dist
staticx --debug a b

and the debug message is:

INFO:root:Running StaticX version 0.13.6
INFO:root:Libraries:
INFO:root:  elftools: 0.28
DEBUG:root:External tools:
INFO:root:  ldd: /usr/bin/ldd: ldd (Ubuntu GLIBC 2.35-0ubuntu3) 2.35
INFO:root:  objcopy: /usr/bin/objcopy: GNU objcopy (GNU Binutils for Ubuntu) 2.38
INFO:root:  strip: /usr/bin/strip: GNU strip (GNU Binutils for Ubuntu) 2.38
INFO:root:  patchelf: /usr/bin/patchelf: patchelf 0.14.3
DEBUG:root:Arguments:
DEBUG:root:  prog:      'a'
DEBUG:root:  output:    'b'
DEBUG:root:  libs:      None
DEBUG:root:  strip:     False
DEBUG:root:  compress:  True
DEBUG:root:  debug:     True
INFO:root:Using XZ BCJ filter FILTER_X86
DEBUG:root:Bootloader: bootloader version 0.13.6 compiled Dec  2 2021 at 21:42:39 by musl-gcc version 10.2.1 20210110
INFO:root:Program interpreter: /lib64/ld-linux-x86-64.so.2
DEBUG:root:Running ['patchelf', '--remove-rpath', '/tmp/staticx-prog-aej0xxmq']
DEBUG:root:Running ['patchelf', '--set-interpreter', 'iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii', '--set-rpath', 'rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr', '--force-rpath', '/tmp/staticx-prog-aej0xxmq']
DEBUG:root:Running ['patchelf', '--no-default-lib', '/tmp/staticx-prog-aej0xxmq']
INFO:root:Opened PyInstaller archive!
DEBUG:root:Extracting to /tmp/staticx-pyi-sv0xc9mf/lib-dynload/_bz2.cpython-310-x86_64-linux-gnu.so
DEBUG:root:Extracting to /tmp/staticx-pyi-sv0xc9mf/lib-dynload/_codecs_cn.cpython-310-x86_64-linux-gnu.so
DEBUG:root:Extracting to /tmp/staticx-pyi-sv0xc9mf/lib-dynload/_codecs_hk.cpython-310-x86_64-linux-gnu.so
DEBUG:root:Extracting to /tmp/staticx-pyi-sv0xc9mf/lib-dynload/_codecs_iso2022.cpython-310-x86_64-linux-gnu.so
DEBUG:root:Extracting to /tmp/staticx-pyi-sv0xc9mf/lib-dynload/_codecs_jp.cpython-310-x86_64-linux-gnu.so
DEBUG:root:Extracting to /tmp/staticx-pyi-sv0xc9mf/lib-dynload/_codecs_kr.cpython-310-x86_64-linux-gnu.so
DEBUG:root:Extracting to /tmp/staticx-pyi-sv0xc9mf/lib-dynload/_codecs_tw.cpython-310-x86_64-linux-gnu.so
DEBUG:root:Extracting to /tmp/staticx-pyi-sv0xc9mf/lib-dynload/_contextvars.cpython-310-x86_64-linux-gnu.so
DEBUG:root:Extracting to /tmp/staticx-pyi-sv0xc9mf/lib-dynload/_decimal.cpython-310-x86_64-linux-gnu.so
DEBUG:root:Extracting to /tmp/staticx-pyi-sv0xc9mf/lib-dynload/_hashlib.cpython-310-x86_64-linux-gnu.so
DEBUG:root:Extracting to /tmp/staticx-pyi-sv0xc9mf/lib-dynload/_lzma.cpython-310-x86_64-linux-gnu.so
DEBUG:root:Extracting to /tmp/staticx-pyi-sv0xc9mf/lib-dynload/_multibytecodec.cpython-310-x86_64-linux-gnu.so
DEBUG:root:Extracting to /tmp/staticx-pyi-sv0xc9mf/lib-dynload/_opcode.cpython-310-x86_64-linux-gnu.so
DEBUG:root:Extracting to /tmp/staticx-pyi-sv0xc9mf/lib-dynload/resource.cpython-310-x86_64-linux-gnu.so
DEBUG:root:Extracting to /tmp/staticx-pyi-sv0xc9mf/libbz2.so.1.0
DEBUG:root:Extracting to /tmp/staticx-pyi-sv0xc9mf/libcrypto.so.3
DEBUG:root:Extracting to /tmp/staticx-pyi-sv0xc9mf/libexpat.so.1
DEBUG:root:Extracting to /tmp/staticx-pyi-sv0xc9mf/liblzma.so.5
DEBUG:root:Extracting to /tmp/staticx-pyi-sv0xc9mf/libmpdec.so.3
DEBUG:root:Extracting to /tmp/staticx-pyi-sv0xc9mf/libpython3.10.so.1.0
DEBUG:root:Extracting to /tmp/staticx-pyi-sv0xc9mf/libz.so.1
DEBUG:root:Running ['ldd', '/tmp/staticx-pyi-sv0xc9mf/lib-dynload/_bz2.cpython-310-x86_64-linux-gnu.so']
DEBUG:root:Ignoring synthetic library: linux-vdso.so.1
DEBUG:root:libbz2.so.1.0 already in pyinstaller archive
INFO:root:Processing library libc.so.6 (/lib/x86_64-linux-gnu/libc.so.6)
INFO:root:Adding /lib/x86_64-linux-gnu/libc.so.6 as libc.so.6
INFO:root:Processing library ld-linux-x86-64.so.2 (/lib64/ld-linux-x86-64.so.2)
INFO:root:Adding Symlink ld-linux-x86-64.so.2 => ld-linux-x86-64.so.2
INFO:root:Adding /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 as ld-linux-x86-64.so.2
Traceback (most recent call last):
  File "/usr/local/bin/staticx", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/staticx/__main__.py", line 49, in main
    generate(args.prog, args.output,
  File "/usr/local/lib/python3.10/dist-packages/staticx/api.py", line 320, in generate
    gen.generate(output=output)
  File "/usr/local/lib/python3.10/dist-packages/staticx/api.py", line 134, in generate
    run_hooks(self)
  File "/usr/local/lib/python3.10/dist-packages/staticx/hooks/__init__.py", line 12, in run_hooks
    hook(sx)
  File "/usr/local/lib/python3.10/dist-packages/staticx/hooks/pyinstaller.py", line 27, in process_pyinstaller_archive
    h.process()
  File "/usr/local/lib/python3.10/dist-packages/staticx/hooks/pyinstaller.py", line 57, in process
    self._add_required_deps(binary)
  File "/usr/local/lib/python3.10/dist-packages/staticx/hooks/pyinstaller.py", line 135, in _add_required_deps
    self.sx.add_library(deppath, exist_ok=True)
  File "/usr/local/lib/python3.10/dist-packages/staticx/api.py", line 210, in add_library
    raise InternalError("libname {} absent from _added_libs but"
staticx.errors.InternalError: libname ld-linux-x86-64.so.2 absent from _added_libs but library ld-linux-x86-64.so.2 present

and the output of ldd a is :

root@ddce636ec402:/dist# ldd a
	linux-vdso.so.1 (0x00007ffd393e9000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f37af5b9000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f37af59d000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f37af598000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f37af370000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f37af5c3000)

@JonathonReinhart
Copy link
Owner

JonathonReinhart commented Jul 31, 2022 •
edited
Loading

With this extra logging:

DEBUG:root:Running ['ldd', '/tmp/libnssfix-ajnvtj4s.so']
DEBUG:root:Ignoring synthetic library: linux-vdso.so.1
INFO:root:Processing library libc.so.6 (/lib/x86_64-linux-gnu/libc.so.6)
INFO:root:Adding /lib/x86_64-linux-gnu/libc.so.6 as libc.so.6
INFO:root:Processing library libnss_dns.so.2 (/lib/x86_64-linux-gnu/libnss_dns.so.2)
INFO:root:Adding /lib/x86_64-linux-gnu/libnss_dns.so.2 as libnss_dns.so.2
INFO:root:Processing library libnss_files.so.2 (/lib/x86_64-linux-gnu/libnss_files.so.2)
INFO:root:Adding /lib/x86_64-linux-gnu/libnss_files.so.2 as libnss_files.so.2
INFO:root:Processing library ld-linux-x86-64.so.2 (/lib64/ld-linux-x86-64.so.2)
INFO:root:Adding Symlink ld-linux-x86-64.so.2 => ld-linux-x86-64.so.2
INFO:root:Adding /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 as ld-linux-x86-64.so.2
INFO:root:_added_libs is: {'ld-linux-x86-64.so.2': None,
 'libc.so.6': '/lib/x86_64-linux-gnu/libc.so.6',
 'libnss_dns.so.2': '/lib/x86_64-linux-gnu/libnss_dns.so.2',
 'libnss_files.so.2': '/lib/x86_64-linux-gnu/libnss_files.so.2'}
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/jreinhart/code/staticx/staticx/__main__.py", line 63, in <module>
    main()
  File "/home/jreinhart/code/staticx/staticx/__main__.py", line 49, in main
    generate(args.prog, args.output,
  File "/home/jreinhart/code/staticx/staticx/api.py", line 322, in generate
    gen.generate(output=output)
  File "/home/jreinhart/code/staticx/staticx/api.py", line 135, in generate
    run_hooks(self)
  File "/home/jreinhart/code/staticx/staticx/hooks/__init__.py", line 12, in run_hooks
    hook(sx)
  File "/home/jreinhart/code/staticx/staticx/hooks/glibc.py", line 39, in process_glibc_prog
    sx.add_library(libpath, exist_ok=True)
  File "/home/jreinhart/code/staticx/staticx/api.py", line 212, in add_library
    raise InternalError("libname {} absent from _added_libs but"
staticx.errors.InternalError: libname ld-linux-x86-64.so.2 absent from _added_libs but library ld-linux-x86-64.so.2 present

Note that /lib64/ld-linux-x86-64.so.2 ➡️ /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2.

@JonathonReinhart
Copy link
Owner

JonathonReinhart commented Jul 31, 2022 •
edited
Loading

I think this is actually the problem:

INFO:root:Adding Symlink ld-linux-x86-64.so.2 => ld-linux-x86-64.so.2

The intended use case of adding symlinks is to handle library version numbers, e.g. libfoo.so.1 ➡️ libfoo.so.1.2 ➡️ libfoo.so.1.2.3.

The code (mistakenly) doesn't account for symlinks with the same basename, .e.g. /lib64/libfoo.so ➡️ /lib/libfoo.so.

The problem is that the paths are stripped any only the symlink name is kept. So we end up adding a self-referential symlink. I think we can just skip these.

@JonathonReinhart JonathonReinhart mentioned this issue Aug 6, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
awaiting feedback bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Ignore the case where a library symlink has the same basename as its target JonathonReinhart/staticx
3 participants
@JonathonReinhart @jmsmkn @liujin1993