Add an option to ignore missing shared libraries

[pavel-kirienko]

I am building a Python GUI application based on PyQt5. The application is packaged into a native binary with PyInstaller, which generally works well, except that the resulting application ends up being dependent on a particular version of libGL which is not always available, which results in serious portability issues. Additionally, the native binary is unusable with systems that use older GLIBC, but this is a well-known property of PyInstaller so there is nothing to complain about.

Shared libraries are generally a pain to deal with, so I decided to try StaticX to build a fully static executable. Upon running it against my binary generated by PyInstaller, I encountered several failures where StaticX would find some obscure binary shipped with PyQt5 and then complain that it refers to a shared library which cannot be found. For example, libmysqlclient.so.18, libQt53DQuickRenderer.so.5, and several dozen others. This behavior is not helpful because none of the missing libraries are actually needed for my application to run.

Hence, in my case it would help greatly if I could just tell StaticX to skip missing shared libraries. I made a minimal PoC by ignoring the ldd output lines that contain not found in them, and verified that it works and the final executable is usable:

diff --git a/staticx/elf.py b/staticx/elf.py
index 279c1eb..0c28515 100644
--- a/staticx/elf.py
+++ b/staticx/elf.py
@@ -101,7 +101,10 @@ def get_shobj_deps(path, libpath=[]):
             # Some shared objs might have no DT_NEEDED tags (see issue #67)
             if line == '\tstatically linked':
                 break
-            raise ToolError('ldd', "Unexpected line in ldd output: " + line)
+            elif 'not found' in line.lower():
+                continue
+            else:
+                raise ToolError('ldd', "Unexpected line in ldd output: " + line)
         libname  = m.group(1)
         libpath  = m.group(2)
         baseaddr = int(m.group(3), 16)

I am not sure what is the best way to integrate this feature into upstream. I have several ideas:

• Log every such occurrence at the WARNING level, do nothing else. This may be incompatible with the case when the user actually wants to ensure that all referred shared objects are found and packaged.
• Refactor the get_shobj_deps() function to take an additional argument specifying the missing library handling policy: either throw or ignore. This seems rather messy because a lot of other code will be affected as well.
• Use a global configuration object.

[JonathonReinhart]

Hi @pavel-kirienko, thanks for opening this issue! Unfortunately, I think you're going to have to dig a bit deeper before I can consider this change.

To be honest, I'm not sure when your proposed change would ever be the correct behavior. That change could lead to necessary libraries getting missed, and an application that crashes at runtime.

AFAIK StaticX can't be overly aggressive in discovering "obscure" library dependencies. It simply runs ldd on the application provided by the user and grabs any libraries that the application indicates it needs. In the case of apps built with PyInstaller, it opens the PYI archive, and runs ldd on all of those libraries, too. So if ldd says a library is not found, well, that's probably a problem. Now perhaps there's something special about QT5 that causes extra libraries to be picked up by PyInstaller (I haven't yet dug through their hooks). But even if that's the case, those libraries will fail to load if someone tries to do so, and your patch is in place.

The patch you've shown covers your use-case, but doesn't really help to understand why you're running into missing libraries. What would be really helpful for me is a minimal reproducible test case; some python code and a set of build steps that I could run to reproduce your problem, to fully understand the issue and ensure the right changes are made.

If you'd like to dig into the issue, you could use pyi-archive_viewer to identify which libraries are included in your PyInstaller application, and run ldd on them to determine their dependencies. Or you could run staticx --loglevel=DEBUG to see everything that's going on. You may find that staticx says that ldd returned 'not found, but when you run ldd` manually on the library (in its installed location), everything is found.

Let me know if I can help track this down.

[pavel-kirienko]

Jonathon, thank you for the suggestions. I will have to push back on this for a few months since I am currently engaging other matters, but I will be back to you once I'm back to work on this particular application.

[JonathonReinhart]

Pavel, no problem. I'm going to close this issue for now, because I don't currently see a valid use-case for this proposal. However, if you can provide some more details and make a stronger case, please re-open this issue and I would be happy to reconsider. Thanks again!