Segmentation fault (core dumped) when pip installing inside nested docker container #485
Comments
|
I am unable to add "v0.4.1" and "Bug" labels. |
|
@iisojunn, thanks for the very detailed description and reproduction steps! We have fixed a couple of bugs in this space over the last few weeks. I believe that this one in particular addresses the problem that you are referring to. I'm not able to reproduce the problem with our latest code anymore. These fixes will be included in our next release within a couple of weeks. For now, please build sysbox binaries from scratch and let us know if that fixes the issue. Find the instructions here. |
|
Hi, thanks for the fast response! I just tested with binaries build from the following commit: Pip install within nested container seems to work! I guess, I'll continue with the workaround until the next release is out. |
|
Any updates on when the new release is due to come out? |
|
@damienrj, we expect the new release to come out by the end of the week. |
Segmentation fault (core dumped) occurs when executing
python -m pip install <package>insidepython:3.8.0-slimcontainer running insidenestybox/alpine-dockerwhich utilizessysbox-runc. The issue seems to be related how pip uninstalls packages. The fault only occurs inside nested (second level) docker container, not on first level docker container (sysbox-runc in use in both cases). It is unclear for me if root cause for the issue exists insysbox-runcorpip.The issue was found during docker-in-docker image build CI job. It does not matter if the command
python -m pip install <package>is executed inside the container manually or when building image and running Dockerfile commandRUN python -m pip install <package>.The issue is not limited on
python:3.8.0-slimimage, also newer python versions and not slim images cause same symptoms. Imagepython:3.7.0-slimdid not seem to have this problem.In addition to Segmentation fault (error code 139), there has also been been Bus Error (error code 135) visible, but just couple times.
The segmentation fault does not occur every time. With different pip packages pip install sometimes succeeds and sometimes not. However, there are also consistent ways for producing Segmentation fault every time.
When executing different pip commands (varying package etc.), the segmentation fault occurs at different places in code. This means that tracebacks/backtraces are not similar in every case. Some occurred segfault locations are: visit_decref, pymalloc_alloc, _PyObject_IS_GC.
Workaround
The segmentation fault does seem to occur when pip installs package which has already different version installed. Somehow it might be related to pip uninstall logic with folder stashing (setuptools -> ~etuptools) and rollback possibility. It seems that one can avoid segmentation fault if pip packages are installed with
--ignore-installed. (Also--userflag and--target <empty-folder>seem to work, probably because in the target folder there is no packages to be uninstalled.)Generating the fault
Starting first level docker container on host.
docker run -it --runtime=sysbox-runc nestybox/alpine-docker sh -c "dockerd >/dev/null 2>&1 & sh"Consistent Segmentation fault due to
python -m pip install --upgrade setuptoolswhich is executed inside second level docker container.Same as earlier, but with gdb debugger attached and with python debugging symbols (python line numbers seemed to be off for some reason).
Segmentation fault every now and then caused by
python -m pip uninstall setuptools -yDebug logs
Debug logs
Host computer
The text was updated successfully, but these errors were encountered: