MallocCrash on Ubuntu with 4.20 when importing module

[slartibaartfast]

Hi,
I can't figure out why importing some modules causes a crash.

The OS is Pop!_OS 18.04 LTS, and have also reproduced this on Ubuntu 18.04.
The UE4 version is 4.20
UnrealEnginePython branch is master.
Python version is 3.6.6

The system has been modified by adding a symlink /usr/include/python3.6/x86_64-linux-gnu/python3.6m/pyconfig.h which points to /usr/include/x86_64-linux-gnu/python3.6m/pyconfig.h, per issue #477

sudo ln -s /usr/include/x86_64-linux-gnu/python3.6m/pyconfig.h .

When working in the UE4 Editor, importing some modules, such as asyncio, causes a crash. It might only crash when importing a module that depends on a shared library. Running with gdb, the error looks like:

[2018.10.14-17.11.59:120][449]LogTemp: >>> import asyncio
free(): invalid pointer
Thread 1 "UE4Editor" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) continue
Continuing.
Signal 6 caught.
Malloc Size=131076 LargeMemoryPoolOffset=131092
CommonUnixCrashHandler: Signal=6
Malloc Size=65535 LargeMemoryPoolOffset=196655
Malloc Size=1312160 LargeMemoryPoolOffset=1508832
Malloc Size=68112 LargeMemoryPoolOffset=1576960
Malloc Size=145512 LargeMemoryPoolOffset=1722488
Realloc PtrSize=68112 NewSize=145512 PooledPtr=0x00007fff495605f0
Malloc Size=253872 LargeMemoryPoolOffset=1976384
Realloc PtrSize=145512 NewSize=253872 PooledPtr=0x00007fff49571010

Thread 1 "UE4Editor" received signal SIGTRAP, Trace/breakpoint trap.
0x00007ffff71a32b6 in FUnixPlatformMisc::DebugBreakInternal () at /home/trota/Code/UnrealEngine/Engine/Source/Runtime/Core/Private/Unix/UnixPlatformMisc.cpp:820
820 asm volatile("int $0x03");

The backtrace is available in this gist. Lines 17 through 20, and 223 look most relevant.

The python path looks like:

[2018.10.14-17.09.26:344][753]LogTemp: >>> import sys
[2018.10.14-17.09.31:936][424]LogTemp: >>> for p in sys.path:
[2018.10.14-17.09.37:528][ 95]LogTemp: ... print(p)
[2018.10.14-17.09.38:045][157]LogPython:
[2018.10.14-17.09.38:045][157]LogPython:
[2018.10.14-17.09.38:045][157]LogPython: ../../../../../Documents/UnrealProjects/test420/Content/Scripts
[2018.10.14-17.09.38:045][157]LogPython:
[2018.10.14-17.09.38:045][157]LogPython: ../../../../../Documents/UnrealProjects/test420/Plugins/UnrealEnginePython
[2018.10.14-17.09.38:045][157]LogPython:
[2018.10.14-17.09.38:045][157]LogPython: ../../../../../Documents/UnrealProjects/test420/Content/ue_python.zip
[2018.10.14-17.09.38:045][157]LogPython:
[2018.10.14-17.09.38:045][157]LogPython: /home/trota/Documents/UnrealProjects/test420
[2018.10.14-17.09.38:045][157]LogPython:
[2018.10.14-17.09.38:045][157]LogPython: /usr/lib/python36.zip
[2018.10.14-17.09.38:045][157]LogPython:
[2018.10.14-17.09.38:045][157]LogPython: /usr/lib/python3.6
[2018.10.14-17.09.38:045][157]LogPython:
[2018.10.14-17.09.38:045][157]LogPython: /usr/lib/python3.6/lib-dynload
[2018.10.14-17.09.38:045][157]LogPython:
[2018.10.14-17.09.38:045][157]LogPython: /home/trota/.local/lib/python3.6/site-packages
[2018.10.14-17.09.38:046][157]LogPython:
[2018.10.14-17.09.38:046][157]LogPython: /usr/local/lib/python3.6/dist-packages
[2018.10.14-17.09.38:046][157]LogPython:
[2018.10.14-17.09.38:046][157]LogPython: /usr/lib/python3/dist-packages
[2018.10.14-17.09.38:046][157]LogPython:

Pip can see asyncio:

[2018.10.14-17.10.23:865][855]LogTemp: >>> import subprocess
[2018.10.14-17.11.14:041][586]LogTemp: >>> subprocess.check_call([sys.executable, '-m', 'pip', 'list'])
asn1crypto (0.24.0)
asyncio (3.4.3)
...

To reproduce this on an Ubuntu 18.04 or Pop!_OS 18.04 machine with Python3.6 and asyncio module:

1. Clone and make UnrealEngine 4.20 by running ./Setup, ./GenerateProjectFiles.sh, and make
2. Create a new C++ project
3. Clone UnrealEnginePython into the new project's plugins directory
4. Open the new project, let it compile the plugin
5. Open the python console
6. enter import asyncio

Is the problem that not all modules are being loaded into the dictionaries? I don't know how to work around this and would really appreciate any suggestions. Thanks in advance!

[slartibaartfast]

In the UE4.20 installation at UnrealEngine/Engine/Build/BatchFiles/Linux/SetupToolchain.sh they are setting the clang version to 5.

TOOLCHAIN_VERSION=v11_clang-5.0.0-centos7

It looks like 4.21 will use clang 6.0.1, so I will try building with that.

[slartibaartfast]

This error also occurs in 4.21 with the v12_clang-6.0.1 toolchain.

@ishyesh Have you run into this problem - not being able to import modules such as asyncio or requests?

[slartibaartfast]

If the module that is being imported is in a directory, like asyncio, then this error occurs. If it is a file, like six.py, then there is no error.
From within the editor, I tried loading modules with imp.load_module and the behaviour is the same - invalid pointer.
@rdeioris do you know what I could do to fix this? I thought it might be a conversion problem, but nothing I've tried has worked so far.

[slartibaartfast]

The changes in PR #644 fix some of the package import issues, or at least work around them.

Another problem was due to my using the wrong OpenSSL version (1.1.0, with _blake2.cpython-37m-x86_64-linux-gnu.so), which was corrected by building Python and UnrealEnginePython using this method.

[friolero]

The changes in PR #644 fix some of the package import issues, or at least work around them.

Another problem was due to my using the wrong OpenSSL version (1.1.0, with _blake2.cpython-37m-x86_64-linux-gnu.so), which was corrected by building Python and UnrealEnginePython using this method.

Hi just encountered the same problem here. Have tried building an empty c++ project with default python 3.6 or explicitly defined anaconda python 3.6, both had the same issue for malloc crash when import module like numpy.

Also tried to update the UnrealEnginePython.cpp as your PR and problem still persists. Could you please shed some light here on any cause or solution? Thanks!

[slartibaartfast]

Hi @frlolero. The solution that worked for me was to use Conan to apply specific versions of OpenSSL (OpenSSL/1.0.2r@conan/stable) and zlib during the build of my project. Adam Rhen wrote some tools to make this easier and he talks about them in this article.
I forked his conan recipe repo here if you'd like to use any of my recipes.
I think the cause was that when building using the UE4 toolchain, if modules like zlib and openssl aren't the same version, there is a conflict.