Chemfiles-v0.9.2+0 makes Julia crash on 64-bit Windows

[giordano]

Continuing from JuliaPackaging/Yggdrasil#372 and https://github.com/JuliaBinaryWrappers/Chemfiles_jll.jl/issues/1

---

Simply Libdl.dlopening libchemfiles.dll for x86_64-w64-mingw32 (v0.9.2+0) makes Julia crash. Expanding C++ string ABIs seems to fix the issue.

Vaguely sounds like #539, but for this issue we don't have a backtrace from gdb, yet.

[giordano]

@visr you seem to have used gdb on Windows for #539, could you please tell @Luthaf how to get it?

[visr]

Yeah I install and run gdb with Cygwin.
As far as I remember for this to work you also need to build julia from source in Cygwin as well, here are the instructions for that: https://github.com/JuliaLang/julia/blob/master/doc/build/windows.md

And then in the Cygwin prompt I effectively do

gdb --args julia try.jl
r
bt

Where r stands for run and bt for backtrace.

[Luthaf]

Compilling julia from source is going to take a long time, could you share the one you have, or is there a pre-built version of julia-debug for windows somewhere?

[giordano]

Compilling julia from source is going to take a long time

Out of curiosity, why? I never compiled Julia on Windows, but on my GNU/Linux system the process is pretty straightforward and it doesn't take more than fifteen-twenty minutes for a a full compile.

[Luthaf]

Well, last time I compiled julia from source (it was in the 0.5-0.6 era, mind you) it took 2h between LLVM and all dependencies. It might have improved since then though =)

[giordano]

You don't need to build binary dependencies, BinaryBuilder-powered ones are provided by default 😉

[Luthaf]

BinaryBuilder-powered ones are provided by default

That is a great improvement!

---

I got the following backtraces with julia 1.3.0

julia-debug.exe tmp.jl

#0  0x000000006fcb2e45 in libstdc++-6!_ZNSs4_Rep10_M_disposeERKSaIcE ()
   from C:\julia-1.3.0\usr\bin\libstdc++-6.dll
#1  0x00000001200c6122 in __static_initialization_and_destruction_0(int, int) [clone .constprop.79] ()
   from C:\Users\luthaf\.julia\artifacts\791ede0adee17616fa62fefebb07c0e988b142c3\bin\libchemfiles.dll
#2  0x000000011ffee445 in __do_global_ctors () at /workspace/srcdir/mingw-w64-v6.0.0/mingw-w64-crt/crt/gccmain.c:67
#3  0x000000011fe3129e in __DllMainCRTStartup (hDllHandle=0x11fe30000, dwReason=1, lpreserved=0x0)
    at /workspace/srcdir/mingw-w64-v6.0.0/mingw-w64-crt/crt/crtdll.c:193
#4  0x00007ffa43a08f07 in ntdll!RtlAnsiStringToUnicodeString () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x00007ffa43a04865 in ntdll!RtlCompareUnicodeStrings () from C:\Windows\SYSTEM32\ntdll.dll
#6  0x00007ffa43a04648 in ntdll!RtlCompareUnicodeStrings () from C:\Windows\SYSTEM32\ntdll.dll
#7  0x00007ffa43a03431 in ntdll!RtlFreeUnicodeString () from C:\Windows\SYSTEM32\ntdll.dll
#8  0x00007ffa43a09311 in ntdll!RtlAnsiStringToUnicodeString () from C:\Windows\SYSTEM32\ntdll.dll
#9  0x00007ffa43a06408 in ntdll!RtlCreateUnicodeStringFromAsciiz () from C:\Windows\SYSTEM32\ntdll.dll
#10 0x00007ffa43a058b4 in ntdll!LdrLoadDll () from C:\Windows\SYSTEM32\ntdll.dll
#11 0x00007ffa403dee71 in LoadLibraryExW () from C:\Windows\System32\KernelBase.dll
#12 0x0000000065145b02 in jl_dlopen (
    filename=0xcabba0 "C:\\Users\\luthaf\\.julia\\artifacts\\791ede0adee17616fa62fefebb07c0e988b142c3\\bin\\libchemfiles.dll",
    flags=68) at /cygdrive/c/julia-1.3.0/src/dlload.c:85
#13 0x0000000065145e23 in jl_load_dynamic_library (
    modname=0x10b8b2af8 "C:\\Users\\luthaf\\.julia\\artifacts\\791ede0adee17616fa62fefebb07c0e988b142c3\\bin\\libchemfiles.dll",
    flags=68, throw_err=1) at /cygdrive/c/julia-1.3.0/src/dlload.c:189
#14 0x0000000101677606 in julia_#dlopen#3_8661 (throw_error=1 '\001', s=<optimized out>, flags=68)
    at C:\julia-1.3.0\usr\share\julia\stdlib\v1.3\Libdl\src\Libdl.jl:109
#15 0x000000011aa61158 in dlopen () at C:\julia-1.3.0\usr\share\julia\stdlib\v1.3\Libdl\src\Libdl.jl:109
#16 julia_dlopen_17052 (s=<error reading variable: Cannot access memory at address 0x16fd12300>)
    at C:\julia-1.3.0\usr\share\julia\stdlib\v1.3\Libdl\src\Libdl.jl:109
#17 0x000000011aa611da in jfptr_dlopen_17053 ()
#18 0x000000006512b4c9 in _jl_invoke (F=0x1027bed20 <jl_system_image_data+11709856>, args=0xcac0d8, nargs=1,
    mfunc=0x102764270 <jl_system_image_data+11338480>, world=26787) at /cygdrive/c/julia-1.3.0/src/gf.c:2141
#19 0x000000006512bc0a in jl_apply_generic (F=0x1027bed20 <jl_system_image_data+11709856>, args=0xcac0d8, nargs=1)
    at /cygdrive/c/julia-1.3.0/src/gf.c:2305
#20 0x000000011aa506c1 in julia___init___16959 ()
    at C:\Users\luthaf\.julia\packages\Chemfiles_jll\p8FYb\src\wrappers\x86_64-w64-mingw32.jl:37
#21 0x000000011aa50d34 in jfptr___init___16960 ()
#22 0x000000006512b4c9 in _jl_invoke (F=0x10a780040, args=0xcac330, nargs=0, mfunc=0x10a775fd0, world=26787)
    at /cygdrive/c/julia-1.3.0/src/gf.c:2141
#23 0x000000006512bc0a in jl_apply_generic (F=0x10a780040, args=0xcac330, nargs=0)
    at /cygdrive/c/julia-1.3.0/src/gf.c:2305
#24 0x0000000065167648 in jl_apply (args=0xcac328, nargs=1) at /cygdrive/c/julia-1.3.0/src/julia.h:1631
#25 0x000000006516795f in jl_module_run_initializer (m=0x10a62a230) at /cygdrive/c/julia-1.3.0/src/toplevel.c:74
#26 0x000000006515bd8b in jl_init_restored_modules (init_order=0x10ac29330)
    at /cygdrive/c/julia-1.3.0/src/dump.c:2477
#27 0x00000001014c808b in japi1__include_from_serialized_3263 (path=0x41, depmods=0x10a768050) at loading.jl:692
#28 0x00000001014c9cef in japi1__require_search_from_serialized_2862 (pkg=..., sourcepath=0x48) at loading.jl:776
#29 0x00000001014d8ea7 in japi1__require_2211 (pkg=...) at loading.jl:1001
#30 0x00000001014db89d in japi1_require_2198 (uuidkey=...) at loading.jl:922
#31 0x000000010150ae9a in japi1_require_4320 (into=..., mod=0x0) at loading.jl:917
#32 0x000000006512a7e6 in jl_fptr_args (f=0x10428bb30 <jl_system_image_data+39812016>, args=0xcadaa8, nargs=2,
    m=0x10428c3d0 <jl_system_image_data+39814224>) at /cygdrive/c/julia-1.3.0/src/gf.c:1915
#33 0x000000006512b476 in _jl_invoke (F=0x10428bb30 <jl_system_image_data+39812016>, args=0xcadaa8, nargs=2,
#33 0x000000006512b476 in _jl_invoke (F=0x10424bb30 <jl_system_image_data+39812016>, args=0xcadaa8, nargs=2, mfunc=0x1020c9240 <jl_system_image_data+4674240>, world=26786)
    at /cygdrive/c/julia-1.3.0/src/gf.c:2135
#34 0x000000006512bc0a in jl_apply_generic (F=0x10424bb30 <jl_system_image_data+39812016>, args=0xcadaa8, nargs=2) at /cygdrive/c/julia-1.3.0/src/gf.c:2305
#35 0x0000000065167648 in jl_apply (args=0xcadaa0, nargs=3) at /cygdrive/c/julia-1.3.0/src/julia.h:1631
#36 0x0000000065168bdc in call_require (mod=0x101cbe6d0 <jl_system_image_data+436048>, var=0x10eb0d340) at /cygdrive/c/julia-1.3.0/src/toplevel.c:399
#37 0x0000000065168db0 in eval_import_path (where=0x101cbe6d0 <jl_system_image_data+436048>, from=0x0, args=0x10a6cf250, name=0xcadbc8, keyword=0x6545df74 <szclass_table+1076> "using")
    at /cygdrive/c/julia-1.3.0/src/toplevel.c:436
#38 0x0000000065169c82 in jl_toplevel_eval_flex (m=0x101cbe6d0 <jl_system_image_data+436048>, e=0x10a6c7fb0, fast=1, expanded=1) at /cygdrive/c/julia-1.3.0/src/toplevel.c:656
#39 0x00000000651359be in jl_parse_eval_all (fname=0x10a6c8e18 "C:\\julia-1.3.0\\tmp.jl", content=0x0, contentlen=0, inmodule=0x101cbe6d0 <jl_system_image_data+436048>)
    at /cygdrive/c/julia-1.3.0/src/ast.c:873
#40 0x000000006516aac7 in jl_load (module=0x101cbe6d0 <jl_system_image_data+436048>, fname=0x10a6c8e18 "C:\\julia-1.3.0\\tmp.jl")
    at /cygdrive/c/julia-1.3.0/src/toplevel.c:878
#41 0x000000006516ab19 in jl_load_ (module=0x101cbe6d0 <jl_system_image_data+436048>, str=0x10a6c8e10) at /cygdrive/c/julia-1.3.0/src/toplevel.c:885
#42 0x0000000101497276 in include () at boot.jl:328
#43 japi1_include_relative_3601 (mod=..., _path=0x6) at loading.jl:1105
#44 0x00000001014a4805 in japi1_include_3798 (mod=..., path=0x6) at Base.jl:31
#45 0x000000006512a7e6 in jl_fptr_args (f=0x104a82f50 <jl_system_image_data+48426960>, args=0xcaf438, nargs=2, m=0x104a82ba0 <jl_system_image_data+48426016>)
    at /cygdrive/c/julia-1.3.0/src/gf.c:1915
#46 0x000000006512b476 in _jl_invoke (F=0x104a82f50 <jl_system_image_data+48426960>, args=0xcaf438, nargs=2, mfunc=0x101f03500 <jl_system_image_data+2815360>, world=26786)
    at /cygdrive/c/julia-1.3.0/src/gf.c:2135
#47 0x000000006512bc0a in jl_apply_generic (F=0x104a82f50 <jl_system_image_data+48426960>, args=0xcaf438, nargs=2) at /cygdrive/c/julia-1.3.0/src/gf.c:2305
#48 0x00000001014a396a in julia_exec_options_2168 (opts=...) at client.jl:287
#49 0x00000001014a43d1 in julia__start_2120 () at client.jl:460
#50 0x00000001014a458f in jfptr.start_2121 () from C:\julia-1.3.0\usr\lib\julia\sys-debug.dll
#51 0x000000006512b476 in _jl_invoke (F=0x101ffdde0 <jl_system_image_data+3841632>, args=0xcafa38, nargs=0, mfunc=0x101ffdac0 <jl_system_image_data+3840832>, world=26786)
    at /cygdrive/c/julia-1.3.0/src/gf.c:2135
#52 0x000000006512bc0a in jl_apply_generic (F=0x101ffdde0 <jl_system_image_data+3841632>, args=0xcafa38, nargs=0) at /cygdrive/c/julia-1.3.0/src/gf.c:2305
#53 0x0000000000401621 in jl_apply (args=0xcafa30, nargs=1) at /cygdrive/c/julia-1.3.0/src/julia.h:1631
#54 0x00000000004019d2 in true_main (argc=1, argv=0x408d8f8) at /cygdrive/c/julia-1.3.0/ui/repl.c:96
#55 0x000000000040217c in wmain (argc=1, argv=0x408d8f8, envp=0x4094430) at /cygdrive/c/julia-1.3.0/ui/repl.c:217
#56 0x00000000004013df in __tmainCRTStartup () at /usr/src/debug/mingw64-x86_64-runtime-7.0.0-1/crt/crtexe.c:334
#57 0x000000000040151b in mainCRTStartup () at /usr/src/debug/mingw64-x86_64-runtime-7.0.0-1/crt/crtexe.c:223

julia.exe tmp.jl

#0  0x00007ffa43abafd3 in ntdll!RtlIsNonEmptyDirectoryReparsePointAllowed () from C:\Windows\SYSTEM32\ntdll.dll
#1  0x00007ffa43ac389e in ntdll!RtlpNtMakeTemporaryKey () from C:\Windows\SYSTEM32\ntdll.dll
#2  0x00007ffa43ac3baa in ntdll!RtlpNtMakeTemporaryKey () from C:\Windows\SYSTEM32\ntdll.dll
#3  0x00007ffa43a5ecb1 in ntdll!RtlRaiseStatus () from C:\Windows\SYSTEM32\ntdll.dll
#4  0x00007ffa43a6ce22 in ntdll!memset () from C:\Windows\SYSTEM32\ntdll.dll
#5  0x00007ffa410f9d0c in msvcrt!free () from C:\Windows\System32\msvcrt.dll
#6  0x000000006e156122 in __static_initialization_and_destruction_0(int, int) [clone .constprop.79] ()
   from C:\Users\luthaf\.julia\artifacts\791ede0adee17616fa62fefebb07c0e988b142c3\bin\libchemfiles.dll
#7  0x000000006e07e445 in __do_global_ctors () at /workspace/srcdir/mingw-w64-v6.0.0/mingw-w64-crt/crt/gccmain.c:67
#8  0x000000006dec129e in __DllMainCRTStartup (hDllHandle=0x6dec0000, dwReason=1, lpreserved=0x0) at /workspace/srcdir/mingw-w64-v6.0.0/mingw-w64-crt/crt/crtdll.c:193
#9  0x00007ffa43a08f07 in ntdll!RtlAnsiStringToUnicodeString () from C:\Windows\SYSTEM32\ntdll.dll
#10 0x00007ffa43a04865 in ntdll!RtlCompareUnicodeStrings () from C:\Windows\SYSTEM32\ntdll.dll
#11 0x00007ffa43a04648 in ntdll!RtlCompareUnicodeStrings () from C:\Windows\SYSTEM32\ntdll.dll
#12 0x00007ffa43a03431 in ntdll!RtlFreeUnicodeString () from C:\Windows\SYSTEM32\ntdll.dll
#13 0x00007ffa43a09311 in ntdll!RtlAnsiStringToUnicodeString () from C:\Windows\SYSTEM32\ntdll.dll
#14 0x00007ffa43a06408 in ntdll!RtlCreateUnicodeStringFromAsciiz () from C:\Windows\SYSTEM32\ntdll.dll
#15 0x00007ffa43a058b4 in ntdll!LdrLoadDll () from C:\Windows\SYSTEM32\ntdll.dll
#16 0x00007ffa403dee71 in LoadLibraryExW () from C:\Windows\System32\KernelBase.dll
#17 0x000000006e90134a in jl_dlopen (filename=filename@entry=0xcac930 "C:\\Users\\luthaf\\.julia\\artifacts\\791ede0adee17616fa62fefebb07c0e988b142c3\\bin\\libchemfiles.dll", flags=flags@entry=68)
    at /cygdrive/c/julia-1.3.0/src/dlload.c:85
#18 0x000000006e901540 in jl_load_dynamic_library (modname=0x1278a008 "C:\\Users\\luthaf\\.julia\\artifacts\\791ede0adee17616fa62fefebb07c0e988b142c3\\bin\\libchemfiles.dll", flags=68,
    throw_err=110485456) at /cygdrive/c/julia-1.3.0/src/dlload.c:189
#19 0x0000000007fb091f in julia_#dlopen#3_8630 () at C:\julia-1.3.0\usr\share\julia\stdlib\v1.3\Libdl\src\Libdl.jl:109
#20 0x00000000204212c2 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Notice the Backtrace stopped: previous frame inner to this frame (corrupt stack?) in the julia.exe backtrace.

---

Other than that, it seems to fail when running constructors for global object, so the root cause might be that chemfiles have a few global objects containing std::string, and the ABI of libstdc++ loaded by julia do not match the ABI used to build chemfiles, somehow creating this issue. The symbols should be different for the two ABI though.

[giordano]

@staticfloat with

nm libchemfiles.so.0.9.2 | cut -d' ' -f3 | c++filt | grep -i string

I see a few references to std::basic_string, can that be a problem?

[staticfloat]

It shouldn't but there is clearly a windows-centric C++ issue that I don't fully understand yet.

[staticfloat]

By it shouldn't what I mean is that only std::string should have changed between cxx03 and cxx11. You can read more about it here: https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html

[staticfloat]

Oh; actually, that page mentions that std::list can be a problem. Are there any lists?

[Luthaf]

No, I don't use any std::list, and neither does any dependency (confirmed by nm)

What does expand_cxxstring_abis does exactly? Build the code twice with and without _GLIBCXX_USE_CXX11_ABI? And how is the right version picked by Pkg after that?

[staticfloat]

Build the code twice with and without _GLIBCXX_USE_CXX11_ABI?

Precisely. It stores them in tarballs with target triplets such as x86_64-linux-gnu-cxx03 and x86_64-linux-gnu-cxx11.

And how is the right version picked by Pkg after that?

Pkg looks at how Julia was built; it inspects libLLVM, looking for tell-tale signs of being built either with a cxx03-style string ABI, or a cxx11-style string ABI, and chooses binaries to match.

[Luthaf]

Could it be that upon loading a DLL windows runs the global constructors, dynamically trying to resolve any symbols? Then if julia uses the cxx11 abi, and the DLL uses std::string in global object (this is the case for chemfiles), the process crashes since it can not find std::basic_string (only std::__cxx11::basic_string is defined)?

That would explain the stacktrace and why expand_cxxstring_abis fixes the issues. This does not explain the difference between 32 and 64-bit windows though =/

[giordano]

Ok, I've built locally Chemfiles currently on master with lastest BinaryBuilder and I get

[ Info: /home/mose/repo/builders/Yggdrasil/C/Chemfiles/build/x86_64-linux-gnu/EEoXEMQ3/destdir/lib/libchemfiles.so.0.9.2 locks us to cxx03
┌ Warning: /home/mose/repo/builders/Yggdrasil/C/Chemfiles/build/x86_64-linux-gnu/EEoXEMQ3/destdir/lib/libchemfiles.so.0.9.2 contains std::string values!  This causes incompatibilities across the GCC 4/5 version boundary.  To remedy this, you must build a tarball for both GCC 4 and GCC 5.  To do this, immediately after your `products` definition in your `build_tarballs.jl` file, add the line:
│ 
│     products = expand_cxx_versions(products)
└ @ BinaryBuilder ~/.julia/dev/BinaryBuilder/src/auditor/compiler_abi.jl:189

Thus JuliaPackaging/Yggdrasil#372 looks the way to go

[giordano]

For the record, it looks like std::basic_string is indeed a problem: for a different library, GEOS, in JuliaPackaging/Yggdrasil#293 (comment) I found a lot of symbols like

0000000000170c10 T geos::triangulate::quadedge::LocateFailureException::LocateFailureException(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
0000000000170c10 T geos::triangulate::quadedge::LocateFailureException::LocateFailureException(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
0000000000107f00 T geos::io::ParseException::ParseException(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
0000000000108ae0 T geos::io::ParseException::ParseException(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, double)
00000000001080c0 T geos::io::ParseException::ParseException(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
0000000000107f00 T geos::io::ParseException::ParseException(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
0000000000108ae0 T geos::io::ParseException::ParseException(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, double)
00000000001080c0 T geos::io::ParseException::ParseException(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
00000000001090a0 T geos::io::StringTokenizer::StringTokenizer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
00000000001090a0 T geos::io::StringTokenizer::StringTokenizer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)

Luckily, now these situations are caught by the audit.