Windows nightly binaries statically link against LLVM

[maleadt]

$ ./julia-1.0.1-bin/bin/julia -e "using InteractiveUtils; versioninfo();
                                  using Libdl; println(filter(x -> occursin(\"LLVM\", x), Libdl.dllist()))"
Julia Version 1.0.1
Commit 0d713926f8 (2018-09-29 19:05 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin14.5.0)
  CPU: Intel(R) Core(TM) i5-4308U CPU @ 2.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, haswell)
AbstractString["/Users/julia/Julia/julia-1.0.1-bin/lib/julia//libLLVM.dylib"]

$ ./julia-dev-bin/bin/julia -e "using InteractiveUtils; versioninfo();
                                using Libdl; println(filter(x -> occursin(\"LLVM\", x), Libdl.dllist()))"
Julia Version 1.1.0-DEV.646
Commit 648c3690a5 (2018-11-09 07:15 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin14.5.0)
  CPU: Intel(R) Core(TM) i5-4308U CPU @ 2.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, haswell)
AbstractString[]

i.e. the nightly binaries do not show LLVM.dylib in the output of Libdl.dllist()...
The library is definitely there though, and suspiciously libjulia has grown with 40M:

$ ls -lah julia*-bin/lib/{libjulia.1.*,julia/libLLVM}.dylib
-rwxr-xr-x  1 julia  staff    39M Sep 29 15:22 julia-1.0.1-bin/lib/julia/libLLVM.dylib
-rwxr-xr-x  1 julia  staff   3.0M Sep 29 15:22 julia-1.0.1-bin/lib/libjulia.1.0.dylib
-rwxr-xr-x  1 julia  staff    40M Nov  9 02:24 julia-dev-bin/lib/julia/libLLVM.dylib
-rwxr-xr-x  1 julia  staff    43M Nov  9 02:25 julia-dev-bin/lib/libjulia.1.1.dylib

Looks like we've reverted to statically linking LLVM, presumably unintendedly since we're still shipping LLVM.dylib?

Discovered while debugging LLVM.jl, which requires a dynamically-linked libLLVM and finds that library by looking at the output of Libdl.dllist().

I can do a bisect on the csail macmini, but that will take a while, and maybe somebody already knows what's up by the looks of this issue.

cc @staticfloat

[maleadt]

Bisected to 391f2dd/#29446. On the macOS system, llvm-config --libfiles just lists all static archives, while on my Linux system that outputs only libLLVM.so...

[maleadt]

$ ./bin/llvm-config --shared-mode
static

$ ./bin/llvm-config --libfiles --link-shared
llvm-config: error: libLLVM-6.0.dylib is missing

$ find . -name "libLLVM*.dylib"  
./lib/libLLVM.dylib

$ ln -s lib/libLLVM.dylib lib/libLLVM-6.0.dylib

$ ./bin/llvm-config --libfiles --link-shared
/Users/julia/Julia/julia-dev/deps/scratch/llvm-6.0.1/build_Release/lib/libLLVM-6.0.dylib

Anybody knows what's up?

[maleadt]

This should have been closed with #30033, but I just discovered the same issue applies to Windows:

tbesard@phoenix-win64 MINGW64 ~/Julia                                                                                                                                                                                                        
$ ./julia-1.0.2/bin/julia.exe                                                                                                                                                                                                                
               _                                                                                                                                                                                                                             
   _       _ _(_)_     |  Documentation: https://docs.julialang.org                                                                                                                                                                          
  (_)     | (_) (_)    |                                                                                                                                                                                                                     
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.                                                                                                                                                                              
  | | | | | | |/ _` |  |                                                                                                                                                                                                                     
  | | |_| | | | (_| |  |  Version 1.0.2 (2018-11-08)                                                                                                                                                                                         
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release                                                                                                                                                                            
|__/                   |                                                                                                                                                                                                                     
                                                                                                                                                                                                                                             
julia> using Libdl                                                                                                                                                                                                                           
                                                                                                                                                                                                                                             
julia> filter(x -> occursin("LLVM", x), Libdl.dllist())                                                                                                                                                                                      
1-element Array{AbstractString,1}:                                                                                                                                                                                                           
 "C:\\Users\\tbesard\\Julia\\julia-1.0.2\\bin\\LLVM.dll"                                                                                                                                                         

                                                                                                                                                                                                                                             
tbesard@phoenix-win64 MINGW64 ~/Julia                                                                                                                                                                                                        
$ ./julia-nightly/bin/julia.exe                                                                                                                                                                                                              
               _                                                                                                                                                                                                                             
   _       _ _(_)_     |  Documentation: https://docs.julialang.org                                                                                                                                                                          
  (_)     | (_) (_)    |                                                                                                                                                                                                                     
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.                                                                                                                                                                              
  | | | | | | |/ _` |  |                                                                                                                                                                                                                     
  | | |_| | | | (_| |  |  Version 1.1.0-DEV.786 (2018-12-03)                                                                                                                                                                                 
 _/ |\__'_|_|_|\__'_|  |  Commit 71748da03b (0 days old master)                                                                                                                                                                              
|__/                   |                                                                                                                                                                                                                     
                                                                                                                                                                                                                                             
julia> using Libdl                                                                                                                                                                                                                           
                                                                                                                                                                                                                                             
julia> filter(x -> occursin("LLVM", x), Libdl.dllist())                                                                                                                                                                                      
0-element Array{AbstractString,1} 

$ ls -lah julia-*/bin/{libjulia,LLVM}.dll                                                                                                                                                                                                    
-rwxr-xr-x 1 tbesard None 42M Nov  8 23:31 julia-1.0.2/bin/libjulia.dll                                                                                                                                                                      
-rwxr-xr-x 1 tbesard None 52M Nov  8 23:22 julia-1.0.2/bin/LLVM.dll                                                                                                                                                                          
-rwxr-xr-x 1 tbesard None 82M Dec  3 12:34 julia-nightly/bin/libjulia.dll                                                                                                                                                                    
-rwxr-xr-x 1 tbesard None 53M Dec  3 12:30 julia-nightly/bin/LLVM.dll

[maleadt]

This one seems more troublesome. Running llvm-config from LLVMBuilder with --shared-mode --link-shared reveals that LLVM-6.0.dll is not found. So far so good, but tracing with ProcMon shows that it looks in C:\workspace\destdir\tools\ and nowhere else...
It figures that path out by looking at LLVM_TOOLS_INSTALL_DIR, which is hard coded in https://github.com/staticfloat/LLVMBuilder/blob/3b43a1dda7148cd44db6542b4c88ffaed097901c/build_tarballs.jl#L201

[JeffBezanson]

Tentatively adding to the 1.1 milestone since this is a regression since 1.0. @staticfloat @eli-schwartz @vchuravy @vtjnash any ideas?

[vtjnash]

I think Tim's findings are why we hardcoded the correct value (prior to #29446)

[eli-schwartz]

Ouch. Seems like it would make more sense, though, to ensure the llvm toolchain is in healthy working order. If you're moving things around by hand in the llvm build configuration, it would likely be a good idea to fix it too... is it simply a matter of creating the symlinks on every OS?

BTW looking at the source for llvm-config.cpp it looks like this hardcodes a PREFIX{bin,lib,include} style layout. Maybe it should respect compile-time paths for this.

[vtjnash]

Yeah, the problem we found is that the llvm toolchain doesn't have a health working order. I agree we should try to fix this upstream as you mentioned. In the meantime, we can put your fix behind a ifneq ($(USE_SYSTEM_LLVM),0) test.

[nalimilan]

Maybe related: I made a PR upstream two years ago to restore support for custom install dirs. See https://reviews.llvm.org/D28234.

[Nosferican]

Any update on this as the last issue blocking 1.1 release?

[maleadt]

Fixed (worked around) by #30459.