Windows nightly 64-bit release exe does not start

[ihnorton]

The release exe does not start in the latest Win64 nightly (ef06211). The -debug version and both 32-bit exes are fine. --help does work, but julia -e "println(\"hello, world\")" does not.

There are no messages when run in cmd, and nothing obviously wrong in Dependency Walker. I don't have gdb available right now, but in WinDbg I get:

...
ModLoad: 000007fe`ff610000 000007fe`ff65d000   C:\Windows\system32\WS2_32.dll
(1613d0.1613a4): Break instruction exception - code 80000003 (first chance)
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for ntdll.dll - 
ntdll!CsrSetPriorityClass+0x40:
00000000`77bfcb70 cc              int     3
0:000> g
ModLoad: 000007fe`ff5e0000 000007fe`ff60e000   C:\Windows\system32\IMM32.DLL
ModLoad: 000007fe`ff170000 000007fe`ff279000   C:\Windows\system32\MSCTF.dll
ModLoad: 000007fe`fd000000 000007fe`fd055000   C:\Windows\system32\mswsock.dll
ModLoad: 000007fe`fca30000 000007fe`fca37000   C:\Windows\System32\wshtcpip.dll
ModLoad: 000007fe`fd100000 000007fe`fd107000   C:\Windows\System32\wship6.dll
HEAP[julia.exe]: Invalid address specified to RtlFreeHeap( 0000000000F30000, 00000000001A47B0 )
(1613d0.1613a4): Break instruction exception - code 80000003 (first chance)
ntdll!EtwpCreateEtwThread+0x1721:
00000000`77bff0c1 cc              int     3
0:000> k
Child-SP          RetAddr           Call Site
00000000`00d5f3e8 00000000`77be8331 ntdll!EtwpCreateEtwThread+0x1721
00000000`00d5f3f0 00000000`77c19a39 ntdll!MD5Final+0x6d81
00000000`00d5f430 00000000`77bbdb50 ntdll!LdrGetFileNameFromLoadAsDataTable+0x1949
00000000`00d5f490 00000000`77ba40fd ntdll!RtlIsDosDeviceName_U+0x14130
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\Windows\system32\msvcrt.dll - 
00000000`00d5f7d0 000007fe`ffa510c8 ntdll!RtlAllocateHeap+0xd9d
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for D:\Software\Julia-0.4.0-dev-ef062117ba\bin\libjulia.dll - 
00000000`00d5f850 00000000`6bf9adea msvcrt!free+0x1c
00000000`00d5f880 00000000`6bfa47ed libjulia!jl_f_new_expr+0x33a
00000000`00d5fb10 00000000`6bf9dcee libjulia!jl_preload_sysimg_so+0x8d
00000000`00d5fba0 00000000`6bf9ec1d libjulia!jl_get_builtin_hooks+0x5ee
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for julia.exe - 
00000000`00d5fd50 00000000`0040245f libjulia!julia_init+0xd
00000000`00d5fd90 00000000`004013ca julia+0x245f
00000000`00d5fe60 00000000`004014f8 julia+0x13ca
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\Windows\system32\kernel32.dll - 
00000000`00d5ff30 00000000`77a459ed julia+0x14f8
00000000`00d5ff60 00000000`77b7c541 kernel32!BaseThreadInitThunk+0xd
00000000`00d5ff90 00000000`00000000 ntdll!RtlUserThreadStart+0x21
0:000> g
HEAP[julia.exe]: Invalid address specified to RtlFreeHeap( 0000000000F30000, 00000000001A4990 )
(1613d0.1613a4): Break instruction exception - code 80000003 (first chance)
ntdll!EtwpCreateEtwThread+0x1721:
00000000`77bff0c1 cc              int     3
0:000> k
Child-SP          RetAddr           Call Site
00000000`00d5f3e8 00000000`77be8331 ntdll!EtwpCreateEtwThread+0x1721
00000000`00d5f3f0 00000000`77c19a39 ntdll!MD5Final+0x6d81
00000000`00d5f430 00000000`77bbdb50 ntdll!LdrGetFileNameFromLoadAsDataTable+0x1949
00000000`00d5f490 00000000`77ba40fd ntdll!RtlIsDosDeviceName_U+0x14130
00000000`00d5f7d0 000007fe`ffa510c8 ntdll!RtlAllocateHeap+0xd9d
00000000`00d5f850 00000000`6bf9adea msvcrt!free+0x1c
00000000`00d5f880 00000000`6bfa47ed libjulia!jl_f_new_expr+0x33a
00000000`00d5fb10 00000000`6bf9dcee libjulia!jl_preload_sysimg_so+0x8d
00000000`00d5fba0 00000000`6bf9ec1d libjulia!jl_get_builtin_hooks+0x5ee
00000000`00d5fd50 00000000`0040245f libjulia!julia_init+0xd
00000000`00d5fd90 00000000`004013ca julia+0x245f
00000000`00d5fe60 00000000`004014f8 julia+0x13ca
00000000`00d5ff30 00000000`77a459ed julia+0x14f8
00000000`00d5ff60 00000000`77b7c541 kernel32!BaseThreadInitThunk+0xd
00000000`00d5ff90 00000000`00000000 ntdll!RtlUserThreadStart+0x21

(in WinDbg I can continue through these errors and get to the REPL prompt)

Based on seeing jl_get_builtin_hooks, I'm mildly suspicious of #10714, but I can't justify the suspicion since we should not be compiling the new readlink function at startup. Given the timing, the tuple overhaul seems more likely, but I did build and run ef06211 in release mode last night.

[ihnorton]

(I won't be able to debug further until tonight, but opening this up in case someone has time to debug or sees something obvious)

[pao]

Confirmed (using c384fce build downloaded from the bot minutes ago).

[tkelman]

Something is wrong in building the docs (python packaging is eternally aggravating) so I can't do make install in a source build to try to reproduce this.

edit: changing this line to "$(shell ./find_python2)" $< --no-setuptools $@ fixed my build issue, for whatever reason. Still can't reproduce the problem in a source build.

[tkelman]

Reproduced locally via make dist, the root cause is not having sys.dll present. We currently don't, but should, test on AppVeyor that Julia can start without sys.dll present. Once this is fixed I'll add a test for that, like we do on Travis.

This was introduced by #10832, cc @garrison, bisect log:

Tony@TK-samsung ~/julia
$ git bisect bad
ada46544e3016a247eff895f3ba66be91b9189bb is the first bad commit
commit ada46544e3016a247eff895f3ba66be91b9189bb
Author: Jim Garrison <jim@garrison.cc>
Date:   Wed Apr 15 08:21:55 2015 -0700

    Fix #10829

    The change in 11a0f43 made the assumption that `handle->errmsg`
    would be initialized to NUL.  This is true within julia itself, but
    BinDeps violates this assumption.  The current commit changes things
    to fix the memory leak at its actual source, in `jl_load_dynamic_library_`.

:040000 040000 48ffb34fe1f4864956f1a40eb2fbb9ddba23b45e 3ec5e0e0e72475474d9d3d331c2d76a50868463a M src

Tony@TK-samsung ~/julia
$ git bisect log
git bisect start
# bad: [c384fce796b41d08c4e530f04b3166bf4e8d96f7] Merge branch 'master' of github.com:JuliaLang/julia
git bisect bad c384fce796b41d08c4e530f04b3166bf4e8d96f7
# good: [2226a6d52e9c2556367a42ab0499d5d0c22cc8cf] Really nix ambiguity warning in new indexing behavior
git bisect good 2226a6d52e9c2556367a42ab0499d5d0c22cc8cf
# good: [b2565251315bc02fbca231fa32afccdc4897c6fa] Merge pull request #10845 from PalladiumConsulting/sg/llvm-ostream
git bisect good b2565251315bc02fbca231fa32afccdc4897c6fa
# bad: [58cef56f523698018a9e81519022689dd87e06db] Merge pull request #10714 from peter1000/libuv_readlink_function
git bisect bad 58cef56f523698018a9e81519022689dd87e06db
# good: [c40bbc60460715c4c91b044f73dd5bc7bc337852] grisu ignores n in SHORTEST mode, so _show needs to also
git bisect good c40bbc60460715c4c91b044f73dd5bc7bc337852
# bad: [3cda7a5fe774876511639090d54c538cb1b0700b] Merge pull request #10832 from garrison/fix-10829
git bisect bad 3cda7a5fe774876511639090d54c538cb1b0700b
# bad: [ada46544e3016a247eff895f3ba66be91b9189bb] Fix #10829
git bisect bad ada46544e3016a247eff895f3ba66be91b9189bb
# first bad commit: [ada46544e3016a247eff895f3ba66be91b9189bb] Fix #10829

[garrison]

2949b1e looks good to me -- sorry about this.

[garrison]

Actually, I am surprised the problem did not show up beginning with 11a0f43. But the fix looks correct.