Nightly -pc-windows-gnu missing DLL crash

[retep998]

First found on appveyor:
https://ci.appveyor.com/project/retep998/winapi-rs/build/1.0.724/job/qiwg0jsw19fyb0b4
It appears that the libstdc++-6.dll which rustc_llvm-fd663c41.dll depends on has picked up a dependency on libwinpthread-1.dll.
Issue does not affect -msvc targets.
Last known good version was rustc 1.8.0-nightly (c894ff6b1 2016-02-20).
First version where issue was observed rustc 1.8.0-nightly (c92e910c1 2016-02-21).
Offending diff range is c894ff6...c92e910

cc @alexcrichton @vadimcn

[retep998]

Welp, it's just a missing DLL.

I'm also tempted to blame rust-lang-deprecated/rust-buildbot@8faa011

[retep998]

It is specifically libstdc++-6.dll which is to blame.

[vadimcn]

Did the mingw toolchain get changed from "win32" to "pthreads" flavor?

[alexcrichton]

Yes it looks like the wrong libstdc++-6.dll is getting slurped up by make-win-dist.py. On the bots there are two copies installed:

c:\mingw-w64\i686-4.9.1-win32-dwarf-rt_v3-rev1\mingw32\bin\libstdc++-6.dll
c:\msys64\mingw32\bin\libstdc++-6.dll

The former is the one we want to ship, where the latter is what we're accidentally shipping (verified by just checking the md5sum of what's on the bots and what's installed locally). I don't understand why we're slurping up this copy, however. When the dist step is run, the PATH on the bots looks like:

  PATH=c:\mingw-w64\i686-4.9.1-win32-dwarf-rt_v3-rev1\mingw32\bin;c:\msys64\mingw32\bin;c:\msys64\usr\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\msys64\usr\bin;c:\python27;c:\python27\scripts;c:\program files (x86)\inno setup 5;c:\program files (x86)\CMake\bin;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\;C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files (x86)\Microsoft SDKs\TypeScript\1.0\

When we try to find the libstdc++-6.dll file we look it up in bin_path which should just be a copy of PATH.

@vadimcn do you have any ideas for what might be going on here? I recently changed buildbot to move the c:\msys64\mingw32\bin\ path farther forward in PATH. That was done to ensure that rustbuild could use a non-broken python (the one in c:\msys64\usr\bin), but the gcc path is still up front, so I have no idea why it's picking up the wrong one.

Do you know if this normpath call somehow doesn't like non-msys-rooted paths or something like that?

[alexcrichton]

Whelp, yet another reason to burn mingw with fire:

$ PATH=a /mingw32/bin/python -c 'import os; print os.environ["PATH"]'
C:\msys64\mingw32\bin;a

Looks like mingw is doing us a favor and prepending to path for us! The other python installed on the system is:

$ PATH=a /c/Python27/python -c 'import os; print os.environ["PATH"]'
a

Which must also explain why we didn't run into this before. Good lord the pain with mingw never ends...

[alexcrichton]

Should be fixed by rust-lang-deprecated/rust-buildbot@77185c8

[tbourvon]

The issue is back at least starting from rustc 1.18.0-nightly (2b4c91158 2017-04-25)

It was working fine 3 days ago, not sure what the last good version is.

[alexcrichton]

@burningmind do you have logs and/or a description of the environment that causes this? We've changed a few things recently which may result in this, but it's definitely not an intentional regression!

[tbourvon]

@alexcrichton Woops, where are my manners...

On Windows 10, not yet updated to Creators Update, using rustup from the default Windows command line.

I simply updated using rustup update, then the problem started appearing.

[While installing cargo-check]

error: linking with `gcc` failed: exit code: 3221225781
  |
  = note: "gcc" "-Wl,--enable-long-section-names" "-fno-use-linker-plugin" "-Wl,--nxcompat" "-nostdlib" "-m64" "C:\\Users\\Tristan\\.rustup\\toolchains\\nightly-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\crt2.o" "C:\\Users\\Tristan\\.rustup\\toolchains\\nightly-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\rsbegin.o" "-L" "C:\\Users\\Tristan\\.rustup\\toolchains\\nightly-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib" "C:\\Users\\Tristan\\AppData\\Local\\Temp\\cargo-install.KxQfRVp3dGHX\\release\\deps\\cargo_check-2596dae3cd925d41.0.o" "-o" "C:\\Users\\Tristan\\AppData\\Local\\Temp\\cargo-install.KxQfRVp3dGHX\\release\\deps\\cargo_check-2596dae3cd925d41.exe" "-Wl,--gc-sections" "-nodefaultlibs" "-L" "C:\\Users\\Tristan\\AppData\\Local\\Temp\\cargo-install.KxQfRVp3dGHX\\release\\deps" "-L" "C:\\Users\\Tristan\\.rustup\\toolchains\\nightly-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib" "-Wl,-Bstatic" "C:\\Users\\Tristan\\AppData\\Local\\Temp\\cargo-install.KxQfRVp3dGHX\\release\\deps\\libserde_json-bc1ee7fea83c8cc1.rlib" "C:\\Users\\Tristan\\AppData\\Local\\Temp\\cargo-install.KxQfRVp3dGHX\\release\\deps\\libitoa-424ed7f85b6e27db.rlib" "C:\\Users\\Tristan\\AppData\\Local\\Temp\\cargo-install.KxQfRVp3dGHX\\release\\deps\\libnum_traits-fa047955b7cf35d8.rlib" "C:\\Users\\Tristan\\AppData\\Local\\Temp\\cargo-install.KxQfRVp3dGHX\\release\\deps\\libdtoa-97fc67b59f6b10cd.rlib" "C:\\Users\\Tristan\\AppData\\Local\\Temp\\cargo-install.KxQfRVp3dGHX\\release\\deps\\libserde-9de4b0bb905ab369.rlib" "C:\\Users\\Tristan\\.rustup\\toolchains\\nightly-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\libstd-e2451a256f66ac19.rlib" "C:\\Users\\Tristan\\.rustup\\toolchains\\nightly-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\libpanic_unwind-2d4bf02140c11dcb.rlib" "C:\\Users\\Tristan\\.rustup\\toolchains\\nightly-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\libunwind-add7a84d7e82d084.rlib" "C:\\Users\\Tristan\\.rustup\\toolchains\\nightly-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\liblibc-84688accbc86d6b7.rlib" "C:\\Users\\Tristan\\.rustup\\toolchains\\nightly-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\librand-c279a51d66700350.rlib" "C:\\Users\\Tristan\\.rustup\\toolchains\\nightly-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\libcollections-d7bf31a4ca1ea637.rlib" "C:\\Users\\Tristan\\.rustup\\toolchains\\nightly-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\liballoc-fe2e68b21f0bdd7a.rlib" "C:\\Users\\Tristan\\.rustup\\toolchains\\nightly-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\liballoc_system-7fc0381594c93f56.rlib" "C:\\Users\\Tristan\\.rustup\\toolchains\\nightly-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\libstd_unicode-d367c3ba0db49600.rlib" "C:\\Users\\Tristan\\.rustup\\toolchains\\nightly-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\libcore-ea9d77e7c23fe65c.rlib" "C:\\Users\\Tristan\\.rustup\\toolchains\\nightly-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\libcompiler_builtins-91b619d34dd1f5aa.rlib" "-Wl,-Bdynamic" "-l" "advapi32" "-l" "ws2_32" "-l" "userenv" "-l" "shell32" "-Wl,-Bstatic" "-l" "gcc_eh" "-l" "pthread" "-Wl,-Bdynamic" "-lmingwex" "-lmingw32" "-lgcc" "-lmsvcrt" "-luser32" "-lkernel32" "C:\\Users\\Tristan\\.rustup\\toolchains\\nightly-x86_64-pc-windows-gnu\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\rsend.o"
  = note:

error: aborting due to previous error

error: failed to compile `cargo-check v0.2.2`, intermediate artifacts can be found at `C:\Users\Tristan\AppData\Local\Temp\cargo-install.KxQfRVp3dGHX`

Caused by:
  Could not compile `cargo-check`.

To learn more, run the command again with --verbose.

[retep998]

Yep, the bundled mingw does not come with the necessary DLLs.

[alexcrichton]

Thanks for the info @burningmind! Can you confirm as well that you're using rustup?

It looks like we're shipping the right dlls but when we invoke the gcc.exe that's bundled the right dlls aren't in PATH, so they're not found correctly.

[tbourvon]

@alexcrichton I am indeed using rustup. (unless that was directed to @retep998 )

[alexcrichton]

Thanks @burningmind.

I'll work on getting this fixed on nightly. In the meantime, if you'd like to get things working again, I'd recommend one of:

• Pin to an older nightly
• Cross-compile from MSVC to GNU inside of an MSYS shell (w/ gcc installed from elsewhere)
• Manually add the nightly toolchain's bin folder to your PATH, but this would likely cause lots of other problems so I'd go with one of the prior two if I were you

[alexcrichton]

@burningmind I've opened #41571 for this