Prebuilt powerpc64-unknown-linux-gnu binaries require Power ISA v.2.07

[zeldin]

The prebuild rustc binary for powerpc64-unknown-linux-gnu uses the lxvw4x instruction only available in Power ISA v.2.07 and newer. I'm on a PA6T system which implements Power ISA v.2.04, and thus I get an Illegal instruction when trying to run the binary. For bootstrapping purposes it would be good if the prebuilt binaries used only Power ISA v.2.03 instructions so that they work on all 64-bit implementations.

[alexcrichton]

I'm not particularly familiar with the powerpc64 architecture myself, but this is likely due to how we configure powerpc64 by default and LLVM has this feature enabled by default. We can differ from the LLVM defaults, however, if older systems are more common!

[zeldin]

Hm. In clang (3.8.1) you need to specify either -mcpu=power8 or -mpower8-vector to enable these instructions. The default is -mcpu=ppc64 and -mno-power8-vector. But maybe defaults are different when using LLVM directly?

[zeldin]

Looking at Target/PowerPC/PPC.td in the LLVM sources, the instruction is enabled by the ProcessorModels "pwr8" and "ppc64le", but not by "ppc64". No idea how the default ProcessorModel is chosen if none is specified...

[sanxiyn]

If you look at the file @alexcrichton linked, base.cpu is set to "ppc64". This is what is used to choose ProcessorModel. base.cpu defaults to "generic".

[zeldin]

Then it's weird that the instruction is generated, since that ProcessorModel does not have the required FeatureVSX. Well, I'm going to cross compile rustc on Linux/AArch64 (which has a working beta available to bootstrap the process) so that I can investigate more directly.

[zeldin]

Ok, it's becoming more clear now. The offending instruction does not come from compilation of Rust code at all, it is in fact C++ code from src/llvm/lib/Support/CommandLine.cpp. This file is compiled with

powerpc-linux-gnu-g++ -DGTEST_HAS_RTTI=0 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I.../build/build/powerpc64-unknown-linux-gnu/llvm/build/lib/Support -I.../src/llvm/lib/Support -I.../build/build/powerpc64-unknown-linux-gnu/llvm/build/include -I.../src/llvm/include -ffunction-sections -fdata-sections -m64 -fPIC -fPIC -fvisibility-inlines-hidden -Wall -W -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wno-maybe-uninitialized -Wdelete-non-virtual-dtor -Wno-comment -Werror=date-time -std=c++11 -ffunction-sections -fdata-sections -O3 -DNDEBUG -fno-exceptions -fno-rtti -o CMakeFiles/LLVMSupport.dir/CommandLine.cpp.o -c .../src/llvm/lib/Support/CommandLine.cpp

For me this does not produce any VSX instructions, so I suspect the cross toolchain on the build server has been configure with inappropriate defaults.

[alexcrichton]

@zeldin interesting! Our toolchains come from apt repos, so maybe one of those have different defaults?

[zeldin]

Fascinating. I downloaded the g++-5-powerpc64-linux-gnu package from Debian testing, using the following entry in /etc/apt/sources.list:

deb http://ftp.se.debian.org/debian/ testing main non-free contrib

It had -mppc64 as the default:

*asm_default:
-mppc%{!m32:64}

(excerpt of the output of powerpc64-linux-gnu-g++-5 -dumpspecs), and when cross-compiling rustc I did not get any VSX instructions, and the resulting rustc-1.15.0-dev-powerpc64-unknown-linux-gnu.tar.gz actually runs on my PA6T system (well, if I try to compile anything it gets an assertion failure, but that seems like a separate problem)...

What apt repo are you using? Could you check the output from powerpc64-linux-gnu-g++-5 -dumpspecs on the build machine?

[alexcrichton]

@zeldin the build machine is actually that docker image which was most recently built as alexcrichton/rust-slave-linux-cross:2016-11-11 so when running

docker run --entrypoint bash alexcrichton/rust-slave-linux-cross:2016-11-11 -c 'powerpc64-linux-gnu-g++-5 -dumpspecs'

I get this output

[zeldin]

Most peculiar. I don't see anything in there that would turn on POWER8 instructions by default...

[zeldin]

Well, at least I can reproduce the issue by running the g++ command inside the docker image. I also tried powerpc64-linux-gnu-g++-5 --version and got the following

powerpc64-linux-gnu-g++-5 (Ubuntu/IBM 5.4.0-6ubuntu1~16.04.1) 5.4.0 20160609

Ok, so apparently some IBM patches have been applied to the toolchain? Might be relevant...

[zeldin]

I can also confirm that passing -mcpu=powerpc64 explicitly to g++ fixes the issue. So the defaults have indeed been tampered with.

[zeldin]

So, I guess my proposal would be to add

CFLAGS_powerpc64_unknown_linux_gnu=-mcpu=powerpc64

to the environment in the build script. That should fix it.

[zeldin]

Created a pull-request rust-lang-deprecated/rust-buildbot#145 for this solution.

[zeldin]

rust-lang-deprecated/rust-buildbot#149 appears to have resolved the root issue, just waiting for the updated builds to hit the regular download page now.

[zeldin]

Normal nightlies are now working (tested with 1.17.0-nightly (668864d9e 2017-02-16)), closing the issue.