[DoNotMergeYet] fix cross compiling to ARM when using cmake

[japaric]

The first commit fixes point (c) of rust-lang/rust#32194, and the second commit fixes point (b). See commit messages for details.

The first commit should let us use rustbuild to cross compile rustc for arm(v7)-unknown-linux-gnueabihf and the second commit should let us cross compile for arm-unknown-linux-gnueabi.

I'm still testing rustc cross compilation in the linux-cross docker image, so don't merge it just yet!

r? @alexcrichton

P.S. I wasn't sure to which branch send this PR, so feel free to cherry pick these commits to other branch if I got the wrong one.

[japaric]

Ok, so after cross compiling rustc for the three ARM targets mentioned above, this is the result:

• Cross compilation succeeded for arm-unknown-linux-gnueabi(hf), but failed for armv7-unknown-linux-gnueabihf with:

error: linking with `arm-linux-gnueabihf-gcc` failed: exit code: 1
note: "arm-linux-gnueabihf-gcc" (...)
note: /buildslave/build/build/x86_64-unknown-linux-gnu/stage2-rustc/armv7-unknown-linux-gnueabihf/release/build/rustc_llvm-7ff0c6eea8a7c03c/out/librustllvm.a: member /buildslave/build/build/x86_64-unknown-linux-gnu/stage2-rustc/armv7-unknown-linux-gnueabihf/release/build/rustc_llvm-7ff0c6eea8a7c03c/out/librustllvm.a(ExecutionEngineWrapper.o) in archive is not an object

Which looks like something I've seen before but I don't think I ever figured out why it happens.

• I tested the rustc cross compiled for arm-unknown-linux-gnueabihf by compiling hello world, clap and hyper. Only hello world worked, the other two failed with this LLVM assertion near the beginning of the cargo build:

rustc: /buildslave/rust/src/llvm/lib/IR/Attributes.cpp:876: llvm::AttributeSet llvm::AttributeSet::removeAttributes(llvm::LLVMContext&, unsigned int, llvm::AttributeSet) const: Assertion `!Attrs.hasAttribute(Index, Attribute::Alignment) && "Attempt to change alignment!"' failed.

Backtrace:

#0  __libc_do_syscall () at ../ports/sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:44
#1  0xb67b7f0e in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#2  0xb67ba766 in __GI_abort () at abort.c:89
#3  0xb67b3150 in __assert_fail_base (fmt=0x1 <error: Cannot access memory at address 0x1>, assertion=0xb48f9014 "!Attrs.hasAttribute(Index, Attribute::Alignment) && \"Attempt to change alignment!\"", assertion@entry=0x0, file=0xb48f8684 "/buildslave/rust/src/llvm/lib/IR/Attributes.cpp",
    file@entry=0xb1bff270 "\001", line=876, line@entry=3062325420,
    function=function@entry=0xb48f96b8 <llvm::AttributeSet::removeAttributes(llvm::LLVMContext&, unsigned int, llvm::AttributeSet) const::__PRETTY_FUNCTION__> "llvm::AttributeSet llvm::AttributeSet::removeAttributes(llvm::LLVMContext&, unsigned int, llvm::AttributeSet) const") at assert.c:92
#4  0xb67b31e6 in __GI___assert_fail (assertion=0x0, file=0xb1bff270 "\001", line=3062325420,
    function=0xb48f96b8 <llvm::AttributeSet::removeAttributes(llvm::LLVMContext&, unsigned int, llvm::AttributeSet) const::__PRETTY_FUNCTION__> "llvm::AttributeSet llvm::AttributeSet::removeAttributes(llvm::LLVMContext&, unsigned int, llvm::AttributeSet) const") at assert.c:101
#5  0xb3a5c41c in llvm::AttributeSet::removeAttributes(llvm::LLVMContext&, unsigned int, llvm::AttributeSet) const () from /home/japaric/stage2/lib/librustc_llvm-fbe9f3f0bb11f26d.so
#6  0xb3a9ba34 in LLVMRemoveFunctionAttr () from /home/japaric/stage2/lib/librustc_llvm-fbe9f3f0bb11f26d.so
#7  0xb6166770 in trans::attributes::from_fn_attrs::h90d28c04db141a87Olh () from /home/japaric/stage2/lib/librustc_trans-9ca1d0119dec280b.so
#8  0xb6197058 in trans::base::register_method::h519d6b0fa5938c1d0Wk () from /home/japaric/stage2/lib/librustc_trans-9ca1d0119dec280b.so
#9  0xb6191e0c in trans::base::get_item_val::h1dec98050df0274cmJk () from /home/japaric/stage2/lib/librustc_trans-9ca1d0119dec280b.so
#10 0xb61901f8 in trans::base::trans_item::h0c4d97fad9923c0blpk () from /home/japaric/stage2/lib/librustc_trans-9ca1d0119dec280b.so
#11 0xb61a97e8 in trans::base::TransItemsWithinModVisitor$LT$$u27$a$C$$u20$$u27$tcx$GT$.Visitor$LT$$u27$v$GT$::visit_item::h79cf8d53e2c99a5frpl () from /home/japaric/stage2/lib/librustc_trans-9ca1d0119dec280b.so
#12 0xb61992ec in trans::base::trans_crate::h6d010e65c6a8c4b2R8k () from /home/japaric/stage2/lib/librustc_trans-9ca1d0119dec280b.so
#13 0xb6eb4f58 in driver::phase_4_translate_to_llvm::hd215fe33d54395bdK2a () from /home/japaric/stage2/lib/librustc_driver-c44e3ef3742242cd.so
#14 0xb6eb3bec in driver::compile_input::_$u7b$$u7b$closure$u7d$$u7d$::closure.31135 () from /home/japaric/stage2/lib/librustc_driver-c44e3ef3742242cd.so
#15 0xb6eb0bf4 in driver::phase_3_run_analysis_passes::_$u7b$$u7b$closure$u7d$$u7d$::closure.30157 () from /home/japaric/stage2/lib/librustc_driver-c44e3ef3742242cd.so
#16 0xb6eab7d4 in middle::ty::context::TyCtxt$LT$$u27$tcx$GT$::create_and_enter::h18081228826689638672 () from /home/japaric/stage2/lib/librustc_driver-c44e3ef3742242cd.so
#17 0xb6ea8d18 in driver::phase_3_run_analysis_passes::h18385739884896382119 () from /home/japaric/stage2/lib/librustc_driver-c44e3ef3742242cd.so
#18 0xb6e82224 in driver::compile_input::h315a4573d98af515Pca () from /home/japaric/stage2/lib/librustc_driver-c44e3ef3742242cd.so
#19 0xb6e73bec in run_compiler::h86e1e8e04684a5e9JPc () from /home/japaric/stage2/lib/librustc_driver-c44e3ef3742242cd.so
#20 0xb6e714bc in sys_common::unwind::try::try_fn::h12880112494064645751 () from /home/japaric/stage2/lib/librustc_driver-c44e3ef3742242cd.so
#21 0xb69526d4 in __rust_try () from /home/japaric/stage2/lib/libstd-0399f70ae701a18b.so
#22 0xb695265c in sys_common::unwind::inner_try::h260d452bf9836e26a4t () from /home/japaric/stage2/lib/libstd-0399f70ae701a18b.so
#23 0xb6e71d10 in boxed::F.FnBox$LT$A$GT$::call_box::h15383368847685676869 () from /home/japaric/stage2/lib/librustc_driver-c44e3ef3742242cd.so
#24 0xb695c6c4 in sys::thread::Thread::new::thread_start::h5c227de00475bfdaL5y () from /home/japaric/stage2/lib/libstd-0399f70ae701a18b.so
#25 0xb2476fbc in start_thread (arg=0xb1bff270) at pthread_create.c:314

I don't know where this assertion came from, this was working fine last week! Could the cause be:

• downgrading gcc/glibc versions? I think this is unlikely
• changes in gcc default configuration (armv7-a -> armv6, vfp3-d16 -> vfp)? That sounds like it would change the instructions generated for the libstdc++ and llvm part of rustc but shouldn't change llvm behavior.
• perhaps, that the LLVM submodule got updated?
• some recent change in trans?

I checked if the unofficial rustc binaries have the same problem and they don't', but their latest release is 18 days old so they may not contain the change that could be causing the problem I'm seeing here.

[japaric]

Okay, about the second problem: testing the cross compiled rustc on ARM. I just remember that rustc on ARM doesn't work when it's built with LLVM assertions enabled because it always hits that LLVM assertion about the attributes. That's the reason why, when I was running the ruststrap service, I disabled LLVM assertions on my builds. After the LLVM assertions were disabled rustc was able to build working binaries AFAICT.

Now, rustc on ARM should work even when LLVM assertions are enabled but that's a regression that I never managed to bisect because building ARM compilers took such a long time.

I'm going to confirm that the LLVM assertions are the cause of the problem I'm seeing here and report back.

[alexcrichton]

Thanks @japaric! This is getting significant enough that perhaps we can try to upstream some of these changes? Our fork seems to be diverging a little too much from upstream :(

[japaric]

I'm going to confirm that the LLVM assertions are the cause of the problem I'm seeing here and report back.

Disabling the LLVM assertions "fixed" the rustc for arm-unknown-linux-gnueabihf, I can cargo test hyper natively on ARM.

I can't test the rustc for arm-unknown-linux-gnueabi because I can't find a rootfs for soft float ARM; all of ubuntu core rootfs-es are for hard float ARM. (I wonder how many people out there are running a linux distro using the soft float ABI)

As for armv7-unknown-linux-gnueabihf, I think the problem is that gcc now defaults to -march=armv6 because this didn't happen when gcc was configured with -march=armv7-a. This probably indicates that we are not passing -march to gcc when building some C library, most likely when cross-compiling LLVM.

This is getting significant enough that perhaps we can try to upstream some of these changes? Our fork seems to be diverging a little too much from upstream :(

Does that mean that we have to wait until these changes land upstream? I was hoping to get official rustc binaries for ARM by the end of this week.

[alexcrichton]

bah oh well, if this fixes problems locally it's all pretty easy to rebase so we can deal with if we ever need to. That and once we're gating on this it'll naturally fall out of any upgrade as well :)

Regardless, I shall merge! Do you wanna send a PR to rust-lang/rust to update the submodules?

[japaric]

Do you wanna send a PR to rust-lang/rust to update the submodules?

Done in rust-lang/rust#32280

[parched]

@japaric Doesn't 8169706 break softfp? I haven't tested it, but I assume if you are using softfp and add two doubles it would make a call to __adddf3vfp which would be missing now.

[japaric]

@parched Before that commit we were using Makefiles to build compiler-rt for ARM and those Makefiles didn't compile the adddf3vfp intrinsic. In other words, libcompiler-rt.a for arm*-unknown-linux-gnueabi never contained the __adddf3vfp intrinsic.

if you are using softfp and add two doubles

That operation lowers to __aeabi_dadd. I don't know what operation (if any) lowers to adddf3vfp shrugs.

[parched]

@japaric

That operation lowers to __aeabi_dadd

It shouldn't, provided a double precision fpu is specified. However, from my testing it doesn't use __adddf3vfp, but rather a vmul.fp64 directly. I guess what I mean is then, that these vfp intrinsics are just as valid for *eabi as *eabihf even though they may not get used in either case.

[japaric]

Doesn't 8169706 break softfp?

Oh, I misread. I thought you meant "soft float". None of the arm*-unknown-linux-gnueabi* targets use the softfp ABI though. By softfp were you referring to the arm(v7)-linux-androideabi targets? (I think those are softfp; the presence of Tag_FP_arch and the lack of Tag_ABI_VFP_args in an ELF file cross compiled for that target appears to indicate so).

I guess what I mean is then, that these vfp intrinsics are just as valid for *eabi as *eabihf even though they may not get used in either case.

Reading the LLVM source, it appears these *vfp* intrinsics are only used for "MachO" targets. As far as Rust targets go, these ARM "MachO" targets could be the armv7(s)-apple-ios targets (?).

[parched]

Reading that source it's only for thumb, indeed testing I find llc -mtriple=thumb--macho -mattr=+vfp2 does generate these calls. So such a target would be broken here, but I don't think a target like this would be added to rust. I do think, nevertheless, that it is best work around the problems this is solving another way.

[parched]

Actually I think this can affect current rust, with armv7-apple-ios if you pass +thumb-mode to llvm I think it will generate one of these calls.