Run MIR-libstd tests for other architectures #202

RalfJung · 2017-06-21T21:03:23Z

Currently, Travis runs MIR-libstd tests only for the host architecture. Ideally, this should also be done for other architectures.

oli-obk · 2017-08-02T13:41:03Z

I think this is blocked on rust-lang/rust#36501, because we can't build apple/windows jemalloc on linux

RalfJung · 2017-08-02T18:36:52Z

Right so we want to build libstd with the system allocator, so the only code that has to be cross-compiled is Rust code.
xargo says that needs something like rust-lang/rust#37975.

oli-obk · 2017-08-03T14:37:16Z

If rust-lang/rust#43628 goes through in its current form, we'd be all set, no more need for xargo

RalfJung · 2017-08-04T00:09:01Z

However, I will soon want for miri to be tested against a libstd that also uses -Zmir-emit-validate=1, so this wouldn't really help.

oli-obk · 2017-08-04T05:34:21Z

We can use lld as the linker and we inject our own jemalloc crate which doesn't do anything.

oli-obk · 2017-08-04T05:45:48Z

Or we use features = ["force_alloc_system"] in xargo.toml.

RalfJung · 2017-08-04T05:48:24Z

Or we use features = ["force_alloc_system"] in xargo.toml.

According to xargo README, that doesn't actually work, exactly due to lack of rust-lang/rust#37975. I did not investigate this further yet.

RalfJung · 2017-08-04T05:55:48Z

Hm, I suppose force_alloc_system is newer. However, just setting that in Xargo does not work:

    Updating registry `https://github.com/rust-lang/crates.io-index`
   Compiling gcc v0.3.51
   Compiling libc v0.2.29
   Compiling core v0.0.0 (file:///home/r/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore)
   Compiling libc v0.0.0 (file:///home/r/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/rustc/libc_shim)
   Compiling unwind v0.0.0 (file:///home/r/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libunwind)
   Compiling filetime v0.1.10
   Compiling build_helper v0.1.0 (file:///home/r/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/build_helper)
   Compiling compiler_builtins v0.0.0 (file:///home/r/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/rustc/compiler_builtins_shim)
   Compiling std v0.0.0 (file:///home/r/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd)
   Compiling cmake v0.1.24
   Compiling rustc_lsan v0.0.0 (file:///home/r/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/librustc_lsan)
   Compiling rustc_asan v0.0.0 (file:///home/r/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/librustc_asan)
   Compiling rustc_msan v0.0.0 (file:///home/r/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/librustc_msan)
   Compiling rustc_tsan v0.0.0 (file:///home/r/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/librustc_tsan)
   Compiling std_unicode v0.0.0 (file:///home/r/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd_unicode)
   Compiling rand v0.0.0 (file:///home/r/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/librand)
   Compiling panic_abort v0.0.0 (file:///home/r/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libpanic_abort)
   Compiling alloc v0.0.0 (file:///home/r/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/liballoc)
   Compiling panic_unwind v0.0.0 (file:///home/r/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libpanic_unwind)
   Compiling alloc_system v0.0.0 (file:///home/r/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/liballoc_system)
   Compiling collections v0.0.0 (file:///home/r/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcollections)
error[E0463]: can't find crate for `alloc_jemalloc`

error: aborting due to previous error

error: Could not compile `std`.

RalfJung · 2017-08-04T06:24:00Z

I reported this as rust-lang/rust#43637. I feel like something fishy is going on...

RalfJung · 2017-08-19T11:40:10Z

The good news is that I locally managed to cross-compile libstd without having a C toolchain for the target! The bad news is that I have no idea how to make that work with the upstream rust-src component... here's what I did:

Compile rustc stage1 locally (currently you have to also apply use gcc::Build rather than deprecated gcc::Config rust#43975 to not fail later).
Make that stage1 the toolchain we use.
Patch the rustc source folder: In $RUSTDIR/src/rustc/compiler_builtins_shim/Cargo.toml, remove "c" from the default features. In $RUSTDIR/src/libstd/Cargo.toml, remove "dylib" from the crate types.
In miri's source, cd xargo. Edit Xargo.toml to remove the "jemalloc" feature. (You can remove "panic_unwind" as well, it doesn't make a difference.)
XARGO_RUST_SRC=$RUSTDIR/src/ RUSTFLAGS='-Zalways-encode-mir -Zmir-emit-validate=1' xargo build --target i686-unknown-linux-gnu should succeed now!

Unfortunately, it seems cross-compiling a dylib does need a C toolchain for the target, which is why I had to not build libstd as a dylib. It also doesn't seem to be possible to change the crate type depending on features (though maybe build scripts can do something here?), which means there currently is no way to upstream this.

libstd being a dylib really is a huge pain. :/ The dylib build fails with

     Running `rustc --crate-name std /home/r/src/rust/rustc.2/src/libstd/lib.rs --crate-type dylib --crate-type rlib --emit=dep-info,link -C prefer-dynamic -C opt-level=3 --cfg 'feature="force_alloc_system"' --cfg 'feature="panic_unwind"' -C metadata=6425f2b7e4e24bbc --out-dir /tmp/xargo.Ux1FcrLpYOVz/target/i686-unknown-linux-gnu/release/deps --target i686-unknown-linux-gnu -L dependency=/tmp/xargo.Ux1FcrLpYOVz/target/i686-unknown-linux-gnu/release/deps -L dependency=/tmp/xargo.Ux1FcrLpYOVz/target/release/deps --extern panic_abort=/tmp/xargo.Ux1FcrLpYOVz/target/i686-unknown-linux-gnu/release/deps/libpanic_abort-aca288d1876270cd.rlib --extern compiler_builtins=/tmp/xargo.Ux1FcrLpYOVz/target/i686-unknown-linux-gnu/release/deps/libcompiler_builtins-8121025650d9f1cf.rlib --extern alloc_system=/tmp/xargo.Ux1FcrLpYOVz/target/i686-unknown-linux-gnu/release/deps/liballoc_system-0182c16197ecc71c.rlib --extern libc=/tmp/xargo.Ux1FcrLpYOVz/target/i686-unknown-linux-gnu/release/deps/liblibc-38f9b1efe225e386.rlib --extern alloc=/tmp/xargo.Ux1FcrLpYOVz/target/i686-unknown-linux-gnu/release/deps/liballoc-25e0045b24994582.rlib --extern std_unicode=/tmp/xargo.Ux1FcrLpYOVz/target/i686-unknown-linux-gnu/release/deps/libstd_unicode-8d9513943e1c6875.rlib --extern core=/tmp/xargo.Ux1FcrLpYOVz/target/i686-unknown-linux-gnu/release/deps/libcore-b085860ac99f8331.rlib --extern collections=/tmp/xargo.Ux1FcrLpYOVz/target/i686-unknown-linux-gnu/release/deps/libcollections-b1e405eb98a3aeaf.rlib --extern rand=/tmp/xargo.Ux1FcrLpYOVz/target/i686-unknown-linux-gnu/release/deps/librand-d5ae1183151f520b.rlib --extern unwind=/tmp/xargo.Ux1FcrLpYOVz/target/i686-unknown-linux-gnu/release/deps/libunwind-b966fbb31ba936d3.rlib --extern panic_unwind=/tmp/xargo.Ux1FcrLpYOVz/target/i686-unknown-linux-gnu/release/deps/libpanic_unwind-2820ee3cab700d09.rlib -Zalways-encode-mir -Zmir-emit-validate=1 --sysroot /home/r/.xargo -Z force-unstable-if-unmarked -l dl -l rt -l pthread`
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m32" "-L" "/home/r/.xargo/lib/rustlib/i686-unknown-linux-gnu/lib" "/tmp/xargo.Ux1FcrLpYOVz/target/i686-unknown-linux-gnu/release/deps/std.0.o" "-o" "/tmp/xargo.Ux1FcrLpYOVz/target/i686-unknown-linux-gnu/release/deps/libstd.so" "/tmp/xargo.Ux1FcrLpYOVz/target/i686-unknown-linux-gnu/release/deps/std.crate.metadata.o" "/tmp/xargo.Ux1FcrLpYOVz/target/i686-unknown-linux-gnu/release/deps/std.crate.allocator.o" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-nodefaultlibs" "-L" "/tmp/xargo.Ux1FcrLpYOVz/target/i686-unknown-linux-gnu/release/deps" "-L" "/tmp/xargo.Ux1FcrLpYOVz/target/release/deps" "-L" "/home/r/.xargo/lib/rustlib/i686-unknown-linux-gnu/lib" "-l" "dl" "-l" "rt" "-l" "pthread" "-Wl,-Bstatic" "-Wl,--whole-archive" "/tmp/rustc.QgAxzSnrdFRC/libpanic_unwind-2820ee3cab700d09.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.QgAxzSnrdFRC/libunwind-b966fbb31ba936d3.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.QgAxzSnrdFRC/librand-d5ae1183151f520b.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.QgAxzSnrdFRC/liballoc_system-0182c16197ecc71c.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.QgAxzSnrdFRC/liblibc-38f9b1efe225e386.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.QgAxzSnrdFRC/liballoc-25e0045b24994582.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.QgAxzSnrdFRC/libstd_unicode-8d9513943e1c6875.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.QgAxzSnrdFRC/libcore-b085860ac99f8331.rlib" "-Wl,--no-whole-archive" "/tmp/rustc.QgAxzSnrdFRC/libcompiler_builtins-8121025650d9f1cf.rlib" "-Wl,-Bdynamic" "-l" "gcc_s" "-l" "c" "-l" "m" "-l" "rt" "-l" "pthread" "-l" "util" "-shared"
  = note: /usr/bin/ld: cannot find crti.o: No such file or directory
          /usr/bin/ld: cannot find -ldl
          /usr/bin/ld: cannot find -lrt
          /usr/bin/ld: cannot find -lpthread
          /usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-linux-gnu/6/libgcc.a when searching for -lgcc
          /usr/bin/ld: cannot find -lgcc
          collect2: error: ld returned 1 exit status

RalfJung · 2017-08-19T11:55:24Z

miri still doesn't run unpatched test cases after this:

$ MIRI_SYSROOT=~/.xargo/ cargo run --bin miri -- --target i686-unknown-linux-gnu tests/run-pass-fullmir/heap.rs 
    Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
     Running `target/debug/miri --target i686-unknown-linux-gnu tests/run-pass-fullmir/heap.rs`
error[E0463]: can't find crate for `alloc_jemalloc`

error: aborting due to previous error

This is fixed by adding the following to the test source:

#![feature(alloc_system, allocator_api, global_allocator)]
extern crate alloc_system;
#[global_allocator]
static ALLOC: alloc_system::System = alloc_system::System;

This still fails, but that's because the allocator API emulation in fn_call.rs is too tied to particular symbol names.

I don't think there is a way to set the default allocator from outside the source... so even if we managed to get the right libstd, we'd still have to add this silly prelude to every single test.

oli-obk · 2017-09-16T13:52:47Z

I don't think there is a way to set the default allocator from outside the source... so even if we managed to get the right libstd, we'd still have to add this silly prelude to every single test.

miri can insert that prelude to all compilations

oli-obk · 2017-09-16T13:56:32Z

in the short term we can just add an additional travis matrix for 32 bit

Appveyor already has bot 64 and 32 bit

Macs are blocked on me not having a mac and no mac users being interested enough in miri to get it working with full mir

RalfJung · 2018-12-02T12:03:31Z

in the short term we can just add an additional travis matrix for 32 bit

Travis seems to only support 64bit :/
We could try using Docker though, maybe?

RalfJung · 2018-12-02T13:00:07Z

The allocator trouble is luckily gone now that the system allocator is the default.

The compiler_builtin problem should be solved by rust-lang/rust#56435.

However, that still leaves creating the dylib as a problem: Locally, this works for cross-building from 64bit Linux to 32bit Linux (likely because I have the right gcc toolchain installed), but cross-building to i686-pc-windows-gnu fails saying

error: linking with `gcc` failed: exit code: 1
  |
  = note: "gcc" "-Wl,--enable-long-section-names" "-fno-use-linker-plugin" "-Wl,--nxcompat" "-nostdlib" "-Wl,--large-address-aware" "/home/r/.xargo/lib/rustlib/i686-pc-windows-gnu/lib/dllcrt2.o" "/home/r/.xargo/lib/rustlib/i686-pc-windows-gnu/lib/rsbegin.o" "-L" "/home/r/.xargo/lib/rustlib/i686-pc-windows-gnu/lib" "/tmp/xargo.oIZd2fSCE80c/target/i686-pc-windows-gnu/release/deps/std.std.7jdjnkt3-cgu.0.rcgu.o" "/tmp/xargo.oIZd2fSCE80c/target/i686-pc-windows-gnu/release/deps/std.std.7jdjnkt3-cgu.1.rcgu.o" "/tmp/xargo.oIZd2fSCE80c/target/i686-pc-windows-gnu/release/deps/std.std.7jdjnkt3-cgu.10.rcgu.o" "/tmp/xargo.oIZd2fSCE80c/target/i686-pc-windows-gnu/release/deps/std.std.7jdjnkt3-cgu.11.rcgu.o" "/tmp/xargo.oIZd2fSCE80c/target/i686-pc-windows-gnu/release/deps/std.std.7jdjnkt3-cgu.12.rcgu.o" "/tmp/xargo.oIZd2fSCE80c/target/i686-pc-windows-gnu/release/deps/std.std.7jdjnkt3-cgu.13.rcgu.o" "/tmp/xargo.oIZd2fSCE80c/target/i686-pc-windows-gnu/release/deps/std.std.7jdjnkt3-cgu.14.rcgu.o" "/tmp/xargo.oIZd2fSCE80c/target/i686-pc-windows-gnu/release/deps/std.std.7jdjnkt3-cgu.15.rcgu.o" "/tmp/xargo.oIZd2fSCE80c/target/i686-pc-windows-gnu/release/deps/std.std.7jdjnkt3-cgu.2.rcgu.o" "/tmp/xargo.oIZd2fSCE80c/target/i686-pc-windows-gnu/release/deps/std.std.7jdjnkt3-cgu.3.rcgu.o" "/tmp/xargo.oIZd2fSCE80c/target/i686-pc-windows-gnu/release/deps/std.std.7jdjnkt3-cgu.4.rcgu.o" "/tmp/xargo.oIZd2fSCE80c/target/i686-pc-windows-gnu/release/deps/std.std.7jdjnkt3-cgu.5.rcgu.o" "/tmp/xargo.oIZd2fSCE80c/target/i686-pc-windows-gnu/release/deps/std.std.7jdjnkt3-cgu.6.rcgu.o" "/tmp/xargo.oIZd2fSCE80c/target/i686-pc-windows-gnu/release/deps/std.std.7jdjnkt3-cgu.7.rcgu.o" "/tmp/xargo.oIZd2fSCE80c/target/i686-pc-windows-gnu/release/deps/std.std.7jdjnkt3-cgu.8.rcgu.o" "/tmp/xargo.oIZd2fSCE80c/target/i686-pc-windows-gnu/release/deps/std.std.7jdjnkt3-cgu.9.rcgu.o" "-o" "/tmp/xargo.oIZd2fSCE80c/target/i686-pc-windows-gnu/release/deps/std.dll" "/tmp/xargo.oIZd2fSCE80c/target/i686-pc-windows-gnu/release/deps/std.1uhyejbk9z42l5f3.rcgu.o" "/tmp/xargo.oIZd2fSCE80c/target/i686-pc-windows-gnu/release/deps/std.1pxx9pgyxsed3137.rcgu.o" "-nodefaultlibs" "-L" "/tmp/xargo.oIZd2fSCE80c/target/i686-pc-windows-gnu/release/deps" "-L" "/tmp/xargo.oIZd2fSCE80c/target/release/deps" "-L" "/home/r/.xargo/lib/rustlib/i686-pc-windows-gnu/lib" "-ladvapi32" "-lws2_32" "-luserenv" "-lshell32" "-Wl,-Bstatic" "-Wl,--whole-archive" "/tmp/rustcOs5ByS/libpanic_unwind-68b5125fdea5b3a9.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustcOs5ByS/libunwind-59bf8229558b161a.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustcOs5ByS/liblibc-c2accf2fea9af144.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustcOs5ByS/liballoc-621fa7e1ce74615e.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustcOs5ByS/libcore-79b7f582d7f60a6a.rlib" "-Wl,--no-whole-archive" "/tmp/rustcOs5ByS/libcompiler_builtins-c95cbb7129342495.rlib" "-lgcc_eh" "-lpthread" "-shared" "-Wl,-Bdynamic" "-lmingwex" "-lmingw32" "-lgcc" "-lmsvcrt" "-lmsvcrt" "-luser32" "-lkernel32" "/home/r/.xargo/lib/rustlib/i686-pc-windows-gnu/lib/rsend.o"
  = note: /usr/bin/ld: unrecognized option '--enable-long-section-names'
          /usr/bin/ld: use the --help option for usage information
          collect2: error: ld returned 1 exit status
          

error: aborting due to previous error

error: Could not compile `std`.

RalfJung · 2018-12-02T19:56:04Z

Cc rust-lang/rust#56443

memoryruins · 2018-12-02T20:40:23Z

https://github.com/rust-embedded/cross utilizes docker on travis and runs tests in QEMU.
Although there are some caveats, it enables testing many more targets.

Testing support is more complicated. It relies on QEMU user emulation, so testing may sometimes fail due to QEMU bugs and not because there's a bug in the crate. That being said, cross test is assumed to "work" (test column in the table below) if it can successfully run compiler-builtins test suite.

RalfJung mentioned this issue Jun 23, 2017

Drop support for missing MIR #211

Closed

2 tasks

RalfJung mentioned this issue Jul 18, 2017

Cannot run with libstd MIR on macOS due to stack guards #252

Closed

RalfJung mentioned this issue Aug 14, 2018

Rustup #425

Merged

RalfJung added the C-enhancement Category: a PR with an enhancement or an issue tracking an accepted enhancement label Nov 17, 2018

RalfJung mentioned this issue Dec 10, 2018

Support building and running with full MIR on foreign architectures, drop support for missing MIR #566

Merged

oli-obk closed this as completed in #566 Dec 10, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run MIR-libstd tests for other architectures #202

Run MIR-libstd tests for other architectures #202

RalfJung commented Jun 21, 2017

oli-obk commented Aug 2, 2017

RalfJung commented Aug 2, 2017

oli-obk commented Aug 3, 2017

RalfJung commented Aug 4, 2017

oli-obk commented Aug 4, 2017

oli-obk commented Aug 4, 2017

RalfJung commented Aug 4, 2017

RalfJung commented Aug 4, 2017

RalfJung commented Aug 4, 2017

RalfJung commented Aug 19, 2017 •

edited

Loading

RalfJung commented Aug 19, 2017

oli-obk commented Sep 16, 2017 •

edited

Loading

oli-obk commented Sep 16, 2017

RalfJung commented Dec 2, 2018

RalfJung commented Dec 2, 2018

RalfJung commented Dec 2, 2018

memoryruins commented Dec 2, 2018 •

edited

Loading

Run MIR-libstd tests for other architectures #202

Run MIR-libstd tests for other architectures #202

Comments

RalfJung commented Jun 21, 2017

oli-obk commented Aug 2, 2017

RalfJung commented Aug 2, 2017

oli-obk commented Aug 3, 2017

RalfJung commented Aug 4, 2017

oli-obk commented Aug 4, 2017

oli-obk commented Aug 4, 2017

RalfJung commented Aug 4, 2017

RalfJung commented Aug 4, 2017

RalfJung commented Aug 4, 2017

RalfJung commented Aug 19, 2017 • edited Loading

RalfJung commented Aug 19, 2017

oli-obk commented Sep 16, 2017 • edited Loading

oli-obk commented Sep 16, 2017

RalfJung commented Dec 2, 2018

RalfJung commented Dec 2, 2018

RalfJung commented Dec 2, 2018

memoryruins commented Dec 2, 2018 • edited Loading

RalfJung commented Aug 19, 2017 •

edited

Loading

oli-obk commented Sep 16, 2017 •

edited

Loading

memoryruins commented Dec 2, 2018 •

edited

Loading