Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Disabling jemalloc fails when doing a local rebuild #39054

Closed
aidanhs opened this issue Jan 14, 2017 · 4 comments · Fixed by #39086
Closed

Disabling jemalloc fails when doing a local rebuild #39054

aidanhs opened this issue Jan 14, 2017 · 4 comments · Fixed by #39086

Comments

@aidanhs
Copy link
Member

aidanhs commented Jan 14, 2017 •
edited
Loading

Edit: see #39054 (comment)

~/Desktop/rust/rust-tmp $ rustc --version
rustc 1.16.0-nightly (1a2ed98d3 2017-01-13)
~/Desktop/rust/rust-tmp $ git show | head -n4
commit b0c52c587fe9ba287053359fff5ed886b7edb27c
Merge: 927c55d 3a79f2e
Author: bors <bors@rust-lang.org>
Date:   Fri Jan 13 07:24:53 2017 +0000
~/Desktop/rust/rust-tmp $ ./configure --enable-local-rust --disable-jemalloc --llvm-root=/usr/lib/llvm-3.8 --disable-codegen-tests
[...]
~/Desktop/rust/rust-tmp $ ./x.py build --stage 0 -j3 -v
[...]
     Running `/home/aidanhs/Desktop/rust/rust-tmp/build/bootstrap/debug/rustc --crate-name std src/libstd/lib.rs --crate-type dylib --crate-type rlib --emit=dep-info,li
nk -C prefer-dynamic -C opt-level=2 --cfg 'feature="backtrace"' --cfg 'feature="panic-unwind"' --cfg 'feature="panic_unwind"' -C metadata=d6c751cd700544ff -C extra-file
name=-d6c751cd700544ff --out-dir /home/aidanhs/Desktop/rust/rust-tmp/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps --target x86_64-unk
nown-linux-gnu -L dependency=/home/aidanhs/Desktop/rust/rust-tmp/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps -L dependency=/home/aidanhs/Desktop/rust/rust-tmp/build/x86_64-unknown-linux-gnu/stage0-std/release/deps --extern panic_unwind=/home/aidanhs/Desktop/rust/rust-tmp/build/x86_64-unknown-linux-g
nu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libpanic_unwind-8c67638596b8d567.rlib --extern libc=/home/aidanhs/Desktop/rust/rust-tmp/build/x86_64-unknown-linux-g
nu/stage0-std/x86_64-unknown-linux-gnu/release/deps/liblibc-3eca0ce6ddae7cf0.rlib --extern rand=/home/aidanhs/Desktop/rust/rust-tmp/build/x86_64-unknown-linux-gnu/stage
0-std/x86_64-unknown-linux-gnu/release/deps/librand-3a9e8dccb0763fd8.rlib --extern alloc_system=/home/aidanhs/Desktop/rust/rust-tmp/build/x86_64-unknown-linux-gnu/stage
0-std/x86_64-unknown-linux-gnu/release/deps/liballoc_system-92088ad3ca26d86d.rlib --extern std_unicode=/home/aidanhs/Desktop/rust/rust-tmp/build/x86_64-unknown-linux-gn
u/stage0-std/x86_64-unknown-linux-gnu/release/deps/libstd_unicode-a77984d556d80178.rlib --extern panic_abort=/home/aidanhs/Desktop/rust/rust-tmp/build/x86_64-unknown-li
nux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libpanic_abort-2a59489bf9653084.rlib --extern collections=/home/aidanhs/Desktop/rust/rust-tmp/build/x86_64-unkn
own-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libcollections-4ee10d7be4610b32.rlib --extern compiler_builtins=/home/aidanhs/Desktop/rust/rust-tmp/build
/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libcompiler_builtins-47800f01f94aef26.rlib --extern alloc=/home/aidanhs/Desktop/rust/rust-tmp
/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/liballoc-3b92c602cb553647.rlib --extern core=/home/aidanhs/Desktop/rust/rust-tmp/build/
x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libcore-d0d04b66da2d3e03.rlib --extern unwind=/home/aidanhs/Desktop/rust/rust-tmp/build/x86_64
-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libunwind-c2bfc8772a2653f6.rlib -L native=/home/aidanhs/Desktop/rust/rust-tmp/build/x86_64-unknown-l
inux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/build/std-89e6156878b3b964/out/.libs --cfg cargobuild -l static=backtrace -l dl -l rt -l pthread -L native=/home/ai
danhs/Desktop/rust/rust-tmp/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/build/compiler_builtins-6ba51432c6685f3f/out`
error[E0463]: can't find crate for `alloc_jemalloc`

error: aborting due to previous error

error: Could not compile `std`.

Seems to get past there fine if I remove --enable-local-rust.
@aidanhs
Copy link
Member Author

aidanhs commented Jan 14, 2017 •
edited
Loading

Ok, rustup default beta-2016-12-20 works. That's the version rustbuild will download if left to its own devices. Bisecting built nightlies as my local rust gives this:

  • nightly-2016-12-19 (rustc 1.15.0-nightly (71c06a56a 2016-12-18)) - works
  • [no nightlies]
  • nightly-2016-12-29 - unrelated errors on some i128/u128 stuff before it gets to the relevant point
  • [no nightlies]
  • nightly-2017-01-04 (rustc 1.16.0-nightly (468227129 2017-01-03)) - broken

@aidanhs
Copy link
Member Author

aidanhs commented Jan 14, 2017 •
edited
Loading

Note that this is really quick to reproduce (~1min), as it happens on the initial build of stage0 std. Here's the full log from ./x.py build --stage 0 -j1:

~/Desktop/rust/rust-tmp $ ./x.py build --stage 0 -j1                                                                                                                    
   Compiling build_helper v0.1.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/build_helper)
   Compiling libc v0.2.17
   Compiling gcc v0.3.40
   Compiling rustc-serialize v0.3.19
   Compiling getopts v0.2.14
   Compiling filetime v0.1.10
   Compiling cmake v0.1.18
   Compiling num_cpus v0.2.13
   Compiling toml v0.1.30
   Compiling bootstrap v0.0.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/bootstrap)
    Finished debug [unoptimized] target(s) in 16.84 secs
Synchronising submodule url for 'src/compiler-rt'
Synchronising submodule url for 'src/jemalloc'
Synchronising submodule url for 'src/liblibc'
Synchronising submodule url for 'src/llvm'
Synchronising submodule url for 'src/rt/hoedown'
Synchronising submodule url for 'src/rust-installer'
HEAD is now at a8fc4c1 Merge pull request #28 from xen0n/preprocessor-firefighting
HEAD is now at 9858987 [WIP] sparc64-linux support
HEAD is now at a3736a0 Merge pull request #6 from intelfx/patch-1
HEAD is now at 4f99485 Merge pull request #54 from brson/docdir
Building stage0 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
   Compiling libc v0.0.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/rustc/libc_shim)
   Compiling core v0.0.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/libcore)
   Compiling alloc v0.0.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/liballoc)
   Compiling build_helper v0.1.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/build_helper)
   Compiling rand v0.0.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/librand)
   Compiling gcc v0.3.40
   Compiling std v0.0.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/libstd)
   Compiling panic_abort v0.0.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/libpanic_abort)
   Compiling compiler_builtins v0.0.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/libcompiler_builtins)
   Compiling unwind v0.0.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/libunwind)
   Compiling alloc_system v0.0.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/liballoc_system)
   Compiling panic_unwind v0.0.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/libpanic_unwind)
   Compiling std_unicode v0.0.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/libstd_unicode)
   Compiling collections v0.0.0 (file:///home/aidanhs/Desktop/rust/rust-tmp/src/libcollections)
error[E0463]: can't find crate for `alloc_jemalloc`

error: aborting due to previous error

error: Could not compile `std`.

To learn more, run the command again with --verbose.


command did not execute successfully: "/home/aidanhs/.cargo/bin/cargo" "build" "-j" "1" "--target" "x86_64-unknown-linux-gnu" "--release" "--features" "panic-unwind backtrace" "--manifest-path" "/home/aidanhs/Desktop/rust/rust-tmp/src/rustc/std_shim/Cargo.toml"
expected success, got: exit code: 101

This issue may get a lot more attention once the stage0 downloaded compiler has its version bumped, as I suspect --disable-jemalloc will be totally broken.

Applying #37975 seems to fix it.

@aidanhs
Copy link
Member Author

aidanhs commented Jan 15, 2017

Ah I see, the more recent compiler is being detected as a local rebuild compiler, so rustbuild assumes it has a compiler with stage1 features and therefore performs a stage1 build, which seems to assume that jemalloc is available.

On a positive note, this should never affect someone not using --enable-local-rust.

@aidanhs
Copy link
Member Author

aidanhs commented Jan 15, 2017 •
edited
Loading

Ok I have a more solid understanding of the issue. Currently, a stage0-out libstd will never link against jemalloc. stage1-out is left to its own devices, linking against the default allocator for the compiler that's just been created (stage0-out) which is jemalloc if enabled, system otherwise.

However, a local rebuild uses --cfg stage1 for building the stage0 libstd, which means it uses the default allocator for the local rust. This tends to be jemalloc...except we've disabled jemalloc, so it hasn't been built!

Something along the lines of #37975 seems like the right approach, but I think some elaboration around the logic in this area is necessary in comments or something.

@aidanhs aidanhs changed the title Disabling jemalloc fails with --enable-local-rust Disabling jemalloc fails with when doing a local rebuild Jan 15, 2017
@aidanhs aidanhs mentioned this issue Jan 15, 2017
Merged
@aidanhs aidanhs changed the title Disabling jemalloc fails with when doing a local rebuild Disabling jemalloc fails when doing a local rebuild Jan 17, 2017
bors added a commit that referenced this issue Jan 21, 2017
@bors
Auto merge of #39086 - aidanhs:aphs-local-rebuild-no-jemalloc, r=alex…
633f38a 
…crichton

Make rustbuild force_alloc_system rather than relying on stage0

This 'fixes' jemalloc-less local rebuilds, where we tell cargo that we're actually stage1 (this only fixes the rustbuild path, since I wasn't enthusiastic to dive into the makefiles).

There should be one effect from this PR: `--enable-local-rebuild --disable-jemalloc` will successfully build a stage0 std (rather than erroring). Ideally I think it'd be nice to specify an allocator preference in Cargo.toml/cargo command line (used when an allocator must be picked i.e. dylibs, not rlibs), but since that's not possible we can make do with a force_alloc_system feature. Sadly this locks you into a single allocator in the build libstd, making any eventual implementation of #38575 not quite right in this edge case, but clearly not many people exercise the combination of these two flags.

This PR is also a substitute for #37975 I think. The crucial difference is that the feature name here is distinct from the jemalloc feature (reused in the previous PR) - we don't want someone to be forced into alloc_system just for disabling jemalloc!

Fixes #39054

r? @alexcrichton
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Make rustbuild force_alloc_system rather than relying on stage0 aidanhs/rust
1 participant
@aidanhs