Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not attempt to recompile codegen backend(s) with --keep-stage #52360

Merged
merged 2 commits into from
Jul 15, 2018

Conversation

Mark-Simulacrum
Copy link
Member

Previously we'd attempt to recompile them and that would fail since
we've essentially not built the entire compiler yet, or we're faking
that fact. This commit should make us ignore the codegen backend build
as well.

Unlike the other compile steps, there is no CodegenBackendLink step that
we run here, because that is done later as a part of assembling the
final compiler and as an explicit function call.

r? @alexcrichton

I think this may fix or at least assist with #52174.

cc @RalfJung @tinco -- if you can test this patch locally that'd be
amazing; I don't want to recompile for the next couple hours to test it
locally. I don't think it can make the situation worse, and in fact, if
I've interpreted the cause of the failure correctly then this will fix
your problem.

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 14, 2018
@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading https://files.pythonhosted.org/packages/80/2b/ed30c7941731231afb7d3e97368accd2319047123c75fa7dd043ca39be04/awscli-1.15.59-py2.py3-none-any.whl (1.3MB)
    0% |▎                               | 10kB 6.9MB/s eta 0:00:01
    1% |▌                               | 20kB 1.8MB/s eta 0:00:01
    2% |▊                               | 30kB 2.0MB/s eta 0:00:01
    3% |█                               | 40kB 1.8MB/s eta 0:00:01
---
  Downloading https://files.pythonhosted.org/packages/db/c8/7dcf9dbcb22429512708fe3a547f8b6101c0d02137acbd892505aee57adf/colorama-0.3.9-py2.py3-none-any.whl
Collecting botocore==1.10.58 (from awscli)
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading https://files.pythonhosted.org/packages/cb/6e/4d961f0c98007ba077a64626f4e0056e5351cca54653b15d12bb03c32110/botocore-1.10.58-py2.py3-none-any.whl (4.4MB)
    0% |                                | 10kB 42.9MB/s eta 0:00:01
    0% |▏                               | 20kB 32.4MB/s eta 0:00:01
    0% |▎                               | 30kB 38.5MB/s eta 0:00:01
    0% |▎                               | 40kB 35.1MB/s eta 0:00:01
---

[00:03:57] travis_fold:start:tidy
travis_time:start:tidy
tidy check
[00:03:57] tidy error: /checkout/src/bootstrap/compile.rs:665: line longer than 100 chars
[00:03:58] some tidy checks failed
[00:03:58] 
[00:03:58] 
[00:03:58] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor" "--quiet"
[00:03:58] 
[00:03:58] 
[00:03:58] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:03:58] Build completed unsuccessfully in 0:00:53
[00:03:58] Build completed unsuccessfully in 0:00:53
[00:03:58] Makefile:79: recipe for target 'tidy' failed
[00:03:58] make: *** [tidy] Error 1

The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:06860550
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
---
travis_time:end:02d72e93:start=1531529698253977306,finish=1531529698261640401,duration=7663095
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:316aaa32
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:11721798
travis_time:start:11721798
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:12627504
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

Previously we'd attempt to recompile them and that would fail since
we've essentially not built the entire compiler yet, or we're faking
that fact. This commit should make us ignore the codegen backend build
as well.

Unlike the other compile steps, there is no CodegenBackendLink step that
we run here, because that is done later as a part of assembling the
final compiler and as an explicit function call.
@tinco
Copy link
Contributor

tinco commented Jul 14, 2018

Hi Mark! This does not solve things for me, though it does change the error message. This is the full message:

  rust git:(pull_52360) ./x.py test --keep-stage 0 --stage 1 src/test/pretty ; say "I am done. I am really done. ... Really"
Updating only changed submodules
Submodules updated in 0.09 seconds
    Finished dev [unoptimized] target(s) in 0.30s
Warning: Using a potentially old librustc. This may not behave well.
Warning: Using a potentially old codegen backend. This may not behave well.
Warning: Using a potentially old librustc. This may not behave well.
Warning: Using a potentially old libtest. This may not behave well.
Warning: Using a potentially old librustc. This may not behave well.
Copying stage0 rustc from stage0 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Warning: Using a potentially old codegen backend. This may not behave well.
Assembling stage1 compiler (x86_64-apple-darwin)
Warning: Using a potentially old librustc. This may not behave well.
Copying stage1 rustc from stage1 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Warning: Using a potentially old libtest. This may not behave well.
Copying stage1 test from stage1 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Building stage0 tool compiletest (x86_64-apple-darwin)
    Finished release [optimized] target(s) in 0.36s
Check compiletest suite=pretty mode=pretty (x86_64-apple-darwin -> x86_64-apple-darwin)

running 50 tests
iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiF
failures:

---- [pretty] pretty/issue_12590_c.rs stdout ----

error: pretty-printing failed in round 0 revision None
status: signal: 6
command: "/Users/tinco/Source/rust/build/x86_64-apple-darwin/stage1/bin/rustc" "-" "-Z" "unpretty=expanded" "--target" "x86_64-apple-darwin" "-L" "/Users/tinco/Source/rust/build/x86_64-apple-darwin/test/pretty/issue_12590_c/auxiliary.pretty" "-Crpath" "-O" "-Zunstable-options" "-Lnative=/Users/tinco/Source/rust/build/x86_64-apple-darwin/native/rust-test-helpers"
stdout:
------------------------------------------

------------------------------------------
stderr:
------------------------------------------
dyld: Library not loaded: @rpath/libtest-6187db25f3681564.dylib
  Referenced from: /Users/tinco/Source/rust/build/x86_64-apple-darwin/stage1/bin/rustc
  Reason: image not found

------------------------------------------

thread '[pretty] pretty/issue_12590_c.rs' panicked at 'explicit panic', tools/compiletest/src/runtest.rs:3162:9
note: Run with `RUST_BACKTRACE=1` for a backtrace.


failures:
    [pretty] pretty/issue_12590_c.rs

test result: FAILED. 0 passed; 1 failed; 49 ignored; 0 measured; 0 filtered out

thread 'main' panicked at 'Some tests failed', tools/compiletest/src/main.rs:498:22


command did not execute successfully: "/Users/tinco/Source/rust/build/x86_64-apple-darwin/stage0-tools-bin/compiletest" "--compile-lib-path" "/Users/tinco/Source/rust/build/x86_64-apple-darwin/stage1/lib" "--run-lib-path" "/Users/tinco/Source/rust/build/x86_64-apple-darwin/stage1/lib/rustlib/x86_64-apple-darwin/lib" "--rustc-path" "/Users/tinco/Source/rust/build/x86_64-apple-darwin/stage1/bin/rustc" "--src-base" "/Users/tinco/Source/rust/src/test/pretty" "--build-base" "/Users/tinco/Source/rust/build/x86_64-apple-darwin/test/pretty" "--stage-id" "stage1-x86_64-apple-darwin" "--mode" "pretty" "--target" "x86_64-apple-darwin" "--host" "x86_64-apple-darwin" "--llvm-filecheck" "/Users/tinco/Source/rust/build/x86_64-apple-darwin/llvm/build/bin/FileCheck" "--nodejs" "/usr/local/bin/node" "--host-rustcflags" "-Crpath -O -Zunstable-options " "--target-rustcflags" "-Crpath -O -Zunstable-options  -Lnative=/Users/tinco/Source/rust/build/x86_64-apple-darwin/native/rust-test-helpers" "--docck-python" "/usr/bin/python" "--lldb-python" "/usr/bin/python" "--gdb" "/usr/local/bin/gdb" "--lldb-version" "lldb-902.0.79.7" "--lldb-python-dir" "/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/Python" "--quiet" "--llvm-version" "7.0.0svn\n" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" ""
expected success, got: exit code: 101


failed to run: /Users/tinco/Source/rust/build/bootstrap/debug/bootstrap test --keep-stage 0 --stage 1 src/test/pretty
Build completed unsuccessfully in 0:00:03

The important part being:

dyld: Library not loaded: @rpath/libtest-6187db25f3681564.dylib
  Referenced from: /Users/tinco/Source/rust/build/x86_64-apple-darwin/stage1/bin/rustc
  Reason: image not found

Before running this I did a full test compile once by executing the same command but without --keep-stage 0, to make sure all my artifacts were there.

@tinco
Copy link
Contributor

tinco commented Jul 14, 2018

Keeping stage 1 and running in stage 2 gives a slightly different error:

➜  rust git:(pull_52360) ./x.py test --keep-stage 1 --stage 2 src/test/pretty ; say "I am done. I am really done. ... Really"
Updating only changed submodules
Submodules updated in 0.09 seconds
    Finished dev [unoptimized] target(s) in 0.27s
Warning: Using a potentially old librustc. This may not behave well.
Warning: Using a potentially old codegen backend. This may not behave well.
Warning: Using a potentially old librustc. This may not behave well.
Warning: Using a potentially old libtest. This may not behave well.
Building stage0 std artifacts (x86_64-apple-darwin -> x86_64-apple-darwin)
    Finished release [optimized] target(s) in 0.28s
Copying stage0 std from stage0 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Building stage0 test artifacts (x86_64-apple-darwin -> x86_64-apple-darwin)
    Finished release [optimized] target(s) in 0.26s
Copying stage0 test from stage0 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Building stage0 compiler artifacts (x86_64-apple-darwin -> x86_64-apple-darwin)
    Finished release [optimized] target(s) in 0.34s
Copying stage0 rustc from stage0 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Building stage0 codegen artifacts (x86_64-apple-darwin -> x86_64-apple-darwin, llvm)
    Finished release [optimized] target(s) in 0.34s
Assembling stage1 compiler (x86_64-apple-darwin)
Warning: Using a potentially old librustc. This may not behave well.
Copying stage1 rustc from stage1 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Warning: Using a potentially old codegen backend. This may not behave well.
Assembling stage2 compiler (x86_64-apple-darwin)
Warning: Using a potentially old librustc. This may not behave well.
Copying stage2 rustc from stage2 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
thread 'main' panicked at 'File::open(stamp) failed with No such file or directory (os error 2)', bootstrap/lib.rs:1109:12
note: Run with `RUST_BACKTRACE=1` for a backtrace.
failed to run: /Users/tinco/Source/rust/build/bootstrap/debug/bootstrap test --keep-stage 1 --stage 2 src/test/pretty
Build completed unsuccessfully in 0:00:03

@RalfJung
Copy link
Member

It didn't fully work for me either. I first did ./x.py build src/rustc to get a compiler in stage 2 built (I had to rm build/x86_64-unknown-linux-gnu/stage{1,2}* -rf but that may be because of junk in there from previous attempts).

Then I changed something in rustc_mir a bit, and did ./x.py build src/rustc --keep-stage 1. This failed with the following log:

Updating only changed submodules
Submodules updated in 0.03 seconds
   Compiling bootstrap v0.0.0 (file:///home/r/src/rust/rustc/src/bootstrap)
    Finished dev [unoptimized] target(s) in 5.67s
Warning: Using a potentially old librustc. This may not behave well.
Warning: Using a potentially old codegen backend. This may not behave well.
Warning: Using a potentially old librustc. This may not behave well.
Building stage0 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 0.24s
Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building stage0 test artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 0.22s
Copying stage0 test from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building stage0 compiler artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
   Compiling rustc_mir v0.0.0 (file:///home/r/src/rust/rustc/src/librustc_mir)
   Compiling rustc_passes v0.0.0 (file:///home/r/src/rust/rustc/src/librustc_passes)
   Compiling rustc_borrowck v0.0.0 (file:///home/r/src/rust/rustc/src/librustc_borrowck)
   Compiling rustc_codegen_utils v0.0.0 (file:///home/r/src/rust/rustc/src/librustc_codegen_utils)
   Compiling rustc_lint v0.0.0 (file:///home/r/src/rust/rustc/src/librustc_lint)
   Compiling rustc_driver v0.0.0 (file:///home/r/src/rust/rustc/src/librustc_driver)
   Compiling rustc-main v0.0.0 (file:///home/r/src/rust/rustc/src/rustc)
    Finished release [optimized] target(s) in 14.51s
Copying stage0 rustc from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building stage0 codegen artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu, llvm)
   Compiling rustc_codegen_llvm v0.0.0 (file:///home/r/src/rust/rustc/src/librustc_codegen_llvm)
    Finished release [optimized] target(s) in 7.64s
Assembling stage1 compiler (x86_64-unknown-linux-gnu)
Warning: Using a potentially old librustc. This may not behave well.
Copying stage1 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Warning: Using a potentially old codegen backend. This may not behave well.
Assembling stage2 compiler (x86_64-unknown-linux-gnu)
Warning: Using a potentially old librustc. This may not behave well.
Copying stage2 rustc from stage2 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
thread 'main' panicked at 'File::open(stamp) failed with No such file or directory (os error 2)', bootstrap/lib.rs:1109:12
note: Run with `RUST_BACKTRACE=1` for a backtrace.
failed to run: /home/r/src/rust/rustc/build/bootstrap/debug/bootstrap build src/rustc --keep-stage 1
Build completed unsuccessfully in 0:00:29

@RalfJung
Copy link
Member

FWIW, reverting 9eda4aa (from #52006) fixes the problem.

The best way to build a stage 2 rustc is now probably
  ./x.py build --stage 2 src/rustc # once
  ./x.py build --stage 2 --keep-stage 1 src/rustc
@alexcrichton
Copy link
Member

@bors: r+

@bors
Copy link
Contributor

bors commented Jul 14, 2018

📌 Commit 8eddaba has been approved by alexcrichton

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 14, 2018
@bors
Copy link
Contributor

bors commented Jul 15, 2018

⌛ Testing commit 8eddaba with merge bb09c39...

bors added a commit that referenced this pull request Jul 15, 2018
…, r=alexcrichton

Do not attempt to recompile codegen backend(s) with --keep-stage

Previously we'd attempt to recompile them and that would fail since
we've essentially not built the entire compiler yet, or we're faking
that fact. This commit should make us ignore the codegen backend build
as well.

Unlike the other compile steps, there is no CodegenBackendLink step that
we run here, because that is done later as a part of assembling the
final compiler and as an explicit function call.

r? @alexcrichton

I think this may fix or at least assist with #52174.

cc @RalfJung @tinco -- if you can test this patch locally that'd be
amazing; I don't want to recompile for the next couple hours to test it
locally. I don't think it can make the situation worse, and in fact, if
I've interpreted the cause of the failure correctly then this will fix
your problem.
@bors
Copy link
Contributor

bors commented Jul 15, 2018

☀️ Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing bb09c39 to master...

@bors bors merged commit 8eddaba into rust-lang:master Jul 15, 2018
@Mark-Simulacrum Mark-Simulacrum deleted the fix-keep-stage-for-cg-backends branch June 8, 2019 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants