Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

try to infer linker flavor from linker name and vice versa #52101

Merged
merged 9 commits into from
Aug 20, 2018

Conversation

japaric
Copy link
Member

@japaric japaric commented Jul 6, 2018

This is a second take on PR #50359 that implements the logic proposed in #50359 (review)

With this change it would become possible to link thumb* binaries using GNU's LD on stable as -C linker=arm-none-eabi-ld would be enough to change both the linker and the linker flavor from their default values of arm-none-eabi-gcc and gcc.

To link thumb* binaries using rustc's LLD on stable -Z linker-flavor would need to be stabilized as -C linker=rust-lld -Z linker-flavor=ld.lld are both required to change the linker and the linker flavor, but this PR doesn't propose that. We would probably need some sort of stability guarantee around rust-lld's name and availability to make linking with rustc's LLD truly stable.

With this change it would also be possible to link thumb* binaries using a system installed LLD on stable using the -C linker=ld.lld flag (provided that ld.lld is a symlink to the system installed LLD).

r? @alexcrichton

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 6, 2018
@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-3.9 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:start:test_run-make-fulldeps
Check compiletest suite=run-make-fulldeps mode=run-make (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:27:06] 
[01:27:06] running 187 tests
[01:27:50] ....................i.....................................................................F.........
[01:28:14] thread 'main' panicked at 'Some tests failed', tools/compiletest/src/main.rs:498:22
[01:28:14] .....................................F.................................................
[01:28:14] 
[01:28:14] ---- [run-make] run-make-fulldeps/long-linker-command-lines stdout ----
[01:28:14] 
[01:28:14] error: make failed
[01:28:14] error: make failed
[01:28:14] status: exit code: 2
[01:28:14] command: "make"
[01:28:14] stdout:
[01:28:14] ------------------------------------------
[01:28:14] make[1]: Entering directory '/checkout/src/test/run-make-fulldeps/long-linker-command-lines'
[01:28:14] LD_LIBRARY_PATH="/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/long-linker-command-lines/long-linker-command-lines:/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-bootstrap-tools/x86_64-unknown-linux-gnu/release/deps:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/lib:" '/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc' --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/long-linker-command-lines/long-linker-command-lines -L /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/long-linker-command-lines/long-linker-command-lines  foo.rs -g -O
[01:28:14] RUSTC="/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" LD_LIBRARY_PATH="/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/long-linker-command-lines/long-linker-command-lines:/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-bootstrap-tools/x86_64-unknown-linux-gnu/release/deps:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/lib:" /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/long-linker-command-lines/long-linker-command-lines/foo
[01:28:14] attempt: 100
[01:28:14] Makefile:4: recipe for target 'all' failed
[01:28:14] make[1]: Leaving directory '/checkout/src/test/run-make-fulldeps/long-linker-command-lines'
[01:28:14] ------------------------------------------
[01:28:14] stderr:
[01:28:14] ------------------------------------------
[01:28:14] thread 'main' panicked at 'status: exit code: 101
[01:28:14] thread 'main' panicked at 'status: exit code: 101
[01:28:14] stdout:
[01:28:14] 
[01:28:14] stderr:
[01:28:14] error: couldn't infer linker flavor from specified linker
[01:28:14] error: aborting due to previous error
[01:28:14] 
[01:28:14] ', foo.rs:64:13
[01:28:14] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[01:28:14] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[01:28:14] make[1]: *** [all] Error 101
[01:28:14] ------------------------------------------
[01:28:14] 
[01:28:14] thread '[run-make] run-make-fulldeps/long-linker-command-lines' panicked at 'explicit panic', tools/compiletest/src/runtest.rs:3140:9
[01:28:14] note: Run with `RUST_BACKTRACE=1` for a backtrace.
---
[01:28:14] command: "make"
[01:28:14] stdout:
[01:28:14] ------------------------------------------
[01:28:14] make[1]: Entering directory '/checkout/src/test/run-make-fulldeps/reproducible-build'
[01:28:14] rm -rf /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/reproducible-build/reproducible-build && mkdir /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/reproducible-build/reproducible-build
[01:28:14] LD_LIBRARY_PATH="/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/reproducible-build/reproducible-build:/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-bootstrap-tools/x86_64-unknown-linux-gnu/release/deps:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/lib:" '/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc' --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/reproducible-build/reproducible-build -L /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/reproducible-build/reproducible-build  linker.rs -O
[01:28:14] LD_LIBRARY_PATH="/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/reproducible-build/reproducible-build:/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-bootstrap-tools/x86_64-unknown-linux-gnu/release/deps:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/lib:" '/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc' --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/reproducible-build/reproducible-build -L /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/reproducible-build/reproducible-build  reproducible-build-aux.rs
[01:28:14] LD_LIBRARY_PATH="/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/reproducible-build/reproducible-build:/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-bootstrap-tools/x86_64-unknown-linux-gnu/release/deps:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/lib:" '/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc' --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/reproducible-build/reproducible-build -L /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/reproducible-build/reproducible-build  reproducible-build.rs -C linker=/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/reproducible-build/reproducible-build/linker
[01:28:14] Makefile:12: recipe for target 'smoke' failed
[01:28:14] 
[01:28:14] ------------------------------------------
[01:28:14] stderr:
[01:28:14] ------------------------------------------
[01:28:14] ------------------------------------------
[01:28:14] error: couldn't infer linker flavor from specified linker
[01:28:14] error: aborting due to previous error
[01:28:14] 
[01:28:14] 
[01:28:14] make[1]: *** [smoke] Error 101
[01:28:14] ------------------------------------------
[01:28:14] 
[01:28:14] 
[01:28:14] thread '[run-make] run-make-fulldeps/reproducible-build' panicked at 'explicit panic', tools/compiletest/src/runtest.rs:3140:9
[01:28:14] 
[01:28:14] failures:
[01:28:14]     [run-make] run-make-fulldeps/long-linker-command-lines
[01:28:14]     [run-make] run-make-fulldeps/reproducible-build
[01:28:14]     [run-make] run-make-fulldeps/reproducible-build
[01:28:14] 
[01:28:14] test result: FAILED. 184 passed; 2 failed; 1 ignored; 0 measured; 0 filtered out
[01:28:14] 
[01:28:14] 
[01:28:14] 
[01:28:14] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--rustdoc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustdoc" "--src-base" "/checkout/src/test/run-make-fulldeps" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "run-make" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-3.9/bin/FileCheck" "--host-rustcflags" "-Crpath -O -Zunstable-options " "--target-rustcflags" "-Crpath -O -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "3.9.1\n" "--system-llvm" "--cc" "cc" "--cxx" "c++" "--cflags" "-ffunction-sections -fdata-sections -fPIC -m64" "--llvm-components" "aarch64 aarch64asmparser aarch64asmprinter aarch64codegen aarch64desc aarch64disassembler aarch64info aarch64utils all all-targets amdgpu amdgpuasmparser amdgpuasmprinter amdgpucodegen amdgpudesc amdgpudisassembler amdgpuinfo amdgpuutils analysis arm armasmparser armasmprinter armcodegen armdesc armdisassembler arminfo asmparser asmprinter bitreader bitwriter bpf bpfasmprinter bpfcodegen bpfdesc bpfinfo codegen core coverage debuginfocodeview debuginfodwarf debuginfopdb engine executionengine globalisel hexagon hexagonasmparser hexagoncodegen hexagondesc hexagondisassembler hexagoninfo instcombine instrumentation interpreter ipo irreader libdriver lineeditor linker lto mc mcdisassembler mcjit mcparser mips mipsasmparser mipsasmprinter mipscodegen mipsdesc mipsdisassembler mipsinfo mirparser msp430 msp430asmprinter msp430codegen msp430desc msp430info native nativecodegen nvptx nvptxasmprinter nvptxcodegen nvptxdesc nvptxinfo objcarcopts object objectyaml option orcjit passes powerpc powerpcasmparser powerpcasmprinter powerpccodegen powerpcdesc powerpcdisassembler powerpcinfo profiledata runtimedyld scalaropts selectiondag sparc sparcasmparser sparcasmprinter sparccodegen sparcdesc sparcdisassembler sparcinfo support symbolize systemz systemzasmparser systemzasmprinter systemzcodegen systemzdesc systemzdisassembler systemzinfo tablegen target transformutils vectorize x86 x86asmparser x86asmprinter x86codegen x86desc x86disassembler x86info x86utils xcore xcoreasmprinter xcorecodegen xcoredesc xcoredisassembler xcoreinfo" "--llvm-cxxflags" "-I/usr/lib/llvm-3.9/include -std=c++0x -gsplit-dwarf -Wl,-fuse-ld=gold -fPIC -fvisibility-inlines-hidden -Wall -W -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wno-maybe-uninitialized -Wdelete-non-virtual-dtor -Wno-comment -Werror=date-time -std=c++11 -ffunction-sections -fdata-sections -O2 -g -DNDEBUG  -fno-exceptions -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS" "--ar" "ar" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
[01:28:14] 
[01:28:14] 
[01:28:14] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:28:14] Build completed unsuccessfully in 0:44:53
[01:28:14] Build completed unsuccessfully in 0:44:53
[01:28:14] Makefile:58: recipe for target 'check' failed
[01:28:14] make: *** [check] Error 1

The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:25299ecb
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)


Ok((linker, flavor))
},
(None, None) => otherwise(),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stylistically I might recommend instead of a cps style to instead do something like:

  • Emit only fatal errors here via sess (it doesn't look like the error is ever "caught"?)
  • Return Option instead of Result
  • Instead do something like:
if let Some(ret) = locate_linker(&sess.opts) {
    return Some(ret)
}
if let Some(ret) = locate_linker(&sess.target.target) {
    return Some(ret)
}
sess.fatal("...")

LinkerFlavor::Em => "emcc",
LinkerFlavor::Gcc => "gcc",
LinkerFlavor::Ld => "ld",
LinkerFlavor::Msvc => "link.exe",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is losing the logic of locating link.exe on MSVC which may cause issues there?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic is in get_linker which is called with the values this function returns. I think that function will locate link.exe even with these changes.

(Some(linker), Some(flavor)) => Ok((linker, flavor)),
// only the linker flavor is known; use the default linker for the selected flavor
(None, Some(flavor)) => Ok((PathBuf::from(match flavor {
LinkerFlavor::Em => "emcc",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this may have accidentally lost the logic of emcc.bat on Windows?

LinkerFlavor::Lld(f) => Command::lld(linker, f),
_ => Command::new(linker),

}
};
})();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can probably remove the intermediate closure here, I don't think it's serving much purpose any more

sess
.struct_err(&format!("couldn't infer linker flavor from specified linker"))
.emit();
return Err(());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This runs the risk of being a breaking change by accident if someone's passing in a "flavorful" linker because currently we can't error today I think? Perhaps this could fall back to the target spec's linker flavor and if that's not present emit an error?

@TimNN TimNN added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 10, 2018
@bors
Copy link
Contributor

bors commented Jul 11, 2018

☔ The latest upstream changes (presumably #52268) made this pull request unmergeable. Please resolve the merge conflicts.

@TimNN
Copy link
Contributor

TimNN commented Jul 17, 2018

Ping from triage, @japaric: There are some review comments on your PR.

@japaric
Copy link
Member Author

japaric commented Jul 23, 2018

It'll take me a while to get back to this so I'm going to close the PR in the meantime.

@japaric
Copy link
Member Author

japaric commented Aug 9, 2018

re-r? @alexcrichton I believe I have addressed all your comments.

return ret;
}

sess.fatal("Not enough information provided to determine how to invoke the linker");
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe, today, this is effectively unreachable because linker flavor is a mandatory field of target specifications. Should this be bug instead of fatal?

@japaric
Copy link
Member Author

japaric commented Aug 9, 2018

I realized (too late) that these changes are not enough to use rustc LLD with the thumbv* targets on stable because -C linker=rust-lld doesn't provide information about which LD flavor to use (wasm, LD, macho, or link.exe). Still this change would let us implement the change of default linker (breaking change) proposed in rust-embedded/cortex-m-quickstart#39 (comment) while providing a stable way to switch back to gcc (current default).

What we could do to correctly support -C linker=rust-lld (and -C linker=lld) on stable would be to try to infer the ld flavor from the target specification. Simplest way to do that would be to add a ld_flavor field, which indicates which LD flavor to use when linking with the generic lld binary, to target specifications. @alexcrichton, thoughts on this change (ld_flavor)?

@alexcrichton
Copy link
Member

Ok looks great to me! I agree yeah changing the bug to fatal just to be safe, and for inferring the LLD flavor I think it's probably fine to add a target spec field for that. By default we should assume ld-lld as I think most targets fall into that bucket, but targets could whitelist different flavors like MSVC/OSX/etc

@japaric
Copy link
Member Author

japaric commented Aug 18, 2018

@alexcrichton OK, done. r?

@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
[00:47:24] ....................................................................................................
[00:47:27] ...................................................................................................i
[00:47:30] ....................................................................................................
[00:47:33] ....................................................................................................
[00:47:35] ................................................iiiiiiiii...........................................
[00:47:41] ....................................................................................................
[00:47:44] ....................................................................................................
[00:47:47] .............................i......................................................................
[00:47:50] ...............................................................................i.i..ii..............

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@japaric japaric added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Aug 18, 2018
@alexcrichton
Copy link
Member

@bors: r+

👍

@bors
Copy link
Contributor

bors commented Aug 19, 2018

📌 Commit 4bbedd7 has been approved by alexcrichton

@japaric
Copy link
Member Author

japaric commented Aug 19, 2018

@bors r=alexcrichton

@bors
Copy link
Contributor

bors commented Aug 19, 2018

📌 Commit 98e4cd5 has been approved by alexcrichton

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 19, 2018
@bors
Copy link
Contributor

bors commented Aug 19, 2018

⌛ Testing commit 98e4cd5 with merge 403324fa238c45b1739855f384b945c2c31045e4...

@bors
Copy link
Contributor

bors commented Aug 19, 2018

💔 Test failed - status-travis

@bors bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Aug 19, 2018
@rust-highfive

This comment has been minimized.

@japaric
Copy link
Member Author

japaric commented Aug 19, 2018

expands logs

Broadcast message from root@travis-job-a9b8c329-924c-44b0-b41f-1c50b33472e2
(unknown) at 15:21 ...
The system is going down for power off NOW!

What? What happened?

cc @rust-lang/infra

@kennytm
Copy link
Member

kennytm commented Aug 19, 2018

@bors retry travis-ci/travis-ci#4924

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 19, 2018
@bors
Copy link
Contributor

bors commented Aug 19, 2018

⌛ Testing commit 98e4cd5 with merge e2fdc33267255d50a849a7eae7aa87b56f38b6b8...

@bors
Copy link
Contributor

bors commented Aug 19, 2018

💔 Test failed - status-appveyor

@bors bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Aug 19, 2018
@japaric
Copy link
Member Author

japaric commented Aug 19, 2018

@bors r=alexcrichton

@bors
Copy link
Contributor

bors commented Aug 19, 2018

📌 Commit a6f4ae8 has been approved by alexcrichton

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 19, 2018
@bors
Copy link
Contributor

bors commented Aug 20, 2018

⌛ Testing commit a6f4ae8 with merge d2048b6...

bors added a commit that referenced this pull request Aug 20, 2018
try to infer linker flavor from linker name and vice versa

This is a second take on PR #50359 that implements the logic proposed in #50359 (review)

With this change it would become possible to link `thumb*` binaries using GNU's LD on stable as `-C linker=arm-none-eabi-ld` would be enough to change both the linker and the linker flavor from their default values of `arm-none-eabi-gcc` and `gcc`.

To link `thumb*` binaries using rustc's LLD on stable `-Z linker-flavor` would need to be stabilized as `-C linker=rust-lld -Z linker-flavor=ld.lld` are both required to change the linker and the linker flavor, but this PR doesn't propose that. We would probably need some sort of stability guarantee around `rust-lld`'s name and availability to make linking with rustc's LLD truly stable.

With this change it would also be possible to link `thumb*` binaries using a system installed LLD on stable using the `-C linker=ld.lld` flag (provided that `ld.lld` is a symlink to the system installed LLD).

r? @alexcrichton
@bors
Copy link
Contributor

bors commented Aug 20, 2018

☀️ Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing d2048b6 to master...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants