Add x86_64-unknown-linux-musl build support

[martell]

Closes #1053

[bors]

☔ The latest upstream changes (presumably 6d47fed) made this pull request unmergeable. Please resolve the merge conflicts.

[strfry]

@martell Please regard my additional commit 54b50f5 in #1527 .
The Dockerfile must describe a normal glibc-distro with musl-capabilities for this build system to work.

What else is needed to make rustc appear in x86_64-unknown-linux-musl flavor?

[martell]

Hey @strfry,

First, thanks for the help :)
I thought no one was interested in this but me.

The Dockerfile must describe a normal glibc-distro with musl-capabilities for this build system to work.

EDIT: I see the error now.
bash does not come by default on alpine.
I'll update.

+docker run --entrypoint bash --user 2000:2000 --volume /home/travis/.rustup/toolchains/stable-x86_64-unknown-linux-gnu:/travis-rust:ro --volume /home/travis/build/rust-lang-nursery/rustup.rs:/src:ro --volume /home/travis/build/rust-lang-nursery/rustup.rs/target:/src/target --workdir /src --env TARGET=x86_64-unknown-linux-musl --env SKIP_TESTS=1 x86_64-unknown-linux-musl ci/run-docker.sh
container_linux.go:265: starting container process caused "exec: \"bash\": executable file not found in $PATH"
docker: Error response from daemon: oci runtime error: container_linux.go:265: starting container process caused "exec: \"bash\": executable file not found in $PATH".
ERRO[0000] error waiting for container: context canceled

What else is needed to make rustc appear in x86_64-unknown-linux-musl flavor?

I'm not sure, I upstreamed a few commits from musl / alpine needed changes into llvm 5.0 which was 2 releases ago now in the hopes that later we could get rustup support for alpine.
I think for build systems making minimal alpine rust containers it would be better than using the package manager.

@alexcrichton could you weigh in with some direction/feedback?

Best,
Martell

[strfry]

@martell ,

I started out with just adding bash (and then later rustc and cargo), but the way this is supposed to work is that the rustc is passed in through a Docker volume from the Travis host system. And that doesn't work when the container is a different architecture than the host distro.
What finally worked was to adapt one of the other Dockerfiles and install musl-tools a as the cross-compiler. The build will try call musl-gcc at some point, seems like someone already figured this out ;)

The other part i ment is getting rust itself to compile x86_64-musl as a host architecture, so that it's provided for rustup to download.
I opened rust-lang/rust#55163 and ran into some issues.

[alexcrichton]

@martell I think it's probably not worth doing this until we get a musl host rustc working, otherwise this rustup won't have anything to download! When it works though it'll be easiest to compile a static rustup here, just using cross compilation from a glibc environment to x86_64-unknown-linux-musl which will produce a statically linked binary

[kinnison]

@martell Do you think you want to keep this PR open for now, or should we close it off and perhaps just leave #1053 to track this?

[dwijnand]

Seeing as #1053 is blocked on upstream rust-lang/rust#57439, I think it makes sense to close this PR in the meantime. Please note that a closed PR isn't a rejected PR, the work here can be reused and resubmitted.

[martell]

Sorry for the delay, I'll close this for now until the host support has been resolved.

[mati865]

IIUC this needs cfg similar to the one for MIPS:
https://github.com/rust-lang/rustup.rs/blob/819ddda6b7fc1f18884f027f646cd8714f8c4e9a/src/dist/dist.rs#L178

EDIT: https://github.com/rust-lang/rustup.rs/blob/master/rustup-init.sh would also need to detect if host is musl based. I tested it on Void Linux and Alpine Linux and checking output of ldd --version for musl looks reasonable:

$ ldd --version
musl libc (x86_64)
Version 1.1.21
Dynamic Program Loader
Usage: ldd [options] [--] pathname

[martell]

@alexcrichton soft ping

[alexcrichton]

Ah oops thanks for the ping. I'm not sure the container will work here though since it's primarily used with glibc toolchains, although you can edit .travis.yml to see the failure on the PR. Could the build be configured to do a cross build from a glibc toolchain to a musl toolchain?

[martell]

Finally got some time this weekend to have another look.

@alexcrichton I believe this should be mergeable now.
When you approve I will drop 44ddbe9 from the PR

Thanks again for the suggestion @rye.

[rye]

@martell, no problem, and thank you for doing this! It at least certainly seems that #3396.2 (musl build) was successful and did use the appropriate target! Can't wait to use rustup in my Alpine environments soonish. 😄

[kinnison]

I'm not in a position to review the musl-specific bits, but one comment I do have is regarding the docker machine.

@lzutao put a lot of effort into allowing us to reuse the Rust compiler docker images in order to ensure that we were matching libcs etc wherever possible. Is there a suitable rust-compiler docker for musl to save the creation of the docker from ubuntu?

[tesuji]

@kinnison Yes, Rustc has dist-x86_64-musl dockerfile. I haven't tested it yet
but it's worth a try.

[kinnison]

@lzutao Do you think you could look at what's been done on this PR and then see if you can come up with a proposed patch to reuse the rustc docker if appropriate?

[alexcrichton]

This all looks good to me, thanks @martell!

@kinnison FWIW musl is pretty special in that it doesn't require the same docker containers as other targets. The main difference is that the musl target is by default statically linked instead of having a dependency on a dynamically linked libc. For glibc platforms you need a special toolchain for targeting a really old glibc (which is our goal in rust-lang/rust and what rustup wants to mirror), but for musl there's no need since any toolchain can produce a workable binary!

In that sense this should be good to go IMO w/o docker changes.

[martell]

Awesome, I'm happy to do any docker changes as a follow up if anyone wishes.
I did actually try FROM rust-x86_64-unknown-linux-musl but that did not exist.
We can create one if needed. Anyone have a specific issue they want to create for this?

@alexcrichton I do not have permission to merge.
I did however remove the ci testing commit as promised.
Can you merge?

[tesuji]

The PR might still fail because of missing sccache.

[alexcrichton]

Note that I'm personally just commenting here, I'm leaving the final review up to @kinnison

[kinnison]

If there's no value in using the rustc docker for now, then I'll wait on the sccache comment responses before I make a decision.

[martell]

@kinnison done :)
Same deal as before, once approved I will remove the travis change commit.
Ahh damn, I was a few hours late to make the next rustup release :/

[tesuji]

@martell You could use ci/docker/armv7-unknown-linux-gnueabihf/Dockerfile as an example to
install sccache.

[kinnison]

The appveyor failure is spurious (possibly due to too many parallel tests at once?)

[tesuji]

I pushed some changes to my branch. You could cherry-pick it to your PR.

[tesuji]

Edit ci/cloudfront-invalidation.txt and add musl targets.

[kinnison]

Edit ci/cloudfront-invalidation.txt and add musl targets.

It's not entirely clear if the cloudfront invalidation would be necessary, but it can't hurt to have it amended.

[tesuji]

Ok, I take it at #1882

[martell]

Thanks @lzutao