Spurious error compiling OpenSSL on OSX #40417
Comments
alexcrichton
added
the
A-spurious
|
@sfackler you wouldn't happen to have run into this before, would you have? |
|
I have never seen this, no. |
|
Looks a lot like this. The solutions there are manually patching OpenSSL makefiles, I believe. I don't have any more spare time at the moment. If I get a chance, I'll see if I can figure out what method you are using to compile OpenSSL and see if it can be adjusted. No promises. ;-) |
|
Sure, I'll give it a shot. Would you mind giving me a hint on how to execute said code on my own machine (up-to-date macOS Sierra)? I haven't ever worked on Rust's own implementation, yet, and the I'd rather run it on my own machine until I get it right before committing & making a PR and waiting for the builders to get to it! ;-) |
|
Sure yeah, first you'll want to change this line from |
|
Hmmm. It builds just fine for me. I'll need to get it to fail... |
|
Actually, it doesn't look like it's building openssl at all. I just get this: That's with only this modification: diff --git a/src/bootstrap/step.rs b/src/bootstrap/step.rs
index 6eb12fe..174c71b 100644
--- a/src/bootstrap/step.rs
+++ b/src/bootstrap/step.rs
@@ -491,7 +491,7 @@ pub fn build_rules<'a>(build: &'a Build) -> Rules {
rules.build("test-helpers", "src/rt/rust_test_helpers.c")
.run(move |s| native::test_helpers(build, s.target));
- rules.build("openssl", "path/to/nowhere")
+ rules.build("openssl", "openssl")
.run(move |s| native::openssl(build, s.target));
// Some test suites are run inside emulators, and most of our test binaries |
|
Oh right sorry! That patch looks correct but you'll also need to run the |
|
Okay. That appears to compile openssl, albeit with much, much less output than the builders. But the build succeeds! :-( Any ideas on how I get it to fail? |
|
Ah yeah so the output is much less because the output is suppressed unless it fails (it's a ton and overflows our 4MB limit on Travis). Unfortunately the error is spurious, though, so I don't know how to reproduce :( |
|
Okay, I'm running the build in a loop until it fails. We'll see if it's a race-condition that will occur on my machine. I hope this isn't Travis-specific. |
|
15 builds in a row, no failures yet. |
|
Yeah i'm not sure if it's directly related to the version of tools used on Travis or what :( |
|
I've compiled OpenSSL 50 times locally and it hasn't failed yet. We are using the |
|
I did it about that many times as well, without failure. :-( That's frustrating. I've had similar-in-flaky-nature CI-only problems with pypy on TravisCI and CPython on AppVeyor. My project only has a few commits per week and only takes a minute or two to build, so I just re-run the builds until they pass. Do you think TravisCI would be willing to let us have a copy of their VM to try to debug? |
|
I sent an email to TravisCI last night referencing this Github issue and asking if we could get a copy of a VM. No answer, yet. |
|
I don't think they're going to answer. I'm out of ideas. |
|
I received a reply:
Isolating the failing part and running repeated builds is actually a decent idea. I'll look into setting up TravisCI for my fork of Rust, trim down the |
|
Thanks @CleanCut! |
|
Hmm. I get 48 minutes into the build, and Travis kills it for exceeding the max time limit. Obviously the main Rust project doesn't hit that limit. |
|
Is that basically using the rust repo itself? I'd recommend stripping it down to basically just compiling openssl which in theory could be done with a small shell script as well |
|
You're right. I was just being lazy. ;-) Time to dive down the rabbit hole... |
|
Okay, I traced |
|
Thanks for the continued investigation @CleanCut! Lemme know if you need any help |
|
The build with the real error appears to compile LLVM before the openssl stuff. The LLVM compilation alone causes my build to time out at 48 minutes, so I can't do that in my build. Without it, the build always passes (so far). Any ideas? I spent last evening getting the config of my short build as close as I could to the config of the build that failed in your first post of this issue. But I can't add in building llvm. |
|
Oh in theory compiling OpenSSL shouldn't require LLVM, so that step could just be bypassed right? |
|
Sure. But it compiles just fine without that step. I have run out of ways to make it similar other than compiling LLVM first. And the builds pass. |
|
I'm out of ideas. Perhaps...cross our fingers? |
…hton Overhaul Bootstrap (x.py) Command-Line-Parsing & Help Output While working on rust-lang#40417, I got frustrated with the behavior of x.py and the bootstrap binary it wraps, so I decided to do something about it. This PR should improve documentation, make the command-line-parsing more flexible, and clean up some of the internals. No command that worked before should stop working. At least that's the theory. :-) This should resolve at least rust-lang#40920 and rust-lang#38373. Changes: - No more manual args manipulation -- getopts used everywhere except the one place it's not possible. As a result, options can be in any position, now, even before the subcommand. - The additional options for test, bench, and dist now appear in the help output. - No more single-letter variable bindings used internally for large scopes. - Don't output the time measurement when just invoking `x.py` or explicitly passing `-h` or `--help` - Logic is now much more linear. We build strings up, and then print them. - Refer to subcommands as subcommands everywhere (some places we were saying "command") - Other minor stuff. @alexcrichton This is my first PR. Do I need to do something specific to request reviewers or anything?
…hton Overhaul Bootstrap (x.py) Command-Line-Parsing & Help Output While working on rust-lang#40417, I got frustrated with the behavior of x.py and the bootstrap binary it wraps, so I decided to do something about it. This PR should improve documentation, make the command-line-parsing more flexible, and clean up some of the internals. No command that worked before should stop working. At least that's the theory. :-) This should resolve at least rust-lang#40920 and rust-lang#38373. Changes: - No more manual args manipulation -- getopts used everywhere except the one place it's not possible. As a result, options can be in any position, now, even before the subcommand. - The additional options for test, bench, and dist now appear in the help output. - No more single-letter variable bindings used internally for large scopes. - Don't output the time measurement when just invoking `x.py` or explicitly passing `-h` or `--help` - Logic is now much more linear. We build strings up, and then print them. - Refer to subcommands as subcommands everywhere (some places we were saying "command") - Other minor stuff. @alexcrichton This is my first PR. Do I need to do something specific to request reviewers or anything?
… r=alexcrichton Make sure openssl compiles with only one core This is (hopefully) a fix for the osx openssl spurious failure - #40417. The intermittent failures and failing in different ways made me think of a race condition. But programs are parallel make safe right? [Not openssl](openssl/openssl#298). But we don't do a parallel make on openssl [do we](https://github.com/rust-lang/rust/blob/8c4f2c64c6759a82f143e23964a46a65c67509c9/src/bootstrap/native.rs#L309)? This confused me, except "Waiting for unfinished jobs" is present in the logs...which is evidence of a parallel make! It turns out that when we invoke to top level target [in run.sh](https://github.com/rust-lang/rust/blob/036983201d4e9aeb5c5e56e47c305971972b2569/src/ci/run.sh#L75-L77), make will [pass the flags downwards](https://www.gnu.org/software/make/manual/html_node/Options_002fRecursion.html) in order to take advantage of parallelism in sub-makes. Of course, we don't want this in openssl! Override this by explicitly disabling parallelism on the command line. I don't know why this hasn't happened on anything except OSX. Maybe Linux binutils check if the file is in use? r? @alexcrichton
First seen on https://travis-ci.org/rust-lang/rust/jobs/209693739 appears to look like:
The text was updated successfully, but these errors were encountered: