Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

build pipelining fails #6993

Closed
khuey opened this issue May 29, 2019 · 16 comments · Fixed by #7008
Closed

build pipelining fails #6993

khuey opened this issue May 29, 2019 · 16 comments · Fixed by #7008
Labels
C-bug Category: bug

Comments

@khuey
Copy link
Contributor

khuey commented May 29, 2019

Problem

When running CARGO_BUILD_PIPELINING=true cargo build --all --tests on my project, I get

error: crate `<foo>` required to be available in rlib format, but was not found in this form

Unfortunately this project is not open source, so I'm not sure what else to report.

Notes

Output of cargo version: cargo 1.37.0-nightly (545f354 2019-05-23)
@khuey khuey added the C-bug Category: bug label May 29, 2019
@nnethercote
Copy link
Contributor

CC @alexcrichton

@alexcrichton
Copy link
Member

Thanks for the report! Is it possible to share the project that generated this error?

@khuey
Copy link
Contributor Author

khuey commented May 29, 2019

Probably not. Is there a way to get debugging information out of cargo?

@alexcrichton
Copy link
Member

If you're ok with it you can do a CARGO_LOG=trace cargo build ... and I can try to piece together what went wrong.

@khuey
Copy link
Contributor Author

khuey commented May 30, 2019

I can do that. The log is 67 MB. Can I email you a link to it?

@alexcrichton
Copy link
Member

Sure!

@khuey
Copy link
Contributor Author

khuey commented May 30, 2019

Sent.

@alexcrichton
Copy link
Member

Is this a build where RUSTC_WRAPPER is set in one way or another? The metadata file looks like it's getting produced at the exact same time as the output file which seems like there may be some caching somewhere. If so does this bug reproduce without the caching layer/wrapper?

@khuey
Copy link
Contributor Author

khuey commented Jun 4, 2019

We don't set that anywhere. I think the most interesting thing going on in our build is a linker script. We don't have anything like ccache involved.

alexcrichton added a commit to alexcrichton/cargo that referenced this issue Jun 4, 2019
@alexcrichton
Handle pipelined tests of libraries
0c51d71 
The previous implementation of pipelining accidentally forgot to account
for rlibs being compiled in `--test` mode. The compilations would be
pipelined to where all the dependencies of an rlib might not be
available yet, but `--test` actually performs linking in rustc!

This commit fixes the issue by refactoring slightly and removing
`Target::requires_upstream_objects` (moving it to `TargetKind`) and then
making the source of truth a `Unit::requires_upstream_objects` method
which takes into account the value from `TargetKind` as well as the
`CompileMode`.

Closes rust-lang#6993
@alexcrichton alexcrichton mentioned this issue Jun 4, 2019
Merged
@alexcrichton
Copy link
Member

Ok thanks for the clarification, digging more I believe #7008 should at least fix one of the issues I saw (and was able to reproduce locally). Could you test that out and see if it fixes the issues you're seeing?

@khuey
Copy link
Contributor Author

khuey commented Jun 4, 2019

Unfortunately cargo master (and your PR) dies earlier with error: Unrecognized option: 'json-rendered' when CARGO_BUILD_PIPELINING=true, so I can't actually test it.

@alexcrichton
Copy link
Member

To double check, you're doing the equivalent of rustup run nightly /path/to/cargo/target/debug/cargo build?

@khuey
Copy link
Contributor Author

khuey commented Jun 4, 2019

Apparently not!

That gets me farther, to crate rusoto_ec2 required to be available in rlib format, but was not found in this form which is similar to but different from the original error.

@alexcrichton
Copy link
Member

Can you send me the logs for that as well?

@khuey
Copy link
Contributor Author

khuey commented Jun 4, 2019

After I cleared out the existing target/ directory to get a log from scratch, I haven't been able to reproduce this with your branch.

So it appears that it is in fact fixed \o/

bors added a commit that referenced this issue Jun 5, 2019
@bors
Auto merge of #7008 - alexcrichton:less-pipelining, r=ehuss
afcd5ef 
Handle pipelined tests of libraries

The previous implementation of pipelining accidentally forgot to account
for rlibs being compiled in `--test` mode. The compilations would be
pipelined to where all the dependencies of an rlib might not be
available yet, but `--test` actually performs linking in rustc!

This commit fixes the issue by refactoring slightly and removing
`Target::requires_upstream_objects` (moving it to `TargetKind`) and then
making the source of truth a `Unit::requires_upstream_objects` method
which takes into account the value from `TargetKind` as well as the
`CompileMode`.

Closes #6993
@bors bors closed this as completed in #7008 Jun 5, 2019
@brson brson mentioned this issue Jul 24, 2019
Closed
@brson
Copy link
Contributor

brson commented Jul 24, 2019

Seen in tikv here tikv/tikv#5130

@ctz ctz mentioned this issue Feb 9, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-bug Category: bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Handle pipelined tests of libraries alexcrichton/cargo
4 participants
@alexcrichton @brson @khuey @nnethercote