Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiple matching crates error, if dependency on compiletest is optional #114

Open
hannobraun opened this issue May 23, 2018 · 21 comments
Open

Comments

@hannobraun
Copy link

I'm trying to use compiletest for a #[no_std] crate. Because the crate contains examples that need to be built for an embedded platform where std is not available (thumbv6m-none-eabi), I needed to make the dependency on compiletest optional and add a Cargo feature for it.

Further details:

This works works fine:

$ cargo clean
$ cargo test --features compiletest

This does not work:

$ cargo test
$ cargo test --features compiletest

It results in the following error (excerpt from full build log):

status: exit code: 101
command: "rustc" "tests/compile-fail/swm/assign-function-to-pin-with-unknown-state.rs" "-L" "/tmp" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-C" "prefer-dynamic" "-o" "/tmp/swm/assign-function-to-pin-with-unknown-state.stage-id" "-L" "/home/travis/build/braun-robotics/rust-lpc82x-hal/target/debug/build/cortex-m-rt-e3337e5285e3af86/out" "-L" "/home/travis/build/braun-robotics/rust-lpc82x-hal/target/debug" "-L" "/home/travis/build/braun-robotics/rust-lpc82x-hal/target/debug/deps" "-L" "/home/travis/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-L" "/home/travis/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib" "-L" "/tmp/swm/assign-function-to-pin-with-unknown-state.stage-id.aux" "-A" "unused"
unexpected errors (from JSON output): [
    Error {
        line_num: 1,
        kind: Some(
            Error
        ),
        msg: "1:1: 1:25: multiple matching crates for `lpc82x_hal` [E0464]"
    },
    Error {
        line_num: 1,
        kind: Some(
            Error
        ),
        msg: "1:1: 1:25: found crate `lpc82x_hal` compiled by an incompatible version of rustc [E0514]"
    }
]

I believe the important part is multiple matching crates. I think I understand what happens here: We have two versions of the crate, one with the feature, one without, and rustc doesn't know which one to use. Is this something that could be fixed in clean_rmeta?

@laumann
Copy link
Collaborator

laumann commented May 23, 2018

Yes, this should be something that's fixable by clean_rmeta(), although that function was created to deal with cargo check followed by cargo test. I wonder if this is the same issue as #101

@laumann
Copy link
Collaborator

laumann commented May 23, 2018

Hmm, looking at clean_rmeta(), could it be that the filter:

            .filter(|s| s.ends_with("/deps"))

is the problem?

@laumann
Copy link
Collaborator

laumann commented May 23, 2018

You might also want to try not using link_deps() and just set config.target_rustcflags manually.

@hannobraun
Copy link
Author

Thanks for your replies! Just FYI, I won't be able to follow up here today, and tomorrow I'm off to RustFest. I'll check back in next week, or the week after.

@laumann
Copy link
Collaborator

laumann commented May 24, 2018

No problem 😄 This is a problem that's been a thorn in my side for some time, so it'd be nice to have it solved!

Have fun at RustFest! Say hi to the guys from Seasoned Software if you catch them!

@hannobraun
Copy link
Author

Have fun at RustFest! Say hi to the guys from Seasoned Software if you catch them!

Thank you! I don't know who they are, but I'll keep my eyes open :)

@hannobraun
Copy link
Author

@laumann I did some research based on your comments.

Hmm, looking at clean_rmeta(), could it be that the filter:

       .filter(|s| s.ends_with("/deps"))

is the problem?

I don't think so. I commented it out and it made no difference. In any case, clean_rmeta() only removes .rmeta files, and if my understanding is correct, that shouldn't be relevant in this case. I have multiple rlibs from the different invocations of cargo test, and those are probably the problem.

You might also want to try not using link_deps() and just set config.target_rustcflags manually.

I didn't really know what to set it to, so I checked out what link_deps does, to get a better understanding. It seems that link_deps just gives gives rustc the directories with -L. cargo test on the other hand passes the path of the actual rlibs (e.g. --extern lpc82x_hal=/home/hanno/Projects/rust-lpc82x-hal/target/debug/deps/liblpc82x_hal-9e595ce07b7a25b7.rlib).

Maybe compiletest should do the same, but I don't know how it would know which file is the right one. For reference, I think this is the relevant piece of code for cargo test: https://github.com/rust-lang/cargo/blob/3dbae343acc7b929fa045ed9153179b358bbc116/src/cargo/core/compiler/mod.rs#L923

@laumann
Copy link
Collaborator

laumann commented May 29, 2018

Thanks for looking into this.

Yes, you are correct, link_deps() just adds folders for rustc to find libraries. Usually you'd just set target_rustcflags to -L target/debug or something (the folder where compiled libs are placed).

Sounds like the rlibs might be the problem, as you say. When cargo test fails, do you have multiple versions of the same rlibs? What happens if you then try to specifically remove all rlibs before another cargo test?

@hannobraun
Copy link
Author

When cargo test fails, do you have multiple versions of the same rlibs?

Yes. After a cargo clean && cargo test && cargo test --features compiletest, I have these two rlibs in target/debug/deps:

liblpc82x_hal-9e595ce07b7a25b7.rlib
liblpc82x_hal-d597ff95b0e3d874.rlib

And another one in target/debug:

liblpc82x_hal.rlib

The first cargo test creates the rlib in target/debug and one in target/debug/deps. The cargo test --features compiletest adds the second one in target/debug/deps.

Interestingly, if I run just cargo clean && cargo test --features compiletest, it also creates the rlib in target/debug, in addition to the one in target/debug/deps.

What happens if you then try to specifically remove all rlibs before another cargo test?

If I remove the rlibs (and just the rlibs), both variants of cargo test succeed after that.

@laumann
Copy link
Collaborator

laumann commented May 30, 2018

So we can pretty much say that the "duplicate" rlibs are the problem. Question is then, how do we fix it?

Can we identify which is which? If so, we might be able to just add those as link flags, instead of the entire folder. Otherwise I'm not what to do.

Surely, Cargo has a way of knowing which is which?

@hannobraun
Copy link
Author

Can we identify which is which? If so, we might be able to just add those as link flags, instead of the entire folder. Otherwise I'm not what to do.

Surely, Cargo has a way of knowing which is which?

I've taken a look at the Cargo source trying to find that out, but didn't get very far before I ran out of time. Certainly it must know, but the question is how to get that knowledge out of Cargo and into compiletest.

I plan to take a closer look at the Cargo code next week (if I find the time).

@laumann
Copy link
Collaborator

laumann commented May 30, 2018

OK, thanks again for looking into it 😃

@hannobraun
Copy link
Author

I've taken another look at the Cargo code, but didn't come up with anything useful before running out of time. I've now added a workaround on my side (disabled caching on Travis), and have decided to leave this be for now.

Sorry, I would have liked to fix this properly, but my time is limited and I have to pick my battles.

Keats pushed a commit to Keats/validator that referenced this issue Jun 28, 2018
@MoSal
Copy link
Contributor

MoSal commented Aug 13, 2018

meh, this is kind of annoying. One has to always rm -rf target before running cargo test to make
sure this bug will not be hit.

@aldanor
Copy link

aldanor commented Nov 17, 2018

Just ran into this as well (#147). The .clean_rmeta() indeed removes target/debug/deps/*.rmeta files, however that doesn't seem sufficient, and you still run into both E0463 (crate not found) and E0464 (multiple matching crates) errors, sometimes both at the same time, somehow.

Has anyone figured out what exactly needs to be cleaned so it works in a stable fashion?

@aldanor
Copy link

aldanor commented Nov 17, 2018

@laumann Ok, here's what seems to work for me:

  1. For each foo.rmeta file, also kill foo.* in the same folder (e.g., foo.d).
  2. Don't filter by */deps; do it in all link paths.

Using the cleanup code below (to replace .clean_rmeta()), I can now run cargo check or cargo clippy followed by cargo test and it wouldn't choke on E0463 / E0464.

Disclaimer: there may be some gotchas I'm not aware about but it seems to work so far for me.

use std::fs::{read_dir, remove_file};

fn clean_rlibs(config: &compiletest_rs::Config) {
    if config.target_rustcflags.is_some() {
        for directory in config.target_rustcflags
            .as_ref()
            .unwrap()
            .split_whitespace()
            {
                if let Ok(mut entries) = read_dir(directory) {
                    while let Some(Ok(entry)) = entries.next() {
                        let f = entry.file_name().clone().into_string().unwrap();
                        if f.ends_with(".rmeta") {
                            let prefix = &f[..f.len() - 5];
                            let _ = remove_file(entry.path());
                            if let Ok(mut entries) = read_dir(directory) {
                                while let Some(Ok(entry)) = entries.next() {
                                    let f = entry.file_name().clone().into_string().unwrap();
                                    if f.starts_with(prefix) && !f.ends_with(".rmeta") {
                                        let _ = remove_file(entry.path());
                                    }
                                }
                            }
                        }
                    }
                }
            }
    }
}

@laumann
Copy link
Collaborator

laumann commented Nov 18, 2018

Thanks for looking further into this @aldanor 😄

Without yet having tried it, is this a replacement for clean_rmeta() or an added step? In either case, it looks like something that would go in Config in src/common.rs.

I'm happy to have a PR with this.

@aldanor
Copy link

aldanor commented Nov 18, 2018

@laumann Np. Yea, this is a replacement to the current .clean_rmeta(), basically it just removes more files.

Disclaimer: I don’t have deep understanding of why this works, I just took a snapshot of target folder before and after running a cargo check/clippy and removed all the new files that appeared. That being said, this has been working perfectly fine for me the last few days with no issues.

I could open a PR if that helps :)

@laumann
Copy link
Collaborator

laumann commented Nov 18, 2018

@aldanor A PR would be nice - then we could ask others for feedback. I tried it out on the test-project and it seems to work fine.

@aldanor
Copy link

aldanor commented Nov 18, 2018

@laumann #148

@orium
Copy link
Contributor

orium commented Jul 20, 2023

I've avoided this issue by mimicking the way cargo calls rustc. I set the rustc flags like this:

config.target_rustcflags = Some("--edition=2021 -L dependency=target/debug/deps/ --extern mycrate=<path to mycrate rlib> --extern adependency=<path to adependency rlib>");

where mycrate is the name of your crate and adependency is a crate you depend on and want to access on your tests.

You can see an example of this working here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants