Make sure search paths inside OUT_DIR precede external paths

[vlovich]

If a library exists both in an added folder inside OUT_DIR and in the OS, prefer to use the one within OUT_DIR. Folders within OUT_DIR and folders outside OUT_DIR do not change their relative order between themselves.

This is accomplished by sorting by whether we think the path is inside the search path or outside.

What does this PR try to resolve?

Fixes #15220. If a Rust crates builds a dynamic library & that same dynamic library is installed in the host OS, the result of the build's success & consistent behavior of executed tools depends on whether or not the user has the conflicting dynamic library in the external search path. If they do, then the host OS library will always be used which is unexpected - updates to your Rust dependency will still have you linking & running against an old host OS library (i.e. someone who doesn't have that library has a different silent behavior).

How should we test and review this PR?

This is what I did to verify my issue got resolved but I'm sure there's a simpler example one could construct.

• Make sure Alsa and libllama.so are installed (on Arch I installed alsa-lib and llama.cpp-cuda).
• Clone llama-cpp-2 & init llama.cpp submodule & update the submodule to point to llama : expose llama_model_n_head_kv in the API ggml-org/llama.cpp#11997 instead.
• Add plumbing to expose the new method within llama-cpp-2 as a public facing function on the LlamaModel struct (it's basically the same code as for n_head, just calling n_head_kv from llama.cpp).
• Add cpal as a dependency in crate "foo"
• Add llama-cpp-2 via path as a dependency in crate "foo" and enable the dynamic-link feature.
• Add code using the newly expose n_head_kv method in crate "foo" in main.rs. NOTE: Code just needs to compile & be exported, doesn't have to be correct (fn main is probably easiest.
• Add some basic code that tries to initialize cpal in crate "foo" in fn main.
• Try to build / run crate "foo"

Before my change, it fails with a linker error saying it can't find llama_model_n_head_kv because /usr/lib appears in the search path before the directory that contains the libllama.so that was built internally by the crate. This is because cpal depends on alsa-sys which uses pkg-config which adds /usr/lib to the search path before the llama-cpp-sys-2 build.rs is run.

Additional information

I'm not sure how to add tests so open to some help on that. I wanted to make sure that this approach is even correct. I coded this to change Cargo minimally and defensively since I don't know the internals of Cargo very well (e.g. I don't know if I have to compare against both script_out_dir / script_out_dir_when_generated since I don't know the difference & there's not really any explanation on what they are).

It's possible this over-complicates the implementation so open to any feedback. Additionally, the sort that happens prior to each build up of the rustc environment is not where I'd ideally place it. I think it would be more efficient to have the list of search paths be free-floating and not tied to a BuildOutput so that they could be kept updated live & resorted only on insertion (since it's changed less frequently than rustc is invoked). Additionally, the generalized sort is correct but pessimistic - maintaining the list sorted could be done efficiently with some minor book keeping (i.e. you'd only need to sort the new paths & then could quickly inject into the middle of a VecDeque).

And of course in terms of correctness, I didn't do a thorough job testing across all possible platforms. From first principles this seems directionally correct but it's always possible this breaks someone else's workflow. I'm also uneasy that the relative position of -L / -l arguments changes in this PR & I'm not sure if that's observable behavior or not (i.e. it used to be -L for a crate followed by -l for a crate), but now it's -L for all crates, still grouped by crated internally, followed by -l by crate).

[rustbot]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @ehuss (or someone else) some time within the next two weeks.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

• @rustbot author: the review is finished, PR author should check the comments and take action accordingly
• @rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

[vlovich]

Not sure what the CI failure for "check-version-bump" is but I'm assuming it's some minor book keeping that's validating a version number somewhere needs to be bumped because I made a consequential change? But the output when running locally doesn't make sense to me:

--- failure function_requires_different_generic_type_params: function now requires a different number of generic type parameters ---

Description:
A function now requires a different number of generic type parameters than it used to. Uses of this function that supplied the previous number of generic types will be broken.
        ref: https://doc.rust-lang.org/reference/items/generics.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.39.0/src/lints/function_requires_different_generic_type_params.ron

Failed in:
  function strip_prefix_canonical (1 -> 0 generic types) in /home/vlovich/projects/cargo/crates/cargo-util/src/paths.rs:713

I haven't touched strip_prefix_canonical - running cargo semver-checks check-release --workspace on master seems to indicate this is broken there too so I think this step might just be broken for everyone right now.

[weihanglo]

That CI job failure is completely unrelated to this PR. See #15222.

[weihanglo]

Just note that it is generally recommended an issue to be discussed and labeled as S-accepted and than open a PR, though you are welcome to open this as a draft PR and hack with it :)

https://doc.crates.io/contrib/process/working-on-cargo.html#before-hacking-on-cargo

[vlovich]

Good note but I also wanted to see how difficult such a change would be to help guide any questions that might come up in the issue.

[vlovich]

I think I've resolved all the outstanding review requests. Please let me know if there's any other concerns. I did add a comment to the LibraryPath documentation that may be worth discussing (but also OK just leaving this for future consideration):

/// WARNING: Even though this type implements PartialOrd + Ord, this is a lexicographic ordering.
/// The linker line will require an explicit sorting algorithm. PartialOrd + Ord is derived because
/// BuildOutput requires it but that ordering is different from the one for the linker search path,
/// at least today. It may be worth reconsidering & perhaps it's ok if BuildOutput doesn't have
/// a lexicographic ordering for the library_paths? I'm not sure the consequence of that.

@rustbot review

[weihanglo]

@rustbot author

[vlovich]

@rustbot review

[weihanglo]

Thank you for working with us! I really love how you documented the rationale and how the code is organized.

This looks great. Going to merge it :)