target/debug/deps folder not propagated in LD_LIBRARY_PATH

[Larswad]

Using tarpauling 0.25.1.
I have a crate building a shared library (.so) with c - code (build.rs).
I have another rust-only crate exposing FFI code to that library (calls made in both directions, rust calls into C code, rust expose C FFI functions to be called from the library).
The rust crate has in it's Cargo.toml set a dependency to the dynamic C library:

[dev-dependencies]
my-library.workspace = true

In the rust crate I have a test in the tests folder, loading the C library dynamically using libloading (e.g. dlopen).
Running this with cargo test works fine, with tarpaulin the library is not found.
I inspected the LD_LIBRARY_PATH in both cases, when running tarpaulin the:

/target/debug/deps

folder was missing (but not when running cargo test) and is the reason for the failure.
I am not sure if I have missed anything that I should have given to tarpaulin or if it's a problem not specific to tarpaulin but the inconsistency here bring me the think it's an issue.

To Reproduce
Build a c library using build.rs, it's enough with one C function in there.
Build a rust crate using the function in the C library.
Create a test using libloading that loads up the C library, calling get to resolve the C library's function.
Call it.

Expected behavior
The C library should have been found by the test code when running tarpaulin, just like when using cargo test.
The /target//deps folder should be included in the LD_LIBRARY_PATH during test execution.

[Larswad]

If useful, the:
/target/debug
folder was also missing in the tarpaulin build in LD_LIBRARY_PATH.

[xd009642]

Out of curiosity if you do:

cargo test --no-run
./target/debug/$TEST

Where $TEST is the compiled test, so something like deps/$MODULE_$HASH does it also fail to run with a linker issue? If so that's not really something I can/will fix in tarpaulin - as I can't identify temporary changes to LD_LIBRARY_PATH when building tests that aren't available before running it or after running it.

A bunch of crates using native code will work if you do cargo test but then if you run the tests separately afterwards will require you to set the LD_LIBRARY_PATH yourself.

[Larswad]

Thanks. I am not completely sure what you intend there to check. But I specified the path like:
cargo test --no-run file://target/debug/deps/-hash
(Had to specify as file:// apparantly).
That just doesn't seem to do any meaningful though although nothing fails, hard to describe what I mean, but it doesn't seem to care about the binary I give to it, regardless of --no-run or not.

Just to be clear, it is the environment's LD_LIBRARY_PATH that is not the same for running cargo test as for tarpaulin while running the test (not during building of the test). Since this library is loaded dynamically in runtime (not as a "statically bound" dynamic library if you know what I mean).
So regardless of reason i'm curious why the environment clearly is different, even though i'm not clear on how tarpauling works procedureally in the build/execution process. Does tarpaulin have access to the
target/<buildconfig>/
and
target/<buildconfig>/deps/
folders? If so, I mean, I guess tarpauling could add those path to LD_LIBRARY_PATH by itself, before executing the test.
The .so binary is both present in the target/debug and target/debug/deps folders.

[xd009642]

So you run cargo test --no-run just to compile the tests. And then run one of the compiled tests.

Some native library bindings will actually add to the linker path dynamically via the build script and download the shared objects to a folder in the target dir which may not be the same place. They may also choose not to add that folder to LD_LIBRARY_PATH if a user specifies they want to use a system installed one and not a downloaded one - but still have a downloaded one in their target dir from a previous run.

of course if cargo test always modifies the LD_LIBRARY_PATH in a predictable way I can emulate that - but I have a feeling it doesn't and it's partially on the crate authors doing things in their build.rs files to enable cargos run environment to discover them

[Larswad]

So I misunderstood you, it's too separate things. Sorry.
I first cargo test --no-run
(needed to specify features in my case since we have some mutually exclusive custom features depending on architecture).
Then launch the test by simply executing it manually giving that path target/debug/deps/...testname. Yes, that fails like you suspected.
Only if I prefix with:
LD_LIBRARY_PATH=target/debug it is fine.
If you feel this is kind of stretches things a bit for tarpaulin in a wrong way I'm fine with closing it.
Just don't know how to get out of this situation now. We run tarpaulin in our CI.
We had excluded cfg directives using --avoid-cfg-tarpaulin (found out the reason later, it was because another crate, coap-lite uses an old tag, "skip", that we have no control over, the one that worked for me was #[cfg_attr(tarpaulin, ignore)] (because I'm even fine with ignoring this specific test for tarpaulin.
So, if I ignore it (and remove --avoid-cfg-tarpaulin) I get problems with coap-lite instead.
Bummer.

[Larswad]

I found out that env!() macro in my test gives me the LD_LIBRARY_PATH as it were during build-time (in which the deps folder is actually there). I could in the test loop over the paths, concat with the library name and try loading until one of the paths worked. That solves my issue in the best way.
If you'd ever consider this as an issue, the env!() macro there could be useful to populate the runtime LD_LIBRARY_PATH for tarpaulin. Otherwise, thanks for responding so kindly!

[xd009642]

I don't think the env macro would work for tarpaulin in that case as you're not rebuilding tarpaulin every time you run it. But I will look into what cargo test is doing during run and see if there's something I can add to smooth the workflow.

Personally in $DAY_JOB I set the LD_LIBRARY_PATH for tensorflow and ffmpeg libraries which we install in CI and link to. I prefer that over relying on whatever crate I'm using for bindings downloading one as it has in the past broken things - or resulted in CI testing a different set of dependencies than we're using in prod

[Larswad]

Thanks for the advice. Will discuss it in the team.

[Luthaf]

In general,cargo test will add such directories to the library loading path; and rustdoc recently started to do the same (rust-lang/cargo#8531, fixed in 1.78 by rust-lang/cargo#13490).

It would be nice if cargo-tarpaulin could do the same, so it does not require a separate setup from the other cargo tools!

EDIT: a bit more details: cargo seems to add folders like target/<debug>/build/<crate>-<hash>/out/ to the library search path, NOT target/debug/deps. The first one is where a build script would place it's output when compiling a shared library. Cargo might also just be taking the output of cargo:rustc-link-search=native=<path> from the build script outputs, and not treating target/<debug>/build/<crate>-<hash>/out/ specially. If the information is available, the latter seems like the best way to handle this for tarpaulin.