Inconsistencies when exporting dynamic symbols from executables

[acfoltzer]

I'm working on a project that uses dynamic loading with callbacks back into the Rust executable. Now that the project is big enough to require some of these callbacks to be defined in separate crates, I'm finding the behavior of when a symbol is exported or not to be very inconsistent.

I've tried to shrink the issue by creating a repo that contains several variants that do and don't work. The README describes the variations in more detail: https://github.com/acfoltzer/exported-symbol-example

The upshot is that based on #54451, I would expect that any #[no_mangle] definitions in both the executable crate and any dependencies to be exported from the final executable, assuming --export-dynamic is used for the final link step.

Instead, the exported #[no_mangle] callbacks are eliminated from the final executable in many situations. It's not as simple as whether the callbacks are used/reachable directly from main, but also seems to be influenced by the use of trait objects that happen to share a module with the callbacks. This doesn't seem like the intended behavior.

Meta

Reproducible on nightly and stable:

% rustc --version --verbose
rustc 1.32.0 (9fda7c223 2019-01-16)
binary: rustc
commit-hash: 9fda7c2237db910e41d6a712e9a2139b352e558b
commit-date: 2019-01-16
host: x86_64-unknown-linux-gnu
release: 1.32.0
LLVM version: 8.0

%rustc --version --verbose
rustc 1.34.0-nightly (c1c3c4e95 2019-01-29)
binary: rustc
commit-hash: c1c3c4e95b69dfeaca5c5db6c622d7f90ad30a54
commit-date: 2019-01-29
host: x86_64-unknown-linux-gnu
release: 1.34.0-nightly
LLVM version: 8.0

[acfoltzer]

Thinking about this issue some more, and the constant need to be adding --export-dynamic to RUSTFLAGS, I'm coming around to the idea that we need the ability to mark extern fns and statics as "always exported". This would both address this issue, and prevent all the downsides that come with using RUSTFLAGS. Is this something where I should propose an RFC, or is there another forum to discuss what this should look like?

Possibly related: #34131

[acfoltzer]

Some relevant discussion is occurring on #63909.

[alexcrichton]

@acfoltzer and I debugged this a bit today. My hunch was that this is the problem (here for MSVC). We don't actually tell the linker (unlike we do for wasm for example) that it should export #[no_mangle] pub extern symbols, so it doesn't actually pull it in unless the object is pulled in for some reason.

Apparently though for both the gold and bfd linkers that didn't work. Although --version-script was passed it didn't actually work. It did work, however, to pass --require-defined. Somehow we need to figure out a way to tell the linker "please export these symbols". Maybe we could synthesize an object file with references to the symbols? (unsure)

[acfoltzer]

Related to #47384