Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#[used(compiler)] and #[used(linker)] should work on functions. #94348

Open
AzureMarker opened this issue Feb 25, 2022 · 13 comments
Open

#[used(compiler)] and #[used(linker)] should work on functions. #94348

AzureMarker opened this issue Feb 25, 2022 · 13 comments
Labels
T-lang Relevant to the language team, which will review and decide on the PR/issue.

Comments

@AzureMarker
Copy link
Member

AzureMarker commented Feb 25, 2022

Related to #93798
cc @cynecx since you did the implementation.
cc @Meziu and @ian-h-chamberlain

The #[used(compiler)] and #[used(linker)] should work on functions. They weren't explicitly one of the main motivating use cases brought up in #47384, but expanding them to support functions would be very beneficial. For example, in https://github.com/Meziu/pthread-3ds and https://github.com/Meziu/rust-linker-fix-3ds we have to have an empty "init" function just so the exported functions don't get thrown out.

I was expecting these annotations to fix our problem, so I was excited to try them out when I heard they hit nightly. However, the following fails to compile:

#![feature(used_with_arg)]

#[used(compiler)]
#[no_mangle]
pub unsafe extern "C" fn foo() {}

playground link

error: attribute must be applied to a `static` variable
 --> src/lib.rs:3:1
  |
3 | #[used(compiler)]
  | ^^^^^^^^^^^^^^^^^

Interestingly, changing to #[used(linker)] results in an extra message (Edit: fixed by #94377):

error: `used(compiler)` and `used(linker)` can't be used together
 --> src/lib.rs:3:1
  |
3 | #[used(linker)]
  | ^^^^^^^^^^^^^^^
4 | #[no_mangle]
  | ^^^^^^^^^^^^

Does no_mangle imply used(compiler)?

@cynecx
Copy link
Contributor

cynecx commented Feb 25, 2022

Does no_mangle imply used(compiler)?

no_mangle should not imply used(compiler). This is a diagnostics issue and an oversight. #94377 will fix that.

I think supporting used on functions should be possible, however someone will have to do the impl work (Unfortunately, I don't have much time to work on this).

@cynecx
Copy link
Contributor

cynecx commented Feb 25, 2022

For example, in https://github.com/Meziu/pthread-3ds and https://github.com/Meziu/rust-linker-fix-3ds we have to have an empty "init" function just so the exported functions don't get thrown out.

I might be wrong but I'd imagine something like this should work as a workaround?

#![feature(used_with_arg)]

#[used(compiler)]
pub static _FOO: unsafe extern "C" fn() -> () = foo;

#[no_mangle]
pub unsafe extern "C" fn foo() {}

@nikic
Copy link
Contributor

nikic commented Feb 25, 2022

Agree that supporting #[used] on functions makes sense, and I'd expect that this would mostly "just work" if the restriction to statics is dropped. This applies to #[used] in general though, not really related to the linker/compiler argument.

matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Feb 25, 2022
`check_used` should only look at actual `used` attributes

cc? rust-lang#94348
r? `@nikic`
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Feb 25, 2022
`check_used` should only look at actual `used` attributes

cc? rust-lang#94348
r? ``@nikic``
@AzureMarker
Copy link
Member Author

AzureMarker commented Feb 26, 2022

I might be wrong but I'd imagine something like this should work as a workaround?

Unfortunately that doesn't work because we need the actual functions to be exported. We're polyfilling/implementing well known APIs like pthread. Unless of course that workaround also lets the function symbol stay around, but I assume it only keeps the static's symbol.

@nagisa nagisa added the T-lang Relevant to the language team, which will review and decide on the PR/issue. label Feb 27, 2022
@AzureMarker
Copy link
Member Author

I tested the static workaround and it doesn't seem to work (with used(compiler) or used(linker)).

@carbotaniuman
Copy link
Contributor

@AzureMarker Ok I think I figured it out, and I was kinda XYed. Rust will just ignore your crate if you put it in the dependency list, but don't make another reference to it. Even with the used(compiler)/used(linker) I just implemented. You'd need to add an extern crate xxx to whatever code expects the code to be linked. The used is somewhat of a red herring honestly.

@AzureMarker
Copy link
Member Author

Huh, I thought I already tested that. I'll check again just to make sure.

@carbotaniuman
Copy link
Contributor

As bjorn3 said, #[no_mangle] (and also #[link_name]) already do the right thing. Anybody who needs this is likely running into #47384. I'm not sure if there's any use case for exporting a function with a Rust mangled name, so I've closed my PR, but if anyone has any use cases please let me know.

@AzureMarker
Copy link
Member Author

AzureMarker commented Mar 29, 2022

@carbotaniuman I tested this again with extern crate and commenting out my empty init call (to force the crate/module to be "reachable"), and it has linking errors. So just adding an extern crate doesn't solve it.

The issue I'm seeing is #47384 like you said, but this comment suggested to me that the problem would be fixed by #[used(linker)]. The goal is to get the function not stripped out by the linker. It's already in the compiled object (ex. from #[no_mangle]), but the linker doesn't think it's needed at first, and then fails later when it is needed.

@AzureMarker
Copy link
Member Author

Maybe #93791 will fix this?

@carbotaniuman
Copy link
Contributor

@AzureMarker What crate-type are you building your crate as? Do you have a single repo that I can just build to test it out?

@AzureMarker
Copy link
Member Author

AzureMarker commented Apr 1, 2022

@carbotaniuman I see it with some crates for the Nintendo 3DS (in progress std port) which are compiling/linking to create an ELF file, but I'll try to make a minimal reproduction that runs on Linux.

@AzureMarker
Copy link
Member Author

AzureMarker commented Apr 3, 2022

@carbotaniuman Here is a repository that reproduces the issue:
https://github.com/AzureMarker/reproduce-used-linker-function

The main issue spawns from a requirement in our rustc fork for the 3DS (to be upstreamed eventually). We don't want to link directly to the "libc" library on the 3DS because it's homebrew, and potentially causes legal issues (ex. using the low-level APIs of the 3DS kernel, see #88529 (comment)). Due to this, we have a few crates providing the libc implementation which are pulled in as regular dependencies, not std dependencies. They export well known libc functions which std uses. For example: https://github.com/Meziu/rust-linker-fix-3ds

This reproduction repo has a "pretend-std" crate which represents std in this scenario. It relies on an extern "C" function which is defined in the main crate's lib.rs module (representing the homebrew libc). The main crate's lib.rs module holds the implementation of the extern function. And the main crate's main.rs module uses a "pretend-std" function.

The issue is the link order, and how the linker drops unused symbols. Here is a snippet of RUSTFLAGS="--print link-args" cargo +nightly build:

(main module *.rcgu.o files, plus -L flags)
...
"-Wl,-Bstatic"
"/home/mark/media/Projects/CLionProjects/reproduce-used-linker-function/target/debug/deps/libreproduce_used_linker_function-ecc342a8631bc5ff.rlib"
"/home/mark/media/Projects/CLionProjects/reproduce-used-linker-function/target/debug/deps/libpretend_std-64be2b92808cfdc9.rlib"
"/home/mark/media/Projects/CLionProjects/reproduce-used-linker-function/target/debug/deps/liblibc-04dd900d425ce5fe.rlib"
...
(std rlibs, pthread, etc)

The lib.rs module is linked in first. This module included the implementation of the exported function, but since the exported function hasn't been required yet according to the linker, it's dropped. Second is the "pretend-std" crate, which requires the exported function. At this point the exported function is now required, but the implementation was already dropped. Third is just libc, which is an implementation detail (used in the exported function's signature). If #[used(linker)] supports functions, the hope is that the exported function would survive the linker until it's finally used in "pretend-std".

In the 3DS case, we have multiple exported functions like this which are used by std, but thrown away by the linker before it starts linking in std (and realizes the functions are actually used). I see two possible fixes:

  1. Use the homebrew libc + supplimentary crates (the homebrew libc doesn't support everything Rust's std requires) as explicit dependencies of std for the 3DS target. So far this is not a reasonable option due to legal concerns. Additionally, there are multiple 3DS libc implementations to choose from (we are using libctru for our testing).
  2. Support #[used(linker)] on functions, so they don't get dropped by the linker when it thinks they aren't used.
  3. Add an empty "init" function to each module which exports extern functions, and call it from the main module. This is what we do currently in our 3DS projects. The reproduction crate has this implemented as well, but commented out by default (see main.rs). This doesn't seem to change the link command flags (they are identical), but it does make compilation work. Maybe the main.rs module now requires the lib.rs c_impl module, and the linker keeps the whole c_impl module alive that way? Basically no_mangle/used static is only present in output when in reachable module #47384.

For more detail, here are the symbol tables of the lib.rs module and the pretend-std module (same with/without the init function workaround). This shows that the exported function is exported in the object (i.e. no_mangle implies #[used(compiler)]) and that pretend-std does require it:

objdump -t /home/mark/media/Projects/CLionProjects/reproduce-used-linker-function/target/debug/deps/libreproduce_used_linker_function-ecc342a8631bc5ff.rlib
In archive /home/mark/media/Projects/CLionProjects/reproduce-used-linker-function/target/debug/deps/libreproduce_used_linker_function-ecc342a8631bc5ff.rlib:

lib.rmeta:     file format elf64-x86-64

SYMBOL TABLE:
no symbols



reproduce_used_linker_function-ecc342a8631bc5ff.gv6yiuwmazf51lt.rcgu.o:     file format elf64-x86-64

SYMBOL TABLE:
0000000000000000 l    df *ABS*  0000000000000000 gv6yiuwmazf51lt
0000000000000000 l    d  .text._ZN30reproduce_used_linker_function6c_impl4init17h4984550cceff1353E      0000000000000000 .text._ZN30reproduce_used_linker_function6c_impl4init17h4984550cceff1353E
0000000000000000 l    d  .text.exported_function        0000000000000000 .text.exported_function
0000000000000000 l    d  .debug_abbrev  0000000000000000 .debug_abbrev
0000000000000000 l    d  .debug_info    0000000000000000 .debug_info
0000000000000000 l    d  .debug_ranges  0000000000000000 .debug_ranges
0000000000000000 l    d  .debug_str     0000000000000000 .debug_str
0000000000000000 l    d  .debug_line    0000000000000000 .debug_line
0000000000000000 g     F .text._ZN30reproduce_used_linker_function6c_impl4init17h4984550cceff1353E      0000000000000001 _ZN30reproduce_used_linker_function6c_impl4init17h4984550cceff1353E
0000000000000000 g     F .text.exported_function        0000000000000006 exported_function
0000000000000000  w    O .debug_gdb_scripts     0000000000000022 __rustc_debug_gdb_scripts_section__
objdump -t /home/mark/media/Projects/CLionProjects/reproduce-used-linker-function/target/debug/deps/libpretend_std-64be2b92808cfdc9.rlib
In archive /home/mark/media/Projects/CLionProjects/reproduce-used-linker-function/target/debug/deps/libpretend_std-64be2b92808cfdc9.rlib:

lib.rmeta:     file format elf64-x86-64

SYMBOL TABLE:
no symbols



pretend_std-64be2b92808cfdc9.20vs44k0vbd9kd45.rcgu.o:     file format elf64-x86-64

SYMBOL TABLE:
0000000000000000 l    df *ABS*  0000000000000000 20vs44k0vbd9kd45
0000000000000000 l    d  .text._ZN11pretend_std5value17heb875d6cc6951afdE       0000000000000000 .text._ZN11pretend_std5value17heb875d6cc6951afdE
0000000000000000 l    d  .debug_abbrev  0000000000000000 .debug_abbrev
0000000000000000 l    d  .debug_info    0000000000000000 .debug_info
0000000000000000 l    d  .debug_str     0000000000000000 .debug_str
0000000000000000 l    d  .debug_line    0000000000000000 .debug_line
0000000000000000 g     F .text._ZN11pretend_std5value17heb875d6cc6951afdE       0000000000000011 _ZN11pretend_std5value17heb875d6cc6951afdE
0000000000000000         *UND*  0000000000000000 exported_function
0000000000000000  w    O .debug_gdb_scripts     0000000000000022 __rustc_debug_gdb_scripts_section__

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
T-lang Relevant to the language team, which will review and decide on the PR/issue.
Projects
None yet
Development

No branches or pull requests

5 participants