Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#[global_allocator] in dependent crate can be optimized away #64402

Open
jorendorff opened this issue Sep 12, 2019 · 11 comments
Open

#[global_allocator] in dependent crate can be optimized away #64402

jorendorff opened this issue Sep 12, 2019 · 11 comments
Labels
A-allocators Area: Custom and system allocators A-linkage Area: linking into static, shared libraries and binaries C-bug Category: This is a bug. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Comments

@jorendorff
Copy link
Contributor

jemallocator-global is a library intended to make jemalloc your allocator. It works by declaring a static allocator with #[global_allocator].

However, a downstream crate linked with jemallocator-global nonetheless uses the system malloc on MacOS X.

After experimenting a bit with #[global_allocator], it's behaving as though rustc optimizes away a whole crate if you don't seem to be using it; and it doesn't detect that the global allocator is always used.

@jorendorff
Copy link
Contributor Author

It looks like extern crate jemallocator_global; might be enough to get the allocator to be linked and used. So maybe this is intentional behavior? It's surprising from a user's perspective, though.

@gnzlbg
Copy link
Contributor

gnzlbg commented Sep 12, 2019

I think that #[globall_allocator] should maybe imply #[used], cc @alexcrichton

@alexcrichton
Copy link
Member

@jorendorff to confirm you added jemallocator-global = "..." to Cargo.toml, but the application didn't work as expected to use jemalloc? And then adding extern crate jemallocator_global; fixed the issue?

@jorendorff
Copy link
Contributor Author

First question, yes.

Second question, no. Instead, I made a separate project to play with, and in that project extern crate has this effect. I'll post that as a repo in a sec.

@mati865
Copy link
Contributor

mati865 commented Sep 12, 2019

a downstream crate linked with jemallocator-global nonetheless uses the system malloc on MacOS X.

AFAIK jemallocator-global and #[globall_allocator] are meant to be used in the topmost binary (the one which depends on other crates). Or am I misunderstanding?

@jorendorff
Copy link
Contributor Author

Here's the repo that shows the bug: https://github.com/jorendorff/rust-lang-bug-64402/

The allocator in this repo panics on alloc, so it's easy to tell if it was used or not.

Adding #[used] to the allocator didn't fix the issue.

@jorendorff
Copy link
Contributor Author

jorendorff commented Sep 12, 2019

AFAIK jemallocator-global and #[globall_allocator] are meant to be used in the topmost binary (the one which depends on other crates). Or am I misunderstanding?

jemallocator-global contains a #[global_allocator]: https://docs.rs/jemallocator-global/0.3.2/src/jemallocator_global/lib.rs.html#35

@alexcrichton
Copy link
Member

Ok, thanks for the clarification! In that case, yes, #[used] isn't going to fix this. At this time there's no way to fix this other than using extern crate I believe.

Most crates when linked don't actually have any side effect (e.g. just a bland rlib), but some crates (like the global allocator) when linked actually affect how the compiler structures things. To be conservative the --extern flag on the command line doesn't actually link the crate, it just provides it as a candidate for linkage. If you use anything from the crate that's considered to link it, but if you don't actually use it then the crate isn't linked at all. In this case adding extern crate is explicitly linking the crate. You could perhaps also fix this as use jemallocator_global somewhere (or some path from the crate).

As a bug the compiler could perhaps give special treatment to --extern arguments that have a #[global_allocator], but that's also a tricky path to go down. This is sort of just an unfortunate bug in general, and I'm not sure it has a great solution.

@gnzlbg
Copy link
Contributor

gnzlbg commented Sep 12, 2019

Is this also a bug that, e.g., #[no_std] crates defining a #[panic_handler] that are put in the dependency graph but aren't "used" would have ?

@mati865
Copy link
Contributor

mati865 commented Sep 12, 2019

@jorendorff this is what I mean by topmost crate: https://gist.github.com/mati865/05f96512240e974dec0ca9740316bde6 the same thing should apply for jemallocator-global crate.

@alexcrichton
Copy link
Member

@gnzlbg I don't really know for sure, I'd try it out. If that works then #[global_allocator] is missing treatment that #[panic_handler] has. If it doesn't work then it's probably the same issue.

@Mark-Simulacrum Mark-Simulacrum added A-linkage Area: linking into static, shared libraries and binaries T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. C-bug Category: This is a bug. labels Sep 14, 2019
adetaylor added a commit to adetaylor/rust that referenced this issue Jun 21, 2020
Manishearth added a commit to Manishearth/rust that referenced this issue Jun 23, 2020
kdy1 added a commit to kdy1/swc that referenced this issue May 4, 2021
kdy1 added a commit to kdy1/swc that referenced this issue May 5, 2021
@Enselic Enselic added the A-allocators Area: Custom and system allocators label Dec 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-allocators Area: Custom and system allocators A-linkage Area: linking into static, shared libraries and binaries C-bug Category: This is a bug. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

No branches or pull requests

6 participants