check libc before using SYS_getrandom on Linux

[ahgamut]

Hi,

I was porting HVM to run on Cosmopolitan Libc, and I noticed a small difference between how getrandom is handled by this crate when compared to the Rust standard library.

The Rust standard library has a syscall macro which creates a weak symbol for getrandom, so if the underlying libc provides getrandom, that function is called instead of making a syscall. The standard library uses getrandom for HashMap keys here.

The getrandom crate makes the Linux syscall directly without first checking for getrandom in the libc.

The pattern used in the Rust standard library is flexible to which libc is used, be it glibc, musl, or Cosmopolitan.

Could you change https://github.com/rust-random/getrandom/blob/master/src/linux_android.rs to check if getrandom is provided by the libc first, before using the syscall?

[josephlr]

Good catch, this library aims to have an identical implementation to the standard library (see rust-lang/rust#62079), so we shouldn't deviate unless it's necessary. I think adding this sort of detection would be fine. Our logic would then become:

• Use libc::getrandom if it exists (requires glibc 2.25 or musl 1.1.20)
• Use raw syscall if it exists (requires Linux 3.17 or a backported kernel like RHEL 7)
• Use the /dev/urandom file as the final fallback.

[newpavlov]

Personally, on Linux I prefer using syscalls directly, to the point of thinking that ideally we should not rely on libc::syscall and instead use something like the syscall crate (it's a real shame in my opinion that we do not have a libc-free std/target).

Regardless of the approach used by std, what are the practical motivations for prioritizing libc::getrandom over libc::syscall + libc::SYS_getrandom? On top of my head I can think of the following:

• Support of libc::getrandom on pre-3.17 kernels. IIUC it would simply mean that libc has a similar code to ours for working with /dev/urandom, meaning all the issues with using FD and polling /dev/random. I would prefer to have such code on the Rust side, not in libc.
• Allow interception of getrandom calls. If you are certain that you absolutely need it for some reason, it still should be possible with the libc::syscall-based code. We generally discourage replacing of OS entropy source with something custom.

Assuming we will add the libc::getrandom backend, do we want to keep the libc::syscall-based one?

[briansmith]

The Rust standard library has a syscall macro which creates a weak symbol for getrandom, so if the underlying libc provides getrandom, that function is called instead of making a syscall.

How does the Rust stdlib ensure that the getrandom it finds is actually provided by libc and not by any other library? The getrandom crate intentionally tries to prevent other libraries from overriding the operating system implementation with their own custom implementation, when there is an operating system implementation.

Good catch, this library aims to have an identical implementation to the standard library (see rust-lang/rust#62079), so we shouldn't deviate unless it's necessary.

Reasonably policy. But, let's make sure that the standard library is doing the right thing.

[briansmith]

How does the Rust stdlib ensure that the getrandom it finds is actually provided by libc and not by any other library?

The getrandom crate presently already does this weak symbol lookup for macOS, FreeBSD, Dragonfly, Solaris, and BSDs. It seems to be a bit at odds with the hoops that we go through to make custom implementations only work on platforms that getrandom doesn't already explicitly support.

I am not sure how we test that both code paths (weak symbol found, weak symbol not found and fallback to syscall) are correct in CI/CD.

[josephlr]

@ahgamut after looking at this again, I think the correct approach is to not compile for a Linux target unless you are actually targeting Linux. Given Cosmopolitan's goals of being a compile-once/run-everywhere libc, it shouldn't be using the x86_64-unknown-linux-gnu target (which is specifically Linux-only). If rust supported a x86_64-unknown-unknown-cosmo unix-like target, that would probably be the best bet. Then, we could just add target_env = "cosmo" to our list of targets which just call libc::getrandom.

Given that the Linux fallback logic is already fairly complex, I don't think it makes sense to make it more complex.

An alternative would be to only use the syscall wrapper on Linux targets which lack libc::getrandom, but again the splitting of the implementation is probably not worth it as most Linux targets have a minimum glibc of 2.17 (libc::getrandom was introduced in glibc 2.25).

[briansmith]

I think we should make a plan for how to support x86_64-unknown-linux-none (issue #424) before resolving this, especially before delegating to libc on LInux.

[briansmith]

@ahgamut after looking at this again, I think the correct approach is to not compile for a Linux target unless you are actually targeting Linux. Given Cosmopolitan's goals of being a compile-once/run-everywhere libc, it shouldn't be using the x86_64-unknown-linux-gnu target (which is specifically Linux-only). If rust supported a x86_64-unknown-unknown-cosmo unix-like target, that would probably be the best bet. Then, we could just add target_env = "cosmo" to our list of targets which just call libc::getrandom.

I agree with this. Or, if it is really important for it to be a -linux- target, at a minimum we should have a different target_env that we can use to differentiate it, like uclibc/musl/glibc do.

I spent some time trying to figure out how to make this weak symbol lookup bulletproof, in #429 and privately, and my conclusion is that we should try to avoid the weak symbol lookup whenever practical. There are just too many hazards to do it on the off chance that we might maybe be Cosmopolitan Libc, when we're probably not.

So I think we should close this as Not Planned.

[briansmith]

Also, this is a little bit off topic, but it is likely that especially with the x86_64-unknown-linux-none target now existing, more libraries that are currently using libc are going to start trying to find ways to avoid libc, e.g. by issuing more syscalls directly, and even by issuing syscalls directly in inline/external assembly language code, avoiding even libc::syscall. Only because we are in the early days of this does getrandom currently seem unusual in this respect.

[josephlr]

@briansmith I agree here. While some of the issues with dynamic symbol lookup would be a problem even if we used libc::getrandom without dynamic lookup (say if they binary was dynamically linked), certainly using dlsym makes things more error prone.

If/When getrandom starts being a dep of libstd we can revisit the divergence between our implementation and libstd.