Port from libc to rustix

[notgull]

rustix is a crate which provides I/O-safe wrappers around raw system calls on Linux and libc on other Unixes. By using it we can use raw syscalls instead of libc calls, which has a number of performance, safety and code size advantages.

I am wiling to write a PR for this; however there are a couple of items of discussions we have to go over first.

• This will bump this crate's MSRV to v1.63. Is this acceptable?
• It will also add a couple of (mostly inconsequential) dependencies. Is this acceptable?

[newpavlov]

I considered it (personally, I like rustix and use it instead of libc in my projects), but there are several issues:

• MSRV 1.63 is a bit too high for us. rustix has a laxer MSRV policy, so we probably would have to wait until MSRV-aware resolver will be adopted widely enough (or until next breaking getrandom release which may have sufficiently high MSRV).
• Some users are quite sensitive to additional dependencies, regardless whether they are "mostly inconsequential" or not.
• rustix::rand::getrandom accepts &mut [u8], while we provide API which works over &mut [MaybeUninit<u8>].

So we probably will not be migrating to rustix at the very least until hypothetical getrandom v0.3.

[notgull]

For your last point, getrandom_uninit is exposed in rustix now.

For the rest, I agree. We’ll have to wait.

[newpavlov]

getrandom_uninit is exposed in rustix now.

Ah, good to know!

[josephlr]

@notgull is it the intent of rustix to support all target_family = unix Rust targets? We try to support every rust target, and I would be a pain to deal with both rustix and libc crates.

We could maybe switch just the use_file.rs code to use rustix, which would require Linux, Android, Haiku, Redox, NTO and AIX support.

[newpavlov]

I think it makes sense to use rustix only on Linux and Android for the raw syscall feature. For other targets it's essentially a glorified libc wrapper. We probably can limit its use only to the getrandom path and make it a (disabled by default) crate feature.

[notgull]

@notgull is it the intent of rustix to support all target_family = unix Rust targets? We try to support every rust target, and I would be a pain to deal with both rustix and libc crates.

rustix supports all of the Unix targets, as well as some of the other targets, like Windows (through winsock).

We could maybe switch just the use_file.rs code to use rustix, which would require Linux, Android, Haiku, Redox, NTO and AIX support.

Yes, I think this would work.

I think it makes sense to use rustix only on Linux and Android for the raw syscall feature. For other targets it's essentially a glorified libc wrapper. We probably can limit its use only to the getrandom path and make it a (disabled by default) crate feature.

This sounds like a pain to manage, especially when rustix is present enough in the ecosystem that you're probably going to import it anyways.

[newpavlov]

This sounds like a pain to manage, especially when rustix is present enough in the ecosystem that you're probably going to import it anyways.

We are unlikely to use rustix by default and, as noted above, I don't see reasons to use it on non-Linux platforms in our case. So rustix-based code is likely to be behind a feature flag. Enabling the hypothetical linux-raw feature will be no more painful than enabling optional rustix dependency.

[briansmith]

I looked more into rustix:

• Rustix enables its file I/O functionality by default and it can't be disabled. It has a default implementation of getauxval() that on Linux kernels < 6.4, will do file I/O on "/proc/self/auxv. (Linux 6.4 introduced a syscall to get the auxv vector). Otherwise, it has an option to use libc::getauxval()instead to avoid that file I/O, and it has another option to provide the auxv vector in aninit()call instead. It doesn't matter for us as we don't usegetauxval()` for anything, but it means that we need to be careful about
• I did not see any sanitizer support, especially msan, in rustix or linux-raw-sys, so I think the unpoison in Android/Linux: Support msan; unpoison output of getrandom syscall. #463 would still be necessary. Perhaps we should file a bug (against rustix? and/or against linux-raw-sys?) for them to add it. LLVM's interception support for its libc::getrandom may be useful as a reference: llvm/llvm-project@ac9ee01
• I noticed https://github.com/llvm/llvm-project/blob/bd6e324b67cdadde2593327753e99782146d9bf8/libc/startup/linux/x86_64/tls.cpp#L68-L74 which points out that in general using errno in the implementation of getrandom limits the contexts that getrandom can be used in, though this is mitigated somewhat because Rust doesn't have "life before main" and I guess TLS is always set up before any Rust code can run. Regardless, Rustix doesn't use thread-local errno (not sure if it uses thread-local storage for anything), so that's a benefit of Rustix.
• Rustix MSRV policy is "This crate currently works on the version of Rust on Debian stable, which is currently Rust 1.63. This policy may change in the future, in minor version releases, so users using a fixed version of Rust should pin to a specific version of this crate." The first part seems pretty reasonable. I am not sure why 1.63 would be a problematic MSRV for getrandom but I do see the concern raised above.

[notgull]

It doesn't matter for us as we don't usegetauxval() for anything, but it means that we need to be careful about

Any particular reason for this?

• Perhaps we should file a bug (against rustix? and/or against linux-raw-sys?) for them to add it.

There is already a common function for initializing a written-to buffer. So hopefully we can just add the "unpoison" call there.

• The first part seems pretty reasonable. I am not sure why 1.63 would be a problematic MSRV for getrandom but I do see the concern raised above.

IIRC The main reason is because Debian Stable is considered the "low water mark" for what we reasonably need to support. I'm not sure what the lowest version this crate needs to support it.

It should also be mentioned that the MSRV is bumped to v1.64 if the "std" feature on Rustix is disabled.

[josephlr]

@notgull thanks for demoing how an rustix implementation might work (#470). From what I understand, this would still use libc on non aarch64/arm/x86_64/x86/riscv targets. This is understandable, and is related to #424 (comment) where I noted we couldn't (yet) give up our use of libc::syscall.

Looking at that implementation, I think a lot of the complexity comes from the rustix dependency being optional. If we were going to use rustix, it wouldn't be an optional dependancy; it would entirely replace our use of libc on Linux. As for the use_file implementation, I think a better approach is to use the standard library to provide such an implementation (see #453).

However, I don't think its very likely we would use the rustix backend (at least for now). Given that we would still need to depend on libc for almost all targets (due to libc::syscall), it seems easier for us and our users to depend on libc directly.

[notgull]

However, I don't think its very likely we would use the rustix backend (at least for now). Given that we would still need to depend on libc for almost all targets

I've modified #470 to completely remove libc from the Linux backend. It also makes rustix non-optional on Linux and Linux only.