Implement `io::Entropy` and refactor random data generation #108874

joboet · 2023-03-07T18:18:07Z

Implements the ACP rust-lang/libs-team#159.

Because std has special needs (its OK to use less secure random data for HashMap keys if that means we don't have to block) and the Read API diverges from getrandom (not the whole buffer must be read, but errors should be immediately returned), I decided not to branch out to getrandom, but to refactor the current std implementation, merging elements from getrandom where required. In particular:

On Linux, /dev/random needs to be polled before reading from /dev/urandom when generating secure data
The default source for secure data on UNIXes without special syscalls should be /dev/random/, unless it is known that it is identical to /dev/urandom

As a consequence, this PR is rather large, so please let me know if there is anything I can do to simplify the review process.

Checked on all platforms (that currently are not broken), but tested only on macOS (and the Linux CI).

@rustbot label +S-waiting-on-ACP +T-libs-api

rustbot · 2023-03-07T18:18:14Z

r? @joshtriplett

(rustbot has picked a reviewer for you, use r? to override)

rustbot · 2023-03-07T18:18:17Z

Hey! It looks like you've submitted a new PR for the library teams!

If this PR contains changes to any rust-lang/rust public library APIs then please comment with @rustbot label +T-libs-api -T-libs to tag it appropriately. If this PR contains changes to any unstable APIs please edit the PR description to add a link to the relevant API Change Proposal or create one if you haven't already. If you're unsure where your change falls no worries, just leave it as is and the reviewer will take a look and make a decision to forward on if necessary.

Examples of T-libs-api changes:

Stabilizing library features
Introducing insta-stable changes such as new implementations of existing stable traits on existing stable types
Introducing new or changing existing unstable library APIs (excluding permanently unstable features / features without a tracking issue)
Changing public documentation in ways that create new stability guarantees
Changing observable runtime behavior of library APIs

workingjubilee · 2023-03-08T05:59:03Z

I assume you know that the random number devices have been united in recent Linux kernels and they will not have substantial differences, namely that /dev/random no longer blocks and /dev/urandom will use the same CSPRNG.

joboet · 2023-03-08T07:46:29Z

I assume you know that the random number devices have been united in recent Linux kernels and they will not have substantial differences, namely that /dev/random no longer blocks and /dev/urandom will use the same CSPRNG.

Yes, but polling is (unfortunately) still necessary for compatibility with older versions. The new versions use getrandom anyway, if it's not blocked, so the fast path doesn't go through the file system at all.

workingjubilee · 2023-03-08T11:47:14Z

Hmm, I don't know if it's actually worth it to ever hit /dev/random and not /dev/urandom, seeing as how even on earlier versions the differences were very marginal. But a version note on which kernel we can fully drop the fallback code on is also fine.

joboet · 2023-03-08T13:02:25Z

Hmm, I don't know if it's actually worth it to ever hit /dev/random and not /dev/urandom, seeing as how even on earlier versions the differences were very marginal. But a version note on which kernel we can fully drop the fallback code on is also fine.

I don't want to create security vulnerabilities by not considering the edge cases (/dev/urandom is still used when getrandom is blocked, and if /dev/random is not checked, the effect is the same as getrandom(GRND_INSECURE) which can return bad randomness on embedded devices, even on the newest kennels). Also, getrandom does this, so keeping the source the same means users don't have to reanalyse their security if switching to io::Entropy.

I'll add the version note, that's a good idea.

thomcc · 2023-03-11T17:08:50Z

library/std/src/sys/sgx/entropy.rs

+ unsafe {
+ let mut ret: u64 = 0;
+ for _ in 0..10 {
+ if crate::arch::x86_64::_rdrand64_step(&mut ret) == 1 {


Not a maintainer of sgx but this does not offer high quality entropy. See the "Generating seeds with rdrand" section in https://www.intel.com/content/www/us/en/developer/articles/guide/intel-digital-random-number-generator-drng-software-implementation-guide.html for how to do it (which we probably don't want in std...)

~~I... don't quite get that section, unfortunately.~~

IIUC (please feel free to correct me), RDRAND is like /dev/urandom in that it uses a CSPRNG to generate more random data from a single, truly random seed. Because of the size of the seed, constant reseeding and the strength of the PRNG, the output should not be in any practical way less secure that RDSEED. But that doesn't explain why anyone would use RDSEED...

Edit: Ah, I missed the part about forward and backward prediction resistance... Still, I don't know if it's really, truly necessary here.

I didn't change this, because getrandom uses RDRAND on SGX. Note that it does not consider values 0 or u64::MAX valid because of hardware bugs on AMD devices. Since SGX is Intel-only, I guess it isn't a concern here?

If RDRAND is truly the wrong option here, I guess the alternative is doing what I did for Hermit here and use RDSEED to seed our own PRNG.

CC @raoulstrackx

...the DRNG using the RDRAND instruction is useful for generating high-quality keys for cryptographic protocols, and the RSEED instruction is provided for seeding software-based pseudorandom number generators (PRNGs)

I think that sums it up nicely. If someone needs stronger guarantees for a PRNG, they should take a look at what exactly happens in getrandom. The documentation explicitly mentions the randomness comes from rdrand. To me that looks fine, but I'm not a cryptographic. @zugzwang can you pitch in?

Indeed, AMD hardware bugs have no impact on SGX and the values 0 and u64::MAX should not be handled differently.

Makes sense to me. Intel documentation looks clear, and there is useful discussion on security levels here.

thomcc · 2023-03-11T17:21:01Z

I decided not to branch out to getrandom, but to refactor the current std implementation, merging elements from getrandom where required

std also cannot use getrandom for this because std supports platforms that getrandom does not.

See #104658 where I went through the work of removing use of getrandom from the stdlib test suite, allowing running those tests on several tier3 targets and moving more of the stdlib tests that live in uitests only so they can run on tier3 targets back into the normal test suite.

I would prefer we not undo that improvement.

bors · 2023-03-30T00:37:32Z

☔ The latest upstream changes (presumably #109734) made this pull request unmergeable. Please resolve the merge conflicts.

rustbot · 2023-03-30T10:41:03Z

Hey! It looks like you've submitted a new PR for the library teams!

If this PR contains changes to any rust-lang/rust public library APIs then please comment with @rustbot label +T-libs-api -T-libs to tag it appropriately. If this PR contains changes to any unstable APIs please edit the PR description to add a link to the relevant API Change Proposal or create one if you haven't already. If you're unsure where your change falls no worries, just leave it as is and the reviewer will take a look and make a decision to forward on if necessary.

Examples of T-libs-api changes:

Stabilizing library features
Introducing insta-stable changes such as new implementations of existing stable traits on existing stable types
Introducing new or changing existing unstable library APIs (excluding permanently unstable features / features without a tracking issue)
Changing public documentation in ways that create new stability guarantees
Changing observable runtime behavior of library APIs

joboet · 2023-03-30T10:42:08Z

Rebased to include #107221 and #107387.

bors · 2023-05-08T15:03:55Z

☔ The latest upstream changes (presumably #111346) made this pull request unmergeable. Please resolve the merge conflicts.

joshtriplett · 2024-02-11T02:49:13Z

r? libs

raoulstrackx · 2024-02-13T08:50:13Z

library/std/src/sys/sgx/abi/usercalls/mod.rs

@@ -164,7 +164,8 @@ pub fn wait(event_mask: u64, mut timeout: u64) -> IoResult<u64> {
 // trusted to ensure accurate timeouts.
 if let Ok(timeout_signed) = i64::try_from(timeout) {
 let tenth = timeout_signed / 10;
- let deviation = (rdrand64() as i64).checked_rem(tenth).unwrap_or(0);
+ let rand = rdrand64().unwrap_or_else(|| rtabort!("Failed to obtain random data"));


Nitpick: There's no need to abort here when rdrand64() returns None. This code is just there to avoid developers from relying on the accuracy of timeouts. Using a default (as in the original code) is fine here.
There's even a small availability issue with the new code. The rdrand instruction can be executed by any userspace program. So an attacker may be able to execute this instruction over and over again until the pool of random values is exhausted. This will cause the enclave to abort here. SGX does not provide any security guarantees related to availability, but this attack may be executed to make enclaves abort even when the attacker would not be able to do so through other means (e.g., they don't have the access rights to stop the process running the enclave). On the other hand, when the enclave needs to do a cryptographic operation, there's no other way than to abort. Hence, it's a bit of a nitpick :)

clarfonthey · 2024-02-18T16:11:34Z

library/std/src/collections/hash/map.rs

@@ -3122,7 +3122,17 @@ impl RandomState {
 // increment one of the seeds on every RandomState creation, giving
 // every corresponding HashMap a different iteration order.
 thread_local!(static KEYS: Cell<(u64, u64)> = {
- Cell::new(sys::hashmap_random_keys())
+ if crate::sys::entropy::INSECURE_HASHMAP {
+ Cell::new((1, 2))


This is not blocking, but one thing I was thinking of proposing as a path to make RandomState still work without libstd is allowing users to provide their own entropy function, and it would be nice if this were factored in a way that always took that path, rather than hard-coding in an insecure constant for the state.

Since no one is going to rely explicitly on the (1, 2) state for these cases by design, I think it'd be better to just provide a random eight bytes that get sent through the path instead. That way everything goes through the "entropy" path of decoding the bytes, even if the bytes are static.

clarfonthey · 2024-02-18T16:13:04Z

library/std/src/io/entropy.rs

+///
+/// Be aware that, because the data is of very high quality, reading high amounts
+/// of data can be very slow, and potentially slow down other processes requiring
+/// random data. Use a pseudo-random number generator if speed is important.


Since we do this elsewhere in libstd with other crates, perhaps we could just straight-up recommend the rand crate for this purpose? Especially since this is an org-maintained crate.

We have been trying to not advertise external crates from std docs. And there are other potential options beside rand even if they're more niche.

clarfonthey · 2024-02-18T16:14:17Z

library/std/src/collections/hash/map.rs

+ } else {
+ let mut v = [0u8; 16];
+ let mut entropy = entropy();
+ entropy.set_insecure(true);


Since the purpose is to prevent hash DOS, attacks, wouldn't setting insecure be counter to that? Or am I missing the purpose of this flag?

Dylan-DPC · 2024-03-25T13:46:53Z

~~@joboet any updates on this?~~

Edit: nevermind, just realised this is waiting on the ACP being merged

a1phyr · 2024-05-13T11:56:40Z

library/std/src/io/entropy.rs

+#[derive(Debug)]
+#[unstable(feature = "io_entropy", issue = "none")]
+pub struct Entropy {
+ insecure: bool,


This could probably use sys::Entropy directly here (and define a set_insecure method for it)

joboet · 2024-08-15T16:24:18Z

Closing in favour of #129120. No matter how the API discussion turns out, that PR is a much better starting point than this one. The UNIX refactoring has been done in #128655 already.

rustbot assigned joshtriplett Mar 7, 2023

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-libs Relevant to the library team, which will review and decide on the PR/issue. labels Mar 7, 2023

rustbot added S-waiting-on-ACP Status: PR has an ACP and is waiting for the ACP to complete. T-libs-api Relevant to the library API team, which will review and decide on the PR/issue. labels Mar 7, 2023

thomcc reviewed Mar 11, 2023

View reviewed changes

std: implement io::Entropy, refactor random data generation

f4eeb60

joboet force-pushed the io_entropy branch from f51d4f3 to f4eeb60 Compare March 30, 2023 10:41

Dylan-DPC removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 20, 2023

rustbot assigned Mark-Simulacrum and unassigned joshtriplett Feb 11, 2024

raoulstrackx reviewed Feb 13, 2024

View reviewed changes

clarfonthey reviewed Feb 18, 2024

View reviewed changes

a1phyr reviewed May 13, 2024

View reviewed changes

joboet closed this Aug 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement `io::Entropy` and refactor random data generation #108874

Implement `io::Entropy` and refactor random data generation #108874

joboet commented Mar 7, 2023 •

edited

Loading

rustbot commented Mar 7, 2023

rustbot commented Mar 7, 2023

workingjubilee commented Mar 8, 2023

joboet commented Mar 8, 2023 •

edited

Loading

workingjubilee commented Mar 8, 2023

joboet commented Mar 8, 2023

thomcc Mar 11, 2023

joboet Mar 11, 2023 •

edited

Loading

raoulstrackx Mar 13, 2023

zugzwang Mar 13, 2023

thomcc commented Mar 11, 2023

bors commented Mar 30, 2023

rustbot commented Mar 30, 2023

joboet commented Mar 30, 2023

bors commented May 8, 2023

joshtriplett commented Feb 11, 2024

raoulstrackx Feb 13, 2024

clarfonthey Feb 18, 2024

clarfonthey Feb 18, 2024

ChrisDenton Feb 19, 2024

clarfonthey Feb 18, 2024

Dylan-DPC commented Mar 25, 2024 •

edited

Loading

a1phyr May 13, 2024

joboet commented Aug 15, 2024

Implement io::Entropy and refactor random data generation #108874

Implement io::Entropy and refactor random data generation #108874

Conversation

joboet commented Mar 7, 2023 • edited Loading

rustbot commented Mar 7, 2023

rustbot commented Mar 7, 2023

workingjubilee commented Mar 8, 2023

joboet commented Mar 8, 2023 • edited Loading

workingjubilee commented Mar 8, 2023

joboet commented Mar 8, 2023

Choose a reason for hiding this comment

joboet Mar 11, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thomcc commented Mar 11, 2023

bors commented Mar 30, 2023

rustbot commented Mar 30, 2023

joboet commented Mar 30, 2023

bors commented May 8, 2023

joshtriplett commented Feb 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Dylan-DPC commented Mar 25, 2024 • edited Loading

Choose a reason for hiding this comment

joboet commented Aug 15, 2024

Implement `io::Entropy` and refactor random data generation #108874

Implement `io::Entropy` and refactor random data generation #108874

joboet commented Mar 7, 2023 •

edited

Loading

joboet commented Mar 8, 2023 •

edited

Loading

joboet Mar 11, 2023 •

edited

Loading

Dylan-DPC commented Mar 25, 2024 •

edited

Loading