-
Notifications
You must be signed in to change notification settings - Fork 36.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feed environment data into RNG initializers #17270
Conversation
0971ae9
to
0c6d5cb
Compare
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
Errors on MacOSX (looks like it doesn't have
|
c197258
to
4b12381
Compare
Concept ACK: good idea |
0b1d233
to
eed8cfe
Compare
I think I've addressed a compatibility/build issues. Can someone try this on OSX, and perhaps some BSD flavor? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Concept ACK, nice commit sequence.
I get the following error when building eed8cfe on OpenBSD 6.6:
|
Log on FreeBSD 12.0-RELEASE-p10 (same complaint, about
|
Building freeBSD here: https://cirrus-ci.com/task/5690364417015808 Edit: Same error: https://cirrus-ci.com/task/5690364417015808?command=make#L734 |
Does the comment in |
e604e52
to
8470cbb
Compare
New freeBSD build: https://cirrus-ci.com/task/4712584309112832 |
d1c0277 Report amount of data gathered from environment (Pieter Wuille) 64e1e02 Use thread-safe atomic in perfmon seeder (Pieter Wuille) d61f2bb Run background seeding periodically instead of unpredictably (Pieter Wuille) 483b942 Add information gathered through getauxval() (Pieter Wuille) 11793ea Feed CPUID data into RNG (Pieter Wuille) a81c494 Use sysctl for seeding on MacOS/BSD (Pieter Wuille) 2554c1b Gather additional entropy from the environment (Pieter Wuille) c2a262a Seed randomness with process id / thread id / various clocks (Pieter Wuille) 723c796 [MOVEONLY] Move cpuid code from random & sha256 to compat/cpuid (Pieter Wuille) cea3902 [MOVEONLY] Move perfmon data gathering to new randomenv module (Pieter Wuille) b51bae1 doc: minor corrections in random.cpp (fanquake) Pull request description: This introduces a new `randomenv` module that queries varies non-cryptographic (and non-RNG) sources of entropy available on the system; things like user IDs, system configuration, time, statistics, CPUID data. The idea is that these provide a fallback in scenarios where system entropy is somehow broken (note that if system entropy *fails* we will abort regardless; this is only meant to function as a last resort against undetected failure). It includes some data sources OpenSSL currently uses, and more. The separation between random and randomenv is a bit arbitrary, but I felt that all this "non-essential" functionality deserved to be separated from the core random module. ACKs for top commit: TheBlueMatt: utACK d1c0277. Certainly no longer measuring the time elapsed between a 1ms sleep (which got removed in the latest change) is a fair tradeoff for adding about 2 million other actually-higher-entropy bits :). laanwj: ACK d1c0277 Tree-SHA512: d290a8db6538a164348118ee02079e4f4c8551749ea78fa44b2aad57f5df2ccbc2a12dc7d80d8f3e916d68cdd8e204faf9e1bcbec15f9054eba6b22f17c66ae3
f93fc61 Put bounds on the number of CPUID leaves explored (Pieter Wuille) ba2c5fe Fix CPUID subleaf iteration (Pieter Wuille) Pull request description: This fixes bitcoin#17523. The code to determine which CPUID subleaves to explore was incorrect in bitcoin#17270. The new code here is based on Intel's reference documentation for CPUID (a document called "Intel® Processor Identification and the CPUID Instruction - Application Note 485", which I cannot actually find on their own website). ACKs for top commit: laanwj: ACK f93fc61 jonatack: ACK f93fc61 code review, tested rebased on current master bb862d7 with Debian 4.19 x86_64 mzumsande: ACK f93fc61, reviewed code and compared with the intel doc, tested on an AMD and an Intel processor. Tree-SHA512: 2790b326fa397b736c0f39f25807bea57de2752fdd58bf6693d044b8cb26df36c11cce165a334b471f8e33724f10e3b76edab5cc4e0e7776601aabda13277245
…around rv64 toolchain issue eafd259 build: Add NX workaround for RV64 (Wladimir J. van der Laan) f6e4225 build: Allow export of environ symbols (Wladimir J. van der Laan) Pull request description: This export was introduced in bitcoin#17270 which added ``` //! Necessary on some platforms extern char** environ; ``` This should (finally) make the gitian build pass again (fix issue bitcoin#17525.). Built on top of bitcoin#17538 which should be merged first. Top commit has no ACKs. Tree-SHA512: 5c2054d52d0957aec3dc945b76d8e219187d22dc03889e7a88fb76049bf8e4a3e9f4da00dd1e9dd0351211f8e70d1a1b8ad7244f0348dab698e9d14b9d0c0bd4
… noexcept 55b2cb1 random: mark RandAddPeriodic and SeedPeriodic as noexcept (fanquake) 461e547 doc: correct random.h docs after bitcoin#17270 (fanquake) Pull request description: The usage of `MilliSleep()` in SeedPeriodic (previously SeedSleep) was [removed](bitcoin@d61f2bb) in bitcoin#17270, meaning it, and its users can now be marked `noexcept`. This also corrects the docs in random.h for some of the changes in bitcoin#17270. ACKs for top commit: practicalswift: ACK 55b2cb1 laanwj: ACK 55b2cb1 sipa: ACK 55b2cb1 Tree-SHA512: 672d369796e7c4f9b4d98dc545e5454999fa1bef373871994a26041d6163c58909e2255e4f820d3ef011679aa3392754eb57477306a89f5fd3d57e2bd7f0811a
dc9305b random: don't special case clock usage on macOS (fanquake) Pull request description: `clock_gettime()`, `CLOCK_MONOTONIC` and `CLOCK_REALTIME` are all available for use on macOS (now that we require macOS >=10.12 and build against 10.14). Use them rather than the [deprecated](https://developer.apple.com/library/archive/documentation/Darwin/Conceptual/KernelProgramming/Mach/Mach.html) `mach_timespec_t` time API. I mentioned the possibility for this change [in bitcoin#17270](bitcoin#17270 (comment)). [master](1dbf335): ```bash 2019-12-23T20:49:43Z Feeding 216 bytes of dynamic environment data into RNG 2019-12-23T20:50:43Z Feeding 216 bytes of dynamic environment data into RNG ``` This PR: ```bash 2019-12-23T20:32:41Z Feeding 232 bytes of dynamic environment data into RNG 2019-12-23T20:33:42Z Feeding 232 bytes of dynamic environment data into RNG ``` ~~Depends on bitcoin#16392.~~ Merged. ACKs for top commit: laanwj: ACK dc9305b Tree-SHA512: 18c2f336ea628f9cf7339b817381d230a18893fd9c0351bf99a39ca6f45c5b0a20af9d599d48d6c09515627d5edafa91337c17f9f790264251d2cdcb3763bbd5
This export was introduced in #17270 which added ``` //! Necessary on some platforms extern char** environ; ```
The usage of MilliSleep() in SeedPeriodic (previously SeedSleep) was removed in #17270, meaning it, and its users can now be marked noexcept.
The usage of MilliSleep() in SeedPeriodic (previously SeedSleep) was removed in bitcoin#17270, meaning it, and its users can now be marked noexcept.
The usage of MilliSleep() in SeedPeriodic (previously SeedSleep) was removed in bitcoin#17270, meaning it, and its users can now be marked noexcept.
The usage of MilliSleep() in SeedPeriodic (previously SeedSleep) was removed in bitcoin#17270, meaning it, and its users can now be marked noexcept.
The usage of MilliSleep() in SeedPeriodic (previously SeedSleep) was removed in bitcoin#17270, meaning it, and its users can now be marked noexcept.
cecbf6c Use secure.h header for secure allocators (Fuzzbawls) d9f67da net: add ifaddrs.h include (fanquake) e906436 build: check if -lsocket is required with *ifaddrs (fanquake) 414f405 rand: only try and use freeifaddrs if available (fanquake) 3a039d6 build: avoid getifaddrs when unavailable (Cory Fields) 77bddd7 Use GetStrongRandBytes in gmp bignum initialization (Fuzzbawls) b70b26f Fix typo in comment in randomenv.cpp (Fuzzbawls) fec460c Put bounds on the number of CPUID leaves explored (Pieter Wuille) 41ab1ff Fix CPUID subleaf iteration (Pieter Wuille) 8a9bbb1 Move events_hasher into RNGState() (Pieter Wuille) 88c2ae5 random: mark RandAddPeriodic and SeedPeriodic as noexcept (fanquake) 81d382f doc: correct random.h docs after bitcoin#17270 (fanquake) f363ea9 Seed RNG with precision timestamps on receipt of net messages. (Matt Corallo) 7d6ddcb Run background seeding periodically instead of unpredictably (Pieter Wuille) 4679181 Add information gathered through getauxval() (Pieter Wuille) 88d97d0 Feed CPUID data into RNG (Pieter Wuille) 8f5b9c9 Use sysctl for seeding on MacOS/BSD (Pieter Wuille) 67de246 Gather additional entropy from the environment (Pieter Wuille) 6142e1f Seed randomness with process id / thread id / various clocks (Pieter Wuille) 7bde8b7 [MOVEONLY] Move cpuid code from random to compat/cpuid (Fuzzbawls) 52b5336 [MOVEONLY] Move perfmon data gathering to new randomenv module (Pieter Wuille) 27cf995 doc: minor corrections in random.cpp (fanquake) fccd2b8 doc: correct function name in ReportHardwareRand() (fanquake) 909473e Fix FreeBSD build by including utilstrencodings.h (Fuzzbawls) 630931f break circular dependency: random/sync -> util -> random/sync (Fuzzbawls) 5eed08c random: remove call to RAND_screen() (Windows only) (fanquake) ada9868 gui: remove OpenSSL PRNG seeding (Windows, Qt only) (fanquake) 22a7121 Fix non-deterministic coverage of test DoS_mapOrphans (Fuzzbawls) 79e7fd3 Add ChaCha20 bench (Jonas Schnelli) 6966aa9 Add ChaCha20 encryption option (XOR) (Jonas Schnelli) 28c9cdb tests: Add script checking for deterministic line coverage (practicalswift) c82e359 test: Make bloom tests deterministic (MarcoFalke) 7b33223 Document strenghtening (Pieter Wuille) 0190dec Add hash strengthening to the RNG (Pieter Wuille) 67e336d Use RdSeed when available, and reduce RdRand load (Pieter Wuille) 4ffda1f Document RNG design in random.h (Pieter Wuille) 2b6381e Use secure allocator for RNG state (Pieter Wuille) 080deb3 Encapsulate RNGState better (Pieter Wuille) 787d72f DRY: Implement GetRand using FastRandomContext::randrange (Pieter Wuille) 5bc2583 Sprinkle some sweet noexcepts over the RNG code (Pieter Wuille) 774899f Remove hwrand_initialized. (Pieter Wuille) 698d133 Switch all RNG code to the built-in PRNG. (Pieter Wuille) 038a45a Integrate util/system's CInit into RNGState (Fuzzbawls) 5f20e62 Abstract out seeding/extracting entropy into RNGState::MixExtract (Pieter Wuille) 298f97c Add thread safety annotations to RNG state (Pieter Wuille) 2326535 Rename some hardware RNG related functions (Pieter Wuille) d76ee83 Automatically initialize RNG on first use. (Pieter Wuille) 1a5dbc5 Don't log RandAddSeedPerfmon details (Pieter Wuille) 32e6c42 Simplify testing RNG code (Fuzzbawls) 972effa Make unit tests use the insecure_rand_ctx exclusively (Fuzzbawls) af52bf5 Use a FastRandomContext in LimitOrphanTxSize (Fuzzbawls) 746d466 Introduce a Shuffle for FastRandomContext and use it in wallet (Fuzzbawls) 1cdf124 Use a local FastRandomContext in a few more places in net (Fuzzbawls) e862564 Make addrman use its local RNG exclusively (Fuzzbawls) 94b2ead Make FastRandomContext support standard C++11 RNG interface (Pieter Wuille) Pull request description: This is a collection of upstream PRs that have been backported to bring our RNG (`src/random`) code more up-to-date. The following upstream PRs have been included here: - bitcoin#12742 - bitcoin#14624 - some of this had already been merged previously - bitcoin#14955 - bitcoin#15250 - bitcoin#15224 - bitcoin#15324 - bitcoin#15296 - bitcoin#15512 - bitcoin#16878 - bitcoin#17151 - bitcoin#17191 - bitcoin#13236 - bitcoin#13314 - bitcoin#17169 - bitcoin#17270 - omitted last commit as our testing framework doesn't support it currently - omitted bitcoin@64e1e02, to be pulled in after our time utility is updated in a separate PR - bitcoin#17573 - bitcoin#17507 - bitcoin#17670 - bitcoin#17527 - bitcoin#14127 - bitcoin#21486 ACKs for top commit: furszy: ACK cecbf6c with a minor nit that can be easily tackled later. random-zebra: rebase utACK cecbf6c and merging... Tree-SHA512: 3463b693cc9bddc1ec15228d264a794f5c2f159073fafa2ccf6e2563abfeb4369e49505f97ca84f2478ca792bd07b66d2cd83c58044d6a0cae6af42d22f5784b
This export was introduced in bitcoin#17270 which added ``` //! Necessary on some platforms extern char** environ; ```
This export was introduced in bitcoin#17270 which added ``` //! Necessary on some platforms extern char** environ; ```
This export was introduced in bitcoin#17270 which added ``` //! Necessary on some platforms extern char** environ; ```
This export was introduced in bitcoin#17270 which added ``` //! Necessary on some platforms extern char** environ; ```
This export was introduced in bitcoin#17270 which added ``` //! Necessary on some platforms extern char** environ; ```
This export was introduced in bitcoin#17270 which added ``` //! Necessary on some platforms extern char** environ; ```
This export was introduced in bitcoin#17270 which added ``` //! Necessary on some platforms extern char** environ; ```
Summary: > No longer needed after [[bitcoin/bitcoin#17270 | core#17270]] This is a backport of [[bitcoin/bitcoin#19090 | core#19090]] [1/6] bitcoin/bitcoin@fa609c4 Backport note: the original PR has 7 commits, but the last one is not relevant to our codebase. Test Plan: With tsan: `ninja check` Reviewers: #bitcoin_abc, majcosta Reviewed By: #bitcoin_abc, majcosta Differential Revision: https://reviews.bitcoinabc.org/D9955
…around rv64 toolchain issue eafd259 build: Add NX workaround for RV64 (Wladimir J. van der Laan) f6e4225 build: Allow export of environ symbols (Wladimir J. van der Laan) Pull request description: This export was introduced in bitcoin#17270 which added ``` //! Necessary on some platforms extern char** environ; ``` This should (finally) make the gitian build pass again (fix issue bitcoin#17525.). Built on top of bitcoin#17538 which should be merged first. Top commit has no ACKs. Tree-SHA512: 5c2054d52d0957aec3dc945b76d8e219187d22dc03889e7a88fb76049bf8e4a3e9f4da00dd1e9dd0351211f8e70d1a1b8ad7244f0348dab698e9d14b9d0c0bd4
…around rv64 toolchain issue eafd259 build: Add NX workaround for RV64 (Wladimir J. van der Laan) f6e4225 build: Allow export of environ symbols (Wladimir J. van der Laan) Pull request description: This export was introduced in bitcoin#17270 which added ``` //! Necessary on some platforms extern char** environ; ``` This should (finally) make the gitian build pass again (fix issue bitcoin#17525.). Built on top of bitcoin#17538 which should be merged first. Top commit has no ACKs. Tree-SHA512: 5c2054d52d0957aec3dc945b76d8e219187d22dc03889e7a88fb76049bf8e4a3e9f4da00dd1e9dd0351211f8e70d1a1b8ad7244f0348dab698e9d14b9d0c0bd4
//! Necessary on some platforms | ||
extern char** environ; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@sipa
While compiling natively on Windows with MSVC, receiving C4273
warnings:
randomenv.cpp
C:\Users\hebasto\bitcoin\src\randomenv.cpp(60,22): warning C4273: '__p__environ': inconsistent dll linkage [C:\Users\hebasto\bitcoin\build\src\util\bitcoin_util.vcxproj]
C:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\ucrt\stdlib.h(1158,29): message : see previous definition of '__p__environ' [C:\Users\hebasto\bitcoin\build\src\util\bitcoin_util.
vcxproj]
randomenv.cpp
C:\Users\hebasto\bitcoin\src\randomenv.cpp(60,22): warning C4273: '__p__environ': inconsistent dll linkage [C:\Users\hebasto\bitcoin\build\src\kernel\bitcoinkernel.vcxproj]
C:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\ucrt\stdlib.h(1158,29): message : see previous definition of '__p__environ' [C:\Users\hebasto\bitcoin\build\src\kernel\bitcoinkern
el.vcxproj]
Considering https://docs.microsoft.com/en-us/cpp/c-runtime-library/environ-wenviron, is this code correct for MSVC?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nm, no issues with the latest build.
This introduces a new
randomenv
module that queries varies non-cryptographic (and non-RNG) sources of entropy available on the system; things like user IDs, system configuration, time, statistics, CPUID data.The idea is that these provide a fallback in scenarios where system entropy is somehow broken (note that if system entropy fails we will abort regardless; this is only meant to function as a last resort against undetected failure). It includes some data sources OpenSSL currently uses, and more.
The separation between random and randomenv is a bit arbitrary, but I felt that all this "non-essential" functionality deserved to be separated from the core random module.