Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RandomSource update - Validated #190

Merged
merged 6 commits into from
May 31, 2021
Merged

RandomSource update - Validated #190

merged 6 commits into from
May 31, 2021

Conversation

c19x
Copy link
Contributor

@c19x c19x commented May 17, 2021

  • Enables developers to supply their own custom RandomSource
    • Same four existing sources are still available
    • Updated RandomSource test to ensure test for randomness should always pass
  • PseudoDeviceAddress (PDA) now use SHA256 to cryptographically separate random address from random source
    • Existing non-blocking random source made it impractical to use observed addresses for tracking
      • Call time is inherently random due to real-world events as entropy source
      • Shared random seed source is inherently random as system initialisation time is unpredictable
      • Random values follow a uniform distribution, thus all values may be generated by all seeds
      • Random sequence follow a uniform distribution, thus knowledge of prior values offer minimal gain in reducing the total number of seeds that could have generated the consecutive addresses
    • Introduction of SHA256 hash and data trimming offer additional assurance that the random seed is cryptographically challenging to identify given observed addresses
      • Developers are free to adopt their own customer RandomSource that meets their specific requirements
      • Recommendation is to use the non-blocking random source in contact tracing applications as the impact of blocking, leading to non-detection of disease transmission, is far greater than the theoretical risk of tracking
  • Passes all unit tests

Signed-off-by: c19x support@c19x.org

closes #189

@c19x
RandomSource update - Validated
12cf38f 
- Enables developers to supply their own custom RandomSource
  - Same four existing sources are still available
- Updated RandomSource test to ensure test for randomness should always pass
- Updated PseudoDeviceAddress to use SHA256 hash to cryptographically separate random address from random source
- Passes all unit tests

Signed-off-by: c19x <support@c19x.org>
@adamfowleruk
Copy link
Collaborator

Would it be possible to add a higher level test for PseudoMacAddress that validated each byte value for the mac had an even distribution and was comparable in every byte position? (I.e. to ensure that the first byte being, say, 00 was just as likely as its other values, and that the frequency was comparable with bytes 2-6's frequency of 00 too).

@c19x
Copy link
Contributor Author

c19x commented May 18, 2021

Thanks for review.

  • Current tests have a high tolerance as they aim to test for obviously not random output (i.e. due to coding error), rather than confirm the output is random and uniformly distributed
  • Good idea to incorporate uniformity tests, considering incorporating existing standalone simulation code, but need to make it run within reasonable time
  • Also have an idea for non-blocking secure random, shall provide update, please hold PR

c19x added 2 commits May 18, 2021 13:52
@c19x
Non-blocking CSPRNG - Validated
6b56f19 
- New random source that is non-blocking and uses application domain specific truly random entropy source as seeds
  - Uses SHA256 and truncation and separate one-time use PRNG from seed
  - Solution mitigates blocking caused by use of SecureRandom on idle systems, while offering similar security design for the application domain
- Added uniform distribution tests for sequence value and initial value from one-time use PRNGs
  - Added performance test which showed new CSPRNG offers the fastest performance
  - Tests show all current solutions offer uniform distribution, thus tracking based on limited observations is impractical given there are too many candidate seeds that can yield the observations

Signed-off-by: c19x <support@c19x.org>
@c19x
Documentation for NonBlockingCSPRNG
580ee27 
- Added explanation of design concept
- Introduced debug logging (commented out) to enable manual visual inspection
- Validated by inspection
- Tested on Pixel 3 and A20, and also passed cross-platform unit tests

Signed-off-by: c19x <support@c19x.org>
@c19x
Copy link
Contributor Author

c19x commented May 18, 2021

  • Added comment to clarify why the tolerance is so high for existing tests (to catch obviously not random output)
  • Added new uniformity tests with just 10% threshold for confirming if all values are equally likely in a sequence and as initial value given a seed
    • Uniformity test can run at lower threshold given more samples, current parameters have been set to ensure the automated tests should run quickly and passes almost all the time given uniformly distributed random sources
  • Introduced new Non-blocking CSPRNG based on the same concepts as SecureRandom except entropy is gathered from sources that won't exhaust in the target application environment. This is a simplified and quantified version of existing Non-blocking PRNG, enhanced with SHA256 for separating the observations (random values) generated by the one-time use PRNG from its seed that has been derived from a mixture of entropy sources, and also separating the seed source from the entropy source.
    • Passes all new unit tests
    • Delivers the best speed performance across all available random sources.
    • This is now the recommended default random source.
    • Validated on Pixel 3 and A20
  • Running overnight test to confirm new Non-Blocking CSPRNG really is non-blocking on idle devices
    • Code analysis has not identified any blocking calls in the new random source.
    • Planning to migrate uniformity test to server for statistical tests as separate standalone background task.

@c19x
Copy link
Contributor Author

c19x commented May 19, 2021 •
edited
Loading

  • Conducted 18hr+ test on Pixel3 and A20 to confirm Non-Blocking CSPRNG is non-blocking
  • New design idea for CSPRNG without reliance on LCG, pending tests. Please hold PR.

@c19x
Non-blocking secure random - Validated
fd97aed 
- Cryptographically secure pseudo random number generator based on SecureRandom, SHA256, xor, truncation, and external entropy
  - Seeded by SecureRandom data on app initialisation, as system entropy is unlikely to be exhausted on an active device
  - Internal 256-bit state advanced by SHA256 hash of current state and truly random external entropy sources
  - Blocking avoided by using a background thread to replace internal state with SecureRandom data at a low rate of 6-24 hours
    - If SecureRandom does block, the non-blocking cryptographically secure state updates shall enable continuous operation
  - Eliminated reliance on LCG based PRNG
  - Internal state updates incorporate truly unpredictable external data, and also utilises cryptographically secure lossy
    and one-way functions, thus making discovery of internal state and exploitation of discovered state challenging
- Passes all unit tests, including test for uniform distribution

Signed-off-by: c19x <support@c19x.org>
@c19x
Copy link
Contributor Author

c19x commented May 20, 2021 •
edited
Loading

  • NonBlockingCSPRNG confirmed to be non-blocking after 44hr test on idle devices
  • Battery usage was 84% (Samsung A20) and 82% (Google Pixel 3) after 44 hours continuous use
  • Added new NonBlockingSecureRandom which is the recommended random source for the future as all reliance on LCG has been removed. The solution is based entirely on cryptographically secure functions only, yet remains non-blocking.
  • Conducting long running test to confirm NonBlockingSecureRandom now.

@c19x
Copy link
Contributor Author

c19x commented May 22, 2021

  • NonBlockingSecureRandom FAILED to perform continuously without any negative impact on wider systems
    • Test shows any call to SecureRandom is liable to exhaust system entropy which prevents BLE operation
    • Developing new algorithm to remove reliance on SecureRandom

@c19x
NonBlockingSecureRandom - Validated
012c7c7 
- New implementation of cryptographically secure non-blocking random source that does not rely on SecureRandom
  - Experiments with previous solution (blocking SecureRandom runs in background thread) shows *ANY* call to SecureRandom on idle device is liable to cause blocking in wider subsystems
  - A single call to create a new instance of SecureRandom for reseeding the internal state was sufficient to interrupt continuous BLE operation for several minutes
  - New solution eliminates reliance on SecureRandom for reseeding by using thread scheduling as truly unpredictable random source that is guaranteed to be non-blocking
  - This concept is similar to that employed by Oracle and OpenJDK implementation of SecureRandom where the ThreadSeedGenerator acts as a fallback random source when /dev/random is exhausted

Signed-off-by: c19x <support@c19x.org>
@c19x
Copy link
Contributor Author

c19x commented May 24, 2021 •
edited
Loading

  • New implementation of NonBlockingSecureRandom is guaranteed to be non-blocking
    • Eliminated reliance on SecureRandom for reseeding
    • Using a time-based truly unpredictable random source for reseeding
      • Solution is based on thread scheduling which is inherently unpredictable and determined by wider system events
      • Same concept as ThreadSeedGenerator that is used by Oracle and OpenJDK implementations of SecureRandom that acts as backup random source when /dev/random is exhausted
  • Running overnight test under stress testing condition to confirm solution is non-blocking
    • No logging to avoid generating storage events to minimise system entropy
    • Advert refreshes every 3 minutes to trigger generation of new pseudo device address
    • RandomSource gathering external entropy from just one target device (test conducted with 2 phones in a faraday bag)
    • NonBlockingSecureRandom set to reseed every minute

closes #118

@c19x
Copy link
Contributor Author

c19x commented May 24, 2021

  • Tests have confirmed NonBlockingSecureRandom is non-blocking under continuous background use
    • Advert refresh set to once every 3 minutes to trigger unusually frequent use of random source
    • Reseed set to once every minute to trigger unusually frequent use of entropy source
    • Logging set to OFF to minimise system activities
    • Pixel3 and A20 placed in a single Faraday bag to eliminate external entropy
    • 100% continuity in 5 hour test, shall continue testing for 24 hours
  • PR ready for merge as new solution is now validated and proven to be non-blocking
    • Design builds on accepted seed generation method in SecureRandom and addresses known vulnerabilities
    • 2048 bit internal state, SHA256 based state updates, incorporates external truly random data when available

closes #189
closes #118
closes #110

@c19x
NonBlockingSecureRandom - Validated
591fbd4 
- Added checks to ensure thread based entropy source is not vulnerable to system load attacks
- Tested on A20 and Pixel3 in Faraday bag to confirm non-blocking even when there is no external influences
  - Advert refresh set to once every 3 minutes
  - Reseed set to once every minute
  - Logging set to OFF to minimise system activities
  - Test shows 100% continuity for 27 hours in idle state

Signed-off-by: c19x <support@c19x.org>
@c19x
Copy link
Contributor Author

c19x commented May 25, 2021

Ready for merge

  • 27hr test shows 100% continuity

@c19x c19x mentioned this pull request May 25, 2021
Closed
@adamfowleruk adamfowleruk merged commit 6c084b9 into theheraldproject:develop May 31, 2021
This was referenced May 31, 2021
Closed
Closed
Open
@adamfowleruk adamfowleruk added this to the v2.0 Move to LFPH milestone May 31, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@adamfowleruk adamfowleruk adamfowleruk left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v2.0 Move to LFPH
Development

Successfully merging this pull request may close these issues.

RandomSource update
2 participants
@c19x @adamfowleruk