Random Test Tool (RTT)

Random Test Tool (also refered RTT) is a Python script designed for testing the randomness of sequences of integers or bits. RTT serves the purpose of evaluating random number generators.

This project was primarily motivated by the following objectives:

• Evaluate series of integers: RTT allows the assessment of series of integers generated by programs based on generators or directly by generators.

• Facilitate random outputs controls during security audits: RTT provides a structured set of instructions and assessment outputs to enhance the interpretation of statistical results obtained during security audits.

• Simplify manipulation of test inputs, tests themselves and test outputs: The tool offers an easy-to-use Python implementation, enabling users to manipulate test inputs, outputs, and tests effortlessly.

🏗️ Installation

For installation instructions, see: INSTALL.md.

▶️ Usage

Running Random Test Tool

Random Test Tool can be executed using the following commands:

random-test-tool -i <file_path>
random-test-tool -d <dir_path>

For example:

random-test-tool -i random_generator_samples/python_random_integer/20230816-105301_RANDOM_NUMBERS.txt

Random Test Tool supports three input formats (within 20230816-105301_RANDOM_NUMBERS.txt):

1. "bitstring" series, i.e., a sequence of 0s and 1s:

01010111010011001...

2. list of integers starting at 0 or 1 separated by commas, spaces, semicolons or line breaks:

25
12
1
4

3. "bytestring" series, a sequence of bytes

Random Test Tool is capable of testing multiple files in a row:

random-test-tool -i test_file_1.txt test_file_2.txt

Additionally, it can test all files within a directory:

random-test-tool -d test_files

By default, each file will be treated separately. However, the -C option can be used:

random-test-tool -C 10

This option will split the whole input into the number of data chunks provided.

Outputs

By default, Random Test Tool returns results in the terminal.

You can use the -o option to specify a return file or generate graphs.

Configuration file

A yaml configuration file can also be used (see base_run_config.yaml).

random-test-tool -c config_file.yaml

Other options

For a comprehensive understanding of available options, use the following command:

random-test-tool -h 

Script testing the randomness of a serie of integers bits via statistical statistical_tests.

optional arguments:
  -h, --help            show this help message and exit
  -i INPUT_FILES [INPUT_FILES ...], --input_files INPUT_FILES [INPUT_FILES ...]
                        List of files to test.
  -d INPUT_DIR, --input_dir INPUT_DIR
                        Input directory, statistical_tests will be launched on each file.
  -o {terminal,csv,graph,html,all}, --output {terminal,csv,graph,html,all}
                        Output report options.
  -O OUTPUT_DIR, --output_dir OUTPUT_DIR
                        Output directory.
  -j {1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31}, --n_cores {1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31}
                        Number of processes used, 1 by default, maximum 31
  -t [STATISTICAL_TESTS ...], --test [STATISTICAL_TESTS ...]
                        Specifies which statistical_tests to launch. By default all statistical_tests are launched.
  -dt {int,bits,bytes}, --data_type {int,bits,bytes}
                        Used to select data type of sample, by default integer (int)
  -s {\n, ,,,;}, --separator {\n, ,,,;}
                        Separator used for integer files.
  -ll {ALL,DEBUG,INFO,WARN,ERROR,FATAL,OFF,TRACE}, --log_level {ALL,DEBUG,INFO,WARN,ERROR,FATAL,OFF,TRACE}
                        Log level (default: INFO).
  -c CONFIG, --config CONFIG
                        Configuration file of the run, if used will override other options.
  -C CHUNKS, --chunks CHUNKS
                        If this option is given, the inputs will bemerged and splited into n chunks then processed independently. 

Usage examples

For usage examples, check EXAMPLES.md

🎒 Interpretation of the generated results

The tests in this tool calculate a p-value.

The p-value represents the probability of obtaining a distribution at least as extreme as that observed.

We compare this p-value to a threshold (0.01). If the p-value is lower than this threshold, it implies that the probability of the observed behavior occurring by chance 1 - 0.01 (99%).

For this example (p-value < 0.01), a perfectly random sample is expected to fail this test only 1% of the time.

To account for this behavior, the recommended use of Random Test Tool is:

1. Run it on multiple samples from the same source;
2. consider a test to fail if it fails on a sufficient number of samples compared to the expected number (1% of samples in the example).

For example, if we run the tests on 100 samples and the binary rank test fails 12 times, as 12% > 1% (threshold of 0.01), we can consider that the source fails the test in question.

Conversely, if there are only 1 failure out of 100, the test would be considered a success.

In RTT we follow the following default classification:

Label	p-value	Interpretation
OK	p in [0.1; 0.99]	Test successful
SUSPECT	p in [0.01; 0.1] U [0.9; 0.99]	Indicates the necessity to re-test to confirm or infirm the null hypothesis.
KO	p < 0.01 and p > 0.99	If ocurring in the conidtion described above, indicates test failure

The p-values limits of 0.1 and 0.01 are by default and can be modified using the yaml configuration file see base_run_config.yaml.

↗️ ↘️ Comparison with Dieharder, NIST Test Suite and TestU01

The primary motivation for implementing this Random Test Tool was to provide to the community a tool highly user friendly, easy to manipulate, capable of delivering clear and explicit results data, requiring minimal configuration.

While there are, of course, other tools that implement statistical tests:

• Diehard (mainly in C and Fortran languages)
• NIST Statistical Test Suite (mainly in C language)
• TestU01 (mainly in Clanguage)

Implemented statistical tests comparison

The table below lists the implemented statistical tests by Tool.

Statistical test	Diehard	NIST Test Suite	Dieharder	TestU01	Random Test Tool
Monobit (Chi2)	✅	✅	✅	✅	✅
Frequency in block	✅	✅	✅	✅	✅
Run Test	✅	✅	✅	✅	✅
Longest run of Ones		✅	✅	✅	✅
Binary Rank	✅	✅	✅	✅	✅
DFT		✅	✅	✅	✅
Non-overlapping template matching	✅	✅	✅	✅
Overlapping template matching	✅	✅	✅	✅
Maurer test		✅	✅	✅	✅
Lempel-Ziv		✅	✅	✅
Linear Complexity		✅	✅	✅	✅
Serial		✅	✅	✅	✅
Approximate entropy		✅	✅	✅
Cumulative Sums		✅	✅	✅
Random excursions		✅	✅	✅	✅
Birthday Spacing	✅		✅	✅
5-Permutation	✅		✅	✅
OPSO/OQSO	✅		✅	✅
DNA	✅		✅	✅
Parking Lot	✅		✅	✅
Minimum Distance	✅		✅	✅
3-D Spheres	✅		✅	✅
Craps	✅		✅	✅
Squeeze	✅		✅	✅
Other "Crush Tests" (multiple tests)				✅
Other "BigCrush" tests (multiple tests)				✅

It is important to note that TestU01 implements three test batteries: SmallCrush, Crush, and BigCrush. Due to the substantial number of tests within these batteries, not all of them are listed in the above table.

From a purely statistical perspective, TestU01 is currently the most comprehensive test suite available .

Usage comparison

The table below provides a comparison of the previously discussed tools, focusing on criteria and features chosen by auditors (from an audit perspective rather than a statistical one).

The aim of this comparison is to help one user to identify the most relevant tool to pick based on the users' needs.

Theme	Control	Dieharder	NIST Test Suite	TestU01	Random Test Tool
Ease of installation	Via native operating system package manager	✅	❌	❌	❌
"	Via non-native package manager	❌	❌	❌	✅ (pip)
"	Via distributed executable	❌	❌	❌	❌
"	Via code compilation	✅	✅	✅	✅
"	Simple configuration (no/few non-standard installation tasks)	✅	✅	❌	✅
Sharing / Transparency	Open-source tool	✅	✅	✅	✅
Documentation	Quality of installation documentation	✅	❌	✔️	✅
"	Quality of "quick start" / "out-of-the-box" usage documentation	✔️	❌	❌	✅
"	Documentation usability quality (ease of search and presentation of topics)	✔️	❌	✔️	✔️
"	Quality of mathematical documentation (precise description of statistical tests)	✔️	✅	✔️	❌
Ease of use	Overall ease of use / intuitiveness	✔️	✅	✔️	✅
"	Automatable (e.g., no interactivity required)	✅	❌	✅	✅
Inputs/Outputs	Inputs: ASCII binary	✅	✅	✔️	✅
"	Inputs: Raw binary (files)	✅	✅	✔️	✅
"	Inputs: Floating numbers [0; 1]	❌	❌	✅	❌
"	Inputs: Integers / Range of integers	❌	❌	❌	✅
"	Outputs: Results within the terminal	✅	✅	✅	✅
"	Outputs: Structured outputs	✅	✅	✔️	✅
"	Outputs: Interpreted statistical results	✅	❌	✅	✅
"	Outputs: Measurement and numerical results returned	✅	✅	✔️	✅
Relevance of statistical tests	Completeness of tests algorithms and precision of the configuration	✔️	✔️	✅	✔️

Table caption

Color	Qualification
❌	Nonexistent / Difficult to identify / Complex
✔️	Partially addressed
✅	Complete / Adequate

Please note that the provided qualification represents a subjective perspective based on few hours of usage/research for each tool, within the context of random-related technical audits.

It's important to highlight that no performance comparison was conducted between the tools

👥 Contribute to this project!

Feedback, contributions and ideas are very Welcome 🙂!

Need some feature or encountering a bug?

Please open an issue and describe the encountered bug and or share any awesome ideas you may have related to the Random Test Tool project.

Steps for submitting code

1. Fork the current repository.

2. Write your feature. Please follow-up the PEP 8 coding style.

3. Send a GitHub Pull Request on the develop branch. Contributions will be merged after a code review. Branches will be moved to main when required.

📚 Related work

French blog posts

• Des dés aux données : de l'importance des nombres aléatoires (partie 1/2)
• Des dés aux données : de l'importance des nombres aléatoires (partie 2/2)

English blog posts

We intend to translate both of the above blog posts into English in the near future once our English blog is available 🙂.