Skip to content

Commit

Permalink
Update benchmark documentation in tool/readme.md (#1812)
Browse files Browse the repository at this point in the history
### Description of changes: 
Update the documentation to use the more flexible `BENCHMARK_LIBS`
option that supports more libraries than the previous options.

### Testing:
Ran all the commands locally to ensure they work as expected.

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license and the ISC license.
  • Loading branch information
andrewhop authored Sep 9, 2024
1 parent f506b4b commit 8835f07
Showing 1 changed file with 60 additions and 15 deletions.
75 changes: 60 additions & 15 deletions tool/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,9 @@ The speed subtool of `bssl` runs a performance test for a number of cryptographi
## Benchmarking Tools
When compiled, AWS-LC will generate separate benchmarking tools when provided with corresponding compiler flags. These tools take the same arguments as `bssl speed` tool.

The `awslc_bm` tool is expected to be used when benchmarking an installation of AWS-LC with a different speed tool: e.g.
This tool also supports building a newer benchmark with an older version of AWS-LC's code: e.g.
build and install AWS-LC FIPS from 2021 but run the latest benchmark tool from main. To benchmark the AWS-LC libcrypto
from the current folder it is recomended to run `bssl speed` which executes the same code as other benchmarks: e.g.
`ossl_1_1_bm`.
from the current folder it is recommended to run `bssl speed` which executes the same code as other benchmarks.

Additionally, the speed tool now prints a message when it is benchmarking a non-release build of AWS-LC instead of a release build of the project.

Expand All @@ -28,24 +27,70 @@ Running each tool without any options (e.g. `./awslc_bm`) will run all available
Additionally, there are a number of arguments that enable different functionality:
* `-filter` provides a filter on the benchmarking tests to be run.
* `-timeout` sets the number of seconds each test is run for (default 1).
* `-threads` is a comma-separated list of thread counts to run multithreaded benchmarks (default is "1,2,4,8,16,32,64")
* `-chunks` is a comma-separated list of input sizes to run tests at (default is "
"16,256,1350,8192,16384)
* `-json` has the tool print the output of each benchmark in JSON format.

For example, `./awslc_bm -filter AES -timeout 10 -chunks 16, 256, -json` will run all AES-related tests with input sizes of 16 and 256 for 10 seconds, and output the results in JSON format.

## Setup
In order to build the above-mentioned benchmarking tools, absolute paths to each libaries' install location must be provided via compiler flags.

### Compiler Flags
| Tool Name | Compiler Flag |
| ------------- | ------------- |
| bssl speed | (none) |
| awslc_bm | -DAWSLC_INSTALL_DIR |
| bssl_bm | -DBORINGSSL_INSTALL_DIR |
| ossl_1_0_bm | -DOPENSSL_1_0_INSTALL_DIR |
| ossl_1_1_bm | -DOPENSSL_1_1_INSTALL_DIR |
| ossl_3_0_bm | -DOPENSSL_3_0_INSTALL_DIR |
## Comparison Setup
The AWS-LC benchmark supports building and running with other common libcryptos.
Build and install the other libcryptos you would like to test with locally, for
example building AWS-LC 2022 FIPS branch and OpenSSL 3.3 to compare with AWS-LC
main branch:

```bash
mkdir ~/aws-lc-benchmark && pushd ~/aws-lc-benchmark

git clone -b openssl-3.3 --depth 1 https://github.com/openssl/openssl.git openssl-3.3-src
pushd openssl-3.3-src
./config --prefix="${HOME}/aws-lc-benchmark/openssl-3.3-install" --openssldir="${HOME}/aws-lc-benchmark/openssl-3.3-install"
make -j
make install_sw
popd


git clone -b fips-2022-11-02 --depth 1 https://github.com/aws/aws-lc.git aws-lc-fips-src
pushd aws-lc-fips-src
cmake -DCMAKE_INSTALL_PREFIX="${HOME}/aws-lc-benchmark/aws-lc-fips-install" -DCMAKE_BUILD_TYPE=Release -DFIPS=1 -DBUILD_SHARED_LIBS=1
make -j install
popd && popd
```

To build the main branch speed.cc against other libraries pass in the
BENCHMARK_LIBS option when running CMake. BENCHMARK_LIBS is a list of tuples,
the format is `executable_name:install_path`. `executable_name` is the name for
the benchmark that AWS-LC will build with whatever library is in `install_path`.
Multiple libraries can be specified with a semicolon between them:
`executable1_name:executable1_install_path;executable1_name:executable1_install_path;`

To build AWS-LC main speed.cc against the two previously built libcrypto libraries
(AWS-LC FIPS 2022 and OpenSSL 3.3):
```bash
pushd ~/aws-lc-benchmark
git clone -b main --depth 1 https://github.com/aws/aws-lc.git aws-lc-main-src
pushd aws-lc-main-src
cmake -DCMAKE_BUILD_TYPE=Release -DBENCHMARK_LIBS="\
aws-lc-fips-2022:${HOME}/aws-lc-benchmark/aws-lc-fips-install;\
openssl-3-3:${HOME}/aws-lc-benchmark/openssl-3.3-install;"
make -j
popd && popd
```

This will build 3 relevant binaries:
* `~/aws-lc-benchmark/aws-lc-main-src/tool/bssl` is the complete tool build with the main branch of code, `speed` is required to run the benchmark
* `~/aws-lc-benchmark/aws-lc-main-src/tool/aws-lc-fips-2022` is the main branch of speed.cc built with the AWS-LC FIPS 2022 install
* `~/aws-lc-benchmark/aws-lc-main-src/tool/openssl-3-3` is the main branch of speed.cc built with the OpenSSL 3.3 install

Not all benchmarks will be available with all libraries, for example OpenSSL 3.3
does not support ML-KEM.

```
~/aws-lc-benchmark/aws-lc-main-src/tool/bssl speed -filter P-256
~/aws-lc-benchmark/aws-lc-main-src/tool/aws-lc-fips-2022 -filter P-256
~/aws-lc-benchmark/aws-lc-main-src/tool/openssl-3-3 -filter P-256
```

### Expected Directory Structure
Additionally, the benchmarking tools expects specific directory structures for the provided install locations for each
Expand Down

0 comments on commit 8835f07

Please sign in to comment.