The evaluation of NeoBFT consists of three parts:
- AOM micro-benchmarks (section 6.1, figure 4-6)
- four replica performance benchmarks (section 6.2, 6.4 and 6.5, figure 7, 9 and 10)
- AWS deployment (section 6.3, figure 8).
The first two parts should be conducted on a hardware-accessible cluster. For part one, you will need a Tofino1 switch, a Xillinx AU50 FPGA and a server machine. For part two, you will need the switch and the FPGA same as part 1, and five server machines, and the first four of which must have identical specification.
For part three, you will need an AWS account (and pay for the bill after evaluation).
For all three parts, you will also need a development environment that builds artifacts, controls the evaluation runs and collect results. Refer to the preparation guide for detail setup.
During the evaluation, you will only need to issue commands from switch, the first server machine and development environment.
Open three terminals, the first two on switch and the third on the first server.
In the first terminal, start switchd for HMAC program
switch:~$ $SDE/run_switchd.sh -p neo_hmac_bench
You may be prompted to enter password for sudo access.
Wait until the switchd prompt bfshell> is shown.
In the second terminal, set up switch program
switch:~$ cd neo-switch
switch:~/neo-switch$ ./setup_hmac.sh
Start packet generation script, input "99" for load
switch:~/neo-switch$ ./pktgen_hmac.sh
... (output omitted)
load: 99
In the third terminal, capture packets (assuming network interface named ens1f0)
server1:~$ sudo tcpdump -i ens1f0 -e | grep -e "00:00:00:" > extracted.log
In the second terminal, start packet generation session
PD(neo_hmac_bench)[0]>>> pktgen.app_enable(1)
In the first terminal, check switch's instantaneous throughput (there may be some output after the prompt which is safe to be ignored)
bfshell> ucli
Starting UCLI from bf-shell
Cannot read termcap database;
using dumb terminal settings.
bf-sde> pm rate-period 1
bf-sde> pm rate-show
-----+----+---+----+-------+---+-------+-------+---------+---------+----+----
PORT |MAC |D_P|P/PT|SPEED |RDY|RX Mpps|TX Mpps|RX Mbps |TX Mbps |RX %|TX %
-----+----+---+----+-------+---+-------+-------+---------+---------+----+----
9/0 |15/0| 0|0/ 0|100G |UP | 0.00| 76.25| 0.00| 54904.47| 0%| 54%
... (output omitted)
bf-sde> exit
As shown in Tx Mpps column, the throughput is 76.25Mpps.
After ~10 seconds, stop packet generation
PD(neo_hmac_bench)[0]>>> pktgen.app_disable(1)
Enter ctrl-d to exit session.
In the third terminal, ctrl-c to stop packet capturing.
Save MAC addresses into result file
server1:~$ awk -F ' ' '{print $6}' extracted.log > results_hmac99.log
Repeat the process above, input "50" for load.
Save the capturing results to results_hmac50.log.
You don't need to check for throughput in the first terminal.
Repeat the process above, input "25" for load.
Save the capturing results to results_hmac25.log.
You don't need to check for throughput in the first terminal.
In the first terminal, ctrl-\ to stop switchd.
To collect data for public key cryptography variant, repeat the same process while replacing hmac with fpga, i.e. neo_hmac_bench to neo_fpga_bench, ./setup_hmac.sh to ./setup_fpga.sh, ./pktgen_hmac.sh to ./pktgen_fpga.sh.
The throughput shown in the first terminal is 1.11Mpps.
Save the results to results_fpgaXX.log files.
Open three terminals, the first two on switch and the third on dev.
In the first terminal, start switchd for HMAC program
switch:~$ $SDE/run_switchd.sh -p neo_hmac
You may be prompted to enter password for sudo access.
Wait until the switchd prompt bfshell> is shown.
In the second terminal, set up switch program
switch:~$ cd neo-switch
switch:~/neo-switch$ ./setup_hmac.sh
In the third terminal, reload servers and run benchmarks
dev:~$ cd neobft-artifact
dev:~/neobft-artifact$ cargo -q run -p reload
Finished release [optimized] target(s) in 0.16s
* server started on server1
dev:~/neobft-artifact$ cargo -q run -p control -- hmac
The results will be saved to saved-hmac.csv.
The complete run takes minutes.
If the evaluating is crashed or killed manually, it will resume in the next run.
Just make sure to reload before running it again.
After running, in the first terminal enter ctrl-\ to stop switchd.
Then repeat the process with hmac replaced to fpga, i.e. neo_hmac to neo_fpga, ./setup_hmac.sh to ./setup_fpga.sh, and hmac to fpga in the control script command.
The results are saved to saved-fpga.csv.
Open one terminal on dev.
Create AWS cluster
dev:~$ cd neobft-artifact
dev:~/neobft-artifact$ terraform -chdir=scripts/neo-aws apply
Enter "yes" when prompted.
Initialize the cluster
dev:~/neobft-artifact$ cargo -q run -p neo-aws
Reload servers and run control script
dev:~/neobft-artifact$ cargo -q run -p reload --features aws
Finished release [optimized] target(s) in 0.07s
* server started on ip-x-x-x-x.ap-east-1.compute.amazonaws.com
dev:~/neobft-artifact$ cargo -q run -p control --features aws -- aws
The results will be saved to saved-aws.csv.
After running, destroy AWS cluster
dev:~/neobft-artifact$ terraform -chdir=scripts/neo-aws destroy
Enter "yes" when prompted.
Collect all result files into data directory and refer to the notebooks.
This is a complete overhaul of the original version.
The exact code the produces camera-ready results can be found in master-legacy branch.
Noticeable implementation differences:
- Replace public key cryptography library
secp256k1tok256 - Remove cryptography offloading and worker threads
- Apply the same pace-based adaptive batching strategy to all protocols universally
- The Zyzzyva implementation is currently buggy under the new strategy, but its faulty variant seems normal
These changes help produce more realistic results, better explore protocol's best and typical latency, and reduce resource utilization.