feat(primitives) : Chunk Implementation for Parallel Sender Recovery

[Arindam2407]

Added chunk implementation to TransactionSigned::recover_signers using stream wrappers to handle parallel sender recovery using rayon. Replaced into_par_iter method handling parallel sender recovery for one signed transaction at a time. Partially resolving #5189 issue.

[onbjerg]

Thanks for taking this on! How does this impl compare perf wise to the current impl on main?

[rkrasiuk]

supportive of @onbjerg comments:

• locks are unnecessary here. the better API would be to return the stream wrapper around receiver channels to guarantee the order of recovered signers
• we need benchmarks to make sure this is an improvement

[joshieDo]

these constants seem a bit too arbitrary/random

[Arindam2407]

Can definitely change that

[joshieDo]

can we remove this unwrap? even if we have to change to Result<Option<...>>

[rakita]

@onbjerg @rkrasiuk please check the benchmark tests I have added. The chunk implementation outperforms the current into_par_iter method after around 100 transactions.

Just a note here, #[test] are run unoptimized with debug so this measurement is not valid

[Arindam2407]

Fixed the arbitrary constants. Chunk implementation amended to pass previously failing consensus tests. into_par_iter slightly outperforms chunk impl in unoptimized tests but the overhead of work stealing with into_par_iter might increase with the number of transactions.

[gakonst]

Can you re-run this with mainnet SenderRecovery stage via reth stage run senders and measure the time before/after this PR?

[gakonst]

Hi @Arindam2407 wonder if you had a chance to compare performance before/after?

[Arindam2407]

Hi @Arindam2407 wonder if you had a chance to compare performance before/after?

Hi @gakonst, I was not able to run the mainnet SenderRecovery stage because of my device specs. Instead, I ran optimized tests using cargo test --package reth-primitives --release to get more reliable measurement as pointed out by @rakita. I compared the performance before/after (without/with chunks) for recovering 100,000 transactions.

The test:

Additional crates used in mod tests:

use std::{fs::File, io::Write, time::Instant};
use rayon::prelude::*;

    #[test]
    fn benchmark_chunks_parallel_sender_recovery_100000(txes in proptest::collection::vec(proptest::prelude::any::<Transaction>(), *PARALLEL_SENDER_RECOVERY_THRESHOLD * 20000)) {
        let mut rng =rand::thread_rng();
        let secp = Secp256k1::new();
        let txes: Vec<TransactionSigned> = txes.into_iter().map(|mut tx| {
             if let Some(chain_id) = tx.chain_id() {
                // Otherwise we might overflow when calculating `v` on `recalculate_hash`
                tx.set_chain_id(chain_id % (u64::MAX / 2 - 36));
            }

            let key_pair = KeyPair::new(&secp, &mut rng);

            let signature =
                sign_message(B256::from_slice(&key_pair.secret_bytes()[..]), tx.signature_hash()).unwrap();

            TransactionSigned::from_transaction_and_signature(tx, signature)
        }).collect();

        let mut file = File::create("benchmark.txt")?;

        let start_chunk = Instant::now();
        let _par_chunk_senders = TransactionSigned::recover_signers(&txes, txes.len());
        let elapsed_chunk = start_chunk.elapsed();

        let start = Instant::now();
        let _par_senders = txes.into_par_iter().map(|tx| tx.recover_signer()).collect::<Option<Vec<_>>>();
        let elapsed = start.elapsed();

        write!(file, "Time taken for parallel recovery of 100000 addresses: {:.2?} (with chunks) and {:.2?} (without chunks)", elapsed_chunk, elapsed)?;

    }

The results:

Chunk implementation takes 20-30% less time than the current implementation in all runs. It would be great if anyone else could run it and compare the performance. Since they are all optimized test runs, I think the estimates are reliable.