Reduce ConnectionPool size used by send-transaction-service

[lijunwangs]

Problem

Reduce pool size for ConnectionCache used by send-transaction-service

Summary of Changes

Reduce pool size for ConnectionCache used by send-transaction-service to 2 from 4. No significant slow down of performance from bench-tps testing using rpc-client which is used by send-transaction-service. This will reduce active connections maintained both on the server and client. This will enable us to cache connections for more nodes.

Fixes #

[lijunwangs]

size=2

Highest TPS: 24064.27 sampling period 1s max transactions: 208421 clients: 1 drop rate: 0.60

[2023-10-04T21:35:55.736701590Z INFO solana_bench_tps::bench] Average TPS: 6890.304

Highest TPS: 21028.71 sampling period 1s max transactions: 225191 clients: 1 drop rate: 0.56

[2023-10-04T22:42:48.290458451Z INFO solana_bench_tps::bench] Average TPS: 7473.9106

Highest TPS: 22264.54 sampling period 1s max transactions: 248191 clients: 1 drop rate: 0.51

[2023-10-04T22:53:46.822453533Z INFO solana_bench_tps::bench] Average TPS: 7998.1733

Highest TPS: 24731.69 sampling period 1s max transactions: 197774 clients: 1 drop rate: 0.61

[2023-10-04T23:00:28.413413709Z INFO solana_bench_tps::bench] Average TPS: 6373.1846

Highest TPS: 25901.69 sampling period 1s max transactions: 194674 clients: 1 drop rate: 0.60

[2023-10-04T23:05:54.484348552Z INFO solana_bench_tps::bench] Average TPS: 6471.919

size=4

Highest TPS: 21053.14 sampling period 1s max transactions: 218945 clients: 1 drop rate: 0.57

[2023-10-05T15:22:16.308344351Z INFO solana_bench_tps::bench] Average TPS: 7056.3384

Highest TPS: 27498.80 sampling period 1s max transactions: 240015 clients: 1 drop rate: 0.53

[2023-10-05T15:26:18.136180982Z INFO solana_bench_tps::bench] Average TPS: 7965.6826

Highest TPS: 34923.98 sampling period 1s max transactions: 265108 clients: 1 drop rate: 0.50

[2023-10-05T15:36:39.110740734Z INFO solana_bench_tps::bench] Average TPS: 8542.928

Highest TPS: 36179.60 sampling period 1s max transactions: 223725 clients: 1 drop rate: 0.56

[2023-10-05T16:01:57.050631312Z INFO solana_bench_tps::bench] Average TPS: 7443.8564

Highest TPS: 37270.70 sampling period 1s max transactions: 246068 clients: 1 drop rate: 0.52

[2023-10-05T17:20:10.456737141Z INFO solana_bench_tps::bench] Average TPS: 8129.8066

[lijunwangs]

Just an FYI, when using tpu-client, the pool size still made big difference:

size=2

 Highest TPS: 7636.77 sampling period 1s max transactions: 185344 clients: 1 drop rate: 0.00

[2023-10-04T19:50:29.126351162Z INFO solana_metrics::metrics] datapoint: bench-tps-lamport_balance balance=0i
[2023-10-04T19:50:29.126357351Z INFO solana_bench_tps::bench] Average TPS: 5970.13

Highest TPS: 10785.58 sampling period 1s max transactions: 210987 clients: 1 drop rate: 0.00

[2023-10-04T19:51:55.739417025Z INFO solana_bench_tps::bench] Average TPS: 7004.1636

Highest TPS: 9268.18 sampling period 1s max transactions: 202348 clients: 1 drop rate: 0.00

[2023-10-04T19:54:56.555236496Z INFO solana_bench_tps::bench] Average TPS: 6518.2456

size=4:

Highest TPS: 16863.32 sampling period 1s max transactions: 334348 clients: 1 drop rate: 0.00

[2023-10-04T20:23:44.664669703Z INFO solana_bench_tps::bench] Average TPS: 10771.607

Highest TPS: 14725.02 sampling period 1s max transactions: 316948 clients: 1 drop rate: 0.00

[2023-10-04T20:24:56.718196712Z INFO solana_bench_tps::bench] Average TPS: 10509.352

Highest TPS: 16251.96 sampling period 1s max transactions: 313519 clients: 1 drop rate: 0.00

[2023-10-04T20:26:10.040882712Z INFO solana_bench_tps::bench] Average TPS: 10405.963

[lijunwangs]

The send-transaction-service itself is not using tpu-client... Maybe we should consider utilizing that? @CriesofCarrots

[codecov]

Codecov Report

Merging #33548 (0f805e4) into master (6ea5128) will decrease coverage by 0.1%.
Report is 67 commits behind head on master.
The diff coverage is 100.0%.

@@            Coverage Diff            @@
##           master   #33548     +/-   ##
=========================================
- Coverage    81.9%    81.9%   -0.1%     
=========================================
  Files         799      799             
  Lines      217339   217341      +2     
=========================================
- Hits       178163   178105     -58     
- Misses      39176    39236     +60