metadata refresh is really expensive!

[rukai]

I am using scylla-rust-driver as the driver in a cassandra benchmark.
When that benchmark reaches 60s passed, throughput roughly halves for a few seconds and I have tracked this down to the metadata refresh.
I am able to eliminate the loss in throughput by changing this to a very large number : https://github.com/scylladb/scylla-rust-driver/blob/4efc84dfbc7bb204b49a8564378537e35cfe3ad1/scylla/src/transport/cluster.rs#L485`

I would like to raise two issues as a result:

issue 1

The metadata refresh should be made more performant.
I did a quick investigation and found it was MetadataReader::read_metadata that was impacting throughput.
The atomic swapping of metadata results seems to be working fine as removing the swap did not improve throughput in anyway.
I'm not sure if the cause is cassandra slowing down, the client slowing down due to running queries or the client slowing down due to processing results of queries, or a combination of these or something else entirely.

So I think it would be a good idea for the scylla-rust-driver team to give this a thorough investigation as it seems like this would cause a dip in production performance every 60s.
However for the needs of my project I think all I will need is what I describe in issue 2

issue 2

We need a way to disable and/or change the timing of the metadata refresh.
As I am writing a benchmark from which I can guarantee no other client is altering the schema or topology, I would like a way to completely disable such background work so I can evaluate the average throughput alone.

[rishabharyal]

Hi @rukai There is a PR active to manually set the topology refresh interval.
#776

However, other changes are not addressed in that pull request.

[rukai]

Oh, sorry for duplicating your pr, I'll go ahead and close mine.

It would make semantic sense to be able to set None in order to completely disable the refresh but I can always just set the value to 1000000 for effectively the same result.

[avelanarius]

@rukai Could you describe the setup of your benchmark in more detail?

I'm not able to reproduce the throughput dip around 60s. For my testing, I'm using cql-stress which uses Rust Driver, ./target/release/cql-stress-scylla-bench -mode write -workload sequential -nodes "127.0.0.1:9042" --partition-count 1000000. I tried it with both Scylla 5.4.0 (master) and Cassandra 3.11.15 and I don't see a throughput dip:

Logs

$ ./target/release/cql-stress-scylla-bench -mode write -workload sequential -nodes "127.0.0.1:9042" --partition-count 1000000
Configuration
Mode:			 write
Workload:		 sequential
Timeout:		 5.0s
Consistency level:	 quorum
Partition count:	 1000000
Clustering rows:	 100
Clustering row size:	 Fixed(4)
Rows per request:	 1
Page size:		 1000
Concurrency:		 16
Maximum rate:		 unlimited
Client compression:	 true
time        ops/s  rows/s errors    max 99.9th   99th   95th   90th median   mean
1.0s        62790   62790      0 1.60ms  716μs  442μs  354μs  326μs  248μs  254μs
2.0s        70961   70961      0 36.1ms  549μs  322μs  290μs  274μs  214μs  225μs
3.0s        70589   70589      0 30.9ms  424μs  331μs  295μs  278μs  217μs  226μs
4.0s        70173   70173      0 22.7ms  414μs  331μs  297μs  280μs  220μs  227μs
5.0s        71800   71800      0 23.7ms  934μs  396μs  291μs  272μs  212μs  222μs
6.0s        74249   74249      0 1.95ms  412μs  317μs  287μs  271μs  212μs  215μs
7.0s        71489   71489      0 29.4ms  351μs  316μs  288μs  273μs  215μs  223μs
8.0s        75378   75378      0  718μs  349μs  313μs  284μs  268μs  209μs  211μs
9.0s        71557   71557      0 36.0ms  820μs  337μs  287μs  271μs  211μs  223μs
10.0s       72476   72476      0 35.0ms  388μs  315μs  284μs  269μs  210μs  220μs
11.0s       74803   74803      0 1.46ms  358μs  313μs  284μs  269μs  211μs  213μs
12.0s       72413   72413      0 34.3ms  369μs  319μs  287μs  270μs  210μs  220μs
13.0s       74634   74634      0 1.24ms  359μs  314μs  285μs  270μs  212μs  213μs
14.0s       71706   71706      0 35.6ms  370μs  316μs  287μs  272μs  213μs  222μs
15.0s       71063   71063      0 34.0ms  755μs  346μs  290μs  273μs  213μs  224μs
16.0s       73064   73064      0 1.45ms  397μs  323μs  292μs  276μs  216μs  218μs
17.0s       72503   72503      0 33.4ms  368μs  316μs  285μs  270μs  210μs  220μs
18.0s       73485   73485      0 1.34ms  352μs  316μs  287μs  272μs  213μs  214μs
19.0s       71586   71586      0 34.8ms  411μs  332μs  294μs  276μs  215μs  225μs
20.0s       70933   70933      0 34.2ms  547μs  327μs  290μs  274μs  214μs  225μs
21.0s       72102   72102      0  515μs  375μs  325μs  296μs  281μs  219μs  221μs
22.0s       70153   70153      0 32.6ms  880μs  338μs  295μs  278μs  216μs  227μs
23.0s       72741   72741      0 1.35ms  352μs  311μs  284μs  270μs  211μs  213μs
24.0s       71337   71337      0 34.5ms  403μs  329μs  297μs  281μs  220μs  230μs
25.0s       71712   71712      0 35.0ms  378μs  317μs  288μs  272μs  212μs  222μs
26.0s       73984   73984      0 1.85ms  463μs  323μs  290μs  273μs  213μs  215μs
27.0s       72069   72069      0 35.8ms  366μs  316μs  285μs  270μs  212μs  221μs
28.0s       72660   72660      0 34.0ms  354μs  312μs  283μs  268μs  210μs  219μs
29.0s       72374   72374      0 1.59ms  870μs  344μs  292μs  276μs  216μs  220μs
30.0s       72031   72031      0 33.9ms  375μs  319μs  288μs  271μs  212μs  221μs
31.0s       74030   74030      0 1.84ms  488μs  326μs  289μs  273μs  212μs  215μs
32.0s       72400   72400      0 33.3ms  369μs  315μs  285μs  270μs  210μs  220μs
33.0s       71191   71191      0 34.8ms  382μs  322μs  290μs  274μs  214μs  224μs
34.0s       73433   73433      0  573μs  349μs  316μs  289μs  274μs  215μs  217μs
35.0s       70170   70170      0 37.4ms  781μs  347μs  293μs  276μs  215μs  227μs
36.0s       73203   73203      0 1.36ms  356μs  318μs  290μs  275μs  216μs  218μs
37.0s       71726   71726      0 34.5ms  440μs  319μs  286μs  271μs  213μs  222μs
38.0s       70500   70500      0 34.3ms  373μs  324μs  292μs  276μs  217μs  226μs
39.0s       74087   74087      0 1.43ms  367μs  319μs  288μs  272μs  213μs  215μs
40.0s       68335   68335      0 35.3ms  403μs  338μs  302μs  285μs  223μs  233μs
41.0s       68565   68565      0 27.1ms  412μs  336μs  302μs  286μs  225μs  232μs
42.0s       70874   70874      0 22.8ms 1.15ms  482μs  294μs  275μs  212μs  225μs
43.0s       71805   71805      0 28.0ms  471μs  322μs  289μs  273μs  214μs  222μs
44.0s       73421   73421      0 1.39ms  384μs  328μs  293μs  276μs  215μs  217μs
45.0s       72422   72422      0 33.7ms  361μs  314μs  285μs  269μs  211μs  220μs
46.0s       74150   74150      0 3.16ms  384μs  315μs  287μs  272μs  212μs  215μs
47.0s       72288   72288      0 35.0ms  368μs  313μs  284μs  269μs  211μs  220μs
48.0s       71578   71578      0 33.5ms  361μs  317μs  287μs  272μs  214μs  223μs
49.0s       73275   73275      0 2.47ms  880μs  333μs  290μs  273μs  213μs  217μs
50.0s       71695   71695      0 33.4ms  360μs  314μs  286μs  271μs  213μs  222μs
51.0s       74182   74182      0 1.34ms  356μs  316μs  288μs  272μs  213μs  215μs
52.0s       71606   71606      0 34.6ms  369μs  319μs  289μs  273μs  213μs  223μs
53.0s       72331   72331      0 36.0ms  360μs  313μs  284μs  268μs  211μs  220μs
54.0s       73054   73054      0  679μs  405μs  322μs  291μs  276μs  216μs  218μs
55.0s       70644   70644      0 34.3ms  686μs  346μs  292μs  275μs  213μs  226μs
56.0s       71554   71554      0 33.2ms  674μs  319μs  289μs  273μs  213μs  223μs
57.0s       74404   74404      0 1.30ms  351μs  314μs  286μs  271μs  212μs  214μs
58.0s       71700   71700      0 33.1ms  375μs  318μs  288μs  273μs  213μs  222μs
59.0s       72182   72182      0 1.68ms  384μs  326μs  296μs  280μs  219μs  221μs
1m0.0s      71219   71219      0 32.2ms  433μs  323μs  290μs  274μs  214μs  224μs
1m1.0s      71140   71140      0 33.2ms  379μs  319μs  289μs  274μs  215μs  224μs
1m2.0s      72428   72428      0 5.56ms  886μs  342μs  294μs  277μs  216μs  220μs
1m3.0s      71487   71487      0 32.4ms  373μs  322μs  289μs  273μs  214μs  223μs
1m4.0s      72515   72515      0 1.54ms  368μs  323μs  294μs  278μs  218μs  220μs
1m5.0s      71244   71244      0 32.8ms  388μs  327μs  292μs  276μs  214μs  224μs
1m6.0s      71459   71459      0 35.3ms  388μs  319μs  289μs  273μs  213μs  223μs
1m7.0s      73278   73278      0 1.30ms  380μs  322μs  292μs  276μs  215μs  217μs
1m8.0s      71029   71029      0 33.5ms  761μs  338μs  291μs  274μs  213μs  224μs

[rukai]

I tested my bencher again and observed that it doesnt happen locally, only when run on AWS.
Maybe the smaller node size, or the extra latency is the cause?
But I'll do some more investigation myself, when I get the chance.

I have a 3 node cassandra cluster running on 3 aws m6a.large instances
I have a bencher running on aws m6a.large instance.

You should be able to reproduce with:

git clone https://github.com/shotover/shotover-proxy
cd shotover-proxy
cargo windsock --cloud --name cassandra,compression=none,driver=scylla,operation=write_blob,protocol=v4,shotover=none,topology=cluster3 --bench-length-seconds 60

BIG WARNING THOUGH: this will create amazon EC2 instances if you have AWS credentials setup on your machine.
It will attempt to clean up after itself but you should make sure that it succeeds, possibly running cargo windsock --cleanup-cloud-resources to force a cleanup if it panics midway through.
If that sounds scary, fair enough, maybe just setup your own bench manually on cloud infrastructure and see if you can reproduce that way.

The throughput drop happens at about 42s into the bench since the driver is started before benching starts.