State column block cache size

[jbajic]

Introduction

Tracking State column performance after removing filter and index block from block cache with different block cache sizes. After applying these changes we test out if the block cache size has any impact on the performance of the fetching data from the State column. Using the state-perf database tool to check out the performance of reading data from the State column.

This was run on the n2-highcpu-16 GCP node with persistent disk.

The only changes here in the codebase are set_cache_index_and_filter_blocks set to false and block_cache for the State column set to the value in measurements.

Measurements

Column state with block cache size 0MiB:

overall | avg observed_latency: 671.937µs, block_read_time: 624.941µs, samples with merge: 2699 (2.70%)
block_read_count: 0, samples: 7 (0.01%): | avg observed_latency: 20.489µs, block_read_time: 0ns, samples with merge: 7 (100.00%)
block_read_count: 1, samples: 95909 (95.91%): | avg observed_latency: 645.64µs, block_read_time: 599.499µs, samples with merge: 496 (0.52%)
block_read_count: 2, samples: 3750 (3.75%): | avg observed_latency: 1.242928ms, block_read_time: 1.185486ms, samples with merge: 1908 (50.88%)
block_read_count: 3, samples: 299 (0.30%): | avg observed_latency: 1.785408ms, block_read_time: 1.711931ms, samples with merge: 281 (93.98%)
block_read_count: 4, samples: 7 (0.01%): | avg observed_latency: 2.372064ms, block_read_time: 2.28925ms, samples with merge: 7 (100.00%)
block_read_count: 6, samples: 28 (0.03%): | avg observed_latency: 2.121931ms, block_read_time: 826.92µs, samples with merge: 0 (0.00%)

Column state with block cache size 16MiB:

overall | avg observed_latency: 667.613µs, block_read_time: 618.295µs, samples with merge: 2699 (2.70%)
block_read_count: 0, samples: 25 (0.03%): | avg observed_latency: 19.427µs, block_read_time: 0ns, samples with merge: 7 (28.00%)
block_read_count: 1, samples: 95897 (95.90%): | avg observed_latency: 641.485µs, block_read_time: 593.009µs, samples with merge: 497 (0.52%)
block_read_count: 2, samples: 3745 (3.75%): | avg observed_latency: 1.239508ms, block_read_time: 1.179242ms, samples with merge: 1907 (50.92%)
block_read_count: 3, samples: 299 (0.30%): | avg observed_latency: 1.770989ms, block_read_time: 1.695284ms, samples with merge: 282 (94.31%)
block_read_count: 4, samples: 6 (0.01%): | avg observed_latency: 2.395133ms, block_read_time: 2.317414ms, samples with merge: 6 (100.00%)
block_read_count: 6, samples: 28 (0.03%): | avg observed_latency: 2.089112ms, block_read_time: 876.598µs, samples with merge: 0 (0.00%)

Column state with block cache size 32MiB:

overall | avg observed_latency: 676.802µs, block_read_time: 627.189µs, samples with merge: 2699 (2.70%)
block_read_count: 0, samples: 57 (0.06%): | avg observed_latency: 18.474µs, block_read_time: 0ns, samples with merge: 7 (12.28%)
block_read_count: 1, samples: 95870 (95.87%): | avg observed_latency: 650.734µs, block_read_time: 601.967µs, samples with merge: 500 (0.52%)
block_read_count: 2, samples: 3741 (3.74%): | avg observed_latency: 1.252987ms, block_read_time: 1.192053ms, samples with merge: 1905 (50.92%)
block_read_count: 3, samples: 298 (0.30%): | avg observed_latency: 1.792474ms, block_read_time: 1.718403ms, samples with merge: 281 (94.30%)
block_read_count: 4, samples: 6 (0.01%): | avg observed_latency: 2.261001ms, block_read_time: 2.182221ms, samples with merge: 6 (100.00%)
block_read_count: 6, samples: 28 (0.03%): | avg observed_latency: 2.077167ms, block_read_time: 844.535µs, samples with merge: 0 (0.00%)

Column state with block cache size 512MiB:

overall | avg observed_latency: 671.233µs, block_read_time: 620.134µs, samples with merge: 2699 (2.70%)
block_read_count: 0, samples: 836 (0.84%): | avg observed_latency: 18.865µs, block_read_time: 0ns, samples with merge: 18 (2.15%)
block_read_count: 1, samples: 95246 (95.25%): | avg observed_latency: 650.045µs, block_read_time: 599.542µs, samples with merge: 555 (0.58%)
block_read_count: 2, samples: 3615 (3.62%): | avg observed_latency: 1.280448ms, block_read_time: 1.217019ms, samples with merge: 1867 (51.65%)
block_read_count: 3, samples: 269 (0.27%): | avg observed_latency: 1.83444ms, block_read_time: 1.760159ms, samples with merge: 253 (94.05%)
block_read_count: 4, samples: 6 (0.01%): | avg observed_latency: 2.269337ms, block_read_time: 2.189532ms, samples with merge: 6 (100.00%)
block_read_count: 6, samples: 28 (0.03%): | avg observed_latency: 2.048748ms, block_read_time: 831.086µs, samples with merge: 0 (0.00%)

From these results, we can see that there is not much improvement when increasing the block cache size of the State. This needs to be taken with a grain of salt since the access pattern when doing state-perf benchmarks is always unique, but we saw in previous benchmarks that the increase in block cache does improve performance in measurements so we would expect some difference in improvements here but there was none.
But from the aspect of RocksDB we want to cache State in our own Storage layer since we know better which data we will use.

References:

1. Disable RocksDB cache_index_and_filter_blocks for State and FlatState columns #9319