Add new stat rocksdb.table.open.prefetch.tail.read.bytes, rocksdb.table.open.prefetch.tail.{miss|hit}

[hx235]

Context/Summary:
We are adding new stats to measure behavior of prefetched tail size and look up into this buffer

The stat collection is done in FilePrefetchBuffer but only for prefetched tail buffer during table open for now using FilePrefetchBuffer enum. It's cleaner than the alternative of implementing in upper-level call places of FilePrefetchBuffer for table open. It also has the benefit of extensible to other types of FilePrefetchBuffer if needed. See db bench for perf regression concern.

Test:
- Piggyback on existing test
- rocksdb.table.open.prefetch.tail.miss is harder to UT so I manually set prefetch tail read bytes to be small and run db bench.

./db_bench -db=/tmp/testdb -statistics=true -benchmarks="fillseq" -key_size=32 -value_size=512 -num=5000 -write_buffer_size=655 -target_file_size_base=655 -disable_auto_compactions=false -compression_type=none -bloom_bits=3  -use_direct_reads=true

rocksdb.table.open.prefetch.tail.read.bytes P50 : 4096.000000 P95 : 4096.000000 P99 : 4096.000000 P100 : 4096.000000 COUNT : 225 SUM : 921600
rocksdb.table.open.prefetch.tail.miss COUNT : 91
rocksdb.table.open.prefetch.tail.hit COUNT : 1034

- No perf regression observed in db_bench

SETUP command: create same db with ~900 files for pre-change/post-change.

./db_bench -db=/tmp/testdb -benchmarks="fillseq" -key_size=32 -value_size=512 -num=500000 -write_buffer_size=655360  -disable_auto_compactions=true -target_file_size_base=16777216 -compression_type=none

TEST command 60 runs or til convergence: as suggested by @anand1976 and @akankshamahajan15, vary seek_nexts and async_io in testing.

./db_bench -use_existing_db=true -db=/tmp/testdb -statistics=false -cache_size=0 -cache_index_and_filter_blocks=false -benchmarks=seekrandom[-X60] -num=50000 -seek_nexts={10, 500, 1000} -async_io={0|1} -use_direct_reads=true

async io = 0, direct io read = true

	seek_nexts = 10, 30 runs	seek_nexts = 500, 12 runs	seek_nexts = 1000, 6 runs
pre-post change	4776 (± 28) ops/sec; 24.8 (± 0.1) MB/sec	288 (± 1) ops/sec; 74.8 (± 0.4) MB/sec	145 (± 4) ops/sec; 75.6 (± 2.2) MB/sec
post-change	4790 (± 32) ops/sec; 24.9 (± 0.2) MB/sec	288 (± 3) ops/sec; 74.7 (± 0.8) MB/sec	143 (± 3) ops/sec; 74.5 (± 1.6) MB/sec

async io = 1, direct io read = true

	seek_nexts = 10, 54 runs	seek_nexts = 500, 6 runs	seek_nexts = 1000, 4 runs
pre-post change	3350 (± 36) ops/sec; 17.4 (± 0.2) MB/sec	264 (± 0) ops/sec; 68.7 (± 0.2) MB/sec	138 (± 1) ops/sec; 71.8 (± 1.0) MB/sec
post-change	3358 (± 27) ops/sec; 17.4 (± 0.1) MB/sec	263 (± 2) ops/sec; 68.3 (± 0.8) MB/sec	139 (± 1) ops/sec; 72.6 (± 0.6) MB/sec

[facebook-github-bot]

@hx235 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[ajkr]

LGTM, thanks!

[facebook-github-bot]

@hx235 has updated the pull request. You must reimport the pull request before landing.

[facebook-github-bot]

@hx235 has updated the pull request. You must reimport the pull request before landing.

[facebook-github-bot]

@hx235 has updated the pull request. You must reimport the pull request before landing.

[facebook-github-bot]

@hx235 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@hx235 has updated the pull request. You must reimport the pull request before landing.

[hx235]

@ajkr - not official review request but just want to let you know that after adding rocksdb.table.open.prefetch.tail.{miss|hit}, I realized there is a cleaner solution to rocksdb.table.open.prefetch.tail.read.bytes and a bug I made in the original PR. I decided to fix them in the same PR but will request again later since the original PR was accepted.

[facebook-github-bot]

@hx235 has updated the pull request. You must reimport the pull request before landing.

[facebook-github-bot]

@hx235 has updated the pull request. You must reimport the pull request before landing.

Making new changes to the PR that worth another review

[hx235]

@ajkr - Hi it's ready for review! Sorry I decided to make changes to previous approved PR, including adding the new stats as they are too related. For that I have to click "dismiss" your previous approval and request a new review on the PR. Let me know if there is any concern/questions.

Main change: a cleaner solution to collect rocksdb.table.open.prefetch.tail.*.stats in FilePrefetchBuffer level by distinguishing different usage of FilePrefetchBuffer (extensible for future too if we want to know stat about certain FilePrefetchBuffer); fix a bug I made in the original PR by including system prefetch in the buffer byte; variable renaming and better tests

[facebook-github-bot]

@hx235 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@hx235 has updated the pull request. You must reimport the pull request before landing.

[facebook-github-bot]

@ajkr has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[akankshamahajan15]

FilePrefetchBuffer side changes LGTM.

[facebook-github-bot]

@hx235 has updated the pull request. You must reimport the pull request before landing.

[facebook-github-bot]

@hx235 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[hx235]

rebase and re-import (cuz I forgot to do that previously :P)

[ajkr]

LGTM, thanks!

[facebook-github-bot]

@hx235 has updated the pull request. You must reimport the pull request before landing.

[facebook-github-bot]

@hx235 has updated the pull request. You must reimport the pull request before landing.

[facebook-github-bot]

@hx235 has updated the pull request. You must reimport the pull request before landing.

[facebook-github-bot]

@hx235 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@hx235 merged this pull request in bab5f9a.