perf: kv0/kv95 regression around Feb 25 [loosely coupled log truncation]

[tbg]

kv0:

kv95

I'm investigating this.

$ git log origin/master --topo-order --merges --oneline --since '2022-02-24' --until '2022-02-26'

Bisecting via

GCE_PROJECT=andrei-jepsen ./pkg/cmd/roachtest/roachstress.sh -c 5 '^kv95/enc=false/nodes=3/size=64kb$'
for f in artifacts/9dc9eb034d/*/kv95/enc\=false/nodes\=3/size\=64kb/run_*/*run_kv.log; do tail -n 12 $f; done | grep 'read$'

• 8e72482 Merge sql: dropping user with default privilege's error message is more clear #77016
• addf3d4 Merge pull request release: enable bincheck for tag pushes #77038 from rail/bincheck
• 86bb64a Merge sql: extend resolverQueue to support waiter transactions #76719
• 4a8d2f5 Merge kvserver: activate a StoreRebalancer log message without verbosity levels set #76473 opt: fetch virtual columns during cascading delete #77052
• 6a24990 Merge #76086 server: remove query constants in sessions response for VIEWACTIVITYR… #76675 ttl: finalize for v22.1 release #76918 bulk: rename WriteAtRequestTimestamp parameters #76982 roachtest: ignore flaky activerecord test #77042
• 2826900 Merge sql, cli: support basic auto complete for sql keywords #72925 cli: Enable profiles and other debug info for tenants #76539 telemetry,sql: remove redaction from operational sql data #76676 physicalplan: add support for multi-stage execution of corr, covar_samp, sqrdiff, and regr_count aggregate functions. #76754 roachtest: update 22.1 version map to v21.2.6 #76908 spanconfigreconciler{ccl}: apply system span config diffs to the store #76948 sql, geo: Fix upper case geohash parsing and allow NULL arguments #76990 changefeedccl: Fix data race in a test. #77006 log: Set content type header for http sink #77014 sql: update tpcc and tpch stats #77019 bazel: bump rules_go to pick up cockroachdb/rules_go#4 #77045 dev: make sure we inherit stdout and stderr when appropriate #77047 sql: skip flaky schema_changer/drop_database_cascade test #77049
• c3ff362 Merge distsql: adds support for distributed bulk ops in multi-tenant #76566 ci: a couple tweaks to bazel CI #76958 sql: fix error handling for AOST clause #77025 schemachanger: bump size of test #77044
• 0679c4c Merge kv/kvserver: skip TestReplicaCircuitBreaker_RangeFeed #76971
• 715c1eb Merge scpb: add new Status values, add and adopt TargetStatus type #77013
• 95b0430 Merge spanconfigkvsubscriber: teach the KVSubscriber about system span configs  #76942
• 9aae74b Merge spanconfigsqltranslator,jobsprotectedts: add ignore_if_excluded_from_backup to SpanConfig #76831
• dab67a9 Merge pprofui: Increase concurrency for profiles #76266 changefeedccl: allow users to alter changefeed options #76583
• 08bbadf Merge ui: De-duplicate and clarify time selection state #75586 pgwire: fix flaky AOST test #77009
• d6e02c5 Merge kvserver: consider VOTER_INCOMING for lease transfer even if current … #76974 authors: add Xiang Gu to authors #77003
• 94e64ce Merge opt: remove redundant arbiter indexes #76961 sql: use IndexFetchSpec in JoinReader #76963 🔴 ~6.5k read/sec
• 978744d Merge kvserver: enable loosely coupled raft log truncation #76902 🔴 ~6.5k read/sec
• bc8cbe6 Merge sql: add bucket count limit for hash sharded index #77004 🟢 ~7k read/sec
• cdd55fb Merge sql, pgwire: remove flakey part of pgwire aost test #76993 🟢 ~7k read/sec
• 920464f Merge admission: SET default_transaction_quality_of_service=[background,regular,critical]  #76512 ui:increase timeout for statements api call #76739 genbzl: hoist bindata sources #76956 build: use TESTTIMEOUT in teamcity-test #76981 🟢 ~7k read/sec
• 50c5a92 Merge kv/kvserver: skip TestReplicaCircuitBreaker_Follower_QuorumLoss #76894 🟢 ~7k read/sec
• 5b42a98 Merge dev: don't exit immediately in ./dev ui watch #76742 *: enable declarative schema changer #76944
• 9dc9eb0 Merge txnidcache: disable cache by default #76973 🟢 ~7k read/sec
• 653e35c Merge pull request workflow: exempt some KV issues from being marked as stale #76784 from AlexTalks/x-nostale-kv
• 96fe1ea Merge roachtest: skip scbench #76970

Jira issue: CRDB-14119

[tbg]

The "winner" is #76902

 1800.0s        0       11939952         6633.3     16.0     11.0     48.2     71.3   2013.3  read
 1800.0s        0       11980208         6655.7     15.5     10.5     46.1     67.1   1677.7  read
 1800.0s        0       11874980         6597.2     15.9     11.0     48.2     71.3   1140.9  read
 1800.0s        0       10644279         5913.5     18.4     10.0     62.9    100.7   1744.8  read
 1800.0s        0       11759015         6532.8     15.8     11.0     46.1     71.3   1744.8  read

"Without regression" looks consistently like this:

 1800.0s        0       12552535         6973.6     15.0     10.5     44.0     65.0   1946.2  read
 1800.0s        0       12654748         7030.4     15.2     10.5     44.0     65.0   1409.3  read
 1800.0s        0       12558263         6976.8     15.0     10.0     44.0     65.0   2684.4  read
 1800.0s        0       12584422         6991.3     15.4     10.5     46.1     65.0   2147.5  read
 1800.0s        0       12380216         6877.9     15.1     10.0     44.0     67.1   1744.8  read

[blathers-crl]

Hi @tbg, please add branch-* labels to identify which branch(es) this release-blocker affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[sumeerbhola]

I'm trying the kv0/enc=false/nodes=3/size=4kb reproduction.
This is a strange test that does not overload cpu or cause Pebble write amplification to increase. If the bottleneck is the number of workers running the workload, I think it would be worth revising this test since having the throughput be sensitive to small changes in latency is not ideal. The KV transaction throughput slowly decreases during the test run, due to latency increase -- I don't know the root cause of the latency increase (this was also the behavior before this regression). There is ~1% spent in the truncator, mostly in loading the range applied state when there is a miss in the block cache (see screenshot below).

[sumeerbhola]

Based on extra logging, the loosely-coupled truncations are never being merged, isDeltaTrusted is always true, and ~11% of the calls to tryEnactTruncations fail because the raft applied index has not advanced sufficiently. So loosely-coupled truncation is behaving as expected.

[sumeerbhola]

kv95/enc=false/nodes=3/size=64kb is similar in terms of ~10% of tryEnactTruncations failing because of raft applied index being insufficiently advanced. This test does overload the store, so admission control is throttling via IO tokens -- but the truncator writes are not throttled. The cpu profile is similar, and the utilization is ~25%.
Given the low cpu utilization I'm going to ignore the 1% spent in the truncator.

In kv95, there is often ~3s intervals between groups of tryEnactTruncations, which suggests it to be the interval of durability-advanced callbacks (due to Pebble flushes). Perhaps the lag is causing the raft cache to have more misses. I turned off loosely-coupled truncation at 15:38 in the following, and I couldn't spot the difference (the second and third graphs are the entrycache.size and entrycache.bytes)

[tbg]

If you turn the loosely coupled truncations off, do you see the difference in qps? I'm still confused how this is even happening (as are you I assume)

[sumeerbhola]

I think we should try to find the root cause based on kv0/enc=false/nodes=3/size=4kb or any other roachtest which does not overload Pebble, that have regressed.
This is because there is a possibility that the lag in truncation, which causes the number of steady-state raft log entries to be higher, is making the read amp worse: this will be hard to pinpoint, and the probability is low (most of the bytes in the LSM should be in the state machine). And this will just make the investigation harder.

[sumeerbhola]

If you turn the loosely coupled truncations off, do you see the difference in qps? I'm still confused how this is even happening (as are you I assume)

Nothing that I can spot by eyeballing the graphs (following is the kv95 where loosely-coupled truncation was turned off at 15:38)