tests: Support multiple keys in linearizability tests

[serathius]

cc @ahrtr @spzala @tjungblu

[codecov-commenter]

Codecov Report

Merging #14924 (d4c8611) into main (a4c1b3a) will decrease coverage by 0.37%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main   #14924      +/-   ##
==========================================
- Coverage   74.92%   74.55%   -0.38%     
==========================================
  Files         415      415              
  Lines       34276    34276              
==========================================
- Hits        25682    25554     -128     
- Misses       6970     7072     +102     
- Partials     1624     1650      +26     

Flag	Coverage Δ
all	74.55% <ø> (-0.38%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
server/etcdserver/api/etcdhttp/types/errors.go	60.00% <0.00%> (-13.34%)	⬇️
client/pkg/v3/testutil/leak.go	62.83% <0.00%> (-7.08%)	⬇️
client/v3/experimental/recipes/queue.go	58.62% <0.00%> (-6.90%)	⬇️
server/storage/mvcc/watchable_store.go	85.14% <0.00%> (-6.53%)	⬇️
server/lease/lease.go	94.87% <0.00%> (-5.13%)	⬇️
client/v3/leasing/cache.go	87.22% <0.00%> (-4.45%)	⬇️
client/v3/concurrency/session.go	85.10% <0.00%> (-4.26%)	⬇️
server/etcdserver/api/v3rpc/watch.go	84.12% <0.00%> (-4.13%)	⬇️
server/proxy/grpcproxy/watch.go	92.48% <0.00%> (-4.05%)	⬇️
server/storage/mvcc/watcher.go	96.29% <0.00%> (-3.71%)	⬇️
... and 20 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[serathius]

Resolved all comments that do not make sense. Please describe what you would like to change and not send 20 suggestions that will not work.

Suggestions make sense only if:

• They don't affect rest of code
• Change all code that is affected at once https://github.blog/changelog/2020-02-26-multi-line-code-suggestions-beta/

[tjungblu]

clever idea with the state reduction.

[serathius]

Yea, but looks like it's not very readable. Rewrote it to make the code cleaner.

[tjungblu]

lgtm (non-binding)

[ahrtr]

Overall looks good to me. The only concern is the documentation.

The model is the most important part of the linearizablity test, so please add detailed document/comment to describe the overall thought and make sure the document/comment is always up-to-date, especially you are changing the code too fast and frequently. I should raise this comment in your last PR in which you totally changed the implementation. Otherwise others may spend lots of time to understand your thoughts.

[ahrtr]

#14924 (comment)

[serathius]

Decided to separate PR that refactors the model first. Let's discuss documenting model there. #15045

[ahrtr]

Please add self-contained comment in model.go to describe the overall thought. And also add comment something like below for stepState,

`stepState` generates all the possible expected states and expected response based on all the current states. 
The caller can compare the expected response with the actual response, and take action accordingly.

[serathius]

@ahrtr Done, please take a look.