Improve stress test for transactions #9568

riversand963 · 2022-02-15T00:13:49Z

Test only, no change to functionality.
Extremely low risk of library regression.

Update test key generation by maintaining existing and non-existing keys.
Update db_crashtest.py to drive multiops_txn stress test for both write-committed and write-prepared.
Add a make target 'blackbox_crash_test_with_multiops_txn'.

Running the following commands caught the bug exposed in #9571.

$rm -rf /tmp/rocksdbtest/*
$./db_stress -progress_reports=0 -test_multi_ops_txns -use_txn -clear_column_family_one_in=0 \
    -column_families=1 -writepercent=0 -delpercent=0 -delrangepercent=0 -customopspercent=60 \
   -readpercent=20 -prefixpercent=0 -iterpercent=20 -reopen=0 -ops_per_thread=1000 -ub_a=10000 \
   -ub_c=100 -destroy_db_initially=0 -key_spaces_path=/dev/shm/key_spaces_desc -threads=32 -read_fault_one_in=0
$./db_stress -progress_reports=0 -test_multi_ops_txns -use_txn -clear_column_family_one_in=0
   -column_families=1 -writepercent=0 -delpercent=0 -delrangepercent=0 -customopspercent=60 -readpercent=20 \
   -prefixpercent=0 -iterpercent=20 -reopen=0 -ops_per_thread=1000 -ub_a=10000 -ub_c=100 -destroy_db_initially=0 \
   -key_spaces_path=/dev/shm/key_spaces_desc -threads=32 -read_fault_one_in=0

Running the following command caught a bug which will be fixed in #9648 .

$TEST_TMPDIR=/dev/shm make blackbox_crash_test_with_multiops_wc_txn

facebook-github-bot · 2022-02-17T18:30:54Z

@riversand963 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

riversand963 · 2022-02-17T18:33:04Z

cc @lth

facebook-github-bot · 2022-02-24T19:35:58Z

@riversand963 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-02-27T21:17:10Z

@riversand963 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-03-08T23:59:18Z

@riversand963 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-03-10T23:09:44Z

@riversand963 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-03-10T23:11:09Z

@riversand963 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

jay-zhuang

LGTM

jay-zhuang · 2022-03-16T03:19:26Z

db_stress_tool/multi_ops_txns_stress.cc

+// location, and this file will be used by future runs until a new db is
+// created.
+//
+// Create a fresh new dataabse (-destroy_db_initially=1 or there is no database


typo database

jay-zhuang · 2022-03-16T03:49:04Z

db_stress_tool/multi_ops_txns_stress.cc

+DEFINE_int32(lb_a, 0, "(Inclusive) lower bound of A");
+DEFINE_int32(ub_a, 1000, "(Exclusive) upper bound of A");
+DEFINE_int32(lb_c, 0, "(Inclusive) lower bound of C");
+DEFINE_int32(ub_c, 1000, "(Exclusive) upper bound of C");
+
+DEFINE_string(key_spaces_path, "",
+              "Path to file describing the lower and upper bounds of A and C");


What are A key and C key? are they myrocks keys from https://github.com/facebook/mysql-5.6/wiki/MyRocks-record-format
I think we should explain the key format here and how to choose it, why the key range is int.
And we might want to name the stress test blackbox_crash_test_myrocks_multiops_wc

The key format is described in multi_ops_txns_stress.h. I will put a comment here pointing to the header.
The stress test is not only for write-committed (wc)

jay-zhuang · 2022-03-16T03:53:39Z

db_stress_tool/multi_ops_txns_stress.cc

+      std::ostringstream oss;
+      oss << "pk should exist: " << Slice(pk).ToString(true);
+      fprintf(stderr, "%s\n", oss.str().c_str());


I'm wondering if we should use std::cerr instead (maybe a future refactor PR)

jay-zhuang · 2022-03-16T03:59:34Z

db_stress_tool/multi_ops_txns_stress.cc

+// Both [lb_a, ub_a) and [lb_c, ub_c) are partitioned. Each thread operates on
+// one sub range, using KeyGenerators to generate keys.


If each thread is only operating on its own partition, does that mean we're not testing concurrent operations on the same key? Is it to mimic myrocks operation which only has single thread for a key-range?

Even if multiple threads are operating on overlapping key ranges, at the db layer, there will never be concurrent accesses to the same key due to locking at Transaction layer. Having each thread operate on disjoint key ranges makes it easier for us to manage the existing and non-existing keys across different db restarts. This is necessary if we want the majority of read to actually find the keys instead of return not-found, in which case, we are not stressing the database as much.

jay-zhuang · 2022-03-16T04:11:06Z

db_stress_tool/multi_ops_txns_stress.cc

+// data specified via `-lb_a`, `-ub_a`, `-lb_c`, `-ub_c`, etc. Among these, we
+// define the test key spaces as two key ranges: [lb_a, ub_a) and [lb_c, ub_c).
+// The key spaces specification is persisted in a file whose absolute path can
+// be specified via `-key_spaces_path`.


Key spaces specification are just 4 integers, right? Why do we want to persist it in a file instead of providing it from cmd?

Right.
I feel that the key spaces should be persisted across database restarts if destroy_db_initially is false. This is different from the ranges of keys that actually exist in the database, thus we cannot rely on checking the db for this info.

facebook-github-bot · 2022-03-16T17:35:30Z

@riversand963 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

riversand963 · 2022-03-16T17:35:56Z

Thanks @jay-zhuang for the review!

facebook-github-bot · 2022-03-16T18:24:05Z

@riversand963 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-03-16T21:21:45Z

@riversand963 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

This reverts commit f28cd93d680466d4f4190a0d2c19d7c10548bca3.

Summary: Add some extra assertions assuming key spaces and test thread count remain the same across different runs. Will need to relax this in the future.

Summary: Will re-enable other options later

facebook-github-bot · 2022-03-16T23:47:39Z

@riversand963 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-03-16T23:57:53Z

@riversand963 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-03-17T00:00:42Z

@riversand963 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

riversand963 added the WIP Work in progress label Feb 15, 2022

facebook-github-bot added the CLA Signed label Feb 15, 2022

riversand963 force-pushed the config-txn-stress branch 2 times, most recently from f28cd93 to 333614c Compare February 16, 2022 00:23

riversand963 changed the title ~~[WIP] Improve stress test for transactions~~ Improve stress test for transactions Feb 16, 2022

riversand963 force-pushed the config-txn-stress branch 2 times, most recently from 936f45c to 4e73cb8 Compare February 17, 2022 18:14

riversand963 removed the WIP Work in progress label Feb 17, 2022

riversand963 force-pushed the config-txn-stress branch 2 times, most recently from c3d67cf to a60a865 Compare February 17, 2022 18:28

riversand963 marked this pull request as ready for review February 17, 2022 18:30

riversand963 force-pushed the config-txn-stress branch from a60a865 to e19c729 Compare February 24, 2022 19:35

riversand963 force-pushed the config-txn-stress branch from aca55bf to 9fe0efd Compare March 8, 2022 23:59

riversand963 force-pushed the config-txn-stress branch from 9fe0efd to 3c38047 Compare March 10, 2022 23:09

jay-zhuang approved these changes Mar 16, 2022

View reviewed changes

jay-zhuang reviewed Mar 16, 2022

View reviewed changes

riversand963 force-pushed the config-txn-stress branch from 3c38047 to 26f14e3 Compare March 16, 2022 17:20

riversand963 added 2 commits March 16, 2022 16:38

Configure snapshot cache bits and commit cache bits for wp txns

d14c589

Disable secondary

f2a2916

riversand963 added 22 commits March 16, 2022 16:40

Generate WAL during preload and avoid flush afterwards

87a5002

Add assertions for db mutex during recovery

739aa69

[TMP] debug

0694217

Revert "[TMP] debug"

421da19

This reverts commit f28cd93d680466d4f4190a0d2c19d7c10548bca3.

minor

61ae4d0

Combine the logic of key generation for A and C

44e1209

Fix ASAN

75a2701

Add some print info

9def66c

Update db_crashtest and Makefile

afc70e7

Add more debug output when assertion fails

203bf40

Temporarily disallow small commit cache

f065b9d

Add flag disable_auto_compactions

2e47f22

Increase write_buffer size to avoid assertion failure

c05d891

Update verification logic

ee0ade5

Update key generator logic

8647f6d

Summary: Add some extra assertions assuming key spaces and test thread count remain the same across different runs. Will need to relax this in the future.

Restrict tests to fewer options

65978f6

Summary: Will re-enable other options later

Assert false in a txn

fda2807

Reduce test space for wc txns

a1a71be

update db_crashtest.py

6c27f6b

Re-enable two write queus

23d9b86

Address review comments

ff41680

Add targets to crash_test.mk

5597354

riversand963 force-pushed the config-txn-stress branch from 26f14e3 to 5597354 Compare March 16, 2022 23:47

Minor

838faa3

facebook-github-bot closed this in 5894761 Mar 17, 2022

riversand963 deleted the config-txn-stress branch March 17, 2022 02:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve stress test for transactions #9568

Improve stress test for transactions #9568

riversand963 commented Feb 15, 2022 •

edited

Loading

facebook-github-bot commented Feb 17, 2022

riversand963 commented Feb 17, 2022

facebook-github-bot commented Feb 24, 2022

facebook-github-bot commented Feb 27, 2022

facebook-github-bot commented Mar 8, 2022

facebook-github-bot commented Mar 10, 2022

facebook-github-bot commented Mar 10, 2022

jay-zhuang left a comment

jay-zhuang Mar 16, 2022

jay-zhuang Mar 16, 2022

riversand963 Mar 16, 2022 •

edited

Loading

jay-zhuang Mar 16, 2022

jay-zhuang Mar 16, 2022

riversand963 Mar 16, 2022

jay-zhuang Mar 16, 2022

riversand963 Mar 16, 2022

facebook-github-bot commented Mar 16, 2022

riversand963 commented Mar 16, 2022

facebook-github-bot commented Mar 16, 2022

facebook-github-bot commented Mar 16, 2022

facebook-github-bot commented Mar 16, 2022

facebook-github-bot commented Mar 16, 2022

facebook-github-bot commented Mar 17, 2022

		// Both [lb_a, ub_a) and [lb_c, ub_c) are partitioned. Each thread operates on
		// one sub range, using KeyGenerators to generate keys.

Improve stress test for transactions #9568

Improve stress test for transactions #9568

Conversation

riversand963 commented Feb 15, 2022 • edited Loading

facebook-github-bot commented Feb 17, 2022

riversand963 commented Feb 17, 2022

facebook-github-bot commented Feb 24, 2022

facebook-github-bot commented Feb 27, 2022

facebook-github-bot commented Mar 8, 2022

facebook-github-bot commented Mar 10, 2022

facebook-github-bot commented Mar 10, 2022

jay-zhuang left a comment

Choose a reason for hiding this comment

jay-zhuang Mar 16, 2022

Choose a reason for hiding this comment

jay-zhuang Mar 16, 2022

Choose a reason for hiding this comment

riversand963 Mar 16, 2022 • edited Loading

Choose a reason for hiding this comment

jay-zhuang Mar 16, 2022

Choose a reason for hiding this comment

jay-zhuang Mar 16, 2022

Choose a reason for hiding this comment

riversand963 Mar 16, 2022

Choose a reason for hiding this comment

jay-zhuang Mar 16, 2022

Choose a reason for hiding this comment

riversand963 Mar 16, 2022

Choose a reason for hiding this comment

facebook-github-bot commented Mar 16, 2022

riversand963 commented Mar 16, 2022

facebook-github-bot commented Mar 16, 2022

facebook-github-bot commented Mar 16, 2022

facebook-github-bot commented Mar 16, 2022

facebook-github-bot commented Mar 16, 2022

facebook-github-bot commented Mar 17, 2022

riversand963 commented Feb 15, 2022 •

edited

Loading

riversand963 Mar 16, 2022 •

edited

Loading