FB8-267: porting Add exponential backoff for smart restart #7

avbelov23 · 2021-11-10T12:47:34Z

/bin/sh for Ubuntu is dash. dash don't have ** and == operators.
Use POSIX equality operator =.
Use a more universal way to obtain the power of two.

… logic Summary: This diff ports D26559698 (facebook@bd3bd34) & D26217123 (facebook@5eba6ba) to 8.0 [ D26217123 (facebook@5eba6ba) ] It adds gradual throttling logic to the existing replication lag auto throttling system. Note that, It does not affect manual throttling system. Any manual throttling rule will throttle 100% of the requests as soon as it is added. Added a new server variable `write-throttle-rate-step` which determines the step by which throttle rate is stepped up or down. It must be a number between 0-100. It defaults to 100 which means throttle rate is stepped up from 0% to 100% or it is stepped down from 100% to 0% which is the default behavior now. Added a new column to `I_S.write_throttling_rules` to display `throttle_rate` In addition, I also fixed a bug with clearing `auto_throttling_rules`, `currently_throttled_entities` & `currently_monitored_entity` when throttling rules are cleared or auto throttling is turned off. Previously, we were not clearing these data structures properly which would have caused some inconsistencies in the algorithm. [ D26559698 (facebook@bd3bd34) ] It adds a server control to configure which type(dimensions - client/sql_id/user/shard) of entities are allowed to be throttled by the auto throttling system. This is a use case from TAO that they only would like to throttle one or more client_ids in case of replication lag. Similarly, for XDB we may want to throttle a sql_id and then a shard but never throttle a user id. This makes throttling logic more deterministic and easy to reason about. **This variable defines the dimensions that are okay to be throttled and the order in which auto throttling algorithm go through them to find a culprit.** ``` DEFAULT - Do not throttle any dimension SET @GLOBAL.WRITE_THROTTLE_PERMISSIBLE_DIMENSIONS_IN_ORDER = “OFF” For UDB(TAO) - Only throttle client_id SET @GLOBAL.WRITE_THROTTLE_PERMISSIBLE_DIMENSIONS_IN_ORDER = “CLIENT” For some other hostname scheme (XDB) - SET @GLOBAL.WRITE_THROTTLE_PERMISSIBLE_DIMENSIONS_IN_ORDER = “SQL_ID,SHARD” ``` Reviewed By: george-reynya Differential Revision: D27174135 fbshipit-source-id: 0306f0e6c68

Summary: In my testing this reduces the overhead of integer decoding by 2x. The idea of the fix is straight-forward: * Don't call Field* to get information, instead cache as much information as possible in our Rdb_field_packing, similar to InnoDB's row_prebuilt_t. This has a few advantages: * Avoid expensive virtual function calls * More cache friendly * Similarly, don't use field->move_field / move_field_offset to move field to the target buffer and then back - just write to the buffer using pre-calculated field offset directly * Use template integer unpacking function to enable compiler to unroll expensive loops into a constant number of mov instructions. NOTE: FDO data needs to be updated for this diff in order to take full advantage of this change. Reference Patch: facebook@9af9aa5 --- Porting Notes: There are some additional changes required to account for covering TEXT/BLOB changes, especially for blob I need to be able to call into ha_rocksdb->get_blob_buffer. I've added a new struct Rdb_unpack_func_context to be able to pass any additional required arguments without having to update all the unpack functions. For now I'm only passing in the table that we can use to get the ha_rocksdb handler. Reviewed By: Pushapgl Differential Revision: D25800694 fbshipit-source-id: d5f347d0a98

Summary: Similar to integer key decoding improvement, but for values: * Remove dependency to field* and cache those information in Rdb_field_encoder * No more Field::move_field - just write to buffer directly We were originally planning to use Rdb_protocol_value_decoder for MySQL bypass project to eliminate overhead of decoding to integer then convert to string, but in practice it probably makes little difference and it is not going to be needed when we go with bypass RPC (which is thrift binary protocol). So this changes removes Rdb_protocol_value_decoder completely. One thing that worth mentioning is I've unified pack_length and pack_length_in_rec because they are essentially identical in our scenarios and in the case they could be different (more packed BITS format that borrows bits from null byte when storage engine supports it) we don't really support it at all, so unifying it to avoid confusion and assert it. With this MyRocks is now either faster or on par with InnoDB when it comes to SELECT integer columns stored in value. NOTE:FDO data needs to be updated for this diff in order to take full advantage of this change. Reference Patch: facebook@f4cefba Reviewed By: Pushapgl Differential Revision: D25880543 fbshipit-source-id: 19faf536554

Summary: Today in `SELECT count(*)` MyRocks would still decode every single column due to this check, despite the readset being empty: ``` // bitmap is cleared on index merge, but it still needs to decode columns bool field_requested = decode_all_fields || m_verify_row_debug_checksums || bitmap_is_set(field_map, m_table->field[i]->field_index); ``` As a result MyRocks is significantly slower than InnoDB in this particular scenario. Turns out in index merge, when it tries to reset, it calls ha_index_init with an empty column_bitmap, so our field decoders didn't know it needs to decode anything, so the entire query would return nothing. This is discussed in [this commit](facebook@70f2bcd), and [issue 624](facebook#624) and [PR 626](facebook#626). So the workaround we had at that time is to simply treat empty map as implicitly everything, and the side effect is massively slowed down count(*). We have a few options to address this: 1. Fix index merge optimizer - looking at the code in QUICK_RANGE_SELECT::init_ror_merged_scan, it actually fixes up the column_bitmap properly, but after init/reset, so the fix would simply be moving the bitmap set code up. For secondary keys, prepare_for_position will automatically call `mark_columns_used_by_index_no_reset(s->primary_key, read_set)` if HA_PRIMARY_KEY_REQUIRED_FOR_POSITION is set (true for both InnoDB and MyRocks), so we would know correctly that we need to unpack PK when walking SK during index merge. 2. Overriding `column_bitmaps_signal` and setup decoders whenever the bitmap changes - however this doesn't work by itself. Because no storage engine today actually use handler::column_bitmaps_signal this path haven't been tested properly in index merge. In this case, QUICK_RANGE_SELECT::init_ror_merged_scan should call set_column_bitmaps_no_signal to avoid resetting the correct read/write set of head since head is used as first handler (reuses_handler=true) and subsequent place holders for read/write set updates (reuse_handler=false). 3. Follow InnoDB's solution - InnoDB delays it actually initialize its template again in index_read for the 2nd time (relying on `prebuilt->sql_stat_start`), and during index_read `QUICK_RANGE_SELECT::column_bitmap` is already fixed up and the table read/write set is switched to it, so the new template would be built correctly. In order to make it easier to maintain and port, after discussing with Manuel, I'm going with a simplified version of inikep#3 that delays decoder creation until the first read operation (index_*, rnd_*, range_read_*, multi_range_read_*), and setting the delay flag in index_init / rnd_init / multi_range_read_init. Also, I ran into a bug with truncation_partition where Rdb_converter's tbl_def is stale (we only update ha_rocksdb::m_tbl_def), but it is fine because it is not being used after table open. But my change moves the lookup_bitmap initialization into Rdb_converter which takes a dependency on Rdb_converter::m_tbl_def so now we need to reset it properly. Reference Patch: facebook@44d6a8d --------- Porting Note: Due to 8.0's new counting infra (handler::record & handler::record_with_index), this only helps PK counting. Will send out a better fix that works better with 8.0 new counting infra. Reviewed By: Pushapgl Differential Revision: D26265470 fbshipit-source-id: f142be681ab

Summary: 8.0 is taking a new approach with doing counts - MySQL will call handler::records / records_with_index which will then use rnd_init+rnd_next / index_init+index_next to iterate through everything. If you try count(*) you'll see MyRocks 8.0 is actually much slower when iterating the table with secondary keys than 5.6 (the optimizer picks secondary key for whatever reason). This is because MySQL no longer calls `handler::extra(KEY_READ)` so we no longer get the hints to only decode the key - instead we'll see that the SK is not covering so we fall back to PK, which dramatically slows down by *a lot*. This fixes it by adding a m_iteration_only = true hint - with this hint we'll just quickly iterate through everything without any decoding nor falling back to PK. Note even with this fix, if you compare MyRocks 8.0 with InnoDB 8.0, you'll see that InnoDB 8.0 is also much faster - this is because InnoDB overrides the implementation and iterate the records directly in parallel (default to 4). This is something we can do as well. Reviewed By: Pushapgl Differential Revision: D26272733 fbshipit-source-id: 53b1ae328f6

Summary: In 8.0 sizeof(my_core::handler) == 5680. This is a regression from 5.6 where the size was 912 bytes. This is because 8.0 introduced m_random_number_engine field of type std::mt19937 inline. The size of this type is 5000 bytes. This regressed workloads which started taking > 50 MB more as many handlers are initialized when running workloads. m_random_number_engine is only used when generating histogram samples as part of running "ANALYZE TABLE UPDATE HISTOGRAM". Since histograms are not yet widely used and the generator is needed only when updating histogram, convert it into a pointer field. The object of std::mt19937 is now dynamically created when running UPDATE HISTOGRAM, This should be ok as histograms will not be refreshed very often. Reviewed By: yizhang82 Differential Revision: D27251616 fbshipit-source-id: 1c122177a15

… enabled Summary: For secondaries, when enable_super_log_bin_read_only is on and read_only is on, currently it will forbid to install/uninstall plugin during run time. install/uninstall plugin doesn't generate event in binlog, although the thread thd contains OPTION_BIN_LOG flag due to log_slave_updates is on by default in secondaries. It should be safe to execute install/uninstall plugin. the change is to call set_skip_readonly_check() before install/uninstall plugin and call reset_skip_readonly_check()(for completeness) after install/uninstall plugin. BTW, mysql will always call reset_skip_readonly_check() for at the beginning of each statement. thus set_skip_readonly_check() won't affect other statement. ``` #0 THD::reset_skip_readonly_check (this=0x7fb5c1a0b000) at /home/luqun/mysql/mysql-8.0.20/sql/sql_class.h:1754 inikep#1 0x0000000005a500e1 in THD::reset_for_next_command (this=0x7fb5c1a0b000) at /home/luqun/mysql/mysql-8.0.20/sql/sql_parse.cc:5892 inikep#2 0x0000000005a517a5 in mysql_reset_thd_for_next_command (thd=0x7fb5c1a0b000) at /home/luqun/mysql/mysql-8.0.20/sql/sql_parse.cc:5817 inikep#3 0x0000000005a50466 in mysql_parse (thd=0x7fb5c1a0b000, parser_state=0x7fb5f6bb4560, last_timer=0x7fb5f6bb39b0) at /home/luqun/mysql/mysql-8.0.20/sql/sql_parse.cc:6056 inikep#4 0x0000000005a4c7c9 in dispatch_command (thd=0x7fb5c1a0b000, com_data=0x7fb5f6bb4d98, command=COM_QUERY) at /home/luqun/mysql/mysql-8.0.20/sql/sql_parse.cc:2222 inikep#5 0x0000000005a4f991 in do_command (thd=0x7fb5c1a0b000) at /home/luqun/mysql/mysql-8.0.20/sql/sql_parse.cc:1556 inikep#6 0x0000000005ccd4f1 in handle_connection (arg=0x7fb5cc85b740) at /home/luqun/mysql/mysql-8.0.20/sql/conn_handler/connection_handler_per_thread.cc:330 inikep#7 0x00000000078eb95b in pfs_spawn_thread (arg=0x7fb5f8c89720) at /home/luqun/mysql/mysql-8.0.20/storage/perfschema/pfs.cc:2884 inikep#8 0x00007fb5f957020c in start_thread (arg=0x7fb5f6bb6700) at pthread_create.c:479 inikep#9 0x00007fb5f971881f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 ``` Reviewed By: george-reynya Differential Revision: D27213990 fbshipit-source-id: 68a234fb694

Summary: Saw a bunch stage-1 replicaset instance failed due to config change string failure during add/remove instance ``` I0401 00:16:10.894434 1823842 IoCacheUtils.cpp:333] getConfigChangeString eventType = ^G E0401 00:16:10.894451 1823842 IoCacheUtils.cpp:363] Failed to read metadata event body from cache E0401 00:16:10.894456 1823842 MysqlRaft.cpp:659] [replicate] Failed to get config change string from iocache 2021-04-01T00:16:10.894464-07:00 8307 [ERROR] [MY-010207] [Repl] Run function 'before_flush' in plugin 'RPL_RAFT' failed 2021-04-01T00:16:10.894478-07:00 8307 [ERROR] [MY-000000] [Server] Failed to rotate binary log ``` After some investigation, the issue is caused is that calculate metadata event length with config change string but forgot write config change string into event body. Reviewed By: Pushapgl Differential Revision: D27504157 fbshipit-source-id: eddda0671e9

Summary: The change integrates rocksdb::FaultInjectionTestFS with MyRocks code. MyRocks exposes system variable rocksdb_fault_injection_options that takes in options in json format. This is first iteration which exposes write error injection. The injection framework will end up supporting more functionality in the future. ``` rocksdb_fault_injection_options={"retry":true,"onein":1000,"filetypes":["kTableFile","kDescriptorFile"]} ``` Reviewed By: yizhang82 Differential Revision: D27477107 fbshipit-source-id: eff000c3e93

Summary: Setting read only = 1 during shutdown after attempting to kill all connections but before shutting down plugins. This way we can be sure that no writes are being processed when semi-sync/raft plugins are shutdown. Reviewed By: hermanlee Differential Revision: D27539886 fbshipit-source-id: 047ec6a3bd5

Summary: RAFT Instance crash and Tx Logs filling up. rocksdb can fill up txlogs. we should stop restarting mysqld if we have restarted many times in a day Reviewed By: anirbanr-fb Differential Revision: D27372750 (facebook@0b3d1c3) fbshipit-source-id: adf5410c64c

Summary: In `inplace_populate_sk`, `ha_index_init` can return an error if the query has been killed, and if that happens, it's not safe to continue with calling `index_first`/`index_next`. The fix is to check for errors and return early if needed. Reviewed By: luqun Differential Revision: D26960027 (facebook@f12eea3) fbshipit-source-id: eafc511dce4

Summary: In 8.0 query from i_s.tables are *much* slower (~5x) most likely due to table open/close are much slower - profiling data showed that 74% time spent on opening tables and 19.71% time spent on reading histograms, only 2.41% spent on the actual stats inside MyRocks. 8.0 provides a new callback get_table_statistics and get_index_column_cardinality that when "overridden", signal MySQL to go call those functions to fill in stats directly, which should be much faster, with some caveats: * Our stats code path need to work without table open - this fix updates our stat code path to strictly use Rdb_tbl_def and Rdb_key_def. One catch is that we won't be able to update the auto_inc value correctly because we won't know if the table actually has AUTO_INCREMENT without opening it, so there are some corner cases the value might be NULL (such as immediately after creating a table with AUTO_INCREMENT). This actually aligns with ROCKSDB_DDL table behavior and InnoDB behavior as well. Once you open the table, the cached Rdb_tbl_def::m_auto_inc_val should be updated properly. * We also need to make sure the lifetime / locking works properly with regarding to DDL and special cases like drop cf. Fortunately the table_stats code path does take MDL_EXPLICIT lock with the db_name.table_name, so correctness should be straight-forward. One special case if DROP TABLE is running at the same time, the MDL_EXPLICIT lock may succeed *after* drop table succeed, and Rdbl_ddl_manager::find would return null, and we would give default values in the columns and with a error message saying table cannot be found. Note this version only overrides get_table_statistics (and it updates SQL layer to treat them separately) and we can add get_index_column_cardinality in the future if there is a need - right now there is little evidence that we are querying information_schema.statistics much (if not at all). Reviewed By: hermanlee Differential Revision: D26624028 fbshipit-source-id: 3b41f25d61c

Summary: Without allocating the server extension structure, 8.0.20 instances were crashing. This structure maintains compression context information. Moved `mysql_extension_set_server_extn(mysql, &mi->server_extn);` to method `configure_master_connection_options` which is common to both slave stats daemon and replication io thread. Now both threads will be setting the structure properly. Reviewed By: hermanlee Differential Revision: D27486594 fbshipit-source-id: 93fc460440c

Summary: in stage-1 replicaset, when kill a secondary instance, sometime the instance will run into deadlock due to process_raft_queue thread forgot to release its acquired mutex in raft_change_master Reviewed By: Pushapgl, bhatvinay Differential Revision: D27602667 fbshipit-source-id: 11eb07a7576

Summary: When updating index cardinality in InnoDB, the stats are first zero'd out via `dict_stats_empty_index` and then populated. Although there is locking on the write path, when updating index stats, it looks like we usually do not have locking on the read path in innodb. This can lead to bad plans, as we would return bad index stats. The fix is to take a shared lock on the read paths. https://bugs.mysql.com/bug.php?id=103210 Reviewed By: luqun, yizhang82 Differential Revision: D27545419 fbshipit-source-id: 92de08d9197

Summary: When decreasing temptable_max_ram system variable, I am not seeing decrease in memory when running workloads. create_tmp_table is still taking significant amount of memory in temptable APIs. This is because temptable_max_ram limit does not apply to shared block. The shared block is a TLS variable. In multithreaded environment, this takes a significant amount of memory (>150 MB). The shared block is unconditionally allocated from RAM. Also the shared_block is never deallocated once allocated in a thread. The fix introduces a system variable temptable_track_shared_block_ram. When enabled, the shared block memory allocation is also tracked. This tracking enables shared block to be allocated from mmap files in case the total allocated memory by temptable exceeds temptable_max_ram. Also with the fix, shared_block is deallocated after all the chunks are destroyed. This allows us to reclaim memory space more proactively. With this fix, I am seeing savings of >150 MB in our workloads. Reviewed By: yizhang82 Differential Revision: D27520132 fbshipit-source-id: 748b67e6fd2

Summary: Currently, when show slave HOSTS, The Is_semi_sync_slave field is always 0 even for semi-sync slaves. The issue is incorrect is_semi_sync_slave() is called --- there are two overload is_semi_sync_slave() whose purpose is to find whether thd variables contains "rpl_semi_sync_slave". - is_semi_sync_slave(): search "rpl_semi_sync_slave" in current_thd(current session thd) instead, should be only used by semi-sync clients - is_semi_sync_slave(THD, need_lock): search "rpl_semi_sync_slave" in specified THD. should be always used. For show slave hosts scenarios, It should call is_semi_sync_slave(THD, need_lock) and passing semi-sync client THD. Reviewed By: yizhang82 Differential Revision: D27641307 Squash with D19525150 fbshipit-source-id: 0cc0ba1d553

Summary: During promotion(binlog_change_to_binlog), the binary index file path is calculated incorrectly due to opt_binlog_index_name is a file name without path(such as binary-logs-{port}.index) but `mysql_bin_log.open_index_file()` requires either pass full binlog index name as 1st parameter or use 2nd parameter to calculate full binlog index file path. Use 2nd parameter(opt_bin_logname) to calculate full binlog index file path. Reviewed By: bhatvinay Differential Revision: D27703333 fbshipit-source-id: c4827dbc9bc

… MyRocks Manual Compaction Summary: Currently BottommostLevelCompaction is set to default kForceOptimized for manual compaction. This is fine usually. However, for privacy triggered compactions, it does not make much sense, because it compacts Lmax SST files that do not have any overlapping key updates at all. This adds a session variable to specify the option for bottommost level compaction before triggering manual compaction. Reviewed By: yoshinorim, yizhang82 Differential Revision: D27665464 fbshipit-source-id: 2bbdf04ec18

Summary: In 5.6 acl cache is protected by mutex which causes quite a bit of contention (especially if you have lots of acls), so we changed that to a mysql_rwlock_t in 5.6. In 8.0 this is replaced with a singleton MDL lock with namespace = MDL_key::ACL_CACHE with SHARED and EXCLUSIVE lock. However MDL lock has higher overhead than regular mysql_rwlock_t due to the way it maintains the tickets for granting/waiting and another internal mysql_prlock_rwlock (MDL_lock::m_rwlock) to update/traverse them, and therefore is more prone to contentions in high connection scenarios. In this particular case, the deadlock detection for ACL_CACHE leads to bad contention problems in the connection path, because taking rdlock on MDL_lock::m_rwlock while doing deadlock traversal takes longer with more granted and waiting tickets, but the releasing of these granted tickets are themselves blocked on taking wrlock on m_rwlock, meanwhile more connections are trying to come in and grant more tickets and taking rdlock as well in deadlock detection as well, so this becomes a serious lock contention that you can't get out of if you keep new connection come in at a fast pace. A easy way to trigger this is to run FLUSH PRIVILEGES on a server that has lots of incoming connections (with low connection timeout) with lots of connection from acl_check_host / has_global_grant. For ACL_CACHE, the deadlock detection in most cases doesn't provide a ton of value in production, because under ACL_cache lock we only operate on mysql.user/db/etc special system tables, so you'd need to take exclusive lock on them and then go through acl cache locking in order to trigger the deadlock detection, which seems quite unlikely in practice. We didn't see any issues in the past for 5.6 either, which was just a mutex and later we changed it to mysql_rwlock_t and had 0 deadlock detection. Therefore, I'm adding a new switch enable_acl_cache_deadlock_detection to turn it off for ACL_CACHE only. It defaults to on to maintain compat and we can turn it off in production. There are probably other options to fix this issue short term (such as replacing the ACL_cache lock with mysql_rwlock_t lock, moving allow_all_hosts check out of the lock, etc) with different risk trade offs, and longer term fixes that require bigger surgery to MDL locking system which probably should best left to upstream. This seems to be the lowest risk option at the moment. `#ifndef EXTRA_CODE_FOR_UNIT_TESTING` is needed to avoid compilation failure in locking unit tests (which doesn't seem to compile in the sys vars). Reviewed By: yoshinorim Differential Revision: D27740302 fbshipit-source-id: c55e54a466b

Summary: This is port of D26704727 to 8.0.20. MySQL plugin for privacy (D26704751) is build using the AUDIT plugin. The plugin requires access to the from tables of the query. This is extracted from the LEX tree produced by parser after the parse step is complete. This diff adds a function to sql/sql_thd_internal_api.h/.cc: * thd_get_query_tables() - Returns the list of tables extracted from the LEX tree produced by parser. Reviewed By: george-reynya Differential Revision: D27795351 fbshipit-source-id: df25157dd5f

Summary: 1. Change the code to admit new query through `thd_wait_begin/end` callbacks using new `THD_WAIT_ADMIT` wait type. 2. Move yield counting to the callbacks to unify the code between the server and the plugin. `yield_cond` exposes the yield condition to check once the yield counter says that yield is needed. 3. Expose all wait events through `admission_control_wait_events` var. This is especially needed for `USER_LOCK` which still controls innodb thread concurrency yield. 4. Copy and fix up admission control tests to the thread_pool suite. Only the var and table names are changed to refer to thread_pool plugin. The results have been diffed with original tests to make sure they are identical (except for var/table names). Reviewed By: lth Differential Revision: D27593084 fbshipit-source-id: 0a9fb067d02

Summary: - Fix plugin connection handler to stay in memory when reset. - Fix `MYSQL_CALLBACK` to read object and function only once to avoid race when object/function changes between first and second read. - Add `thread_pool_on` functional and stress tests. - Export APIs: - Get current THD mutex. - Set current PSI thread. - Set current PSI thread as THD socket owner. - Set THD priority. Reviewed By: lth Differential Revision: D28250576 fbshipit-source-id: 03fa021eb02

Summary: - Fix `mysql-test-run.pl` to avoid adding the same extra options when `--repeat=N` is used. This breaks `--thread-pool` runs because `--plugin_load_add=mysql_thread_pool.so` option cannot be repeated twice. MTR has a client/server architecture so the server at first generates test case objects and then starts sending them to client workers for execution. After execution the worker sends objects back to server with results. If the test needs to be repeated (in non-parallel case) the server sends this object received from worker back for another execution. So if worker adds options every time it executes the test, then the same options would accumulate in the test case object. In particular, `find_bootstrap_opts()` has a side effect of adding `opt_extra_mysqld_opt` to the passed in array (either `$tinfo->{master_opt}` or `$tinfo->{slave_opt}`). To avoid adding these options multiple times the code already does it only if `bootstrap_master/slave_opt` property of test case is not set. However it could end up being empty array so the check would not prevent repeated calls to `find_bootstrap_opts()`. The fix is to check if the property is defined which covers the case of both empty and non-empty arrays. - Improve error output of MTR to include network errors. - Fix result of `thread_pool.admission_control_hang` that somehow was missed in the initial diff. Reviewed By: lth Differential Revision: D28562017 fbshipit-source-id: 0d7f68451cf

Summary: Currently waits during commit are not instrumented with `thd_wait_begin/thd_wait_end`. As a result, admission control slot is not released during commit waits. Unittestdb/udb frequently gets dogpiles where some connections are in "waiting for commit" state while a lot more are in "waiting for admission" state. This change addresses this problem from multiple angles: - One existing possibility is to filter out COMMIT command with `admission_control_filter` setting. However, if COMMIT is a part of multiquery statement then after admission the sub-statements are not checked against the filter to exit AC. `admission_control_multiquery_filter` variable is added to enable this filtering logic. - In case of autocommit, the commit operation is invoked at the end of the statement so it cannot be filtered as a command. So `THD_WAIT_COMMIT` wait type is added, and the initial idea was to wrap the whole `MYSQL_BIN_LOG::ordered_commit` between `thd_wait_begin/thd_wait_end` calls with the helper class `Thd_wait_scope`. However this approach exposes two issues: 1. if `thd_wait_begin/thd_wait_end` calls are nested then the innermost `thd_wait_end` call will enter AC as opposed to the outermost. This is addressed by tracking the nesting level of these calls on `THD` and only honoring the outermost `thd_wait_end`. 2. `thd_wait_end` call may result in AC timeout. In that case `ordered_commit` would return success however later `thd->is_error()` would be checked and the timeout error would be returned to the client. As a result, the client would expect the command to fail while it would actually succeed. Today this issue in general is not handled well so a similar problem could already happen with any other statement other than COMMIT. The best option would be to wait for readmission with infinite timeout however this may be prone to deadlocks due to bugs in the code or disabled wait events. Another option is to kill the connection on timeout instead of returning an error which would be an acceptable "error" outcome after a successful command. The safer approach though is to limit the scope of the change to just the auto commit after the statement, exit AC before commit, and not re-enter afterwards (similar to how command filtering would work). For this `THD_WAIT_COMMIT` is treated as a wait type that doesn't need readmission after the wait. - `COMMIT` wait event is added. Reviewed By: lth Differential Revision: D28888732 fbshipit-source-id: 3bb2bcf4208

…_filter Summary: - port admission control tests to thread pool suite. - fix a small perf issue in the original port of 5.6 change to 8.0. If `admission_control_multiquery_filter` is enabled and it filtered out a statement, then make sure that `thd_wait_end()` does not try to re-enter AC. Right now it does, however `multi_tenancy_admit_query()` filters out the statement. So it works now but it would be better not to try to enter at all. Reviewed By: lth Differential Revision: D28934346 fbshipit-source-id: 110fea46336

Summary: - Copy a bunch of tests using `thread/sql/one_connection` into thread_pool suite. Fix tests/results to use `thread/thread_pool/worker` instead. - Fix a bug in connection MT: if `max_db_connections` is enabled on the server then the plugin could successfully change to new db while the server later could return MT error. The fix is to check if `max_db_connections` is enabled and switch the order of the callback: do the callback after the server check. - Introduce API to send connection timeout error. Reviewed By: lth Differential Revision: D28436960 fbshipit-source-id: 4b5cfca9322

Summary: - Add `_db_get/set_settings_()` APIs. This is needed to save/restore the dbug settings stack on the worker thread for a connection. The dbug infrastructure has global `init_settings` that every connection initially points to. A connection can also push new empty settings on the stack, add new keywords/etc to the new settings, and pop them later to restore the initial settings. The worker thread maintains the same code state structure for all connections but needs to save the dbug stack per connection and restore it before running any mysql code for that connection. At the end when connection is closed, the dbug stack is popped and freed until it reaches the initial settings. - Fix existing thread_cache test to actually test the thread cache (by validating thread os ids), and use the new thread pool vars. Also added a test case for the idle timeout. - Fix the rest of thread_pool copied tests to use the new `thread_pool/thread/connection` PSI name instead of `worker`. Reviewed By: lth Differential Revision: D29213841 fbshipit-source-id: 93862a7d2cd

Summary: This is mysql side of D28246877. Reviewed By: satya-valluri Differential Revision: D28249028 fbshipit-source-id: 1d37cd8a68d

Summary: asandebug complain there are memory leaks during MYSQL_BIN_LOG open Direct leak of 50 byte(s) in 1 object(s) allocated from: #0 0x67460ef in malloc #1 0x93f0777 in my_raw_malloc(unsigned long, int) #2 0x93f064a in my_malloc(unsigned int, unsigned long, int) #3 0x93f0eb0 in my_strdup(unsigned int, char const*, int) #4 0x8af01a6 in MYSQL_BIN_LOG::open(unsigned int, char const*, char const*, unsigned int) #5 0x8af8064 in MYSQL_BIN_LOG::open_binlog(char const*, char const*, unsigned long, bool, bool, bool, Format_description_log_event*, unsigned int, RaftRotateInfo*, bool) #6 0x8b00c00 in MYSQL_BIN_LOG::new_file_impl(bool, Format_description_log_event*, RaftRotateInfo*) #7 0x8d65e47 in rotate_relay_log(Master_info*, bool, bool, bool, RaftRotateInfo*) #8 0x8d661c0 in rotate_relay_log_for_raft(RaftRotateInfo*) #9 0x8c7696a in process_raft_queue #10 0xa0fa1fd in pfs_spawn_thread(void*) #11 0x7f8c9a12b20b in start_thread release these memory before assign them Reviewed By: Pushapgl Differential Revision: D28819752

Summary: [Porting Notes] We want to dump raft logs to vanilla async replicas regardless of whether it's the relay log or binlog. Effectively after this change we'll dump relay logs on the followers and binlogs on the leader. When the raft role changes, the logs to the dumped are also changed. Dump_log class is introduced as a thin wrapper/continer around mysql_bin_log or rli->relay_log and is inited with mysql_bin_log to emulate vanilla mysql behavior. Dump threads use the global dump_log object instead of mysql_bin_log directly. We switch the log in dump log only when raft role changes (in binlog_change_to_binlog() and binlog_change_to_apply_log()). During raft role change we take all log releated locks (LOCK_log, LOCK_index, LOCK_binlog_end_pos, and dump log lock) to serialize it with other log operations like dumping logs. Related doc - https://fb.quip.com/oTVAAdgEi4zY This diff contains below 7 patches: D23013977 D24766787 D24716539 D24900223 D24955284 D25174166 D25775525 Reviewed By: luqun Differential Revision: D26141496 ------------------------------------------------------------------------------- Passing raw_log pointer to wait_with_heartbeat() and wait_without_heartbeat() Summary: When enable_raft_plugin is OFF Dump_log::lock() is a no-op. Which means that when enable_raft_plugin is OFF there can be a race between log switching and dump threads. This could lead to a scenario where the raw_log that wait_next_event() is working on might be different than what wait_with_heartbeat()/wait_without_heartbeat() is working on. This can cause deadlocks because wait_with_heartbeat()/wait_without_heartbeat()'s mysql_cond_wait would unlock and then lock a different log's LOCK_binlog_end_pos mutex which would then never be unlocked by wait_next_event(). Reviewed By: anirbanr-fb Differential Revision: D32152658 fbshipit-source-id: d96ebcef966 ----------------------------------------------------------------------------------------- Fix rpl_raft_dump_raft_logs Summary: This tests completes but fails because the following warning exists: ``` 2022-08-30T16:28:00.159525Z 11 [ERROR] [MY-013114] [Repl] Slave I/O for channel '': Got fatal error 1236 from master when reading data from binary log: 'Slave has more GTIDs than the master has, using the master's SERVER_UUID. This may indicate that the end of the binary log was truncated or that the last binary log file was lost, e.g., after a power or disk failure when sync_binlog != 1. The master may or may not have rolled back transactions that were already replicated to the slave. Suggest to replicate any transactions that master has rolled back from slave to master, and/or commit empty transactions on master to account for transactions that have been', Error_code: MY-013114 ``` Since the MTR result file is valid, we can suppress this error. Reviewed By: yichenshen Differential Revision: D39141846 fbshipit-source-id: 8e7fdb8 ------------------------------------------------------------------------------- Fix heap overflow in group_relay_log_name handling Summary: We were accessing group_relay_log_name in Query_log_event::do_apply_event_worker() but it's assigned only after the coordinator thread encounters an end event (i.e. xid event or a query event with "COMMIT" or "ROLLBACK" query). This was causing a race between accessing group_relay_log_name in the worker thread and writing it on the coordinator thread. We don't need to set transaction position in events other than end event, so now we set transaction position in query event only if it's an end event. The race is eliminated because group_relay_log_name is set before enqueuing the event to the worker thread (in both dep repl and vanilla mts). Reviewed By: lth Differential Revision: D28767430 ------------------------------------------------------------------------------- fix memory during MYSQL_BIN_LOG::open_existing_binlog Summary: asandebug complain there are memory leaks during MYSQL_BIN_LOG open Direct leak of 50 byte(s) in 1 object(s) allocated from: #0 0x67460ef in malloc #1 0x93f0777 in my_raw_malloc(unsigned long, int) #2 0x93f064a in my_malloc(unsigned int, unsigned long, int) #3 0x93f0eb0 in my_strdup(unsigned int, char const*, int) #4 0x8af01a6 in MYSQL_BIN_LOG::open(unsigned int, char const*, char const*, unsigned int) #5 0x8af8064 in MYSQL_BIN_LOG::open_binlog(char const*, char const*, unsigned long, bool, bool, bool, Format_description_log_event*, unsigned int, RaftRotateInfo*, bool) #6 0x8b00c00 in MYSQL_BIN_LOG::new_file_impl(bool, Format_description_log_event*, RaftRotateInfo*) #7 0x8d65e47 in rotate_relay_log(Master_info*, bool, bool, bool, RaftRotateInfo*) #8 0x8d661c0 in rotate_relay_log_for_raft(RaftRotateInfo*) #9 0x8c7696a in process_raft_queue #10 0xa0fa1fd in pfs_spawn_thread(void*) #11 0x7f8c9a12b20b in start_thread release these memory before assign them Reviewed By: Pushapgl Differential Revision: D28819752

Summary: 1. Rebaseline SHOW INDEX format differences in index_key_block_size 2. Remove descending index from index_key_block_size as it is not relevant to the test and we don't support it - and MySQL 8.0 will properly fail the CREATE TABLE 3. Fix a bug in MyRocks error logging - there are significant changes in the error logging API and as a result sql_print_error no longer takes a level. Given that we can't pass down the va_list to another function, as a result we need to format the message ourselves and call a lower level API to print the actual error message. Without this we'll crash when we turn on logging 4. Remove a unreferenced signal restart_done_during_drop_cf signal as it is causing MySQL 8.0 shutdown to hang. Reviewed By: luqun Differential Revision: D17633174

Summary: [Porting Notes] We want to dump raft logs to vanilla async replicas regardless of whether it's the relay log or binlog. Effectively after this change we'll dump relay logs on the followers and binlogs on the leader. When the raft role changes, the logs to the dumped are also changed. Dump_log class is introduced as a thin wrapper/continer around mysql_bin_log or rli->relay_log and is inited with mysql_bin_log to emulate vanilla mysql behavior. Dump threads use the global dump_log object instead of mysql_bin_log directly. We switch the log in dump log only when raft role changes (in binlog_change_to_binlog() and binlog_change_to_apply_log()). During raft role change we take all log releated locks (LOCK_log, LOCK_index, LOCK_binlog_end_pos, and dump log lock) to serialize it with other log operations like dumping logs. Related doc - https://fb.quip.com/oTVAAdgEi4zY This diff contains below 7 patches: D23013977 D24766787 D24716539 D24900223 D24955284 D25174166 D25775525 Reviewed By: luqun Differential Revision: D26141496 ------------------------------------------------------------------------------- Passing raw_log pointer to wait_with_heartbeat() and wait_without_heartbeat() Summary: When enable_raft_plugin is OFF Dump_log::lock() is a no-op. Which means that when enable_raft_plugin is OFF there can be a race between log switching and dump threads. This could lead to a scenario where the raw_log that wait_next_event() is working on might be different than what wait_with_heartbeat()/wait_without_heartbeat() is working on. This can cause deadlocks because wait_with_heartbeat()/wait_without_heartbeat()'s mysql_cond_wait would unlock and then lock a different log's LOCK_binlog_end_pos mutex which would then never be unlocked by wait_next_event(). Reviewed By: anirbanr-fb Differential Revision: D32152658 fbshipit-source-id: d96ebcef966 ----------------------------------------------------------------------------------------- Fix rpl_raft_dump_raft_logs Summary: This tests completes but fails because the following warning exists: ``` 2022-08-30T16:28:00.159525Z 11 [ERROR] [MY-013114] [Repl] Slave I/O for channel '': Got fatal error 1236 from master when reading data from binary log: 'Slave has more GTIDs than the master has, using the master's SERVER_UUID. This may indicate that the end of the binary log was truncated or that the last binary log file was lost, e.g., after a power or disk failure when sync_binlog != 1. The master may or may not have rolled back transactions that were already replicated to the slave. Suggest to replicate any transactions that master has rolled back from slave to master, and/or commit empty transactions on master to account for transactions that have been', Error_code: MY-013114 ``` Since the MTR result file is valid, we can suppress this error. Reviewed By: yichenshen Differential Revision: D39141846 fbshipit-source-id: 8e7fdb8 ------------------------------------------------------------------------------- Fix heap overflow in group_relay_log_name handling Summary: We were accessing group_relay_log_name in Query_log_event::do_apply_event_worker() but it's assigned only after the coordinator thread encounters an end event (i.e. xid event or a query event with "COMMIT" or "ROLLBACK" query). This was causing a race between accessing group_relay_log_name in the worker thread and writing it on the coordinator thread. We don't need to set transaction position in events other than end event, so now we set transaction position in query event only if it's an end event. The race is eliminated because group_relay_log_name is set before enqueuing the event to the worker thread (in both dep repl and vanilla mts). Reviewed By: lth Differential Revision: D28767430 ------------------------------------------------------------------------------- fix memory during MYSQL_BIN_LOG::open_existing_binlog Summary: asandebug complain there are memory leaks during MYSQL_BIN_LOG open Direct leak of 50 byte(s) in 1 object(s) allocated from: #0 0x67460ef in malloc #1 0x93f0777 in my_raw_malloc(unsigned long, int) #2 0x93f064a in my_malloc(unsigned int, unsigned long, int) #3 0x93f0eb0 in my_strdup(unsigned int, char const*, int) #4 0x8af01a6 in MYSQL_BIN_LOG::open(unsigned int, char const*, char const*, unsigned int) #5 0x8af8064 in MYSQL_BIN_LOG::open_binlog(char const*, char const*, unsigned long, bool, bool, bool, Format_description_log_event*, unsigned int, RaftRotateInfo*, bool) #6 0x8b00c00 in MYSQL_BIN_LOG::new_file_impl(bool, Format_description_log_event*, RaftRotateInfo*) #7 0x8d65e47 in rotate_relay_log(Master_info*, bool, bool, bool, RaftRotateInfo*) #8 0x8d661c0 in rotate_relay_log_for_raft(RaftRotateInfo*) #9 0x8c7696a in process_raft_queue #10 0xa0fa1fd in pfs_spawn_thread(void*) #11 0x7f8c9a12b20b in start_thread release these memory before assign them Reviewed By: Pushapgl Differential Revision: D28819752

Summary: 1. Rebaseline SHOW INDEX format differences in index_key_block_size 2. Remove descending index from index_key_block_size as it is not relevant to the test and we don't support it - and MySQL 8.0 will properly fail the CREATE TABLE 3. Fix a bug in MyRocks error logging - there are significant changes in the error logging API and as a result sql_print_error no longer takes a level. Given that we can't pass down the va_list to another function, as a result we need to format the message ourselves and call a lower level API to print the actual error message. Without this we'll crash when we turn on logging 4. Remove a unreferenced signal restart_done_during_drop_cf signal as it is causing MySQL 8.0 shutdown to hang. Reviewed By: luqun Differential Revision: D17633174

Summary: [Porting Notes] We want to dump raft logs to vanilla async replicas regardless of whether it's the relay log or binlog. Effectively after this change we'll dump relay logs on the followers and binlogs on the leader. When the raft role changes, the logs to the dumped are also changed. Dump_log class is introduced as a thin wrapper/continer around mysql_bin_log or rli->relay_log and is inited with mysql_bin_log to emulate vanilla mysql behavior. Dump threads use the global dump_log object instead of mysql_bin_log directly. We switch the log in dump log only when raft role changes (in binlog_change_to_binlog() and binlog_change_to_apply_log()). During raft role change we take all log releated locks (LOCK_log, LOCK_index, LOCK_binlog_end_pos, and dump log lock) to serialize it with other log operations like dumping logs. Related doc - https://fb.quip.com/oTVAAdgEi4zY This diff contains below 7 patches: D23013977 D24766787 D24716539 D24900223 D24955284 D25174166 D25775525 Reviewed By: luqun Differential Revision: D26141496 ------------------------------------------------------------------------------- Passing raw_log pointer to wait_with_heartbeat() and wait_without_heartbeat() Summary: When enable_raft_plugin is OFF Dump_log::lock() is a no-op. Which means that when enable_raft_plugin is OFF there can be a race between log switching and dump threads. This could lead to a scenario where the raw_log that wait_next_event() is working on might be different than what wait_with_heartbeat()/wait_without_heartbeat() is working on. This can cause deadlocks because wait_with_heartbeat()/wait_without_heartbeat()'s mysql_cond_wait would unlock and then lock a different log's LOCK_binlog_end_pos mutex which would then never be unlocked by wait_next_event(). Reviewed By: anirbanr-fb Differential Revision: D32152658 fbshipit-source-id: d96ebcef966 ----------------------------------------------------------------------------------------- Fix rpl_raft_dump_raft_logs Summary: This tests completes but fails because the following warning exists: ``` 2022-08-30T16:28:00.159525Z 11 [ERROR] [MY-013114] [Repl] Slave I/O for channel '': Got fatal error 1236 from master when reading data from binary log: 'Slave has more GTIDs than the master has, using the master's SERVER_UUID. This may indicate that the end of the binary log was truncated or that the last binary log file was lost, e.g., after a power or disk failure when sync_binlog != 1. The master may or may not have rolled back transactions that were already replicated to the slave. Suggest to replicate any transactions that master has rolled back from slave to master, and/or commit empty transactions on master to account for transactions that have been', Error_code: MY-013114 ``` Since the MTR result file is valid, we can suppress this error. Reviewed By: yichenshen Differential Revision: D39141846 fbshipit-source-id: 8e7fdb8 ------------------------------------------------------------------------------- Fix heap overflow in group_relay_log_name handling Summary: We were accessing group_relay_log_name in Query_log_event::do_apply_event_worker() but it's assigned only after the coordinator thread encounters an end event (i.e. xid event or a query event with "COMMIT" or "ROLLBACK" query). This was causing a race between accessing group_relay_log_name in the worker thread and writing it on the coordinator thread. We don't need to set transaction position in events other than end event, so now we set transaction position in query event only if it's an end event. The race is eliminated because group_relay_log_name is set before enqueuing the event to the worker thread (in both dep repl and vanilla mts). Reviewed By: lth Differential Revision: D28767430 ------------------------------------------------------------------------------- fix memory during MYSQL_BIN_LOG::open_existing_binlog Summary: asandebug complain there are memory leaks during MYSQL_BIN_LOG open Direct leak of 50 byte(s) in 1 object(s) allocated from: #0 0x67460ef in malloc #1 0x93f0777 in my_raw_malloc(unsigned long, int) #2 0x93f064a in my_malloc(unsigned int, unsigned long, int) #3 0x93f0eb0 in my_strdup(unsigned int, char const*, int) #4 0x8af01a6 in MYSQL_BIN_LOG::open(unsigned int, char const*, char const*, unsigned int) #5 0x8af8064 in MYSQL_BIN_LOG::open_binlog(char const*, char const*, unsigned long, bool, bool, bool, Format_description_log_event*, unsigned int, RaftRotateInfo*, bool) #6 0x8b00c00 in MYSQL_BIN_LOG::new_file_impl(bool, Format_description_log_event*, RaftRotateInfo*) #7 0x8d65e47 in rotate_relay_log(Master_info*, bool, bool, bool, RaftRotateInfo*) #8 0x8d661c0 in rotate_relay_log_for_raft(RaftRotateInfo*) #9 0x8c7696a in process_raft_queue #10 0xa0fa1fd in pfs_spawn_thread(void*) #11 0x7f8c9a12b20b in start_thread release these memory before assign them Reviewed By: Pushapgl Differential Revision: D28819752

Summary: [Porting Notes] We want to dump raft logs to vanilla async replicas regardless of whether it's the relay log or binlog. Effectively after this change we'll dump relay logs on the followers and binlogs on the leader. When the raft role changes, the logs to the dumped are also changed. Dump_log class is introduced as a thin wrapper/continer around mysql_bin_log or rli->relay_log and is inited with mysql_bin_log to emulate vanilla mysql behavior. Dump threads use the global dump_log object instead of mysql_bin_log directly. We switch the log in dump log only when raft role changes (in binlog_change_to_binlog() and binlog_change_to_apply_log()). During raft role change we take all log releated locks (LOCK_log, LOCK_index, LOCK_binlog_end_pos, and dump log lock) to serialize it with other log operations like dumping logs. Related doc - https://fb.quip.com/oTVAAdgEi4zY This diff contains below 7 patches: D23013977 D24766787 D24716539 D24900223 D24955284 D25174166 D25775525 Reviewed By: luqun Differential Revision: D26141496 ------------------------------------------------------------------------------- Passing raw_log pointer to wait_with_heartbeat() and wait_without_heartbeat() Summary: When enable_raft_plugin is OFF Dump_log::lock() is a no-op. Which means that when enable_raft_plugin is OFF there can be a race between log switching and dump threads. This could lead to a scenario where the raw_log that wait_next_event() is working on might be different than what wait_with_heartbeat()/wait_without_heartbeat() is working on. This can cause deadlocks because wait_with_heartbeat()/wait_without_heartbeat()'s mysql_cond_wait would unlock and then lock a different log's LOCK_binlog_end_pos mutex which would then never be unlocked by wait_next_event(). Reviewed By: anirbanr-fb Differential Revision: D32152658 ----------------------------------------------------------------------------------------- Fix rpl_raft_dump_raft_logs Summary: This tests completes but fails because the following warning exists: ``` 2022-08-30T16:28:00.159525Z 11 [ERROR] [MY-013114] [Repl] Slave I/O for channel '': Got fatal error 1236 from master when reading data from binary log: 'Slave has more GTIDs than the master has, using the master's SERVER_UUID. This may indicate that the end of the binary log was truncated or that the last binary log file was lost, e.g., after a power or disk failure when sync_binlog != 1. The master may or may not have rolled back transactions that were already replicated to the slave. Suggest to replicate any transactions that master has rolled back from slave to master, and/or commit empty transactions on master to account for transactions that have been', Error_code: MY-013114 ``` Since the MTR result file is valid, we can suppress this error. Reviewed By: yichenshen Differential Revision: D39141846 ------------------------------------------------------------------------------- Fix heap overflow in group_relay_log_name handling Summary: We were accessing group_relay_log_name in Query_log_event::do_apply_event_worker() but it's assigned only after the coordinator thread encounters an end event (i.e. xid event or a query event with "COMMIT" or "ROLLBACK" query). This was causing a race between accessing group_relay_log_name in the worker thread and writing it on the coordinator thread. We don't need to set transaction position in events other than end event, so now we set transaction position in query event only if it's an end event. The race is eliminated because group_relay_log_name is set before enqueuing the event to the worker thread (in both dep repl and vanilla mts). Reviewed By: lth Differential Revision: D28767430 ------------------------------------------------------------------------------- fix memory during MYSQL_BIN_LOG::open_existing_binlog Summary: asandebug complain there are memory leaks during MYSQL_BIN_LOG open Direct leak of 50 byte(s) in 1 object(s) allocated from: #0 0x67460ef in malloc #1 0x93f0777 in my_raw_malloc(unsigned long, int) #2 0x93f064a in my_malloc(unsigned int, unsigned long, int) #3 0x93f0eb0 in my_strdup(unsigned int, char const*, int) #4 0x8af01a6 in MYSQL_BIN_LOG::open(unsigned int, char const*, char const*, unsigned int) #5 0x8af8064 in MYSQL_BIN_LOG::open_binlog(char const*, char const*, unsigned long, bool, bool, bool, Format_description_log_event*, unsigned int, RaftRotateInfo*, bool) #6 0x8b00c00 in MYSQL_BIN_LOG::new_file_impl(bool, Format_description_log_event*, RaftRotateInfo*) #7 0x8d65e47 in rotate_relay_log(Master_info*, bool, bool, bool, RaftRotateInfo*) #8 0x8d661c0 in rotate_relay_log_for_raft(RaftRotateInfo*) #9 0x8c7696a in process_raft_queue #10 0xa0fa1fd in pfs_spawn_thread(void*) #11 0x7f8c9a12b20b in start_thread release these memory before assign them Reviewed By: Pushapgl Differential Revision: D28819752

Rahul-Singhal and others added 30 commits November 9, 2021 11:54

Add required headers for thread pool plugin

d702e53

Summary: This is mysql side of D28246877. Reviewed By: satya-valluri Differential Revision: D28249028 fbshipit-source-id: 1d37cd8a68d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FB8-267: porting Add exponential backoff for smart restart #7

FB8-267: porting Add exponential backoff for smart restart #7

avbelov23 commented Nov 10, 2021

FB8-267: porting Add exponential backoff for smart restart #7

FB8-267: porting Add exponential backoff for smart restart #7

Conversation

avbelov23 commented Nov 10, 2021