FB8-278 - Suppress warning dir exists in test keyring_file_data_qa #11

ldonoso · 2022-03-22T10:31:14Z

The warning is legit as it the test tries to create the dir \ and the
creation fails as the dir already exists.

The same suppression exists in PS/8.0.

Summary: Add primary key information to table map log event. This is useful for incremental backup tool to fetch primary key of a table without querying the server. This is also useful for the backup tool in tracking primary key change due to schema changes. Preserves the order of primary key columns in table map log events. Bitmap used for storing primary key column index in table map log events doesn't preserve the order. While this is completely fine with mysql_backup tool which generates incremental logical dump files, this doesn't work when differentialbackup is used to merge dump files since the sorted order of rows in full logical dumps and in incremental logical dumps generated by mysql_backup are different. Reference Patch: facebook@e236984d7fdb Originally Reviewed By: jtolmer Added a new value `FACEBOOK` for global variable `binlog_row_metadata` to specify FB format metadata. (cherry picked from commit e236984) Reviewed By: lloyd Differential Revision: D9175632

Summary: Adding flag to collect aggregates for assoc/fbobj stats Reference patch: facebook@8200e5e Reference patch: facebook@c6a8bd8 Reference patch: facebook@520394a Differential Revision: D9173860

Summary: mysqlbinlog --find-gtid-position='gtid_set' outputs the latest binlog file name whose previous gtid set is a subset of the input gtid_set. mysqlbinlog errors out if no such binlog is found. Reference patch: facebook@cf3889bf Reviewed By: lloyd Differential Revision: D9306627

Summary: This diff adds a new command SHOW EXECUTED_GTID IN 'log_file_name' FROM 'log_file_pos' This outputs the gtid_executed executed by the server up until log_file_pos offset in binlog file log_file_name. This is used in nrx2.php to set gtid_purged on the newly created slave. Reference patch: facebook@abbe81f Reference patch: facebook@215b84bd8f6 Reviewed By: lloyd Differential Revision: D9368239

Summary: Current behavior of mysqlbinlog with --exclude-gtids requires a binlog file name when streaming from a mysql server. But using the gtid_executed set given in --exclude-gtids server can automatically find the required binlog file name, so no binlog file name to mysqlbinlog is required. Dump thread with auto_position sends a fake rotate event before sending real events to the slave thread. mysqlbinlog assumes this fake rotate event to be the end of file. This behavior is avoided by checking is_artificial_event(). Without this change, mysqlbinlog --exclude-gtid exits without printing any binlog events. This diff fixes an error message. Reference patch: facebook@aa87842 Reviewed By: lloyd Differential Revision: D9305978

Summary: Column names added to table map log events are used by slave to find out the corresponding field in slave's table whose index may be different than the index of the field in master's table due to a schema change. During a schema change on a replica set, slaves may have different schema than master which will break RBR. Porting notes: The functionality to enable column names on a per table basis using ALTER TABLE has not been ported. Instead we'll rely on enabling it globally using `log_column_names` variable. Originally Reviewed By: jtolmer Reference Patches: facebook@c09eb80 facebook@055ba08 facebook@181035a (cherry picked from commit fe89886) Differential Revision: D9438110

Summary: This diff adds an additional command line option --skip-empty-trans to be used in the case when both --database and --skip-gtids options are specified. Further logics for skipping the empty transactions of unrelated databases are added and enabled only when --database is selected and --skip-gtids are enabled. If either --database or --skip-gtids is not specified, mysqlbinlog will output an error message. The fix covers both code paths of the statement based replication as well as the row based replication. The fix essentially uses a database string to keep track of the operating database of the current transaction. Since XID event does not have the corresponding database information, the database tracking string is used for filtering out the COMMIT XID event in both statement and row based replication paths. The fix also ensures that the database will not change in the middle of a transaction and will issue errors if it does. It also ensures that the database string is always cleared after a transaction and reassigned the new value when a new transaction starts. Please note that the flag print_event_info->skipped_event_in_transaction (was set in the code path I touched in the diff) is to tell the next GTID event that there are skipped event previously so that the GTID event can automatically adds a COMMIT. Since --skip-empty-trans can only used with --skip-gtids, the GTID event will be skipped. In other words, the GTID code path checking print_event_info->skipped_event_in_transaction will not be touched in our case. So the flag of print_event_info->skipped_event_in_transaction will not affect the changes in this diff. Also note that the only other place that checks print_event_info->skipped_event_in_transaction is at the very end after printing all the events. print_event_info->skipped_event_in_transaction is checked to print an automatically generated COMMIT message if the previous transaction has skipped events but the transaction itself has not hit a COMMIT event (print_event_info->skipped_event_in_transaction is always set to be false at the COMMIT event). Since --skip-empty-trans only supports a single database within a transaction, the entire transaction is either skipped or not skipped. Therefore, we do not need an automatically generated COMMIT message at the end if a skipped transaction has been interrupted in the middle. In RBR, it's expected the slave's binlog does not carry a session database name for each BEGIN query event. However, this affects the functionality of --skip-empty-trans since this feature was designed to assume that each BEGIN query event should always carries a session database. This diff fixed this issue by relaxing the requirement so that the transaction's database name is set to be the next EVENT 's db name (that carries the database name) after the BEGIN event if the BEGIN doesn't carry the session database name. Also added some more checking and debugging info to be displayed Reference patch: facebook@2bc7ebc Reference Patch: facebook@3c7462a Reviewed By: lloyd Differential Revision: D9310534

Summary: This diff introduces per-user session variable default values. This is for MyRocks and MySQL production debugging. A new global variable 'per-user-session-var-default-val' with type of string is introduced, it defines the default values of multiple session variables for multiple users, e.g. --per-user-session-var-default-val="user1:user2:user3:tx_isolation=read-committed,gap_lock_raise_error=0,user4:user5:big_tables=1,auto_increment_increment=12345,long_query_time=123.45" In this example, the default values of session variables tx_isolation and gap_lock_raise_error for user1, user2, and user3 are defined, the default values of session variables big_tables, auto_increment_increment, and long_query_time for user4 and user 5 are defined. When these users connect to server, the corresponding session variables will be set with these values so that these users don't have to execute SET command to set these session variables before executing queries. The cost of an extra round trip is saved. (1) This global variable is dynamic so it can be set at runtime. (2) Sys-vars have many types. The per-user session variables supported types are enum/bool/int/uint/double, which are the most common types for session variables. Other types like string(charptr)/proxy_user/external_user/key_cache/plugin/struct etc are *not* supported. (3) The users won't be validated when per-user-session-var-default-val is set since users can be changed any time after the global is set (4) The session variables will be validated when per-user-session-var-default-val is set. Invalid command line value will cause server fail to start. Invalid value in SET command will fail the command. Reference patch: facebook@fda980c facebook@a704b91 facebook@337aa9dbe9c facebook@86ce96a192a Differential Revision: D9515197

Summary: This is useful for external applications reading mysql binary logs. Originally Reviewed By: jtolmer Reference Patch: facebook@de7ae77 (cherry picked from commit de7ae77) Differential Revision: D9471053

Summary: mysqlbinlog expects rows_query event only in RBR. It prints an unflushed event warning when it parses a transaction which has rows_query but not a rows events with STMT_END_F flag set. Since there is no direct way to tell the binlog format in mysqlbinlog, we'll just forgo the warning for rows_event which contain metadata. Reference Patch: facebook@89e1c03ac Reviewed By: lloyd Differential Revision: D10054923

Summary: mysqldump in 8.0 queries from information_schema.column_statistics without a server version check. This causes backups from 5.6 servers to fail. Fix by adding a version check. This fixes upstream bug https://bugs.mysql.com/bug.php?id=89825 Differential Revision: D10207120

Summary: Because we skipped 5.7, there was no longer any compatibility code that allows clients to call COM_SHUTDOWN for 5.6 servers. Add it back in until we have finished upgrading all servers to 8.0. Differential Revision: D12984296

Summary: mysqlbinlog verbose mode in case of multi-row BINLOG statement caches events for verbose output and prints it after BINLOG statement is finished. BINLOG statement must not be splitted into multiple statements because row events require Table_map_event Borrowed from MDEV-16372 Differential Revision: D13039422

Summary: With gtid enabled, flushing relay log info is not necessary. The force flushing is required to ensure recovery of MTS in non-gtid mode. The coordinator thread relies on positions in the relay log info repository during recovery. With GTID, this is not required and sql_thread doesn't even need to do any recovery. The slave (io_thread) uses auto-positioning directly. JIRA: https://jira.percona.com/browse/FB8-69 Reference Patch: facebook@ed39d49 Pull Request resolved: facebook#935 Reviewed By: lloyd Differential Revision: D13874140 Pulled By: lth

…ebook#938) Summary: JIRA: https://jira.percona.com/browse/FB8-100 Reference Patch: facebook@e29ff14 The binlog cache file code is designed to handle an error that would cause binlog corruption. It does this by checking the error for a set of values that it expects and when it finds one it marks the cache to indicate an incident has occurred and this mark keeps it from writing corrupt entries to the binlog. Unfortunately, during an out-of-space event, there is a lower level error that gets set which is not being overwritten when we set the expected error. This means that when the code checks for the expected error it is not found - the original lower level error is found. This diff modifies the checking code to scan through all associated conditions (which is a list of all encountered errors) to see if the expected error is present anywhere in the list. This diff can be reverted if/when https://bugs.mysql.com/bug.php?id=72457 is resolved upstream. Pull Request resolved: facebook#938 Reviewed By: lloyd Differential Revision: D13924297 Pulled By: lth

…o timeout Summary: Currently in mysqld when a connection is timed out (defined by wait_timeout) mysqld close the socket directly. Clients won't be able to find out the causes of the lost connection. This diff introduces a new feature to mysqld that when the new added global sys var "send_error_before_closing_timed_out_connection" is truned on server error 2006 with message "Connection closed due to timeout." will be pushed into socket before mysqld closing the connection due to timeout. This error will be delivered and stay in the socket buffer on client side after the connection is closed on server side. Client can read this error and find out the reason of the closed connection is timeout. This feature will help clients improving error handling, to distinguish if it was caused by timeout, or other reasons, e.g. crash, etc. The purpose of forcing the error code to be 2006 is for compatibility since (client) error code 2006 has been used for this error. A new global sys var "send_error_before_closing_timed_out_connection" is added in this diff, its default value is false so by default the timeout error message won't be sent, we will turn it on for testing and once all the clients can correctly consume this error message, we will set the default value to true. To test this feature, a new mysqltest command "dump_timed_out_connection_socket_buffer" is added. Use this command to dump the socket buffer on cilent side after a connection is timeout and verify if the error code / message is valid. Example is as below: set global.wait_timeout = 3; set global.interactive_timeout = 3; connect (con1,localhost,root,,); --echo sleeping 6 sec to make sure the connection get timed out and closed by server... sleep 6; --echo dump the client side socket buffer and verify error 2006 dump_timed_out_connection_socket_buffer; Reference patch: facebook@e04c9e1 facebook@12eeda9 facebook@27c0b0c Differential Revision: D10040191

…cebook#951) (facebook#951) Summary: Jira issue: https://jira.percona.com/browse/FB8-103 Reference patch: facebook@eb9a3db Reference patch: facebook@5803e4f Reference patch: facebook@746c217 Compared to the 5.6 patch: * fixed the rpl_heartbeat_timestamp testcase, which resulted in the test executor killing all servers because of a connection drop * split the rpl_heartbeat_zero_timestamp test into two Heartbeat events should carry the now() timestamp from the master and the last_master_timstamp from the slave. In other words, HB should always carry the lastet master timestamp. This will help services connected to slaves know if they are lagging even when no real events are flowing in the system. Also, for this to work last_master_timestamp should be updated whenever a HB event is received. Let master only send HB timestamp while waiting for new trx, this will fix the SBM being inaccurate when the slave is stopped and trying to catchup from the master. Pull Request resolved: facebook#951 Reviewed By: lloyd Differential Revision: D14034449 Pulled By: lth

… and then updating the index (facebook#943) (facebook#943) Summary: JIRA: https://jira.percona.com/browse/FB8-77 Reference Patch: facebook@832be30 Reference Patch: facebook@0d85250 Reference Patch: facebook@e8be5be Changed the behavior of purging logs. Now deleting logs first and then updating the index. Previously when purging logs, the index was updated before deleting the logs. This creates problems the disk is full. We should first delete logs to free up disk and then update index. When index file is updated, it is first copied to a crash safe index file, updates are made to the crash safe file and finally the crash safe file is renamed. During purge, all this should happen only when at least one log is being removed. Pull Request resolved: facebook#943 Reviewed By: lloyd Differential Revision: D13930811 Pulled By: lth

…) (facebook#939) Summary: JIRA: https://jira.percona.com/browse/FB8-86 Reference Patch: facebook@78fa270 Reference Patch: facebook@72c25d615d8 Reference Patch: facebook@78b6b8bc826 Reference Patch: facebook@00be4805f47 Crash safe index file is created when any change is made to the index file. The index file is first copied over to the crash safe one and then stuff is added/removed from the crash safe index file. Finally, the old index file is deleted and crash safe index file is renamed to the index file. There might be a scenario where mysql crashes after stuff is changed in the crash safe file but before the old index file is deleted. In this case we should delete the crash safe index file at restart. If we don't delete it subsequent changes to the index file can corrupt it because it can be copied to the old crash safe file which already has some data in it. Pull Request resolved: facebook#939 Reviewed By: lloyd Differential Revision: D13924478 Pulled By: lth

…) (facebook#941) Summary: JIRA: https://jira.percona.com/browse/FB8-56 Reference Patch: facebook@9d97282 Create a new mode SEMI_STRICT which ignores HA_ERR_RECORD_CHANGED and HA_ERR_KEY_NOT_FOUND errors. A new status counter will track the usage of this new mode. Logging a warning/error message may cause error log to get flooded with same error message, so I decided against it. Pull Request resolved: facebook#941 Reviewed By: lloyd Differential Revision: D13924607 Pulled By: lth

… plugin log (facebook#948) (facebook#948) Summary: Jira issue: https://jira.percona.com/browse/FB8-96 Reference patch: facebook@3b47dae Reference patch: facebook@87e0683 Pull Request resolved: facebook#948 Reviewed By: lloyd Differential Revision: D13941932 Pulled By: lth

Summary: JIRA: https://jira.percona.com/browse/FB8-52 Reference Patch: facebook@80ca845 Reference Patch: facebook@0fb428b Patch introduces new variable 'innodb_txlog_init_rate' to limit the IO rate of InnoDB redo log file initialization. Pull Request resolved: facebook#931 Reviewed By: lloyd Differential Revision: D13789095 Pulled By: lth

…gin (facebook#934) (facebook#934) Summary: JIRA: https://jira.percona.com/browse/FB8-54 JIRA: https://jira.percona.com/browse/FB8-55 JIRA: https://jira.percona.com/browse/FB8-70 JIRA: https://jira.percona.com/browse/FB8-101 This commit adds the following fields to the generic event in audit log: * query_id * database * affected_rows * connection_certificate Reference Patch: facebook@1def6b7 Reference Patch: facebook@ce95a09 Reference Patch: facebook@588be34 Reference Patch: facebook@ba03c70 Reference Patch: facebook@be8c587 Reference Patch: facebook@22b2508 We need some extra info for the shadowing and security logging. This is a simple first step of info that MariaDB actually also exposes. Now we would have the `query_id` and the database name for general events. Making as few changes as possible to accomplish it, so I'm just taking the information from the TDH and exposing it through `mysql_event_general` struct and as a argument to disconnect. Forward the connection certificate to the audit plugin. The connection certificate can then be parsed by the audit plugin and handled appropriately. It made more sense for the certificate to live in the connection events, since they generally don't change between every general event, so the move was done. This is done by caching a BUF_MEM struct on the THD object. Since it's not possible to change certificates on the same connection, this caching should be correct. The BUF_MEM is released on THD::release_resources. If upstream bumps the MYSQL_AUDIT_INTERFACE_VERSION, we should bump ours to be greater or equal to it. Expose the port current mysqld is running on for the audit plugin. If no port, 0 is used. Pull Request resolved: facebook#934 Reviewed By: lloyd Differential Revision: D13874133 Pulled By: lth

…acebook#940) Summary: JIRA: https://jira.percona.com/browse/FB8-93 Reference Patch: facebook@a2b0c32 Reference Patch: facebook@e82ab5a Instead of just showing the errno when running SHOW SLAVE STATUS command, also printing the error name so that user won't have misunderstanding Pull Request resolved: facebook#940 Reviewed By: lloyd Differential Revision: D13924591 Pulled By: lth

…#937) Summary: https://jira.percona.com/browse/FB8-47 The max user name length is increased to 80 chars (utf8mb3) Since the system tables are InnoDB tables and because Primary key is on the full username (80 *3 = 240 bytes) and other columns, InnoDb initialization with 4k page size fails. This is new limitation. Reference Patch: facebook@3dbbcfc This mitigates upstream bug: http://bugs.mysql.com/3083 User name length limit of 16 was too short, so made it 80. 80 is effectively the limit without making additional changes because of MyISAM's 1000 byte limit on key length and the primary key of mysql.columns_priv. To compute the size of the key for that table, the lengths of the char fields are summed and then multipled by 3, resulting in a limit on the max size of user name lengths being 81 characters. Pull Request resolved: facebook#937 Reviewed By: lloyd Differential Revision: D13890369 Pulled By: lth

Summary: Implements per-database scope read-only setting. The read-only can be turned on in `alter database` statement. Privilege to set read-only follows alter-database ACL. ALTER DATABASE db_name [SUPER_READ_ONLY | READ_ONLY = FALSE | TRUE]] `SUPER_READ_ONLY = TRUE` will prevent all write transactions from committing for any user (including super users). `READ_ONLY = TRUE` will prevent any write transaction from committing for regular users, while super users can still write to the database. `SUPER_READ_ONLY = FALSE` will turn off super_read_only on the database, and the database remains on read_only (for regular users), `READ_ONLY = FALSE` will turn off read_only on the database, The READ_ONLY database status can be shown by `show create database`. mysql> alter database test read_only = true; Query OK, 1 row affected (0.00 sec) mysql> show create database test; +----------+---------------------------------------------------------------------------+ | Database | Create Database | +----------+---------------------------------------------------------------------------+ | test | CREATE DATABASE `test` /*!40100 DEFAULT CHARACTER SET latin1 READ_ONLY */ | +----------+---------------------------------------------------------------------------+ 1 row in set (0.00 sec) mysql> alter database test super_read_only = true; Query OK, 1 row affected (0.00 sec) mysql> show create database test; +----------+---------------------------------------------------------------------------------+ | Database | Create Database | +----------+---------------------------------------------------------------------------------+ | test | CREATE DATABASE `test` /*!40100 DEFAULT CHARACTER SET latin1 SUPER_READ_ONLY */ | +----------+---------------------------------------------------------------------------------+ 1 row in set (0.00 sec) mysql> alter database test super_read_only = false; Query OK, 1 row affected (0.00 sec) mysql> show create database test; +----------+---------------------------------------------------------------------------+ | Database | Create Database | +----------+---------------------------------------------------------------------------+ | test | CREATE DATABASE `test` /*!40100 DEFAULT CHARACTER SET latin1 READ_ONLY */ | +----------+---------------------------------------------------------------------------+ 1 row in set (0.00 sec) Details: * READ_ONLY flag is persisted in db.ops and survives restarts. * Alter database will only return when no write transaction is still committing. Uncommitted write transactions will fail after alter database succeeds. * DB options are stored in each thread (using hash map), so checking the read_only options doesn't grab a global lock. Only db_read_only that is turned on is stored in the local hash map. * The first time thead local hash map is initialized (once), shared db option map will be locked and accessed. Since the db options are not automatically loaded into the shared hash map (cache) at the beginning, we have to explicitly load the db options during the first time any thread initializes its local hash map. * create/alter/delete database DDLs will iterate through thread array to update local db opt (if needed), similar to how show processlist works. * write transactions are checked against local hashmap at commit time (either explicit commit or auto-commit). This will make the explicit commit and auto-commits consistent, and avoid problems such that explicit commit may become a "read-only" transaction if we block write-statements at the parsing time. * The list of databases accessed in a write transaction is obtained through the metadata locks (MDL) that the transaction holds. As I noted in the code, this may rollback some false positives, such as the read-only DB in the list may not actually be modified in the transaction (for cross-db transactions). We do not differentiate such cases. Reference patch: facebook@8194409 facebook@c971129 facebook@e26e9b2 facebook@a9f6f3d facebook@b1ff9a6 facebook@548c7ca Differential Revision: D9379904

…ok#972) (facebook#972) Summary: JIRA ticket: https://jira.percona.com/browse/FB8-120 Reference commit: facebook@2f08e3a Reference commit: facebook@a21e84f Reference commit: facebook@5983ba3181a Reference commit: facebook@414870abeee Added new option db_metadata to the create database `CREATE DATABASE [IF NOT EXISTS] db_name DB_METADATA [=] metadata` The new option is stored in the db.opt file along with other database options like charset and read_only. The options accepts any string value. If not specified, it writes empty string as the default value in the db.opt file. To retrieve the option value for a particular database, the show command can be used as follows: `SHOW CREATE DATABASE db_name` Example Usage: create database test2 charset utf8 read_only = true db_metadata '{\'shard_name\':\'myshard_for_test2\'}'; show create database test2; Output: +----------+--------------------------------------------------------------------------------------------------------------------------+ | Database | Create Database | +----------+--------------------------------------------------------------------------------------------------------------------------+ | test2 | CREATE DATABASE `test2` /*!40100 DEFAULT CHARACTER SET utf8 READ_ONLY DB_METADATA {'shard_name':'myshard_for_test2'} */ | +----------+--------------------------------------------------------------------------------------------------------------------------+ Pull Request resolved: facebook#972 Differential Revision: D14883632 Pulled By: yizhang82

Summary: Add DB_METADATA to INFORMATION_SCHEMA.SCHEMATA for each database for easy query by OLM --- 8.0 porting comments 'SCHEMATA_EXT' Information Schema table is implemented as Data Dictionary system view. Reference Patch: facebook@64bf72c276 Differential Revision: D13510326

…ook#953) (facebook#953) Summary: Jira ticket: https://jira.percona.com/browse/FB8-80 Reference Patch: facebook@564824a Set row_query using the fields unpacked from the row event. Use row_query in show processlist. Pull Request resolved: facebook#953 Reviewed By: lloyd Differential Revision: D14137212 Pulled By: lth

…acebook#965) (facebook#965) Summary: Jira ticket: https://jira.percona.com/browse/FB8-116 Jira ticket: https://jira.percona.com/browse/FB8-152 Jira ticket: https://jira.percona.com/browse/FB8-83 Jira ticket: https://jira.percona.com/browse/FB8-125 Reference commit: facebook@316e0e5 Reference commit: facebook@a85b8d3 Reference commit: facebook@21edb26 Reference commit: facebook@96de6c4 Reference commit: facebook@e1cb53ba0f9 Reference commit: facebook@bd25e29 Reference commit: facebook@e392a18d40e Reference commit: facebook@fa4f588a248 Reference commit: facebook@68f8c9a58fa Reference commit: facebook@4519ab421f8 Reference commit: facebook@42bdfc8 Reference commit: facebook@3950491 Reference commit: facebook@bc17d30 Reference commit: facebook@eb7b1df Reference commit: facebook@6a27bd42dfe Reference commit: facebook@9503743bf59 Reference commit: facebook@7b270a0ad3b Reference commit: facebook@e6f031af712 Reference commit: facebook@7a0da2c9e94 Reference commit: facebook@bb9f4aae4fd Porting notes: The deadlock/ASAN bugfixes were skipped - I wasn't able to reproduce the issue mentioned in the commit messages even without the patches, and the pathes themselves caused early guaranteed crashes in the server. -------- 316e0e5 -------- If enabled, For every trx before the commit event is written we write a rows query event which contains meta data in JSON format wrapped inside a comment. Storing timestamps in the meta data to calculate slave lag in millisecond granularity. -------- a85b8d3 -------- Supporting binlog trx meta data for SBR. The meta data event is written right before every query event in a trx representing a statement. -------- 21edb26 -------- Milli-seconds behind master was not calculated correctly for MTS. With this change we calculate it just like seconds behind master. One difference between these is that sec timestamp is available for every event but msec timestamp is only available once per query/rows event in form of trx metadata inside a Rows_query_event. -------- bd25e29 -------- A separate event for trx metadata was causing an unreasonable increase in binlog size. Also, now we're not writing rows query event if there are no comments in the query and `log_only_query_comments` is enabled. -------- eb7b1df -------- Log meta data event before logging the rows query event for a statement. Pull Request resolved: facebook#965 -------- b8a600a -------- When reset_seconds_behind_master is disabled, in the case where last_master_timestamp is still 0, check to determine if the I/O thread is running before returning 0 or NULL. In the case where replication is broken, the sbm value should be ever increasing, or set to NULL, and not set to 0. Differential Revision: D14652380 Pulled By: yizhang82

…mn families

Summary: [Porting Notes] We want to dump raft logs to vanilla async replicas regardless of whether it's the relay log or binlog. Effectively after this change we'll dump relay logs on the followers and binlogs on the leader. When the raft role changes, the logs to the dumped are also changed. Dump_log class is introduced as a thin wrapper/continer around mysql_bin_log or rli->relay_log and is inited with mysql_bin_log to emulate vanilla mysql behavior. Dump threads use the global dump_log object instead of mysql_bin_log directly. We switch the log in dump log only when raft role changes (in binlog_change_to_binlog() and binlog_change_to_apply_log()). During raft role change we take all log releated locks (LOCK_log, LOCK_index, LOCK_binlog_end_pos, and dump log lock) to serialize it with other log operations like dumping logs. Related doc - https://fb.quip.com/oTVAAdgEi4zY This diff contains below 7 patches: D23013977 D24766787 D24716539 D24900223 D24955284 D25174166 D25775525 Reviewed By: luqun Differential Revision: D26141496 ------------------------------------------------------------------------------- Passing raw_log pointer to wait_with_heartbeat() and wait_without_heartbeat() Summary: When enable_raft_plugin is OFF Dump_log::lock() is a no-op. Which means that when enable_raft_plugin is OFF there can be a race between log switching and dump threads. This could lead to a scenario where the raw_log that wait_next_event() is working on might be different than what wait_with_heartbeat()/wait_without_heartbeat() is working on. This can cause deadlocks because wait_with_heartbeat()/wait_without_heartbeat()'s mysql_cond_wait would unlock and then lock a different log's LOCK_binlog_end_pos mutex which would then never be unlocked by wait_next_event(). Reviewed By: anirbanr-fb Differential Revision: D32152658 fbshipit-source-id: d96ebcef966 ----------------------------------------------------------------------------------------- Fix rpl_raft_dump_raft_logs Summary: This tests completes but fails because the following warning exists: ``` 2022-08-30T16:28:00.159525Z 11 [ERROR] [MY-013114] [Repl] Slave I/O for channel '': Got fatal error 1236 from master when reading data from binary log: 'Slave has more GTIDs than the master has, using the master's SERVER_UUID. This may indicate that the end of the binary log was truncated or that the last binary log file was lost, e.g., after a power or disk failure when sync_binlog != 1. The master may or may not have rolled back transactions that were already replicated to the slave. Suggest to replicate any transactions that master has rolled back from slave to master, and/or commit empty transactions on master to account for transactions that have been', Error_code: MY-013114 ``` Since the MTR result file is valid, we can suppress this error. Reviewed By: yichenshen Differential Revision: D39141846 fbshipit-source-id: 8e7fdb8 ------------------------------------------------------------------------------- Fix heap overflow in group_relay_log_name handling Summary: We were accessing group_relay_log_name in Query_log_event::do_apply_event_worker() but it's assigned only after the coordinator thread encounters an end event (i.e. xid event or a query event with "COMMIT" or "ROLLBACK" query). This was causing a race between accessing group_relay_log_name in the worker thread and writing it on the coordinator thread. We don't need to set transaction position in events other than end event, so now we set transaction position in query event only if it's an end event. The race is eliminated because group_relay_log_name is set before enqueuing the event to the worker thread (in both dep repl and vanilla mts). Reviewed By: lth Differential Revision: D28767430 ------------------------------------------------------------------------------- fix memory during MYSQL_BIN_LOG::open_existing_binlog Summary: asandebug complain there are memory leaks during MYSQL_BIN_LOG open Direct leak of 50 byte(s) in 1 object(s) allocated from: #0 0x67460ef in malloc #1 0x93f0777 in my_raw_malloc(unsigned long, int) #2 0x93f064a in my_malloc(unsigned int, unsigned long, int) #3 0x93f0eb0 in my_strdup(unsigned int, char const*, int) #4 0x8af01a6 in MYSQL_BIN_LOG::open(unsigned int, char const*, char const*, unsigned int) #5 0x8af8064 in MYSQL_BIN_LOG::open_binlog(char const*, char const*, unsigned long, bool, bool, bool, Format_description_log_event*, unsigned int, RaftRotateInfo*, bool) #6 0x8b00c00 in MYSQL_BIN_LOG::new_file_impl(bool, Format_description_log_event*, RaftRotateInfo*) #7 0x8d65e47 in rotate_relay_log(Master_info*, bool, bool, bool, RaftRotateInfo*) #8 0x8d661c0 in rotate_relay_log_for_raft(RaftRotateInfo*) #9 0x8c7696a in process_raft_queue #10 0xa0fa1fd in pfs_spawn_thread(void*) #11 0x7f8c9a12b20b in start_thread release these memory before assign them Reviewed By: Pushapgl Differential Revision: D28819752

…mn families

Summary: [Porting Notes] We want to dump raft logs to vanilla async replicas regardless of whether it's the relay log or binlog. Effectively after this change we'll dump relay logs on the followers and binlogs on the leader. When the raft role changes, the logs to the dumped are also changed. Dump_log class is introduced as a thin wrapper/continer around mysql_bin_log or rli->relay_log and is inited with mysql_bin_log to emulate vanilla mysql behavior. Dump threads use the global dump_log object instead of mysql_bin_log directly. We switch the log in dump log only when raft role changes (in binlog_change_to_binlog() and binlog_change_to_apply_log()). During raft role change we take all log releated locks (LOCK_log, LOCK_index, LOCK_binlog_end_pos, and dump log lock) to serialize it with other log operations like dumping logs. Related doc - https://fb.quip.com/oTVAAdgEi4zY This diff contains below 7 patches: D23013977 D24766787 D24716539 D24900223 D24955284 D25174166 D25775525 Reviewed By: luqun Differential Revision: D26141496 ------------------------------------------------------------------------------- Passing raw_log pointer to wait_with_heartbeat() and wait_without_heartbeat() Summary: When enable_raft_plugin is OFF Dump_log::lock() is a no-op. Which means that when enable_raft_plugin is OFF there can be a race between log switching and dump threads. This could lead to a scenario where the raw_log that wait_next_event() is working on might be different than what wait_with_heartbeat()/wait_without_heartbeat() is working on. This can cause deadlocks because wait_with_heartbeat()/wait_without_heartbeat()'s mysql_cond_wait would unlock and then lock a different log's LOCK_binlog_end_pos mutex which would then never be unlocked by wait_next_event(). Reviewed By: anirbanr-fb Differential Revision: D32152658 fbshipit-source-id: d96ebcef966 ----------------------------------------------------------------------------------------- Fix rpl_raft_dump_raft_logs Summary: This tests completes but fails because the following warning exists: ``` 2022-08-30T16:28:00.159525Z 11 [ERROR] [MY-013114] [Repl] Slave I/O for channel '': Got fatal error 1236 from master when reading data from binary log: 'Slave has more GTIDs than the master has, using the master's SERVER_UUID. This may indicate that the end of the binary log was truncated or that the last binary log file was lost, e.g., after a power or disk failure when sync_binlog != 1. The master may or may not have rolled back transactions that were already replicated to the slave. Suggest to replicate any transactions that master has rolled back from slave to master, and/or commit empty transactions on master to account for transactions that have been', Error_code: MY-013114 ``` Since the MTR result file is valid, we can suppress this error. Reviewed By: yichenshen Differential Revision: D39141846 fbshipit-source-id: 8e7fdb8 ------------------------------------------------------------------------------- Fix heap overflow in group_relay_log_name handling Summary: We were accessing group_relay_log_name in Query_log_event::do_apply_event_worker() but it's assigned only after the coordinator thread encounters an end event (i.e. xid event or a query event with "COMMIT" or "ROLLBACK" query). This was causing a race between accessing group_relay_log_name in the worker thread and writing it on the coordinator thread. We don't need to set transaction position in events other than end event, so now we set transaction position in query event only if it's an end event. The race is eliminated because group_relay_log_name is set before enqueuing the event to the worker thread (in both dep repl and vanilla mts). Reviewed By: lth Differential Revision: D28767430 ------------------------------------------------------------------------------- fix memory during MYSQL_BIN_LOG::open_existing_binlog Summary: asandebug complain there are memory leaks during MYSQL_BIN_LOG open Direct leak of 50 byte(s) in 1 object(s) allocated from: #0 0x67460ef in malloc #1 0x93f0777 in my_raw_malloc(unsigned long, int) #2 0x93f064a in my_malloc(unsigned int, unsigned long, int) #3 0x93f0eb0 in my_strdup(unsigned int, char const*, int) #4 0x8af01a6 in MYSQL_BIN_LOG::open(unsigned int, char const*, char const*, unsigned int) #5 0x8af8064 in MYSQL_BIN_LOG::open_binlog(char const*, char const*, unsigned long, bool, bool, bool, Format_description_log_event*, unsigned int, RaftRotateInfo*, bool) #6 0x8b00c00 in MYSQL_BIN_LOG::new_file_impl(bool, Format_description_log_event*, RaftRotateInfo*) #7 0x8d65e47 in rotate_relay_log(Master_info*, bool, bool, bool, RaftRotateInfo*) #8 0x8d661c0 in rotate_relay_log_for_raft(RaftRotateInfo*) #9 0x8c7696a in process_raft_queue #10 0xa0fa1fd in pfs_spawn_thread(void*) #11 0x7f8c9a12b20b in start_thread release these memory before assign them Reviewed By: Pushapgl Differential Revision: D28819752

Summary: [Porting Notes] We want to dump raft logs to vanilla async replicas regardless of whether it's the relay log or binlog. Effectively after this change we'll dump relay logs on the followers and binlogs on the leader. When the raft role changes, the logs to the dumped are also changed. Dump_log class is introduced as a thin wrapper/continer around mysql_bin_log or rli->relay_log and is inited with mysql_bin_log to emulate vanilla mysql behavior. Dump threads use the global dump_log object instead of mysql_bin_log directly. We switch the log in dump log only when raft role changes (in binlog_change_to_binlog() and binlog_change_to_apply_log()). During raft role change we take all log releated locks (LOCK_log, LOCK_index, LOCK_binlog_end_pos, and dump log lock) to serialize it with other log operations like dumping logs. Related doc - https://fb.quip.com/oTVAAdgEi4zY This diff contains below 7 patches: D23013977 D24766787 D24716539 D24900223 D24955284 D25174166 D25775525 Reviewed By: luqun Differential Revision: D26141496 ------------------------------------------------------------------------------- Passing raw_log pointer to wait_with_heartbeat() and wait_without_heartbeat() Summary: When enable_raft_plugin is OFF Dump_log::lock() is a no-op. Which means that when enable_raft_plugin is OFF there can be a race between log switching and dump threads. This could lead to a scenario where the raw_log that wait_next_event() is working on might be different than what wait_with_heartbeat()/wait_without_heartbeat() is working on. This can cause deadlocks because wait_with_heartbeat()/wait_without_heartbeat()'s mysql_cond_wait would unlock and then lock a different log's LOCK_binlog_end_pos mutex which would then never be unlocked by wait_next_event(). Reviewed By: anirbanr-fb Differential Revision: D32152658 ----------------------------------------------------------------------------------------- Fix rpl_raft_dump_raft_logs Summary: This tests completes but fails because the following warning exists: ``` 2022-08-30T16:28:00.159525Z 11 [ERROR] [MY-013114] [Repl] Slave I/O for channel '': Got fatal error 1236 from master when reading data from binary log: 'Slave has more GTIDs than the master has, using the master's SERVER_UUID. This may indicate that the end of the binary log was truncated or that the last binary log file was lost, e.g., after a power or disk failure when sync_binlog != 1. The master may or may not have rolled back transactions that were already replicated to the slave. Suggest to replicate any transactions that master has rolled back from slave to master, and/or commit empty transactions on master to account for transactions that have been', Error_code: MY-013114 ``` Since the MTR result file is valid, we can suppress this error. Reviewed By: yichenshen Differential Revision: D39141846 ------------------------------------------------------------------------------- Fix heap overflow in group_relay_log_name handling Summary: We were accessing group_relay_log_name in Query_log_event::do_apply_event_worker() but it's assigned only after the coordinator thread encounters an end event (i.e. xid event or a query event with "COMMIT" or "ROLLBACK" query). This was causing a race between accessing group_relay_log_name in the worker thread and writing it on the coordinator thread. We don't need to set transaction position in events other than end event, so now we set transaction position in query event only if it's an end event. The race is eliminated because group_relay_log_name is set before enqueuing the event to the worker thread (in both dep repl and vanilla mts). Reviewed By: lth Differential Revision: D28767430 ------------------------------------------------------------------------------- fix memory during MYSQL_BIN_LOG::open_existing_binlog Summary: asandebug complain there are memory leaks during MYSQL_BIN_LOG open Direct leak of 50 byte(s) in 1 object(s) allocated from: #0 0x67460ef in malloc #1 0x93f0777 in my_raw_malloc(unsigned long, int) #2 0x93f064a in my_malloc(unsigned int, unsigned long, int) #3 0x93f0eb0 in my_strdup(unsigned int, char const*, int) #4 0x8af01a6 in MYSQL_BIN_LOG::open(unsigned int, char const*, char const*, unsigned int) #5 0x8af8064 in MYSQL_BIN_LOG::open_binlog(char const*, char const*, unsigned long, bool, bool, bool, Format_description_log_event*, unsigned int, RaftRotateInfo*, bool) #6 0x8b00c00 in MYSQL_BIN_LOG::new_file_impl(bool, Format_description_log_event*, RaftRotateInfo*) #7 0x8d65e47 in rotate_relay_log(Master_info*, bool, bool, bool, RaftRotateInfo*) #8 0x8d661c0 in rotate_relay_log_for_raft(RaftRotateInfo*) #9 0x8c7696a in process_raft_queue #10 0xa0fa1fd in pfs_spawn_thread(void*) #11 0x7f8c9a12b20b in start_thread release these memory before assign them Reviewed By: Pushapgl Differential Revision: D28819752

abhinav04sharma and others added 30 commits March 15, 2022 17:09

Add --dump-fbobj-assoc-stats flag to mysqldump

4df5eee

Summary: Adding flag to collect aggregates for assoc/fbobj stats Reference patch: facebook@8200e5e Reference patch: facebook@c6a8bd8 Reference patch: facebook@520394a Differential Revision: D9173860

Add unsigned information of columns in table map events

c6d5cde

Summary: This is useful for external applications reading mysql binary logs. Originally Reviewed By: jtolmer Reference Patch: facebook@de7ae77 (cherry picked from commit de7ae77) Differential Revision: D9471053

inikep pushed a commit that referenced this pull request Jun 1, 2023

#11: Column families for RocksDB-SE: support setting options for colu…

9dfcceb

…mn families

inikep pushed a commit that referenced this pull request Jun 1, 2023

Issue #11: Column Family support: address review feedback

d21ef8e

inikep pushed a commit that referenced this pull request Jun 14, 2023

#11: Column families for RocksDB-SE

8a4d538

inikep pushed a commit that referenced this pull request Jun 14, 2023

#11: Column families for RocksDB-SE: support setting options for colu…

3ffd7c9

…mn families

inikep pushed a commit that referenced this pull request Jun 14, 2023

Issue #11: Column Family support: address review feedback

a5445b8

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FB8-278 - Suppress warning dir exists in test keyring_file_data_qa #11

FB8-278 - Suppress warning dir exists in test keyring_file_data_qa #11

ldonoso commented Mar 22, 2022

FB8-278 - Suppress warning dir exists in test keyring_file_data_qa #11

FB8-278 - Suppress warning dir exists in test keyring_file_data_qa #11

Conversation

ldonoso commented Mar 22, 2022