Tokudb 5.7 #1

george-lorch · 2015-12-02T18:33:48Z

First set of TokuDB mtr re-records and bug fix commits.

1 - re-recorded the following test(s) due to changes in the output of the EXPLAIN syntax: tokudb.bugs.1648 tokudb.bugs.1711 tokudb.bugs.1684 tokudb.bugs.5733_innodb tokudb.bugs.5733_tokudb tokudb.bugs.commit_index_end_1 tokudb.bugs.895 tokudb.bugs.bulk_fetch tokudb.bugs.index_read tokudb.bugs.simple_icp tokudb.bugs.1913 tokudb.bugs.2383 tokudb.bugs.2548 tokudb.bugs.2970 tokudb.bugs.2970i tokudb.bugs.3015 tokudb.bugs.5585 * also in 5 tokudb.cluster_update tokudb.cluster_update2 tokudb.cluster_1829 tokudb.cluster_2968-0 tokudb.cluster_2968-1 tokudb.cluster_2968-2 tokudb.cluster_2968-3 tokudb.cluster_create_table * also in 3 tokudb.ext_key_1_innodb * also in 18 tokudb.ext_key_1_tokudb * also in 18 tokudb.ext_key_2_innodb * also in 18 tokudb.ext_key_2_tokudb * also in 18 tokudb.cluster_delete tokudb.cluster_delete2 tokudb.cluster_filter tokudb.cluster_filter_hidden tokudb.cluster_filter_key tokudb.cluster_filter_unpack_varchar tokudb.cluster_filter_unpack_varchar_and_int_hidden tokudb.cluster_filter_unpack_varchar_hidden tokudb.cluster_key tokudb.cluster_query_plan tokudb.mvcc-29 tokudb.mvcc-30 tokudb.mvcc-31 tokudb.mvcc-33 * also in 3 tokudb.mvcc-34 * also in 3 tokudb.mvcc-35 * also in 3 tokudb.mvcc-36 * also in 3 tokudb.mvcc-37 * also in 3 tokudb.mvcc-38 * also in 3 tokudb.mvcc-8 * also in 3 tokudb.replace-ignore tokudb.type_varchar tokudb.add_index.1522 tokudb.add_index.add_index_1 tokudb.add_index.add_index_11 tokudb.add_index.add_index_12 tokudb.add_index.add_index_13 tokudb.add_index.add_index_14 tokudb.add_index.add_index_15 tokudb.add_index.add_index_18 tokudb.add_index.add_index_2 tokudb.add_index.add_index_3 tokudb.add_index.add_index_4 tokudb.add_index.add_index_5 tokudb.add_index.add_index_8 tokudb.add_index.add_index_9 tokudb.alter_table.ai_di tokudb.alter_table.hcad_with_lock_sps tokudb.alter_table.hcr3 * also in 3 2 - re-recorded the following test(s) due to changes in information_schema.processlist: tokudb.bugs.checkpoint_lock 3 - re-recorded the following test(s) due to changes in how CREATE TABLE reports DEFAULT': tokudb.bugs.frm_store3 tokudb.cluster_create_table * also in 1 tokudb.mvcc-21 tokudb.mvcc-22 tokudb.mvcc-23 tokudb.mvcc-25 tokudb.mvcc-3 tokudb.mvcc-33 * also in 1 tokudb.mvcc-34 * also in 1 tokudb.mvcc-35 * also in 1 tokudb.mvcc-36 * also in 1 tokudb.mvcc-37 * also in 1 tokudb.mvcc-38 * also in 1 tokudb.mvcc-4 tokudb.mvcc-5 tokudb.mvcc-6 tokudb.mvcc-7 tokudb.mvcc-8 * also in 1 tokudb.mvcc-9 tokudb.mvcc-10 tokudb.parts.partition_special_tokudb tokudb.parts.partition_exch_qa_1_tokudb tokudb.parts.partition_exchange_tokudb tokudb.alter_table.ai_part tokudb.alter_table.hcr3 * also in 1 tokudb.alter_table.other_alter tokudb.alter_table.other_alter2 tokudb.alter_table.hcad_part 4 - re-recorded the following test(s) due to change in MySQL warning on storing username/password: tokudb.bugs.rpl_row_replace_into tokudb.rpl.rpl_tokudb_mixed_dml * also in 8 5 - re-recorded the following test(s) due to change in 'Note' when inserting a TIMESTAMP value into a DATE column: tokudb.bugs.5585 * also in 1 6 - re-recorded the following test(s) due to include/sync_slave_sql_with_master.inc now appearing in some results: tokudb.rpl.rpl_partition_tokudb 7 - re-recorded the following test(s) due to series of rpl setup statements now not showing in results files as a result of 'include/master-slave.inc': tokudb.rpl.rpl_row_log_tokudb 8 - re-recorded the following test(s) due to several changes in suite/rpl/include/rpl_check*.inc files: tokudb.rpl.rpl_tokudb_mixed_dml * also in 4 9 - re-recorded the following test(s) due to several aditions of IGNORE on INSERTs and UPDATEs in extra/rpl_tests/rpl_extra_col_master.test: tokudb.rpl.rpl_extra_col_master_tokudb 10 - re-recorded the following test(s) due to changes in extra/rpl_tests/rpl_extra_col_slave.test: tokudb.rpl.rpl_extra_col_slave_tokudb 11 - re-recorded the following test(s) due to changes in extra/rpl_tests/rpl_deadlock.test: tokudb.rpl.rpl_deadlock_tokudb 12 - re-recorded the following test(s) due to changes in extra/rpl_tests/rpl_not_null.test: tokudb.rpl.rpl_not_null_tokudb 13 - re-recorded the following test(s) due to slave_transaction_retries warning on multi-threaded slave: tokudb.rpl.rpl_parallel_tokudb_delete_pk tokudb.rpl.rpl_parallel_tokudb_update_pk_uc0_lookup0 tokudb.rpl.rpl_parallel_tokudb_write_pk 14 - re-recorded the following test(s) due to changes in extra/rpl_tests/rpl_row_basic.test: tokudb.rpl.rpl_row_basic_3tokudb 15 - re-recorded the following test(s) due to changes in extra/rpl_tests/rpl_record_compare.test: tokudb.rpl.rpl_row_rec_comp_tokudb 16 - re-recorded the following test(s) due to changes in extra/rpl_tests/rpl_row_tabledefs.test: tokudb.rpl.rpl_row_tabledefs_3tokudb 17 - re-recorded the folowing test(s) due to changes in include/ctype_common.inc: tokudb.ctype_cp1250_ch 18 - re-recorded the following test(s) due to changes in return for 'select @@optimizer_switch': tokudb.ext_key_1_innodb * also in 1 tokudb.ext_key_1_tokudb * also in 1 tokudb.ext_key_2_innodb * also in 1 tokudb.ext_key_2_tokudb * also in 1 19 - re-recorded the following test(s) due to different default sql_mode: tokudb.sql_mode_default 20 - re-recorded the following test(s) due to new SQL warning on duplicate key during insert ignore: tokudb.nested_txn_autocommit tokudb.nested_txn_begin 21 - re-recorded the following test(s) due to changes in suite/parts/inc/partition_mgm_crash.inc: tokudb.parts.partition_debug_tokudb 22 - re-recorded the following test(s) due to changes in suite/parts/inc/partition_bit.inc: tokudb.parts.partition_bit_tokudb 23 - re-recorded the following test(s) due to changes in suite/parts/inc/partition_enum.inc: tokudb.parts.partition_char_tokudb 24 - re-recorded the following test(s) due to new warning : Warning 3090 Changing sql mode 'NO_AUTO_CREATE_USER' is deprecated. It will be removed in a future release: tokudb.parts.partition_datetime_tokudb tokudb.parts.partition_syntax_tokudb tokudb.parts.partition_engine_tokudb tokudb.parts.partition_alter4_tokudb 25 - re-recorded the following test(s) due to changes in suite/parts/inc/partition_auto_increment.inc: tokudb.parts.partition_auto_increment_tokudb 26 - re-recorded the following test(s) due to changes in Note to Warning 1831 Duplicate index '<nr>' defined on the table '<tr>'. This is deprecated and will be disallowed in a future release: tokudb.alter_table.hcad_tmp_tables_56

…mysys/my_malloc.c:128: my_free: Assertion `mh->m_magic == 1234' failed Fixed cross lib free by calling global free and not tokub::memory::free to release keys allocated by FT lib. Re-recorded tests due to new analyze behavior and missed re-records due to crash: tokudb.bugs.db757_part_alter_analyze Fixed minor string size issue in i_s.background_job_manager and re-recorded tokudb.background_job_manager

…alue for 'timestamp' * tokudb.bugs.6684 : - removed DEFAULT TIMESTAMP value from CREATE TABLE as it adds nothing to the function of the test - re-recorded * tokudb.bugs.leak172 : - removed DEFAULT TIMESTAMP value from CREATE TABLE as it adds nothing to the function of the test - fixed up std_data files to change '0' date values to some modern, real date. - re-recorded

…column 'action' at row 1 * tokudb.bugs.db397_delete_trigger - test table used varchar(10) but tried inserting larger strings in, increased the size of the column to allow for the larger strings as this has no bearing on the test or the original bug (DB-397). - re-recorded * tokudb.change_column_bin - hacked up the change_column_bin.py generator to insert mtr "--error ER_DATA_TOO_LONG" before INSERTS that would have previously been truncated - re-generated the test - re-recorded * tokudb.change_column_bin_rename - hacked up the change_column_bin_rename.py generator to insert mtr "--error ER_DATA_TOO_LONG" before INSERTS that would have previously been truncated - re-generated the test - re-recorded * tokudb.change_column_blob_data - hacked up the change_column_blob_data.py generator to insert mtr "--error ER_DATA_TOO_LONG" before INSERTS that would have previously been truncated - re-generated the test - re-recorded

laurynas-biveinis · 2015-12-03T14:20:01Z

storage/tokudb/tokudb_information_schema.cc

@@ -1119,9 +1119,9 @@ void background_job_status_callback(
    table->field[3]->store(type, strlen(type), system_charset_info);
    table->field[4]->store(params, strlen(params), system_charset_info);
    if (user_scheduled)
-        table->field[5]->store("USER", sizeof("USER"), system_charset_info);
+        table->field[5]->store("USER", strlen("USER"), system_charset_info);


Should this go to 5.6 as well? Not right now with an upmerge or anything like that, just to make a note somewhere right now

Yes, I already have a 5.6 branch ready to go since we aren't in GCA mode
yet.

On 12/03/2015 07:20 AM, Laurynas Biveinis wrote:

In storage/tokudb/tokudb_information_schema.cc
#1 (comment):

@@ -1119,9 +1119,9 @@ void background_job_status_callback(
table->field[3]->store(type, strlen(type), system_charset_info);
table->field[4]->store(params, strlen(params), system_charset_info);
if (user_scheduled)

table->field[5]->store("USER", sizeof("USER"), system_charset_info);

table->field[5]->store("USER", strlen("USER"), system_charset_info);

Should this go to 5.6 as well? Not right now with an upmerge or
anything like that, just to make a note somewhere right now

—
Reply to this email directly or view it on GitHub
https://github.com/laurynas-biveinis/percona-server/pull/1/files#r46555614.

George O. Lorch III
Software Engineer, Percona
US/Arizona (GMT -7)
skype: george.ormond.lorch.iii

Tokudb 5.7

Problem: The binary log group commit sync is failing when committing a group of transactions into a non-transactional storage engine while other thread is rotating the binary log. Analysis: The binary log group commit procedure (ordered_commit) acquires LOCK_log during the #1 stage (flush). As it holds the LOCK_log, a binary log rotation will have to wait until this flush stage to finish before actually rotating the binary log. For the #2 stage (sync), the binary log group commit only holds the LOCK_log if sync_binlog=1. In this case, the rotation has to wait also for the sync stage to finish. When sync_binlog>1, the sync stage releases the LOCK_log (to let other groups to enter the flush stage), holding only the LOCK_sync. In this case, the rotation can acquire the LOCK_log in parallel with the sync stage. For commits into transactional storage engine, the binary log rotation checks a counter of "flushed but not yet committed" transactions, waiting until this counter to be zeroed before closing the current binary log file. As the commit of the transactions happen in the #3 stage of the binary log group commit, the sync of the binary log in stage #2 always succeed. For commits into non-transactional storage engine, the binary log rotation is checking the "flushed but not yet committed" transactions counter, but it is zero because it only counts transactions that contains XIDs. So, the rotation is allowed to take place in parallel with the #2 stage of the binary log group commit. When the sync is called at the same time that the rotation has closed the old binary log file but didn't open the new file yet, the sync is failing with the following error: 'Can't sync file 'UNOPENED' to disk (Errcode: 9 - Bad file descriptor)'. Fix: For non-transactional only workload, binary log group commit will keep the LOCK_log when entering #2 stage (sync) if the current group is supposed to be synced to the binary log file.

Description: For ngshell query Crud.Find({ name:$.name, count:count(*) }).GroupBy($.name); an error has been raised: "Expression #1 of SELECT list is not in GROUP BY clause and contains nonaggregated column 'test.coll.doc' which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by" Reviewed-by: Lukasz Kotula <lukasz.kotula@oracle.com> RB:12283

…hutdown) On several testcases (i.e. rpl_gtid_mode), LeakSanitizer diagnoses missed memory deallocation: ================================================================= ==16675==ERROR: LeakSanitizer: detected memory leaks Direct leak of 21 byte(s) in 1 object(s) allocated from: #0 0x7f17748fa54a in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x9854a) #1 0xff7f7f in my_malloc /mnt/workspace/percona-server-5.6-asan-param/BUILD_TYPE/debug-asan/Host/ubuntu-xenial-64bit/mysys/my_malloc.c:38 #2 0x1634b83 in add_pfs_instr_to_array(char const*, char const*) /mnt/workspace/percona-server-5.6-asan-param/BUILD_TYPE/debug-asan/Host/ubuntu-xenial-64bit/storage/perfschema/pfs_server.cc:251 #3 0x58cccf in mysqld_get_one_option /mnt/workspace/percona-server-5.6-asan-param/BUILD_TYPE/debug-asan/Host/ubuntu-xenial-64bit/sql/mysqld.cc:9198 #4 0x10256c6 in my_handle_options /mnt/workspace/percona-server-5.6-asan-param/BUILD_TYPE/debug-asan/Host/ubuntu-xenial-64bit/mysys_ssl/my_getopt.cc:817 #5 0x1025c63 in handle_options /mnt/workspace/percona-server-5.6-asan-param/BUILD_TYPE/debug-asan/Host/ubuntu-xenial-64bit/mysys_ssl/my_getopt.cc:308 #6 0x5963e5 in handle_early_options() /mnt/workspace/percona-server-5.6-asan-param/BUILD_TYPE/debug-asan/Host/ubuntu-xenial-64bit/sql/mysqld.cc:7263 #7 0x5a35a3 in mysqld_main(int, char**) /mnt/workspace/percona-server-5.6-asan-param/BUILD_TYPE/debug-asan/Host/ubuntu-xenial-64bit/sql/mysqld.cc:5613 #8 0x586aae in main /mnt/workspace/percona-server-5.6-asan-param/BUILD_TYPE/debug-asan/Host/ubuntu-xenial-64bit/sql/main.cc:25 #9 0x7f17726cc82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f) This class of errors is already attempted to suppress in valgrind.supp. But these suppressions have been added to work around a bug of racy PFS shutdown, which is not required anymore as pfs_instr_config_array is deallocated exactly once since [1]. Thus, free the elements of this array and remove related suppressions instead.

Use ha_rocksdb::open() should check table->found_next_number_field, not table->next_number_field (like InnoDB does)

Two testcases are empty because RocksDB-SE doesn't support the feature: -autoincrement.test -autoinc_secondary.test

Summary: This adds two features in RocksDB. 1. Supporting START TRANSACTION WITH CONSISTENT SNAPSHOT 2. Getting current binlog position in addition to #1. With these features, mysqldump can take consistent logical backup. The second feature is done by START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT. This is Facebook's extension, and it works like existing START TRANSACTION WITH CONSISTENT INNODB SNAPSHOT. This diff changed some existing codebase/behaviors. - Original Facebook-MySQL always started InnoDB transaction regardless of engine clause. For example, START TRANSACTION WITH CONSISTENT MYISAM SNAPSHOT was accepted but it actually started InnoDB transaction, not MyISAM. This patch does not allow setting engine that does not support consistent snapshot. mysql> start transaction with consistent myisam snapshot; ERROR 1105 (HY000): Consistent Snapshot is not supported for this engine Currently only InnoDB and RocksDB support consistent snapshot. To check engines, I modified sql/sql_yacc.yy, trans_begin() and ha_start_consistent_snapshot() to pass handlerton. - Changed constant name from MYSQL_START_TRANS_OPT_WITH_CONS_INNODB_SNAPSHOT to MYSQL_START_TRANS_OPT_WITH_CONS_ENGINE_SNAPSHOT, because it's no longer InnoDB dependent. - When not setting engine, START TRANSACTION WITH CONSISTENT SNAPSHOT takes both InnoDB and RocksDB snapshots, and both InnoDB and RocksDB participate in transaction. When executing COMMIT, both InnoDB and RocksDB modifications are committed. Remember that XA is not supported yet, so mixing engines is not recommended anyway. - When setting engine, START TRANSACTION WITH CONSISTENT.. takes snapshot for the specified engine only. But it starts both InnoDB and RocksDB transactions. Test Plan: mtr --suite=rocksdb,rocksdb_rpl, --repeat=3 Reviewers: hermanlee4, jonahcohen, jtolmer, tian.xia, maykov, spetrunia Reviewed By: spetrunia Subscribers: steaphan Differential Revision: https://reviews.facebook.net/D32355

Summary: Don't update secondary indexes that do not have columns that are included in table->write_map at query start. Test Plan: mtr Reviewers: maykov, yoshinorim, hermanlee4, jonahcohen Reviewed By: jonahcohen Differential Revision: https://reviews.facebook.net/D33471

Summary: Inside index_next_same() call, we should 1. first check whether the record matches the index lookup prefix, 2. then check pushed index condition. If we try to check #2 without checking #1 first, we may walk off the index lookup prefix and scan till the end of the index. Test Plan: Run mtr Reviewers: hermanlee4, maykov, jtolmer, yoshinorim Reviewed By: yoshinorim Differential Revision: https://reviews.facebook.net/D38769

Summary: MyRocks had two bugs when calculating index scan cost. 1. block_size was not considered. This made covering index scan cost (both full index scan and range scan) much higher 2. ha_rocksdb::records_in_range() may have estimated more rows than the estimated number of rows in the table. This was wrong, and MySQL optimizer decided to use full index scan even though range scan was more efficient. This diff fixes #1 by setting stats.block_size at ha_rocksdb::open(), and fixes #2 by reducing the number of estimated rows if it was larger than stats.records. Test Plan: mtr, updating some affected test cases, and new test case rocksdb_range2 Reviewers: hermanlee4, jkedgar, spetrunia Reviewed By: spetrunia Subscribers: MarkCallaghan, webscalesql-eng Differential Revision: https://reviews.facebook.net/D55869

In WL-included builds ASAN run witnessed missed ~Query_log_event invocation. The destruct-or was not called due to the WL's changes in the error propagation that specifically affect LC MTS. The failure is exposed in particular by rpl_trigger as the following stack: #0 0x9ecd98 in __interceptor_malloc (/export/home/pb2/test/sb_2-22611026-1489061390.32/mysql-commercial-8.0.1-dmr-linux-x86_64-asan/bin/mysqld+0x9ecd98) #1 0x2b1a245 in my_raw_malloc(unsigned long, int) obj/mysys/../../mysqlcom-pro-8.0.1-dmr/mysys/my_malloc.cc:209:12 #2 0x2b1a245 in my_malloc obj/mysys/../../mysqlcom-pro-8.0.1-dmr/mysys/my_malloc.cc:72 #3 0x2940590 in Query_log_event::Query_log_event(char const*, unsigned int, binary_log::Format_description_event const*, binary_log::Log_event_type) obj/sql/../../mysqlcom-pro-8.0.1-dmr/sql/log_event.cc:4343:46 #4 0x293d235 in Log_event::read_log_event(char const*, unsigned int, char const**, Format_description_log_event const*, bool) obj/sql/../../mysqlcom-pro-8.0.1-dmr/sql/log_event.cc:1686:17 #5 0x293b96f in Log_event::read_log_event() #6 0x2a2a1c9 in next_event(Relay_log_info*) Previously before the WL Mts_submode_logical_clock::wait_for_workers_to_finish() had not returned any error even when Coordinator thread is killed. The WL patch needed to refine such behavior, but at doing so it also had to attend log_event.cc::schedule_next_event() to register an error to follow an existing pattern. While my_error() does not take place the killed Coordinator continued scheduling, ineffectively though - no Worker gets engaged (legal case of deferred scheduling), and without noticing its killed status up to a point when it resets the event pointer in apply_event_and_update_pos(): *ptr_ev= NULL; // announcing the event is passed to w-worker The reset was intended for an assigned Worker to perform the event destruction or by Coordinator itself when the event is deferred. As neither is the current case the event gets unattended for its termination. In contrast in the pre-WL sources the killed Coordinator does find a Worker. However such Worker could be already down (errored out and exited), in which case apply_event_and_update_pos() reasonably returns an error and executes delete ev in exec_relay_log_event() error branch. **Fixed** with deploying my_error() call in log_event.cc::schedule_next_event() error branch which fits to the existing pattern. THD::is_error() has been always checked by Coordinator before any attempt to reset *ptr_ev= NULL. In the errored case Coordinator does not reset and destroys the event itself in the exec_relay_log_event() error branch pretty similarly to how the pre-WL sources do. Tested against rpl_trigger and rpl suites to pass. Approved on rb#15667.

… WITH DEVELOPER STUDIO When we do a release type build of the server (with both optimized and debug enabled server/plugins) with Developer Studio, some MTR tests when run with --debug-server will fail in one of two ways: 1. Tests which try to load a plugin into the mysql client fail with missing symbols. This is caused by the plugin having references to functions which do not exist in the non-debug client. 2. Some tests on sparc fail with Thread stack overrun. Fix for issue #1: mtr will have appended /debug to the plugin dir part when running with --debug-server and if there actually is such a directory. The fix is to remove any trailing /debug from the env. variable within the test. This will affect the client only, not the server. Developer builds will not have put the plugins in a subdirectory /debug so it makes no different to those. Fix for issue #2: apparently this thread stack overrun is not feasible to avoid, so just skip the test if running with debug server on sparc; there is already an include file to do that. Also added not_sparc_debug.inc to the "white list" so the tests are skipped even when running mtr --no-skip. (cherry picked from commit 9c79e477261ab252e38def436bca3336ef597603)

Some character sets are designated as MY_CS_STRNXFRM, meaning that sorting needs to go through my_strnxfrm() (implemented by the charset), and some are not, meaning that a client can do the strnxfrm itself based on cs->sort_order. However, most of the logic related to the latter has been removed already (e.g. filesort always uses my_strnxfrm() since 2003), and now it's mostly in the way. The three main uses left are: 1. A microoptimization for constructing sort keys in filesort. 2. A home-grown implementation of Boyer-Moore for accelerating certain LIKE patterns that should probably be handled through FTS. 3. Some optimizations to MyISAM prefix keys. Given that our default collation (utf8mb4_0900_ai_ci) now is a strnxfrm-based collation, the benefits of keeping these around for a narrow range of single-byte locales (like latin1_swedish_ci, cp850 and a bunch of more obscure locales) seems dubious. We seemingly can't remove the flag entirely due to #3 seemingly affecting the on-disk MyISAM structure, but we can remove the code for #1 and #2. Change-Id: If974e490d451b7278355e33ab1fca993f446b792

…l process

Patch #1: When find_child_doms() evaluates an ellipsis path leg, it adds each matching child twice. First once for every immediate child in the top-level call to find_child_doms(), and then again in the recursive call to find_child_doms() on each of the children. This doesn't cause any wrong results, since duplicate elimination prevents them from being added to the result, but it is unnecessary work. This patch makes find_child_dom() stop calling add_if_missing() on the children before it recurses into them, so that they are only added once. Additionally: Make find_child_dom() a free, static function instead of a member of Json_dom. It is a helper function for Json_dom::seek(), and doesn't use any non-public members of Json_dom, so it doesn't have to be part of Json_dom's interface. Remove unnecessary checks for only_need_one in the ellipsis processing. Json_dom::seek() sets the flag to true only in the last path leg, and the JSON path parser rejects paths that end with an ellipsis, so this cannot happen. Added an assert instead. Microbenchmarks (64-bit, Intel Core i7-4770 3.4 GHz, GCC 6.3): BM_JsonDomSearchEllipsis 79880 ns/iter [+32.0%] BM_JsonDomSearchEllipsis_OnlyOne 75872 ns/iter [+35.3%] BM_JsonDomSearchKey 129 ns/iter [ -1.6%] BM_JsonBinarySearchEllipsis 320920 ns/iter [+11.8%] BM_JsonBinarySearchEllipsis_OnlyOne 315458 ns/iter [+12.6%] BM_JsonBinarySearchKey 86 ns/iter [ -2.3%] Change-Id: I865af789b90b820a6e180ad822f2fb68f411516b

Fix misc UBSAN warnings in unit tests. To repeat: export UBSAN_OPTIONS="print_stacktrace=1" ./runtime_output_directory/merge_large_tests-t --gtest_filter='-*DeathTest*' > /dev/null unittest/gunit/gis_algos-t.cc:78:70: runtime error: downcast of address 0x000012dc0be8 which does not point to an object of type 'Gis_polygon_ring' include/sql_string.h:683:35: runtime error: null pointer passed as argument 2, which is declared to never be null #1 0x373e7af in histograms::Value_map<String>::add_values(String const&, unsigned long long) sql/histograms/value_map.cc:149 #2 0x294fcf2 in dd_column_statistics_unittest::add_values(histograms::Value_map<String>&) unittest/gunit/dd_column_statistics-t.cc:62 runtime_output_directory/merge_keyring_file_tests-t --gtest_filter='-*DeathTest*' > /dev/null plugin/keyring/common/keyring_key.cc:82:57: runtime error: null pointer passed as argument 2, which is declared to never be null Change-Id: I2651362e3373244b72e6893f0e22e67402b49a52 (cherry picked from commit 1fe3f72561994da1d912a257689e1b18106f8828)

…_ROW()' FAILED Analysis: When a window with buffering follows a equijoin on a unique index (JT_EQ_REF) , we can get into trouble because windowing modifies the input record, presuming that once the windowing has been handed the record, next time control passes back to the join code a (new) record will be read to the input record. However, this does not hold with JT_EQ_REF, cf. the caching done in join_read_key: From its Doxygen: "Since the eq_ref access method will always return the same row, it is not necessary to read the row more than once, regardless of how many times it is needed in execution. This cache element is used when a row is needed after it has been read once, unless a key conversion error has occurred, or the cache has been disabled." Fix: We solve this problem by reinstating the input record before handing control back from end_write_wf. We optimize: only do this if the window in question follows after such a JOIN, i.e. window #1, and it has actually clobbered the input record. This can only happen if the last qep_tab has type JT_EQ_REF. Another, perhaps better approach, is to refactor to never touch the input record but keep the copying between the out record and the frame table record instead. Left for future refactoring. Added some missing Window method "const"s, and folded a couple of one-liners into window.h (from .cc). Repro added. Change-Id: I33bc43cd99ff79303b17d181abc3805ce226fb85

…TABLE_UPGRADE_GUARD To repeat: cmake -DWITH_ASAN=1 -DWITH_ASAN_SCOPE=1 ./mtr --mem --sanitize main.dd_upgrade_error A few dd tests fail with: ==26861==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7000063bf5e8 at pc 0x00010d4dbe8b bp 0x7000063bda40 sp 0x7000063bda38 READ of size 8 at 0x7000063bf5e8 thread T2 #0 0x10d4dbe8a in Prealloced_array<st_plugin_int**, 16ul>::empty() const prealloced_array.h:186 #1 0x10d406a8b in lex_end(LEX*) sql_lex.cc:560 #2 0x10dae4b6d in dd::upgrade::Table_upgrade_guard::~Table_upgrade_guard() (mysqld:x86_64+0x100f87b6d) #3 0x10dadc557 in dd::upgrade::migrate_table_to_dd(THD*, std::__1::basic_string<char, std::__1::char_traits<char>, Stateless_allocator<char, dd::String_type_alloc, My_free_functor> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, Stateless_allocator<char, dd::String_type_alloc, My_free_functor> > const&, bool) (mysqld:x86_64+0x100f7f557) #4 0x10dad7e85 in dd::upgrade::migrate_plugin_table_to_dd(THD*) (mysqld:x86_64+0x100f7ae85) #5 0x10daec6a1 in dd::upgrade::do_pre_checks_and_initialize_dd(THD*) upgrade.cc:1216 #6 0x10cd0a5c0 in bootstrap::handle_bootstrap(void*) bootstrap.cc:336 Change-Id: I265ec6dd97ee8076aaf03763840c0cdf9e20325b Fix: increase lifetime of 'LEX lex;' which is used by 'table_guard'

AUTH_COMMON.H Description:- Sever crashes due to a NULL pointer de-reference. Analysis:- Sever encounters a NUll pointer de-reference during "acl_load()". Fix:- A check is introduced to avoid the NULL pointer de-reference. This issue is already prevented in 8.0 through Bug#27225806 fix. Therefore, this patch is applicable only for 5.7.

- Remove include/my_global.h from .gitignore - it should be visible, especially when merging from 5.7 (where it's present) to 8.0 (where it's not). This partially fixes bug https://bugs.mysql.com/bug.php?id=91834 (Cleanup .gitignore, e.g. include/my_global.h) CMake / preprocessor symbol changes - Make CMake option WITH_SYSTEM_LIBS configure -DWITH_READLINE=system instead of -DWITH_EDITLINE=system - Remove WITH_SSL CMake settings from build scripts, leaving the implicit default of -DWITH_SSL=system, because the alternative -DWITH_SSL=wolfssl requires more work and is broken by upstream in 8.0.11. - Use MYSQL_ADD_EXECUTABLE instead of ADD_EXECUTABLE in CMake scripts. - Remove HAVE_REPLICATION from diff to match upstream [14]. - Remove WITH_EMBEDDED_SERVER, EMBEDDED_LIBRARY, NO_EMBEDDED_ACCESS_CHECKS, mysql-test/include/not_embedded.inc mentions due to upstream [15], [16], [24] etc. - Remove WITH_PARTITION_STORAGE_ENGINE due to upstream [32]. - Replace MYSQL_CLIENT preprocessor macro uses with !MYSQL_SERVER due to [22]. - Add sql/binlog_crypt_data.cc and sql/event_crypt.cc as mysqlbinlog source files in CMake, to replace the former practice of including them from source files. - Remove setting C++98/03 and C++11 detection from CMake due to upstream converting to C++11 in WL#8896. This includes removing cmake/check_stdcxx11.cmake and its uses from TokuDB / MyRocks. - Make CMake define __STDC_WANT_LIB_EXT1__=1 on macOS to get memset_s. - Define LOG_COMPONENT_TAG for various plugins and unit tests, to accomodate WL#11150 [19]. - Remove uses of UNIV_NONINL due to [35], consequently fold buf0dblwr.ic to buf0dblwr.h. - Revmove UNIV_DEBUG_THREAD_CREATION due to WL#9359 [36] Compilation changes - Fix now-enabled-by-default unused parameter warnings, by removing the unused parameters, or by marking them unused. C to C++ conversion - Convert mysys/system_key.c to C++ mysys/system_key.cc to match upstream [1]. - Convert storage/heap/hp_record.c and storage/heap/hp_dspace.c to C++ hp_record.cc and hp_dspace.cc respectively to match upstream [2]. - Mechanical renames, changes, and partial conversions, incomplete list: - my_bool/FALSE/TRUE/Sys_var_mybool uses to bool/false/true/Sys_var_bool due to upstream my_bool removal at [6]. - #define to [static] const constexpr - Declare local variables at first use - Pointer arguments to const reference ones, where applicable - Convert typedef struct declaration to plain struct, removed redundant struct tags. - NULL/0 to nullptr - compile_time_assert to static_assert - various ad-hoc macros to std::min - various integer types/0/1 to bool/false/true where applicable - Convert mysqldump.cc patches to C++11, remove uses of HASH, make key_type_t an enum class instead of enum. At the same time replace uses of LIST with std::list and std::forward_list, although this was not strictly required. - Remove C_MODE_START/C_MODE_END and extern "C" blocks from C++-only headers. C++03 to C++11 conversion - Replace HASH with std::unordered_map in many patches, because upstream removed HASH in series of commits for "25997748". - Mechanical renames and changes, incomplete list: - ut_snprintf uses to snprintf due to upstream removal of the former at [12]. - my_snprintf to snprintf due to C++11 guaranteeing the presence of the latter. - Applied noexcept to matching functions and methods. - Applied final to matching classes. - boost::movelib::unique_ptr to std::unique_ptr. - insert to emplace for standard containers, where applicable - converted some of std::map uses to std::unordered_map, if ordering is not required - std::auto_ptr to std::unique_ptr - Assignments in constructors and constructor initializer lists to default member initializers. - UNIV_THREAD_LOCAL to thread_local - Replaced uses of compiler builtins with std::atomic for atomic variables. User statistics - Due to HASH replacement with modern C++ collections, cleanup userstat to store objects directly instead of heap-allocated pointers to them. For this, add constructors to TABLE_STATS and INDEX_STATS to replace init_user_stats and init_thread_stats functions. Binlog encryption - Adjust binlog encryption patches for MYSQL_BIN_LOG "append" to "write" rename in cass methods. Parser - Put Percona tokens at the end of the token list due to upstream attempt to ensure token digest value stability ([17], [18]). - Because of upstream bottom-up parser refactoring for CREATE TABLE [23], which includes column attributes, rewrite TokuDB clustered index and InnoDB compression dictionary support: - add apply_zip_dict method to PT_column_attr_base class; - add zip_dict_name arg to the constructor and m_zip_dict_name field to PT_column_format_column_attr class, implement apply_zip_dict; - add m_zip_dict field to PT_field_def_base, get its value in contextualize_attrs method, and pass it to Alter_info instance in PT_column_def::contextualize and PT_alter_table_change_column::contextualize - rename PT_unique_key_column_attr to PT_unique_combo_clustering_key_column_attr, make it handle both UNIQUE and CLUSTERING - move zip_dict_name from LEX to HA_CREATE_INFO in line with the upstream LEX reduction in parser refactoring. - Typedef new alter tablespace option PT_alter_tablespace_option_encryption for general tablespace encryption. Performance Schema - Adjust PFS object declarations for the new volatility and documentation fields, account for PSI_FLAG_GLOBAL rename to PSI_FLAG_SINGLETON. - Move server PSF memory key declarations to sql/psi_memory_key.h and sql/psi_memory_key.cc - Adjust inline_mysql_unix_connect PFS instrumentation in include/mysql/psi/mysql_file.h to modern PFS instrumentation pattern which was introduced back in [3] because the old one stopped compiling in 8.0. While at that, switch from HAVE_PSI_INTERFACE to HAVE_PSI_FILE_INTERFACE. Audit log plugin - Converted the plugin to C++. - Made a lot of declarations in audit_log.cc static and/or const. - Converted hash tables in filter.cc to collation_unordered_set<std::string>. At the same time convert the hash table value struct creator/initialization functions to constructors. Backup locks - Remove everything associated with LOCK BINLOG FOR BACKUP / UNLOCK BINLOG due to upstream performance_schema.log_status table. - Refer to Oracle backup lock as "instance backup lock" and Percona backup lock as "table backup lock" to try to reduce confusion between the two. Expanded fast index creation - Change the type of delayed_key_list container in Alter_info from List to Mem_root_array. Threadpool - Re-implement TP_STATISTICS::num_worker_threads, pool_timer_t::current_microtime and next_timeout_check using C++11 atomics. - Convert Worker_thread_context struct to class with constructor/destructor instead of save/restore methods, to be used as RAII. New data dictionary - Percona features that need to store information in the new data dictionary (CLUSTERED keys for TokuDB; compression dictionary information for InnoDB have not been re-implemented for the new DD yet). For now the legacy implementation is disabled, to be removed together with the new implementation, except for the reading bits, because they are needed for upgrades from 5.7. Specifically, leave SYSTEM_TABLE_NAME patch in, dict_process_sys_zip_dict function Changed page tracking - Move fil_name_process patch not to cause any tablespace in filesystem changes when called by redo log tracking to fil_tablespace_redo_delete and fil_tablespace_redo_create. - Introduce apply arg for fil_tablespace_redo_encryption for redo log parsing mode only. - log_get_tracked_lsn and log_set_tracked_lsn have been remvoed after the tracked LSN conversion to C++11 atomic variable. - Fold log_check_tracking_margin to log0write.cc:log_write_write_buffer. - Move redo log signalling from log_io_complete_checkpoint to log0chkp.cc:log_checkpoint. - Drop log_group_read_log_seg release_mutex arg (which would have been moved to recv_read_log_seg) because of log_sys->mutex removal. - innobase_logs_empty_and_mark_at_shutdown patches moved to srv_log_shutdown. - Due to os_file_dir_t and associated function removals in [13], rewrite their users in log0online.cc to use MY_DIR and my_dir from mysys. - Remove log_group_t mentions from log0online.cc - Add handling for MLOG_COMP_PAGE_CREATE_SDI, MLOG_PAGE_CREATE_SDI, and MLOG_TABLE_DYNAMIC_META MTR types. - Adjust redo log reading code for log_group_read_log_seg rename to recv_read_log_seg, convert the latter from static to public function. - Refactor out redo log tracker-waiting code in fil0fil.cc to a new private static method Fil_system::wait_for_changed_page_tracker. Other changes due to upstream - Remove YaSSL-related source code (mysys_ssl/yassl.cpp, HAVE_YASSL check, build-binary.sh --with-yassl option) - Replace my_global.h includes with the needed smaller ones (my_inttypes.h, mysql/service_mysql_alloc.h, my_macros.h) due to upstream patch series "Bug #24710065: SPLIT MY_GLOBAL.H", and move my_global.h patches to appropriate places: s6_addr32 define to violite.h. - Remove THD::thr_create_utime uses and replace THD::update_server_status calls with THD::update_slow_query_status due to [30]. - Remove Sql_alloc uses due to [26]. - Adjust for class Key rename to Key_spec with declaration moved to sql/key_spec.h in [4]. - Remove NORMAL_PART_NAME uses because of [25]. - Break up the dependency between sql_class.h and handler.h by moving opt_log_slow_admin_statements and opt_log_slow_sp_statements from mysqld.h/mysqld.cc to log.h/log.cc, and including log.h instead of mysqld.h (which then includes handler.h) in sql_class.h. Also move struct fragmentation_stats_t declaration to system_variables.h from sql_thd_internal_api.h, put a forward declaration there, and do not include the latter header from the former. This is done due to [21]. - Remove THD::KILL_BAD_DATA uses due to [28]. - Remove THR_THD uses due to [29]. - Convert uses of dict_table_is_temporary to dict_table_t::is_temporary() due to [7]. - Remove TABLE_SHARE::uniques uses from sql_base.cc Table_Node::Table_Node due to upstream removal at [27]. It is not clear whether the resulting code is correct though. - Replace uses of SRV_LOG_SPACE_FIRST_ID with dict_sys_t::log_space_first_id due to [8] and the n dict_sys_t::s_log_space_first_id due to [9]. - Move srv_parallel_doublewrite_path initialisation to innobase_init_params to reflect InnoDB startup refactoring - Adjust btr_copy_externally_stored_field callers to call lob::btr_copy_externally_stored_field (WL#9141, [34]), and pass index definition to it (WL#8960, [33]). - Move dict_index_get_nth_col_pos patches to dict_index_t::get_col_pos because of [11]. - Adjust get-space-by-space-id code for fil_system mutex removal, sharding, and fil_node_complete_io -> complete_io rename, introduced in WL#8619 [10]. - Remove innodb_use_global_flush_log_at_trx_commit and commit innodb_flush_log_at_trx_commit back to global variable, and remove associated srv_use_global_flush_log_at_trx_commit source code variable and thd_flush_log_at_trx_commit function - Replace struct st_mysql_sys_var occurences with SYS_VAR due to [20]. - Adjust warn_or_info class for InnoDB logging changes at [5], which remove the actual logging from logger::~logger, pass diagnostics level to its constructor, and conditionally disable actual logging using UNIV_NO_ERR_MSGS preprocessor define. - Removed /*=...==*/ comments and replace /**...**//**\n with /** in the InnoDB patch according to [37]. - Replace reserved_system_space_name with dict_sys_t::sys_space_name because of [8], and then to dict_sys_t::s_sys_space_name because of [9]. Misc cleanups - Split assert(condition1 && condition2) to assert(condition1); assert(condition2); - Create type aliases for all the added function pointers in the handlerton interface. - Remove hash table size information leftovers that should have been removed in the 5.7 porting branch - Replace some of the ulint uses with space_id_t where applicable - Remove our #define SRV_MAX_N_PURGE_THREADS in favor of upstream MAX_PURGE_THREADS. - Remove remnants of innodb_flush_method = ALL_O_DIRECT: defines DEFAULT_SRV_LOG_WRITE_AHEAD_SIZE and MAX_SRV_LOG_WRITE_AHEAD_SIZE, xb_simulate_all_o_direct_write_failure, xb_simulate_all_o_direct_read_failure debug points. - Introduce flush_changed_page_bitmaps_t and purge_changed_page_bitmaps_t types for the corresponding handlerton methods. - Remove .is_percona_server_bzr - Replace README.MySQL contents with upstream README - Update man pages from mysql-8.0.11.tar.gz Testsuite These are partial changes only, as there was no goal to run the testsuite at this stage. - Remove uses of have_dynamic_loading.inc, have_symlink.inc, have_innodb.inc, have_perfschema.inc include files due to [31]. TokuDB, MyRocks, PAM authentication These three plugins are not handled in this merge yet. [1]: commit c7a720b Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Thu Sep 24 12:46:44 2015 +0100 Bug#21881278: SWITCH MYSYS LIBRARY FROM C TO C++ This patch switches all mysys/ files from .c to .cc. By doing this we get stricter compile checks (e.g. type checks) and it makes it possible to use C++ features for future bug fixes and extensions inside mysys. [2]: commit 4133e44 Author: Steinar H. Gunderson <steinar.gunderson@oracle.com> Date: Thu Sep 7 15:18:39 2017 +0200 Bug #26781567: REMOVE C LEGACY [noclose] Convert the remaining C files to C++, except for external source code (e.g. memcached) and a few .c files that are there to verify that our client library still builds from C code. The only real exception is libservices/, where ODR violations (services are defined as structs in some files and as void * in others) mean that linking with C++ linkage would break linking on Windows. Change-Id: I61f6a281ffcd5c0fca9f4b9976aeb580dd1b0b65 [3]: commit bd8c99e Author: Marc Alff <marc.alff@oracle.com> Date: Fri Jul 1 10:46:20 2011 +0200 Bug#12346211 PERF IMPACT OF PERFORMANCE-SCHEMA WITH DISABLED INSTRUMENTS/CONSUMERS Performance improvement #1: instrumentation interface. In the instrumentation interface in include/mysql/psi/psi.h, implemented the following changes. 1) Removal of "if (PSI_server != NULL)". Instead of having PSI_server == NULL when the performance schema is disabled, a dummy implementation of the instrumentation interface is provided in mysys/psi_noop.cc. Replacing a test by calling an empty function is similar for the disabled code path. For the instrumented code path (the performance schema is enabled), removing this test completely improves performances. 2) Push down of "if (that->m_psi != NULL)". Testing if an object is instrumented has been pushed down to the implementation of the instrumentation interface. In case of psi_noop.cc, this test is not needed, which improves performances for the disabled case. 3) Removal of 1 "if (locker != NULL)" test. Before the fix, the coding pattern was: locker= ... if (locker != NULL) PSI_server->start_event() <instrumented code> if (locker != NULL) PSI_server->end_event() After the fix, the coding pattern is: locker= ... if (locker != NULL) { PSI_server->start_event() <instrumented code> PSI_server->end_event() } else { <instrumented code> } This saves an extra if() in both the instrumented and non instrumented code path. 4) Packing of helper structures. Unused attributes such as m_src_file and m_src_line have been removed. Attributes have been reordered to: - avoid holes caused by padding - put attributes used very frequently first 5) Use PSI_CALL() Calls to the performance schema implementation have been changed from: PSI_server->function(); to PSI_CALL(function)(); This is cosmetic only, and can help later to make static calls to pfs_function() instead of using the PSI_server pointer. [4]: commit 3f3c259 Author: Tor Didriksen <tor.didriksen@oracle.com> Date: Thu Mar 26 16:56:31 2015 +0100 Bug#20756282 MOVE CLASS KEY AND FOREIGN_KEY OUT OF SQL_CLASS.H Move Key_part_spec, Key_spec (new name for Key) and Foreign_key_spec (new name for Foreign_key) to new files key_spec.h/cc [5]: commit 29e3723 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Feb 16 14:30:31 2018 +1100 WL#11009: Logging services: log writers: add error ID to traditional error log (InnoDB changes) Add a unique ID to all log InnoDB messages. Remove InnoDB: prefix from the pattern matching in the tests. RB#18486 Reviewed by: Ingo, Annamalai and Praveen Hulakund. [6]: commit 6c174bd Author: Steinar H. Gunderson <steinar.gunderson@oracle.com> Date: Tue Feb 21 13:39:24 2017 +0100 Bug #25597667: REMOVE MY_BOOL Replace my_bool with bool everywhere. Change-Id: Ibfe9f9b4a05d8ee82d09ffdc4f1f038ec4555b89 [7]: commit 2238b41 Author: El<C5><BC>bieta Babij <elzbieta.babij@oracle.com> Date: Tue Jul 5 12:22:26 2016 +0200 Bug#23748128 - Add some accessor methods to dict_table_t. dict_table_t::get_n_user_cols(): Get the number of user-specified columns, possibly including a hidden FTS_DOC_ID column. dict_table_t::get_n_sys_cols(): Get the number of internal system columns. dict_table_t::get_n_cols(): Get the number of all columns. dict_table_t::get_col(): Get a column. dict_table_t::get_col_name(): Get a column name. dict_table_t::get_sys_col(): Get an internal system column. [8]: commit cd7649d Author: Allen Lai <zheng.lai@oracle.com> Date: Thu Apr 13 10:59:58 2017 +0800 wl#9525 InnoDB_New_DD: update InnoDB specific metadata into new DD for DDL wl#9534 InnoDB_New_DD: Instantiate InnoDB in-memory metadata with newDD objects wl#9530 InnoDB_New_DD: FTS index support for newDD wl#9531 InnoDB_New_DD: Enable table encryption and transparent compression wl#9532 InnoDB_New_DD: InnoDB_New_DD: Move data dictionary tables to a single tablespace wl#9559 InnoDB_New_DD: Support in-place ALTER PARTITION wl#9537 InnoDB_New_DD: Support InnoDB table import/export for new DD wl#9357<C2><A0>InnoDB: Upgrade steps for new data dictionary wl#9570<C2><A0>Extend SHOW statements to list hidden columns and index information wl#9461<C2><A0>Upgrade for Innodb Internal Dictionary Migration This is the InnoDB side of work to support an unified metadata system between MySQL server and Storage Engines. It removes the InnoDB dependency on InnoDB system tables, that is, all InnoDB metadata are now read from server objects, instead of reading from InnoDB system tables. In these wls, we replaced dict_table_open_* (on_id/on_name) with their newDD counterpart. It replaced metadata instantiation (dict_load_table etc.) with newDD counterpart too. And also, we migrated all related features, like partition, encryption, compression and import/export etc. Upgrade from mysql-5.7 is supported as well. [9]: 3452648 Author: Jimmy Yang <jimmy.yang@oracle.com> AuthorDate: Thu Aug 17 00:34:24 2017 +0200 Commit: Jimmy Yang <jimmy.yang@oracle.com> CommitDate: Thu Aug 17 00:46:42 2017 +0200 Parent: f8076b3 Bug#25727892: Refactor Item::const_item() as a non-virtual function Merged: 5.5 5.6 mysql-5.7.7 Containing: 8.0 mysql-8.0.11-percona-patches mysql-8.0.3-percona-patches mysql-8.0.4-percona-patches (1 more) Follows: mysql-8.0.2 (1261) Precedes: mysql-8.0.3 (144) InnoDB NewDD worklogs(WL#9535 and WL#9536) to support Atomic DDL and completely remove old InnoDB system tables: 1. WL#9535: InnoDB_New_DD: Remove InnoDB System Table and modify the view of their I_S counterparts This worklog obsoletes all InnoDB System tables, which are no longer used and could only be accessed by upgrade. All metadata is now persisted and read from DD system tables in "mysql tablespace" instead of "old" InnoDB System Tables. All corresponding information_schema.innodb_sys_* tables are replaced by new internal system views. These new views have the name similar to the names of old system tables, just removing the 'SYS_' from the middle of the names. 2. WL#9536: InnoDB_New_DD: Support crash-safe DDL This worklog starts to support crash-safe DDL for InnoDB. DDLs like CREATE TABLE, DROP TABLE, RENAME TABLE, TRUNCATE TABLE, ALTER TABLE, CREATE TABLESPACE and DROP TABLESPACE are covered by this worklog. DDLs operations including DD system table updates and corresponding file changes(create, delete, rename etc.) are atomic. Physical data files would be synchronized with the transaction for DDL, thus there should be no physical data files for temporary tables or FTS auxiliary tables, etc. left after recovery. [10]: commit cf63b24 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Dec 8 12:10:42 2017 +1100 WL#8619 - InnoDB: Provide offline database portability without ISL files Summary of changes. 1. Introduce --innodb-directories := "dir1;...;dirN". This is renamed from --innodb-scan-directories which was introduced in WL#9499. Currently this is not dynamic. 2. Revert to 5.6 behaviour where we scan directories and find .ibd files. If you move tablespaces to a new location, add the path to innodb-data-directories so that InnoDB can "discover" the files on startup. 3. Tablespaces can only be created under known directories. This eliminates the need for .isl files. 4. Upgrade redo log format, not backward compatible 5. Change redo log type IDs 6. Code cleanup 7. Shard the fil system data structures for performance 8. Remove WL#9499 files that were used to track open tablespaces. RB#16842 Approved by: Jimmy and Kevin. [11]: commit b3ba94c Author: El<C5><BC>bieta Babij <elzbieta.babij@oracle.com> Date: Tue Jul 26 15:00:04 2016 +0200 Bug #24361023 - Add accessors to indexes. dict_index_t::add_field(): Replaces dict_mem_index_add_field(). dict_index_t::get_field(): Replaces dict_index_get_nth_field(). dict_index_t::get_col(): Replaces dict_index_get_nth_col(). dict_index_t::get_col_no(): Replaces dict_index_get_nth_col_no(). dict_index_t::get_col_pos(): Replaces dict_index_get_nth_col_pos() and dict_index_get_nth_col_or_prefix_pos(). dict_index_t::get_sys_col_pos(): Replaces dict_index_get_sys_col_pos(). RB#13487 Reviewed by Kevin Lewis [12]: commit 98bba73 Author: El<C5><BC>bieta Babij <elzbieta.babij@oracle.com> Date: Mon May 23 10:21:20 2016 +0200 Bug #23329353 - Replace ut_snprint with the C++11 snprintf function. [13]: commit 31350e8 Author: Gopal Shankar <gopal.shankar@oracle.com> Date: Tue Nov 3 13:27:06 2015 +0530 WL#6378 New data dictionary (umbrella). (...) [14]: commit f98b7bc Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Wed Jan 18 10:39:10 2017 +0100 Bug#25413703: REMOVE #IFDEF HAVE_REPLICATION After the removal of embedded server, the HAVE_REPLICATION ifdef is always true and can therefore be removed. [15]: commit 18c6270 Author: Tor Didriksen <tor.didriksen@oracle.com> Date: Tue Nov 8 09:55:35 2016 +0100 WL#9816 Do not build/package/test embedded server - Remove/Cleanup all cmake related code pertaining to embedded server - Remove/Cleanup embedded server from RMP and Debian packages - Remove embedded tests from our mtr test collections - Remove all source code in the libmysqld directory - Remove embedded code from mysql-test-run Change-Id: Ib2432e50043f060f1da64328815cdb9d53981de4 [16]: commit 0a68d5d Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Mon Jan 16 12:35:47 2017 +0100 Bug#25304844: REMOVE EMBEDDED_LIBRARY #IFDEF Remove dead code related to embedded server. Part 5: Remove EMBEDDED_LIBRARY ifdef from client/ and storage/ This patch also removes the --server-arg option from 'mysql'. This option was embedded server specific and undocumented. --server-arg and --server-file are, for the same reason, removed from 'mysqltest'. [17]: commit db3fd68 Author: Gleb Shchepa <gleb.shchepa@oracle.com> Date: Thu Dec 7 16:21:28 2017 +0400 WL#8989: Create an internal API for the MySQL parser to enumerate reserved and non-reserved words Part I. Grammar preparation. * Replace the existent type tag <symbol> with <keyword> for a better readability. * Add the new type tag <keyword> to currently untyped terminal and non-terminal symbols. Part II. Create a simple compile-time program for transforming sql_yacc.yy into a .h file that contains an array of reserved and non-reserved words. * Find %token declarations with the <keyword> type tag by a regexp. * Separate %type declarations are ignored for the simplicity. Part III. Introduce a new system view: INFORMATION_SCHEMA.keywords. * Hardcode the statical list of keywords and reserved words into a view on top of a long JSON_TABLE(...) expression. * Use a trick in the .test file to trace changes in keywords: notify the developer to update the target I_S version in the mysqld binary where necessary -- this will force DD to refresh I_S view declarations on the next run. [18]: commit a2f3a60 Author: Marc Alff <marc.alff@oracle.com> Date: Thu Mar 10 10:10:25 2016 +0100 Bug#22906606 IMPROVE DIGEST STABILITY Before this fix, adding new tokens in the parser caused query digests to change. The root cause was that tokens are typically added in the middle of the list, causing renumbering of tokens. With this fix, the process is now changed and documented: new tokens should be added at the end of the list. The tool gen_lex_token, used during the build, is improved to enforce this process. [19]: commit f9d1c83 Author: Tatiana Azundris Nuernberg <tatjana.nuernberg@oracle.com> Date: Sat Mar 3 13:31:37 2018 +0000 WL#11150: Add <component> to error log messages by default Adds another key/value pair to log events. The new field "subsystem" will contain one of the values "InnoDB" (for errors thrown in the InnoDB storage engine proper), "Repl" (for those thrown by the replication subsystem), or "Server" (otherwise). This value is also added to the traditional log. This field will appear between '[' <error_number> ']' and '[' <error_text> ']'. [20]: commit 545b168 Author: Steinar H. Gunderson <steinar.gunderson@oracle.com> Date: Wed Sep 13 12:57:13 2017 +0200 Bug #26781567: REMOVE C LEGACY Replace the final typedefs to st_* and *_st structs. Note in particular that st_mysql_show_var was called sometimes SHOW_VAR and sometimes STATUS_VAR (by way of duplicated forward-declaring typedefs). This patch changes it so that it's universally SHOW_VAR, which seems to match better with the original type name. Change-Id: Ide483ceac31284d909c42adb46496308dd4d4f85 [21]: commit f58cb36 Author: Steinar H. Gunderson <steinar.gunderson@oracle.com> Date: Tue Oct 10 14:54:53 2017 +0200 Bug #26927386: REDUCE WEIGHT OF SQL_CLASS.H [noclose] Remove field.h, handler.h and sql_profile.h from sql_class.h, and add a simple test to make sure they don't get back. Takes sql_class.h down from approx 1.0 to 0.85 seconds (GCC 7.2.0, -O2 -g). Change-Id: I72618c977c7af330d1215899971e2f9e49ab7b88 [22]: commit be029db Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Thu Jan 19 15:14:58 2017 +0100 Bug#25428042: REPLACE MYSQL_CLIENT IFDEF WITH !MYSQL_SERVER Replace the MYSQL_CLIENT #ifdef with !MYSQL_SERVER to simplify conditional compilation. Also remove unnecessary usage of MYSQL_SERVER and move setting of MYSQL_SERVER from C++ to CMake. [23]: commit f5c32ea Author: Gleb Shchepa <gleb.shchepa@oracle.com> Date: Thu Jul 14 21:48:55 2016 +0400 WL#8067: True bottom-up server parser: refactoring of the CREATE TABLE statement WL#8434: True bottom-up server parser: refactoring of partitioning-related stuff WL#8435: True bottom-up server parser: cleanup and refactoring column definition stuff WL#8433: Separate DD commands from regular SQL queries in the parser grammar WL#7840: Allow parsing a single expression [24]: commit ed007c8 Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Thu Dec 22 11:42:43 2016 +0100 Bug#25304844: REMOVE EMBEDDED_LIBRARY #IFDEF Remove dead code related to embedded server. Part 3: Remove NO_EMBEDDED_ACCESS_CHECKS ifdef [25]: commit 11d6946 Author: Dmitry Lenev <dmitry.lenev@oracle.com> Date: Fri Apr 21 09:52:09 2017 +0300 Removed old partitioning DDL implementation which is no longer in use. After WL#8971 "Deprecate and remove partition engine" and WL#9559 "InnoDB_New_DD: Support in-place ALTER PARTITION" code implementing "fast" partitioning DDL (fast_alter_partition_table() and related functions) became unused. This patch removes this dead code (including DDL_LOG implementation used by it) as a follow-up to the above WLs. [26]: commit 5fecdfa Author: Steinar H. Gunderson <steinar.gunderson@oracle.com> Date: Thu Sep 21 14:40:20 2017 +0200 Bug #26030132: REMOVE SQL_ALLOC Finally remove Sql_alloc, now that 8.0 has been branched off. This is a no-op change for all existing code; there is no code that does new Foo() inheriting from Sql_alloc, since that overload doesn't exist, and similarly no code that deletes such objects, since operator delete is private. After this, it is possible to write such code, and it will do the obvious thing (allocate on the regular heap, and destroy similarly). As usual, code that wants to allocate on a MEM_ROOT must say so explicitly by use of overloaded new. Change-Id: I2e98b39daa06d6dfd8b8c5b1d45a85b1ddc02b3b [27]: commit 4880f97 Author: Guilhem Bichot <guilhem.bichot@oracle.com> Date: Tue Nov 29 15:46:27 2016 +0100 WL#883 non-recursive CTE WL#3634 recursive CTE (...) [28]: commit 597a259 Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Tue Oct 6 14:04:03 2015 +0100 Bug#21944866: REFACTORING: REMOVE KILL_BAD_DATA Remove KILL_BAD_DATA from the THD::killed_state enum. KILL_BAD_DATA was only set by the connection itself when a warning was upgraded to error by STRICT mode. It therefore had nothing to do with the connection being killed by other connections (e.g. KILL QUERY / KILL CONNECTION). This is a refactoring with no changes in behavior. [29]: commit 2bf852b Author: Steinar H. Gunderson <steinar.gunderson@oracle.com> Date: Wed Apr 12 12:26:14 2017 +0200 Bug #25909342: USE C++11 THREAD-LOCAL STORAGE FOR CURRENT_THD Our current implementation of current_thd uses mysys' my_thread_* API, which maps onto pthreads (at least for Linux/UNIX). C++11 supports native TLS, which means we just need two simple MOV instructions to do a TLS fetch instead of a full shared library call; the interface is also much simpler. We probably don't have a lot of performance gain, since we don't pick out current_thd all that often, but it's a nice cleanup to do nevertheless. There are still some uses of the my_thread_* API, particularly in PFS; cleaning this up is for a later patch. Change-Id: Ia16e0c1ccfa52af815882bcddd1eb3be80f77c61 [30]: commit 162e016 Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Tue Mar 15 14:32:39 2016 +0100 Bug#22948828: REFACTOR USAGE OF MY_MICRO_TIME() Remove redundant calls to my_micro_time() to simplify code and reduce the usage of gettimeofdat(). Especially focused on code for handling new connections. sql/conn_handler/connection_handler_one_thread.cc - Removed my_micro_time() call. THD::start_utime has just been set by THD::THD() => THD::init(). THD::thr_create_utime is only needed by connection_handler_per_thread.cc sql/conn_handler/connection_handler_per_thread.cc - Removed my_micro_time() call. THD::start_utime has just been set by THD::THD() => THD::init(). Remove THD::thr_create_utime, we can use THD::start_utime instead. sql/event_queue.cc - Replaced usage of THD::set_current_time() with THD::set_time(). THD::set_time() does the same as long as THD::user_time is NULL which it is for events. Allows the removal of THD::set_current_time() sql/events.cc - Removed THD::set_time() call. THD::set_time() has just been called by THD::THD() => THD::init(). sql/log.cc - Removed default argument value for make_iso8601_timestamp() for clarity. - Reduced critical section of LOCK_logger. - Call my_micro_time() directly instead of the unnecessary THD::current_utime() wrapper. sql/log_event.cc - Use the existing THD::query_start() function instead of accessing THD::start_time directly. - Renamed THD::update_server_status() to THD::check_slow_query() since the old name was somewhat misleading. sql/sql_class.h - Removed THD::thr_create_utime, no longer needed. - Removed redundant inline keyword, added const where possible. - Removed THD::query_start_usec(), not used. - Removed THD::query_start_timeval(), no longer needed. - Removed THD::set_current_time(), no longer needed. - Removed THD::is_valid_time(), code moved to sql_parse.cc - Renamed THD::update_server_status() to THD::check_slow_query() since the old name was somewhat misleading. sql/sql_connect.cc - Use THD::start_utime instead of removed THD::thd_create_utime (had the same value) - Removed call to THD::set_time(). Called by THD::init_for_queries() right afterwards anyway. sql/sql_thd_internal_api.cc - Removed THD::set_time() call. THD::set_time() has just been called by THD::THD() => THD::init(). - Removed call to my_micro_time(). Already done by THD::set_time(). [31] commit 485545b Author: Mohit Joshi <mohit.joshi@oracle.com> Date: Tue Jan 17 11:56:43 2017 +0530 Bug#25166686::REMOVE ALL THE REDUNDANT .INC FILES FROM MTR Description: It is noticed that there are a lot of redundant inc files which are not being used by any of the MTR tests. Also, there are several inc files which are included in the test but are not needed anymore. Fix: Below inc files are not being used by any test and hence removed: 1. include/check_key_reads.inc 2. include/check_key_req.inc 3. include/commandline_option_test.inc 4. include/config_file_option_test.inc 5. include/explain.inc 6. include/finish_option_test.inc 7. include/have_binlog_checksum_off.inc 8. include/have_dynamic_loading.inc 9. include/have_shm.inc 10. include/init_option_test.inc 11. include/not_blackhole.inc 12. include/show_partition_checksums.inc 13. include/show_partition_contents.inc 14. include/test_outfile.inc Below are a few inc files which are used in MTR tests but are no more needed. Hence these inc files along with their references from all MTR tests have been removed. 1. have_symlink.inc 2. have_perfschema.inc 3. have_innodb.inc Note: include/not_valgrind_server.inc is not being used by any MTR test but still not removed as this was introduced as part of Bug#22447330 Reviewed by: Parveez Baig <parveez.baig@oracle.com> Deepa Dixit <deepa.dixit@oracle.com> [32] commit a0d0cab Author: Sivert Sorumgard <sivert.sorumgaard@oracle.com> Date: Thu Jun 30 16:00:31 2016 +0200 WL#8971: Deprecate and remove partitioning storage engine This worklog implements the following main changes: 1. Modify cmake files, do not build partitioning engine. 2. Remove files implementing the partitioning engine. 3. Remove some mtr tests using non-native partitioning. 4. Rewrite some mtr tests to use native partitioning. Various adjustments necessary, e.g. regarding EXPLAIN output. 5. Introduce error handling at the SQL layer to fail if using partitioning related DDL statements for engines not supporting it natively. 6. Remove reference to the DB_TYPE_PARTITIONING_DB internally, but keep the enum entry to avoid disturbing other values. 7. Add an mtr test testing the error handling. 8. Add an mtr test to check for regressions in support for partitioning in ndb. 9. Copy comments from storage/partition/ha_partition.* to sql/handler.*, include/my_base.h etc. [33] commit eda9201 Author: Annamalai Gurusami <annamalai.gurusami@oracle.com> Date: Thu Apr 28 12:29:00 2016 +0530 WL#9141 InnoDB: Refactor uncompressed BLOB code to facilitate partial fetch/update (...) [34] commit eda9201 Author: Annamalai Gurusami <annamalai.gurusami@oracle.com> Date: Thu Apr 28 12:29:00 2016 +0530 WL#9141 InnoDB: Refactor uncompressed BLOB code to facilitate partial fetch/update (...) [35] commit 2eec7f3 Author: Marko M<C3><A4>kel<C3><A4> <marko.makela@oracle.com> Date: Fri Apr 22 10:50:45 2016 +0200 Bug#23150562 REMOVE UNIV_MUST_NOT_INLINE AND UNIV_NONINL Problem: InnoDB used to support a compilation mode that allowed to choose whether the function definitions in .ic files are to be inlined or not. This stopped making sense when InnoDB moved to C++ in MySQL 5.6 (and ha_innodb.cc started to #include .ic files), and more so in MySQL 5.7 when inline methods and functions were introduced in .h files. Solution: Remove all references to UNIV_NONINL and UNIV_MUST_NOT_INLINE from all files, assuming that the symbols are never defined. Remove the files fut0fut.cc and ut0byte.cc which only mattered when UNIV_NONINL was defined. RB: 12470 Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com> [36]: commit 644aafd Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue May 24 19:02:55 2016 +0530 WL#9359 - Code cleanup [37]: commit 59b560e Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Feb 6 22:00:10 2018 +1100 Bug#27496251 REMOVE /*====*/ AND CHANGE /****/ /***/ TO /** .... */

Source tree changes - Remove include/my_global.h from .gitignore - it should be visible, especially when merging from 5.7 (where it's present) to 8.0 (where it's not). This partially fixes bug https://bugs.mysql.com/bug.php?id=91834 (Cleanup .gitignore, e.g. include/my_global.h) CMake / preprocessor symbol changes - Make CMake option WITH_SYSTEM_LIBS configure -DWITH_READLINE=system instead of -DWITH_EDITLINE=system - Remove WITH_SSL CMake settings from build scripts, leaving the implicit default of -DWITH_SSL=system, because the alternative -DWITH_SSL=wolfssl requires more work and is broken by upstream in 8.0.11. - Use MYSQL_ADD_EXECUTABLE instead of ADD_EXECUTABLE in CMake scripts. - Remove HAVE_REPLICATION from diff to match upstream [14]. - Remove WITH_EMBEDDED_SERVER, EMBEDDED_LIBRARY, NO_EMBEDDED_ACCESS_CHECKS, mysql-test/include/not_embedded.inc mentions due to upstream [15], [16], [24] etc. - Remove WITH_PARTITION_STORAGE_ENGINE due to upstream [32]. - Replace MYSQL_CLIENT preprocessor macro uses with !MYSQL_SERVER due to [22]. - Add sql/binlog_crypt_data.cc and sql/event_crypt.cc as mysqlbinlog source files in CMake, to replace the former practice of including them from source files. - Remove setting C++98/03 and C++11 detection from CMake due to upstream converting to C++11 in WL#8896. This includes removing cmake/check_stdcxx11.cmake and its uses from TokuDB / MyRocks. - Make CMake define __STDC_WANT_LIB_EXT1__=1 on macOS to get memset_s. - Define LOG_COMPONENT_TAG for various plugins and unit tests, to accomodate WL#11150 [19]. - Remove uses of UNIV_NONINL due to [35], consequently fold buf0dblwr.ic to buf0dblwr.h. - Revmove UNIV_DEBUG_THREAD_CREATION due to WL#9359 [36] Compilation changes - Fix now-enabled-by-default unused parameter warnings, by removing the unused parameters, or by marking them unused. C to C++ conversion - Convert mysys/system_key.c to C++ mysys/system_key.cc to match upstream [1]. - Convert storage/heap/hp_record.c and storage/heap/hp_dspace.c to C++ hp_record.cc and hp_dspace.cc respectively to match upstream [2]. - Mechanical renames, changes, and partial conversions, incomplete list: - my_bool/FALSE/TRUE/Sys_var_mybool uses to bool/false/true/Sys_var_bool due to upstream my_bool removal at [6]. - #define to [static] const constexpr - Declare local variables at first use - Pointer arguments to const reference ones, where applicable - Convert typedef struct declaration to plain struct, removed redundant struct tags. - NULL/0 to nullptr - compile_time_assert to static_assert - various ad-hoc macros to std::min - various integer types/0/1 to bool/false/true where applicable - Convert mysqldump.cc patches to C++11, remove uses of HASH, make key_type_t an enum class instead of enum. At the same time replace uses of LIST with std::list and std::forward_list, although this was not strictly required. - Remove C_MODE_START/C_MODE_END and extern "C" blocks from C++-only headers. C++03 to C++11 conversion - Replace HASH with std::unordered_map in many patches, because upstream removed HASH in series of commits for "25997748". - Mechanical renames and changes, incomplete list: - ut_snprintf uses to snprintf due to upstream removal of the former at [12]. - my_snprintf to snprintf due to C++11 guaranteeing the presence of the latter. - Applied noexcept to matching functions and methods. - Applied final to matching classes. - boost::movelib::unique_ptr to std::unique_ptr. - insert to emplace for standard containers, where applicable - converted some of std::map uses to std::unordered_map, if ordering is not required - std::auto_ptr to std::unique_ptr - Assignments in constructors and constructor initializer lists to default member initializers. - UNIV_THREAD_LOCAL to thread_local - Replaced uses of compiler builtins with std::atomic for atomic variables. User statistics - Due to HASH replacement with modern C++ collections, cleanup userstat to store objects directly instead of heap-allocated pointers to them. For this, add constructors to TABLE_STATS and INDEX_STATS to replace init_user_stats and init_thread_stats functions. Binlog encryption - Adjust binlog encryption patches for MYSQL_BIN_LOG "append" to "write" rename in cass methods. Parser - Put Percona tokens at the end of the token list due to upstream attempt to ensure token digest value stability ([17], [18]). - Because of upstream bottom-up parser refactoring for CREATE TABLE [23], which includes column attributes, rewrite TokuDB clustered index and InnoDB compression dictionary support: - add apply_zip_dict method to PT_column_attr_base class; - add zip_dict_name arg to the constructor and m_zip_dict_name field to PT_column_format_column_attr class, implement apply_zip_dict; - add m_zip_dict field to PT_field_def_base, get its value in contextualize_attrs method, and pass it to Alter_info instance in PT_column_def::contextualize and PT_alter_table_change_column::contextualize - rename PT_unique_key_column_attr to PT_unique_combo_clustering_key_column_attr, make it handle both UNIQUE and CLUSTERING - move zip_dict_name from LEX to HA_CREATE_INFO in line with the upstream LEX reduction in parser refactoring. - Typedef new alter tablespace option PT_alter_tablespace_option_encryption for general tablespace encryption. Performance Schema - Adjust PFS object declarations for the new volatility and documentation fields, account for PSI_FLAG_GLOBAL rename to PSI_FLAG_SINGLETON. - Move server PSF memory key declarations to sql/psi_memory_key.h and sql/psi_memory_key.cc - Adjust inline_mysql_unix_connect PFS instrumentation in include/mysql/psi/mysql_file.h to modern PFS instrumentation pattern which was introduced back in [3] because the old one stopped compiling in 8.0. While at that, switch from HAVE_PSI_INTERFACE to HAVE_PSI_FILE_INTERFACE. Audit log plugin - Converted the plugin to C++. - Made a lot of declarations in audit_log.cc static and/or const. - Converted hash tables in filter.cc to collation_unordered_set<std::string>. At the same time convert the hash table value struct creator/initialization functions to constructors. Backup locks - Remove everything associated with LOCK BINLOG FOR BACKUP / UNLOCK BINLOG due to upstream performance_schema.log_status table. - Refer to Oracle backup lock as "instance backup lock" and Percona backup lock as "table backup lock" to try to reduce confusion between the two. Expanded fast index creation - Change the type of delayed_key_list container in Alter_info from List to Mem_root_array. Threadpool - Re-implement TP_STATISTICS::num_worker_threads, pool_timer_t::current_microtime and next_timeout_check using C++11 atomics. - Convert Worker_thread_context struct to class with constructor/destructor instead of save/restore methods, to be used as RAII. New data dictionary - Percona features that need to store information in the new data dictionary (CLUSTERED keys for TokuDB; compression dictionary information for InnoDB have not been re-implemented for the new DD yet). For now the legacy implementation is disabled, to be removed together with the new implementation, except for the reading bits, because they are needed for upgrades from 5.7. Specifically, leave SYSTEM_TABLE_NAME patch in, dict_process_sys_zip_dict function Changed page tracking - Move fil_name_process patch not to cause any tablespace in filesystem changes when called by redo log tracking to fil_tablespace_redo_delete and fil_tablespace_redo_create. - Introduce apply arg for fil_tablespace_redo_encryption for redo log parsing mode only. - log_get_tracked_lsn and log_set_tracked_lsn have been remvoed after the tracked LSN conversion to C++11 atomic variable. - Fold log_check_tracking_margin to log0write.cc:log_write_write_buffer. - Move redo log signalling from log_io_complete_checkpoint to log0chkp.cc:log_checkpoint. - Drop log_group_read_log_seg release_mutex arg (which would have been moved to recv_read_log_seg) because of log_sys->mutex removal. - innobase_logs_empty_and_mark_at_shutdown patches moved to srv_log_shutdown. - Due to os_file_dir_t and associated function removals in [13], rewrite their users in log0online.cc to use MY_DIR and my_dir from mysys. - Remove log_group_t mentions from log0online.cc - Add handling for MLOG_COMP_PAGE_CREATE_SDI, MLOG_PAGE_CREATE_SDI, and MLOG_TABLE_DYNAMIC_META MTR types. - Adjust redo log reading code for log_group_read_log_seg rename to recv_read_log_seg, convert the latter from static to public function. - Refactor out redo log tracker-waiting code in fil0fil.cc to a new private static method Fil_system::wait_for_changed_page_tracker. Other changes due to upstream - Remove YaSSL-related source code (mysys_ssl/yassl.cpp, HAVE_YASSL check, build-binary.sh --with-yassl option) - Replace my_global.h includes with the needed smaller ones (my_inttypes.h, mysql/service_mysql_alloc.h, my_macros.h) due to upstream patch series "Bug #24710065: SPLIT MY_GLOBAL.H", and move my_global.h patches to appropriate places: s6_addr32 define to violite.h. - Remove THD::thr_create_utime uses and replace THD::update_server_status calls with THD::update_slow_query_status due to [30]. - Remove Sql_alloc uses due to [26]. - Adjust for class Key rename to Key_spec with declaration moved to sql/key_spec.h in [4]. - Remove NORMAL_PART_NAME uses because of [25]. - Break up the dependency between sql_class.h and handler.h by moving opt_log_slow_admin_statements and opt_log_slow_sp_statements from mysqld.h/mysqld.cc to log.h/log.cc, and including log.h instead of mysqld.h (which then includes handler.h) in sql_class.h. Also move struct fragmentation_stats_t declaration to system_variables.h from sql_thd_internal_api.h, put a forward declaration there, and do not include the latter header from the former. This is done due to [21]. - Remove THD::KILL_BAD_DATA uses due to [28]. - Remove THR_THD uses due to [29]. - Convert uses of dict_table_is_temporary to dict_table_t::is_temporary() due to [7]. - Remove TABLE_SHARE::uniques uses from sql_base.cc Table_Node::Table_Node due to upstream removal at [27]. It is not clear whether the resulting code is correct though. - Replace uses of SRV_LOG_SPACE_FIRST_ID with dict_sys_t::log_space_first_id due to [8] and the n dict_sys_t::s_log_space_first_id due to [9]. - Move srv_parallel_doublewrite_path initialisation to innobase_init_params to reflect InnoDB startup refactoring - Adjust btr_copy_externally_stored_field callers to call lob::btr_copy_externally_stored_field (WL#9141, [34]), and pass index definition to it (WL#8960, [33]). - Move dict_index_get_nth_col_pos patches to dict_index_t::get_col_pos because of [11]. - Adjust get-space-by-space-id code for fil_system mutex removal, sharding, and fil_node_complete_io -> complete_io rename, introduced in WL#8619 [10]. - Remove innodb_use_global_flush_log_at_trx_commit and commit innodb_flush_log_at_trx_commit back to global variable, and remove associated srv_use_global_flush_log_at_trx_commit source code variable and thd_flush_log_at_trx_commit function - Replace struct st_mysql_sys_var occurences with SYS_VAR due to [20]. - Adjust warn_or_info class for InnoDB logging changes at [5], which remove the actual logging from logger::~logger, pass diagnostics level to its constructor, and conditionally disable actual logging using UNIV_NO_ERR_MSGS preprocessor define. - Removed /*=...==*/ comments and replace /**...**//**\n with /** in the InnoDB patch according to [37]. - Replace reserved_system_space_name with dict_sys_t::sys_space_name because of [8], and then to dict_sys_t::s_sys_space_name because of [9]. Misc cleanups - Split assert(condition1 && condition2) to assert(condition1); assert(condition2); - Create type aliases for all the added function pointers in the handlerton interface. - Remove hash table size information leftovers that should have been removed in the 5.7 porting branch - Replace some of the ulint uses with space_id_t where applicable - Remove our #define SRV_MAX_N_PURGE_THREADS in favor of upstream MAX_PURGE_THREADS. - Remove remnants of innodb_flush_method = ALL_O_DIRECT: defines DEFAULT_SRV_LOG_WRITE_AHEAD_SIZE and MAX_SRV_LOG_WRITE_AHEAD_SIZE, xb_simulate_all_o_direct_write_failure, xb_simulate_all_o_direct_read_failure debug points. - Introduce flush_changed_page_bitmaps_t and purge_changed_page_bitmaps_t types for the corresponding handlerton methods. - Remove .is_percona_server_bzr - Replace README.MySQL contents with upstream README - Update man pages from mysql-8.0.11.tar.gz Testsuite These are partial changes only, as there was no goal to run the testsuite at this stage. - Remove uses of have_dynamic_loading.inc, have_symlink.inc, have_innodb.inc, have_perfschema.inc include files due to [31]. TokuDB, MyRocks, PAM authentication These three plugins are not handled in this merge yet. [1]: commit c7a720b Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Thu Sep 24 12:46:44 2015 +0100 Bug#21881278: SWITCH MYSYS LIBRARY FROM C TO C++ This patch switches all mysys/ files from .c to .cc. By doing this we get stricter compile checks (e.g. type checks) and it makes it possible to use C++ features for future bug fixes and extensions inside mysys. [2]: commit 4133e44 Author: Steinar H. Gunderson <steinar.gunderson@oracle.com> Date: Thu Sep 7 15:18:39 2017 +0200 Bug #26781567: REMOVE C LEGACY [noclose] Convert the remaining C files to C++, except for external source code (e.g. memcached) and a few .c files that are there to verify that our client library still builds from C code. The only real exception is libservices/, where ODR violations (services are defined as structs in some files and as void * in others) mean that linking with C++ linkage would break linking on Windows. Change-Id: I61f6a281ffcd5c0fca9f4b9976aeb580dd1b0b65 [3]: commit bd8c99e Author: Marc Alff <marc.alff@oracle.com> Date: Fri Jul 1 10:46:20 2011 +0200 Bug#12346211 PERF IMPACT OF PERFORMANCE-SCHEMA WITH DISABLED INSTRUMENTS/CONSUMERS Performance improvement #1: instrumentation interface. In the instrumentation interface in include/mysql/psi/psi.h, implemented the following changes. 1) Removal of "if (PSI_server != NULL)". Instead of having PSI_server == NULL when the performance schema is disabled, a dummy implementation of the instrumentation interface is provided in mysys/psi_noop.cc. Replacing a test by calling an empty function is similar for the disabled code path. For the instrumented code path (the performance schema is enabled), removing this test completely improves performances. 2) Push down of "if (that->m_psi != NULL)". Testing if an object is instrumented has been pushed down to the implementation of the instrumentation interface. In case of psi_noop.cc, this test is not needed, which improves performances for the disabled case. 3) Removal of 1 "if (locker != NULL)" test. Before the fix, the coding pattern was: locker= ... if (locker != NULL) PSI_server->start_event() <instrumented code> if (locker != NULL) PSI_server->end_event() After the fix, the coding pattern is: locker= ... if (locker != NULL) { PSI_server->start_event() <instrumented code> PSI_server->end_event() } else { <instrumented code> } This saves an extra if() in both the instrumented and non instrumented code path. 4) Packing of helper structures. Unused attributes such as m_src_file and m_src_line have been removed. Attributes have been reordered to: - avoid holes caused by padding - put attributes used very frequently first 5) Use PSI_CALL() Calls to the performance schema implementation have been changed from: PSI_server->function(); to PSI_CALL(function)(); This is cosmetic only, and can help later to make static calls to pfs_function() instead of using the PSI_server pointer. [4]: commit 3f3c259 Author: Tor Didriksen <tor.didriksen@oracle.com> Date: Thu Mar 26 16:56:31 2015 +0100 Bug#20756282 MOVE CLASS KEY AND FOREIGN_KEY OUT OF SQL_CLASS.H Move Key_part_spec, Key_spec (new name for Key) and Foreign_key_spec (new name for Foreign_key) to new files key_spec.h/cc [5]: commit 29e3723 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Feb 16 14:30:31 2018 +1100 WL#11009: Logging services: log writers: add error ID to traditional error log (InnoDB changes) Add a unique ID to all log InnoDB messages. Remove InnoDB: prefix from the pattern matching in the tests. RB#18486 Reviewed by: Ingo, Annamalai and Praveen Hulakund. [6]: commit 6c174bd Author: Steinar H. Gunderson <steinar.gunderson@oracle.com> Date: Tue Feb 21 13:39:24 2017 +0100 Bug #25597667: REMOVE MY_BOOL Replace my_bool with bool everywhere. Change-Id: Ibfe9f9b4a05d8ee82d09ffdc4f1f038ec4555b89 [7]: commit 2238b41 Author: El<C5><BC>bieta Babij <elzbieta.babij@oracle.com> Date: Tue Jul 5 12:22:26 2016 +0200 Bug#23748128 - Add some accessor methods to dict_table_t. dict_table_t::get_n_user_cols(): Get the number of user-specified columns, possibly including a hidden FTS_DOC_ID column. dict_table_t::get_n_sys_cols(): Get the number of internal system columns. dict_table_t::get_n_cols(): Get the number of all columns. dict_table_t::get_col(): Get a column. dict_table_t::get_col_name(): Get a column name. dict_table_t::get_sys_col(): Get an internal system column. [8]: commit cd7649d Author: Allen Lai <zheng.lai@oracle.com> Date: Thu Apr 13 10:59:58 2017 +0800 wl#9525 InnoDB_New_DD: update InnoDB specific metadata into new DD for DDL wl#9534 InnoDB_New_DD: Instantiate InnoDB in-memory metadata with newDD objects wl#9530 InnoDB_New_DD: FTS index support for newDD wl#9531 InnoDB_New_DD: Enable table encryption and transparent compression wl#9532 InnoDB_New_DD: InnoDB_New_DD: Move data dictionary tables to a single tablespace wl#9559 InnoDB_New_DD: Support in-place ALTER PARTITION wl#9537 InnoDB_New_DD: Support InnoDB table import/export for new DD wl#9357<C2><A0>InnoDB: Upgrade steps for new data dictionary wl#9570<C2><A0>Extend SHOW statements to list hidden columns and index information wl#9461<C2><A0>Upgrade for Innodb Internal Dictionary Migration This is the InnoDB side of work to support an unified metadata system between MySQL server and Storage Engines. It removes the InnoDB dependency on InnoDB system tables, that is, all InnoDB metadata are now read from server objects, instead of reading from InnoDB system tables. In these wls, we replaced dict_table_open_* (on_id/on_name) with their newDD counterpart. It replaced metadata instantiation (dict_load_table etc.) with newDD counterpart too. And also, we migrated all related features, like partition, encryption, compression and import/export etc. Upgrade from mysql-5.7 is supported as well. [9]: 3452648 Author: Jimmy Yang <jimmy.yang@oracle.com> AuthorDate: Thu Aug 17 00:34:24 2017 +0200 Commit: Jimmy Yang <jimmy.yang@oracle.com> CommitDate: Thu Aug 17 00:46:42 2017 +0200 Parent: f8076b3 Bug#25727892: Refactor Item::const_item() as a non-virtual function Merged: 5.5 5.6 mysql-5.7.7 Containing: 8.0 mysql-8.0.11-percona-patches mysql-8.0.3-percona-patches mysql-8.0.4-percona-patches (1 more) Follows: mysql-8.0.2 (1261) Precedes: mysql-8.0.3 (144) InnoDB NewDD worklogs(WL#9535 and WL#9536) to support Atomic DDL and completely remove old InnoDB system tables: 1. WL#9535: InnoDB_New_DD: Remove InnoDB System Table and modify the view of their I_S counterparts This worklog obsoletes all InnoDB System tables, which are no longer used and could only be accessed by upgrade. All metadata is now persisted and read from DD system tables in "mysql tablespace" instead of "old" InnoDB System Tables. All corresponding information_schema.innodb_sys_* tables are replaced by new internal system views. These new views have the name similar to the names of old system tables, just removing the 'SYS_' from the middle of the names. 2. WL#9536: InnoDB_New_DD: Support crash-safe DDL This worklog starts to support crash-safe DDL for InnoDB. DDLs like CREATE TABLE, DROP TABLE, RENAME TABLE, TRUNCATE TABLE, ALTER TABLE, CREATE TABLESPACE and DROP TABLESPACE are covered by this worklog. DDLs operations including DD system table updates and corresponding file changes(create, delete, rename etc.) are atomic. Physical data files would be synchronized with the transaction for DDL, thus there should be no physical data files for temporary tables or FTS auxiliary tables, etc. left after recovery. [10]: commit cf63b24 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Dec 8 12:10:42 2017 +1100 WL#8619 - InnoDB: Provide offline database portability without ISL files Summary of changes. 1. Introduce --innodb-directories := "dir1;...;dirN". This is renamed from --innodb-scan-directories which was introduced in WL#9499. Currently this is not dynamic. 2. Revert to 5.6 behaviour where we scan directories and find .ibd files. If you move tablespaces to a new location, add the path to innodb-data-directories so that InnoDB can "discover" the files on startup. 3. Tablespaces can only be created under known directories. This eliminates the need for .isl files. 4. Upgrade redo log format, not backward compatible 5. Change redo log type IDs 6. Code cleanup 7. Shard the fil system data structures for performance 8. Remove WL#9499 files that were used to track open tablespaces. RB#16842 Approved by: Jimmy and Kevin. [11]: commit b3ba94c Author: El<C5><BC>bieta Babij <elzbieta.babij@oracle.com> Date: Tue Jul 26 15:00:04 2016 +0200 Bug #24361023 - Add accessors to indexes. dict_index_t::add_field(): Replaces dict_mem_index_add_field(). dict_index_t::get_field(): Replaces dict_index_get_nth_field(). dict_index_t::get_col(): Replaces dict_index_get_nth_col(). dict_index_t::get_col_no(): Replaces dict_index_get_nth_col_no(). dict_index_t::get_col_pos(): Replaces dict_index_get_nth_col_pos() and dict_index_get_nth_col_or_prefix_pos(). dict_index_t::get_sys_col_pos(): Replaces dict_index_get_sys_col_pos(). RB#13487 Reviewed by Kevin Lewis [12]: commit 98bba73 Author: El<C5><BC>bieta Babij <elzbieta.babij@oracle.com> Date: Mon May 23 10:21:20 2016 +0200 Bug #23329353 - Replace ut_snprint with the C++11 snprintf function. [13]: commit 31350e8 Author: Gopal Shankar <gopal.shankar@oracle.com> Date: Tue Nov 3 13:27:06 2015 +0530 WL#6378 New data dictionary (umbrella). (...) [14]: commit f98b7bc Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Wed Jan 18 10:39:10 2017 +0100 Bug#25413703: REMOVE #IFDEF HAVE_REPLICATION After the removal of embedded server, the HAVE_REPLICATION ifdef is always true and can therefore be removed. [15]: commit 18c6270 Author: Tor Didriksen <tor.didriksen@oracle.com> Date: Tue Nov 8 09:55:35 2016 +0100 WL#9816 Do not build/package/test embedded server - Remove/Cleanup all cmake related code pertaining to embedded server - Remove/Cleanup embedded server from RMP and Debian packages - Remove embedded tests from our mtr test collections - Remove all source code in the libmysqld directory - Remove embedded code from mysql-test-run Change-Id: Ib2432e50043f060f1da64328815cdb9d53981de4 [16]: commit 0a68d5d Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Mon Jan 16 12:35:47 2017 +0100 Bug#25304844: REMOVE EMBEDDED_LIBRARY #IFDEF Remove dead code related to embedded server. Part 5: Remove EMBEDDED_LIBRARY ifdef from client/ and storage/ This patch also removes the --server-arg option from 'mysql'. This option was embedded server specific and undocumented. --server-arg and --server-file are, for the same reason, removed from 'mysqltest'. [17]: commit db3fd68 Author: Gleb Shchepa <gleb.shchepa@oracle.com> Date: Thu Dec 7 16:21:28 2017 +0400 WL#8989: Create an internal API for the MySQL parser to enumerate reserved and non-reserved words Part I. Grammar preparation. * Replace the existent type tag <symbol> with <keyword> for a better readability. * Add the new type tag <keyword> to currently untyped terminal and non-terminal symbols. Part II. Create a simple compile-time program for transforming sql_yacc.yy into a .h file that contains an array of reserved and non-reserved words. * Find %token declarations with the <keyword> type tag by a regexp. * Separate %type declarations are ignored for the simplicity. Part III. Introduce a new system view: INFORMATION_SCHEMA.keywords. * Hardcode the statical list of keywords and reserved words into a view on top of a long JSON_TABLE(...) expression. * Use a trick in the .test file to trace changes in keywords: notify the developer to update the target I_S version in the mysqld binary where necessary -- this will force DD to refresh I_S view declarations on the next run. [18]: commit a2f3a60 Author: Marc Alff <marc.alff@oracle.com> Date: Thu Mar 10 10:10:25 2016 +0100 Bug#22906606 IMPROVE DIGEST STABILITY Before this fix, adding new tokens in the parser caused query digests to change. The root cause was that tokens are typically added in the middle of the list, causing renumbering of tokens. With this fix, the process is now changed and documented: new tokens should be added at the end of the list. The tool gen_lex_token, used during the build, is improved to enforce this process. [19]: commit f9d1c83 Author: Tatiana Azundris Nuernberg <tatjana.nuernberg@oracle.com> Date: Sat Mar 3 13:31:37 2018 +0000 WL#11150: Add <component> to error log messages by default Adds another key/value pair to log events. The new field "subsystem" will contain one of the values "InnoDB" (for errors thrown in the InnoDB storage engine proper), "Repl" (for those thrown by the replication subsystem), or "Server" (otherwise). This value is also added to the traditional log. This field will appear between '[' <error_number> ']' and '[' <error_text> ']'. [20]: commit 545b168 Author: Steinar H. Gunderson <steinar.gunderson@oracle.com> Date: Wed Sep 13 12:57:13 2017 +0200 Bug #26781567: REMOVE C LEGACY Replace the final typedefs to st_* and *_st structs. Note in particular that st_mysql_show_var was called sometimes SHOW_VAR and sometimes STATUS_VAR (by way of duplicated forward-declaring typedefs). This patch changes it so that it's universally SHOW_VAR, which seems to match better with the original type name. Change-Id: Ide483ceac31284d909c42adb46496308dd4d4f85 [21]: commit f58cb36 Author: Steinar H. Gunderson <steinar.gunderson@oracle.com> Date: Tue Oct 10 14:54:53 2017 +0200 Bug #26927386: REDUCE WEIGHT OF SQL_CLASS.H [noclose] Remove field.h, handler.h and sql_profile.h from sql_class.h, and add a simple test to make sure they don't get back. Takes sql_class.h down from approx 1.0 to 0.85 seconds (GCC 7.2.0, -O2 -g). Change-Id: I72618c977c7af330d1215899971e2f9e49ab7b88 [22]: commit be029db Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Thu Jan 19 15:14:58 2017 +0100 Bug#25428042: REPLACE MYSQL_CLIENT IFDEF WITH !MYSQL_SERVER Replace the MYSQL_CLIENT #ifdef with !MYSQL_SERVER to simplify conditional compilation. Also remove unnecessary usage of MYSQL_SERVER and move setting of MYSQL_SERVER from C++ to CMake. [23]: commit f5c32ea Author: Gleb Shchepa <gleb.shchepa@oracle.com> Date: Thu Jul 14 21:48:55 2016 +0400 WL#8067: True bottom-up server parser: refactoring of the CREATE TABLE statement WL#8434: True bottom-up server parser: refactoring of partitioning-related stuff WL#8435: True bottom-up server parser: cleanup and refactoring column definition stuff WL#8433: Separate DD commands from regular SQL queries in the parser grammar WL#7840: Allow parsing a single expression [24]: commit ed007c8 Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Thu Dec 22 11:42:43 2016 +0100 Bug#25304844: REMOVE EMBEDDED_LIBRARY #IFDEF Remove dead code related to embedded server. Part 3: Remove NO_EMBEDDED_ACCESS_CHECKS ifdef [25]: commit 11d6946 Author: Dmitry Lenev <dmitry.lenev@oracle.com> Date: Fri Apr 21 09:52:09 2017 +0300 Removed old partitioning DDL implementation which is no longer in use. After WL#8971 "Deprecate and remove partition engine" and WL#9559 "InnoDB_New_DD: Support in-place ALTER PARTITION" code implementing "fast" partitioning DDL (fast_alter_partition_table() and related functions) became unused. This patch removes this dead code (including DDL_LOG implementation used by it) as a follow-up to the above WLs. [26]: commit 5fecdfa Author: Steinar H. Gunderson <steinar.gunderson@oracle.com> Date: Thu Sep 21 14:40:20 2017 +0200 Bug #26030132: REMOVE SQL_ALLOC Finally remove Sql_alloc, now that 8.0 has been branched off. This is a no-op change for all existing code; there is no code that does new Foo() inheriting from Sql_alloc, since that overload doesn't exist, and similarly no code that deletes such objects, since operator delete is private. After this, it is possible to write such code, and it will do the obvious thing (allocate on the regular heap, and destroy similarly). As usual, code that wants to allocate on a MEM_ROOT must say so explicitly by use of overloaded new. Change-Id: I2e98b39daa06d6dfd8b8c5b1d45a85b1ddc02b3b [27]: commit 4880f97 Author: Guilhem Bichot <guilhem.bichot@oracle.com> Date: Tue Nov 29 15:46:27 2016 +0100 WL#883 non-recursive CTE WL#3634 recursive CTE (...) [28]: commit 597a259 Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Tue Oct 6 14:04:03 2015 +0100 Bug#21944866: REFACTORING: REMOVE KILL_BAD_DATA Remove KILL_BAD_DATA from the THD::killed_state enum. KILL_BAD_DATA was only set by the connection itself when a warning was upgraded to error by STRICT mode. It therefore had nothing to do with the connection being killed by other connections (e.g. KILL QUERY / KILL CONNECTION). This is a refactoring with no changes in behavior. [29]: commit 2bf852b Author: Steinar H. Gunderson <steinar.gunderson@oracle.com> Date: Wed Apr 12 12:26:14 2017 +0200 Bug #25909342: USE C++11 THREAD-LOCAL STORAGE FOR CURRENT_THD Our current implementation of current_thd uses mysys' my_thread_* API, which maps onto pthreads (at least for Linux/UNIX). C++11 supports native TLS, which means we just need two simple MOV instructions to do a TLS fetch instead of a full shared library call; the interface is also much simpler. We probably don't have a lot of performance gain, since we don't pick out current_thd all that often, but it's a nice cleanup to do nevertheless. There are still some uses of the my_thread_* API, particularly in PFS; cleaning this up is for a later patch. Change-Id: Ia16e0c1ccfa52af815882bcddd1eb3be80f77c61 [30]: commit 162e016 Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Tue Mar 15 14:32:39 2016 +0100 Bug#22948828: REFACTOR USAGE OF MY_MICRO_TIME() Remove redundant calls to my_micro_time() to simplify code and reduce the usage of gettimeofdat(). Especially focused on code for handling new connections. sql/conn_handler/connection_handler_one_thread.cc - Removed my_micro_time() call. THD::start_utime has just been set by THD::THD() => THD::init(). THD::thr_create_utime is only needed by connection_handler_per_thread.cc sql/conn_handler/connection_handler_per_thread.cc - Removed my_micro_time() call. THD::start_utime has just been set by THD::THD() => THD::init(). Remove THD::thr_create_utime, we can use THD::start_utime instead. sql/event_queue.cc - Replaced usage of THD::set_current_time() with THD::set_time(). THD::set_time() does the same as long as THD::user_time is NULL which it is for events. Allows the removal of THD::set_current_time() sql/events.cc - Removed THD::set_time() call. THD::set_time() has just been called by THD::THD() => THD::init(). sql/log.cc - Removed default argument value for make_iso8601_timestamp() for clarity. - Reduced critical section of LOCK_logger. - Call my_micro_time() directly instead of the unnecessary THD::current_utime() wrapper. sql/log_event.cc - Use the existing THD::query_start() function instead of accessing THD::start_time directly. - Renamed THD::update_server_status() to THD::check_slow_query() since the old name was somewhat misleading. sql/sql_class.h - Removed THD::thr_create_utime, no longer needed. - Removed redundant inline keyword, added const where possible. - Removed THD::query_start_usec(), not used. - Removed THD::query_start_timeval(), no longer needed. - Removed THD::set_current_time(), no longer needed. - Removed THD::is_valid_time(), code moved to sql_parse.cc - Renamed THD::update_server_status() to THD::check_slow_query() since the old name was somewhat misleading. sql/sql_connect.cc - Use THD::start_utime instead of removed THD::thd_create_utime (had the same value) - Removed call to THD::set_time(). Called by THD::init_for_queries() right afterwards anyway. sql/sql_thd_internal_api.cc - Removed THD::set_time() call. THD::set_time() has just been called by THD::THD() => THD::init(). - Removed call to my_micro_time(). Already done by THD::set_time(). [31] commit 485545b Author: Mohit Joshi <mohit.joshi@oracle.com> Date: Tue Jan 17 11:56:43 2017 +0530 Bug#25166686::REMOVE ALL THE REDUNDANT .INC FILES FROM MTR Description: It is noticed that there are a lot of redundant inc files which are not being used by any of the MTR tests. Also, there are several inc files which are included in the test but are not needed anymore. Fix: Below inc files are not being used by any test and hence removed: 1. include/check_key_reads.inc 2. include/check_key_req.inc 3. include/commandline_option_test.inc 4. include/config_file_option_test.inc 5. include/explain.inc 6. include/finish_option_test.inc 7. include/have_binlog_checksum_off.inc 8. include/have_dynamic_loading.inc 9. include/have_shm.inc 10. include/init_option_test.inc 11. include/not_blackhole.inc 12. include/show_partition_checksums.inc 13. include/show_partition_contents.inc 14. include/test_outfile.inc Below are a few inc files which are used in MTR tests but are no more needed. Hence these inc files along with their references from all MTR tests have been removed. 1. have_symlink.inc 2. have_perfschema.inc 3. have_innodb.inc Note: include/not_valgrind_server.inc is not being used by any MTR test but still not removed as this was introduced as part of Bug#22447330 Reviewed by: Parveez Baig <parveez.baig@oracle.com> Deepa Dixit <deepa.dixit@oracle.com> [32] commit a0d0cab Author: Sivert Sorumgard <sivert.sorumgaard@oracle.com> Date: Thu Jun 30 16:00:31 2016 +0200 WL#8971: Deprecate and remove partitioning storage engine This worklog implements the following main changes: 1. Modify cmake files, do not build partitioning engine. 2. Remove files implementing the partitioning engine. 3. Remove some mtr tests using non-native partitioning. 4. Rewrite some mtr tests to use native partitioning. Various adjustments necessary, e.g. regarding EXPLAIN output. 5. Introduce error handling at the SQL layer to fail if using partitioning related DDL statements for engines not supporting it natively. 6. Remove reference to the DB_TYPE_PARTITIONING_DB internally, but keep the enum entry to avoid disturbing other values. 7. Add an mtr test testing the error handling. 8. Add an mtr test to check for regressions in support for partitioning in ndb. 9. Copy comments from storage/partition/ha_partition.* to sql/handler.*, include/my_base.h etc. [33] commit eda9201 Author: Annamalai Gurusami <annamalai.gurusami@oracle.com> Date: Thu Apr 28 12:29:00 2016 +0530 WL#9141 InnoDB: Refactor uncompressed BLOB code to facilitate partial fetch/update (...) [34] commit eda9201 Author: Annamalai Gurusami <annamalai.gurusami@oracle.com> Date: Thu Apr 28 12:29:00 2016 +0530 WL#9141 InnoDB: Refactor uncompressed BLOB code to facilitate partial fetch/update (...) [35] commit 2eec7f3 Author: Marko M<C3><A4>kel<C3><A4> <marko.makela@oracle.com> Date: Fri Apr 22 10:50:45 2016 +0200 Bug#23150562 REMOVE UNIV_MUST_NOT_INLINE AND UNIV_NONINL Problem: InnoDB used to support a compilation mode that allowed to choose whether the function definitions in .ic files are to be inlined or not. This stopped making sense when InnoDB moved to C++ in MySQL 5.6 (and ha_innodb.cc started to #include .ic files), and more so in MySQL 5.7 when inline methods and functions were introduced in .h files. Solution: Remove all references to UNIV_NONINL and UNIV_MUST_NOT_INLINE from all files, assuming that the symbols are never defined. Remove the files fut0fut.cc and ut0byte.cc which only mattered when UNIV_NONINL was defined. RB: 12470 Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com> [36]: commit 644aafd Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue May 24 19:02:55 2016 +0530 WL#9359 - Code cleanup [37]: commit 59b560e Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Feb 6 22:00:10 2018 +1100 Bug#27496251 REMOVE /*====*/ AND CHANGE /****/ /***/ TO /** .... */

Source tree changes - Remove include/my_global.h from .gitignore - it should be visible, especially when merging from 5.7 (where it's present) to 8.0 (where it's not). This partially fixes bug https://bugs.mysql.com/bug.php?id=91834 (Cleanup .gitignore, e.g. include/my_global.h) - Remove storage/innobase/.clang-format due to upstream reformatting CMake / preprocessor symbol changes - Make CMake option WITH_SYSTEM_LIBS configure -DWITH_READLINE=system instead of -DWITH_EDITLINE=system - Remove WITH_SSL CMake settings from build scripts, leaving the implicit default of -DWITH_SSL=system, because the alternative -DWITH_SSL=wolfssl requires more work and is broken by upstream in 8.0.11. - Use MYSQL_ADD_EXECUTABLE instead of ADD_EXECUTABLE in CMake scripts. - Remove HAVE_REPLICATION from diff to match upstream [14]. - Remove WITH_EMBEDDED_SERVER, EMBEDDED_LIBRARY, NO_EMBEDDED_ACCESS_CHECKS, mysql-test/include/not_embedded.inc mentions due to upstream [15], [16], [24] etc. - Remove WITH_PARTITION_STORAGE_ENGINE due to upstream [32]. - Replace MYSQL_CLIENT preprocessor macro uses with !MYSQL_SERVER due to [22]. - Add sql/binlog_crypt_data.cc and sql/event_crypt.cc as mysqlbinlog source files in CMake, to replace the former practice of including them from source files. - Remove setting C++98/03 and C++11 detection from CMake due to upstream converting to C++11 in WL#8896. This includes removing cmake/check_stdcxx11.cmake and its uses from TokuDB / MyRocks. - Make CMake define __STDC_WANT_LIB_EXT1__=1 on macOS to get memset_s. - Define LOG_COMPONENT_TAG for various plugins and unit tests, to accomodate WL#11150 [19]. - Remove uses of UNIV_NONINL due to [35], consequently fold buf0dblwr.ic to buf0dblwr.h. - Revmove UNIV_DEBUG_THREAD_CREATION due to WL#9359 [36] Compilation changes - Fix now-enabled-by-default unused parameter warnings, by removing the unused parameters, or by marking them unused. C to C++ conversion - Convert mysys/system_key.c to C++ mysys/system_key.cc to match upstream [1]. - Convert storage/heap/hp_record.c and storage/heap/hp_dspace.c to C++ hp_record.cc and hp_dspace.cc respectively to match upstream [2]. - Mechanical renames, changes, and partial conversions, incomplete list: - my_bool/FALSE/TRUE/Sys_var_mybool uses to bool/false/true/Sys_var_bool due to upstream my_bool removal at [6]. - #define to [static] const constexpr - Declare local variables at first use - Pointer arguments to const reference ones, where applicable - Convert typedef struct declaration to plain struct, removed redundant struct tags. - NULL/0 to nullptr - compile_time_assert to static_assert - various ad-hoc macros to std::min - various integer types/0/1 to bool/false/true where applicable - Convert mysqldump.cc patches to C++11, remove uses of HASH, make key_type_t an enum class instead of enum. At the same time replace uses of LIST with std::list and std::forward_list, although this was not strictly required. - Remove C_MODE_START/C_MODE_END and extern "C" blocks from C++-only headers. C++03 to C++11 conversion - Replace HASH with std::unordered_map in many patches, because upstream removed HASH in series of commits for "25997748". - Mechanical renames and changes, incomplete list: - ut_snprintf uses to snprintf due to upstream removal of the former at [12]. - my_snprintf to snprintf due to C++11 guaranteeing the presence of the latter. - Applied noexcept to matching functions and methods. - Applied final to matching classes. - boost::movelib::unique_ptr to std::unique_ptr. - insert to emplace for standard containers, where applicable - converted some of std::map uses to std::unordered_map, if ordering is not required - std::auto_ptr to std::unique_ptr - Assignments in constructors and constructor initializer lists to default member initializers. - UNIV_THREAD_LOCAL to thread_local - Replaced uses of compiler builtins with std::atomic for atomic variables. User statistics - Due to HASH replacement with modern C++ collections, cleanup userstat to store objects directly instead of heap-allocated pointers to them. For this, add constructors to TABLE_STATS and INDEX_STATS to replace init_user_stats and init_thread_stats functions. Binlog encryption - Adjust binlog encryption patches for MYSQL_BIN_LOG "append" to "write" rename in cass methods. Parser - Put Percona tokens at the end of the token list due to upstream attempt to ensure token digest value stability ([17], [18]). - Because of upstream bottom-up parser refactoring for CREATE TABLE [23], which includes column attributes, rewrite TokuDB clustered index and InnoDB compression dictionary support: - add apply_zip_dict method to PT_column_attr_base class; - add zip_dict_name arg to the constructor and m_zip_dict_name field to PT_column_format_column_attr class, implement apply_zip_dict; - add m_zip_dict field to PT_field_def_base, get its value in contextualize_attrs method, and pass it to Alter_info instance in PT_column_def::contextualize and PT_alter_table_change_column::contextualize - rename PT_unique_key_column_attr to PT_unique_combo_clustering_key_column_attr, make it handle both UNIQUE and CLUSTERING - move zip_dict_name from LEX to HA_CREATE_INFO in line with the upstream LEX reduction in parser refactoring. - Typedef new alter tablespace option PT_alter_tablespace_option_encryption for general tablespace encryption. Performance Schema - Adjust PFS object declarations for the new volatility and documentation fields, account for PSI_FLAG_GLOBAL rename to PSI_FLAG_SINGLETON. - Move server PSF memory key declarations to sql/psi_memory_key.h and sql/psi_memory_key.cc - Adjust inline_mysql_unix_connect PFS instrumentation in include/mysql/psi/mysql_file.h to modern PFS instrumentation pattern which was introduced back in [3] because the old one stopped compiling in 8.0. While at that, switch from HAVE_PSI_INTERFACE to HAVE_PSI_FILE_INTERFACE. Audit log plugin - Converted the plugin to C++. - Made a lot of declarations in audit_log.cc static and/or const. - Converted hash tables in filter.cc to collation_unordered_set<std::string>. At the same time convert the hash table value struct creator/initialization functions to constructors. Backup locks - Remove everything associated with LOCK BINLOG FOR BACKUP / UNLOCK BINLOG due to upstream performance_schema.log_status table. - Refer to Oracle backup lock as "instance backup lock" and Percona backup lock as "table backup lock" to try to reduce confusion between the two. Expanded fast index creation - Change the type of delayed_key_list container in Alter_info from List to Mem_root_array. Threadpool - Re-implement TP_STATISTICS::num_worker_threads, pool_timer_t::current_microtime and next_timeout_check using C++11 atomics. - Convert Worker_thread_context struct to class with constructor/destructor instead of save/restore methods, to be used as RAII. New data dictionary - Percona features that need to store information in the new data dictionary (CLUSTERED keys for TokuDB; compression dictionary information for InnoDB have not been re-implemented for the new DD yet). For now the legacy implementation is disabled, to be removed together with the new implementation, except for the reading bits, because they are needed for upgrades from 5.7. Specifically, leave SYSTEM_TABLE_NAME patch in, dict_process_sys_zip_dict function Changed page tracking - Move fil_name_process patch not to cause any tablespace in filesystem changes when called by redo log tracking to fil_tablespace_redo_delete and fil_tablespace_redo_create. - Introduce apply arg for fil_tablespace_redo_encryption for redo log parsing mode only. - log_get_tracked_lsn and log_set_tracked_lsn have been remvoed after the tracked LSN conversion to C++11 atomic variable. - Fold log_check_tracking_margin to log0write.cc:log_write_write_buffer. - Move redo log signalling from log_io_complete_checkpoint to log0chkp.cc:log_checkpoint. - Drop log_group_read_log_seg release_mutex arg (which would have been moved to recv_read_log_seg) because of log_sys->mutex removal. - innobase_logs_empty_and_mark_at_shutdown patches moved to srv_log_shutdown. - Due to os_file_dir_t and associated function removals in [13], rewrite their users in log0online.cc to use MY_DIR and my_dir from mysys. - Remove log_group_t mentions from log0online.cc - Add handling for MLOG_COMP_PAGE_CREATE_SDI, MLOG_PAGE_CREATE_SDI, and MLOG_TABLE_DYNAMIC_META MTR types. - Adjust redo log reading code for log_group_read_log_seg rename to recv_read_log_seg, convert the latter from static to public function. - Refactor out redo log tracker-waiting code in fil0fil.cc to a new private static method Fil_system::wait_for_changed_page_tracker. Other changes due to upstream - Remove YaSSL-related source code (mysys_ssl/yassl.cpp, HAVE_YASSL check, build-binary.sh --with-yassl option) - Replace my_global.h includes with the needed smaller ones (my_inttypes.h, mysql/service_mysql_alloc.h, my_macros.h) due to upstream patch series "Bug #24710065: SPLIT MY_GLOBAL.H", and move my_global.h patches to appropriate places: s6_addr32 define to violite.h. - Remove THD::thr_create_utime uses and replace THD::update_server_status calls with THD::update_slow_query_status due to [30]. - Remove Sql_alloc uses due to [26]. - Adjust for class Key rename to Key_spec with declaration moved to sql/key_spec.h in [4]. - Remove NORMAL_PART_NAME uses because of [25]. - Break up the dependency between sql_class.h and handler.h by moving opt_log_slow_admin_statements and opt_log_slow_sp_statements from mysqld.h/mysqld.cc to log.h/log.cc, and including log.h instead of mysqld.h (which then includes handler.h) in sql_class.h. Also move struct fragmentation_stats_t declaration to system_variables.h from sql_thd_internal_api.h, put a forward declaration there, and do not include the latter header from the former. This is done due to [21]. - Remove THD::KILL_BAD_DATA uses due to [28]. - Remove THR_THD uses due to [29]. - Convert uses of dict_table_is_temporary to dict_table_t::is_temporary() due to [7]. - Remove TABLE_SHARE::uniques uses from sql_base.cc Table_Node::Table_Node due to upstream removal at [27]. It is not clear whether the resulting code is correct though. - Replace uses of SRV_LOG_SPACE_FIRST_ID with dict_sys_t::log_space_first_id due to [8] and the n dict_sys_t::s_log_space_first_id due to [9]. - Move srv_parallel_doublewrite_path initialisation to innobase_init_params to reflect InnoDB startup refactoring - Adjust btr_copy_externally_stored_field callers to call lob::btr_copy_externally_stored_field (WL#9141, [34]), and pass index definition to it (WL#8960, [33]). - Move dict_index_get_nth_col_pos patches to dict_index_t::get_col_pos because of [11]. - Adjust get-space-by-space-id code for fil_system mutex removal, sharding, and fil_node_complete_io -> complete_io rename, introduced in WL#8619 [10]. - Remove innodb_use_global_flush_log_at_trx_commit and commit innodb_flush_log_at_trx_commit back to global variable, and remove associated srv_use_global_flush_log_at_trx_commit source code variable and thd_flush_log_at_trx_commit function - Replace struct st_mysql_sys_var occurences with SYS_VAR due to [20]. - Adjust warn_or_info class for InnoDB logging changes at [5], which remove the actual logging from logger::~logger, pass diagnostics level to its constructor, and conditionally disable actual logging using UNIV_NO_ERR_MSGS preprocessor define. - Removed /*=...==*/ comments and replace /**...**//**\n with /** in the InnoDB patch according to [37]. - Replace reserved_system_space_name with dict_sys_t::sys_space_name because of [8], and then to dict_sys_t::s_sys_space_name because of [9]. Misc cleanups - Split assert(condition1 && condition2) to assert(condition1); assert(condition2); - Create type aliases for all the added function pointers in the handlerton interface. - Remove hash table size information leftovers that should have been removed in the 5.7 porting branch - Replace some of the ulint uses with space_id_t where applicable - Remove our #define SRV_MAX_N_PURGE_THREADS in favor of upstream MAX_PURGE_THREADS. - Remove remnants of innodb_flush_method = ALL_O_DIRECT: defines DEFAULT_SRV_LOG_WRITE_AHEAD_SIZE and MAX_SRV_LOG_WRITE_AHEAD_SIZE, xb_simulate_all_o_direct_write_failure, xb_simulate_all_o_direct_read_failure debug points. - Introduce flush_changed_page_bitmaps_t and purge_changed_page_bitmaps_t types for the corresponding handlerton methods. - Remove .is_percona_server_bzr - Replace README.MySQL contents with upstream README - Update man pages from mysql-8.0.11.tar.gz Testsuite These are partial changes only, as there was no goal to run the testsuite at this stage. - Remove uses of have_dynamic_loading.inc, have_symlink.inc, have_innodb.inc, have_perfschema.inc include files due to [31]. TokuDB, MyRocks, PAM authentication These three plugins are not handled in this merge yet. [1]: commit c7a720b Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Thu Sep 24 12:46:44 2015 +0100 Bug#21881278: SWITCH MYSYS LIBRARY FROM C TO C++ This patch switches all mysys/ files from .c to .cc. By doing this we get stricter compile checks (e.g. type checks) and it makes it possible to use C++ features for future bug fixes and extensions inside mysys. [2]: commit 4133e44 Author: Steinar H. Gunderson <steinar.gunderson@oracle.com> Date: Thu Sep 7 15:18:39 2017 +0200 Bug #26781567: REMOVE C LEGACY [noclose] Convert the remaining C files to C++, except for external source code (e.g. memcached) and a few .c files that are there to verify that our client library still builds from C code. The only real exception is libservices/, where ODR violations (services are defined as structs in some files and as void * in others) mean that linking with C++ linkage would break linking on Windows. Change-Id: I61f6a281ffcd5c0fca9f4b9976aeb580dd1b0b65 [3]: commit bd8c99e Author: Marc Alff <marc.alff@oracle.com> Date: Fri Jul 1 10:46:20 2011 +0200 Bug#12346211 PERF IMPACT OF PERFORMANCE-SCHEMA WITH DISABLED INSTRUMENTS/CONSUMERS Performance improvement #1: instrumentation interface. In the instrumentation interface in include/mysql/psi/psi.h, implemented the following changes. 1) Removal of "if (PSI_server != NULL)". Instead of having PSI_server == NULL when the performance schema is disabled, a dummy implementation of the instrumentation interface is provided in mysys/psi_noop.cc. Replacing a test by calling an empty function is similar for the disabled code path. For the instrumented code path (the performance schema is enabled), removing this test completely improves performances. 2) Push down of "if (that->m_psi != NULL)". Testing if an object is instrumented has been pushed down to the implementation of the instrumentation interface. In case of psi_noop.cc, this test is not needed, which improves performances for the disabled case. 3) Removal of 1 "if (locker != NULL)" test. Before the fix, the coding pattern was: locker= ... if (locker != NULL) PSI_server->start_event() <instrumented code> if (locker != NULL) PSI_server->end_event() After the fix, the coding pattern is: locker= ... if (locker != NULL) { PSI_server->start_event() <instrumented code> PSI_server->end_event() } else { <instrumented code> } This saves an extra if() in both the instrumented and non instrumented code path. 4) Packing of helper structures. Unused attributes such as m_src_file and m_src_line have been removed. Attributes have been reordered to: - avoid holes caused by padding - put attributes used very frequently first 5) Use PSI_CALL() Calls to the performance schema implementation have been changed from: PSI_server->function(); to PSI_CALL(function)(); This is cosmetic only, and can help later to make static calls to pfs_function() instead of using the PSI_server pointer. [4]: commit 3f3c259 Author: Tor Didriksen <tor.didriksen@oracle.com> Date: Thu Mar 26 16:56:31 2015 +0100 Bug#20756282 MOVE CLASS KEY AND FOREIGN_KEY OUT OF SQL_CLASS.H Move Key_part_spec, Key_spec (new name for Key) and Foreign_key_spec (new name for Foreign_key) to new files key_spec.h/cc [5]: commit 29e3723 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Feb 16 14:30:31 2018 +1100 WL#11009: Logging services: log writers: add error ID to traditional error log (InnoDB changes) Add a unique ID to all log InnoDB messages. Remove InnoDB: prefix from the pattern matching in the tests. RB#18486 Reviewed by: Ingo, Annamalai and Praveen Hulakund. [6]: commit 6c174bd Author: Steinar H. Gunderson <steinar.gunderson@oracle.com> Date: Tue Feb 21 13:39:24 2017 +0100 Bug #25597667: REMOVE MY_BOOL Replace my_bool with bool everywhere. Change-Id: Ibfe9f9b4a05d8ee82d09ffdc4f1f038ec4555b89 [7]: commit 2238b41 Author: El<C5><BC>bieta Babij <elzbieta.babij@oracle.com> Date: Tue Jul 5 12:22:26 2016 +0200 Bug#23748128 - Add some accessor methods to dict_table_t. dict_table_t::get_n_user_cols(): Get the number of user-specified columns, possibly including a hidden FTS_DOC_ID column. dict_table_t::get_n_sys_cols(): Get the number of internal system columns. dict_table_t::get_n_cols(): Get the number of all columns. dict_table_t::get_col(): Get a column. dict_table_t::get_col_name(): Get a column name. dict_table_t::get_sys_col(): Get an internal system column. [8]: commit cd7649d Author: Allen Lai <zheng.lai@oracle.com> Date: Thu Apr 13 10:59:58 2017 +0800 wl#9525 InnoDB_New_DD: update InnoDB specific metadata into new DD for DDL wl#9534 InnoDB_New_DD: Instantiate InnoDB in-memory metadata with newDD objects wl#9530 InnoDB_New_DD: FTS index support for newDD wl#9531 InnoDB_New_DD: Enable table encryption and transparent compression wl#9532 InnoDB_New_DD: InnoDB_New_DD: Move data dictionary tables to a single tablespace wl#9559 InnoDB_New_DD: Support in-place ALTER PARTITION wl#9537 InnoDB_New_DD: Support InnoDB table import/export for new DD wl#9357<C2><A0>InnoDB: Upgrade steps for new data dictionary wl#9570<C2><A0>Extend SHOW statements to list hidden columns and index information wl#9461<C2><A0>Upgrade for Innodb Internal Dictionary Migration This is the InnoDB side of work to support an unified metadata system between MySQL server and Storage Engines. It removes the InnoDB dependency on InnoDB system tables, that is, all InnoDB metadata are now read from server objects, instead of reading from InnoDB system tables. In these wls, we replaced dict_table_open_* (on_id/on_name) with their newDD counterpart. It replaced metadata instantiation (dict_load_table etc.) with newDD counterpart too. And also, we migrated all related features, like partition, encryption, compression and import/export etc. Upgrade from mysql-5.7 is supported as well. [9]: 3452648 Author: Jimmy Yang <jimmy.yang@oracle.com> AuthorDate: Thu Aug 17 00:34:24 2017 +0200 Commit: Jimmy Yang <jimmy.yang@oracle.com> CommitDate: Thu Aug 17 00:46:42 2017 +0200 Parent: f8076b3 Bug#25727892: Refactor Item::const_item() as a non-virtual function Merged: 5.5 5.6 mysql-5.7.7 Containing: 8.0 mysql-8.0.11-percona-patches mysql-8.0.3-percona-patches mysql-8.0.4-percona-patches (1 more) Follows: mysql-8.0.2 (1261) Precedes: mysql-8.0.3 (144) InnoDB NewDD worklogs(WL#9535 and WL#9536) to support Atomic DDL and completely remove old InnoDB system tables: 1. WL#9535: InnoDB_New_DD: Remove InnoDB System Table and modify the view of their I_S counterparts This worklog obsoletes all InnoDB System tables, which are no longer used and could only be accessed by upgrade. All metadata is now persisted and read from DD system tables in "mysql tablespace" instead of "old" InnoDB System Tables. All corresponding information_schema.innodb_sys_* tables are replaced by new internal system views. These new views have the name similar to the names of old system tables, just removing the 'SYS_' from the middle of the names. 2. WL#9536: InnoDB_New_DD: Support crash-safe DDL This worklog starts to support crash-safe DDL for InnoDB. DDLs like CREATE TABLE, DROP TABLE, RENAME TABLE, TRUNCATE TABLE, ALTER TABLE, CREATE TABLESPACE and DROP TABLESPACE are covered by this worklog. DDLs operations including DD system table updates and corresponding file changes(create, delete, rename etc.) are atomic. Physical data files would be synchronized with the transaction for DDL, thus there should be no physical data files for temporary tables or FTS auxiliary tables, etc. left after recovery. [10]: commit cf63b24 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Dec 8 12:10:42 2017 +1100 WL#8619 - InnoDB: Provide offline database portability without ISL files Summary of changes. 1. Introduce --innodb-directories := "dir1;...;dirN". This is renamed from --innodb-scan-directories which was introduced in WL#9499. Currently this is not dynamic. 2. Revert to 5.6 behaviour where we scan directories and find .ibd files. If you move tablespaces to a new location, add the path to innodb-data-directories so that InnoDB can "discover" the files on startup. 3. Tablespaces can only be created under known directories. This eliminates the need for .isl files. 4. Upgrade redo log format, not backward compatible 5. Change redo log type IDs 6. Code cleanup 7. Shard the fil system data structures for performance 8. Remove WL#9499 files that were used to track open tablespaces. RB#16842 Approved by: Jimmy and Kevin. [11]: commit b3ba94c Author: El<C5><BC>bieta Babij <elzbieta.babij@oracle.com> Date: Tue Jul 26 15:00:04 2016 +0200 Bug #24361023 - Add accessors to indexes. dict_index_t::add_field(): Replaces dict_mem_index_add_field(). dict_index_t::get_field(): Replaces dict_index_get_nth_field(). dict_index_t::get_col(): Replaces dict_index_get_nth_col(). dict_index_t::get_col_no(): Replaces dict_index_get_nth_col_no(). dict_index_t::get_col_pos(): Replaces dict_index_get_nth_col_pos() and dict_index_get_nth_col_or_prefix_pos(). dict_index_t::get_sys_col_pos(): Replaces dict_index_get_sys_col_pos(). RB#13487 Reviewed by Kevin Lewis [12]: commit 98bba73 Author: El<C5><BC>bieta Babij <elzbieta.babij@oracle.com> Date: Mon May 23 10:21:20 2016 +0200 Bug #23329353 - Replace ut_snprint with the C++11 snprintf function. [13]: commit 31350e8 Author: Gopal Shankar <gopal.shankar@oracle.com> Date: Tue Nov 3 13:27:06 2015 +0530 WL#6378 New data dictionary (umbrella). (...) [14]: commit f98b7bc Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Wed Jan 18 10:39:10 2017 +0100 Bug#25413703: REMOVE #IFDEF HAVE_REPLICATION After the removal of embedded server, the HAVE_REPLICATION ifdef is always true and can therefore be removed. [15]: commit 18c6270 Author: Tor Didriksen <tor.didriksen@oracle.com> Date: Tue Nov 8 09:55:35 2016 +0100 WL#9816 Do not build/package/test embedded server - Remove/Cleanup all cmake related code pertaining to embedded server - Remove/Cleanup embedded server from RMP and Debian packages - Remove embedded tests from our mtr test collections - Remove all source code in the libmysqld directory - Remove embedded code from mysql-test-run Change-Id: Ib2432e50043f060f1da64328815cdb9d53981de4 [16]: commit 0a68d5d Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Mon Jan 16 12:35:47 2017 +0100 Bug#25304844: REMOVE EMBEDDED_LIBRARY #IFDEF Remove dead code related to embedded server. Part 5: Remove EMBEDDED_LIBRARY ifdef from client/ and storage/ This patch also removes the --server-arg option from 'mysql'. This option was embedded server specific and undocumented. --server-arg and --server-file are, for the same reason, removed from 'mysqltest'. [17]: commit db3fd68 Author: Gleb Shchepa <gleb.shchepa@oracle.com> Date: Thu Dec 7 16:21:28 2017 +0400 WL#8989: Create an internal API for the MySQL parser to enumerate reserved and non-reserved words Part I. Grammar preparation. * Replace the existent type tag <symbol> with <keyword> for a better readability. * Add the new type tag <keyword> to currently untyped terminal and non-terminal symbols. Part II. Create a simple compile-time program for transforming sql_yacc.yy into a .h file that contains an array of reserved and non-reserved words. * Find %token declarations with the <keyword> type tag by a regexp. * Separate %type declarations are ignored for the simplicity. Part III. Introduce a new system view: INFORMATION_SCHEMA.keywords. * Hardcode the statical list of keywords and reserved words into a view on top of a long JSON_TABLE(...) expression. * Use a trick in the .test file to trace changes in keywords: notify the developer to update the target I_S version in the mysqld binary where necessary -- this will force DD to refresh I_S view declarations on the next run. [18]: commit a2f3a60 Author: Marc Alff <marc.alff@oracle.com> Date: Thu Mar 10 10:10:25 2016 +0100 Bug#22906606 IMPROVE DIGEST STABILITY Before this fix, adding new tokens in the parser caused query digests to change. The root cause was that tokens are typically added in the middle of the list, causing renumbering of tokens. With this fix, the process is now changed and documented: new tokens should be added at the end of the list. The tool gen_lex_token, used during the build, is improved to enforce this process. [19]: commit f9d1c83 Author: Tatiana Azundris Nuernberg <tatjana.nuernberg@oracle.com> Date: Sat Mar 3 13:31:37 2018 +0000 WL#11150: Add <component> to error log messages by default Adds another key/value pair to log events. The new field "subsystem" will contain one of the values "InnoDB" (for errors thrown in the InnoDB storage engine proper), "Repl" (for those thrown by the replication subsystem), or "Server" (otherwise). This value is also added to the traditional log. This field will appear between '[' <error_number> ']' and '[' <error_text> ']'. [20]: commit 545b168 Author: Steinar H. Gunderson <steinar.gunderson@oracle.com> Date: Wed Sep 13 12:57:13 2017 +0200 Bug #26781567: REMOVE C LEGACY Replace the final typedefs to st_* and *_st structs. Note in particular that st_mysql_show_var was called sometimes SHOW_VAR and sometimes STATUS_VAR (by way of duplicated forward-declaring typedefs). This patch changes it so that it's universally SHOW_VAR, which seems to match better with the original type name. Change-Id: Ide483ceac31284d909c42adb46496308dd4d4f85 [21]: commit f58cb36 Author: Steinar H. Gunderson <steinar.gunderson@oracle.com> Date: Tue Oct 10 14:54:53 2017 +0200 Bug #26927386: REDUCE WEIGHT OF SQL_CLASS.H [noclose] Remove field.h, handler.h and sql_profile.h from sql_class.h, and add a simple test to make sure they don't get back. Takes sql_class.h down from approx 1.0 to 0.85 seconds (GCC 7.2.0, -O2 -g). Change-Id: I72618c977c7af330d1215899971e2f9e49ab7b88 [22]: commit be029db Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Thu Jan 19 15:14:58 2017 +0100 Bug#25428042: REPLACE MYSQL_CLIENT IFDEF WITH !MYSQL_SERVER Replace the MYSQL_CLIENT #ifdef with !MYSQL_SERVER to simplify conditional compilation. Also remove unnecessary usage of MYSQL_SERVER and move setting of MYSQL_SERVER from C++ to CMake. [23]: commit f5c32ea Author: Gleb Shchepa <gleb.shchepa@oracle.com> Date: Thu Jul 14 21:48:55 2016 +0400 WL#8067: True bottom-up server parser: refactoring of the CREATE TABLE statement WL#8434: True bottom-up server parser: refactoring of partitioning-related stuff WL#8435: True bottom-up server parser: cleanup and refactoring column definition stuff WL#8433: Separate DD commands from regular SQL queries in the parser grammar WL#7840: Allow parsing a single expression [24]: commit ed007c8 Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Thu Dec 22 11:42:43 2016 +0100 Bug#25304844: REMOVE EMBEDDED_LIBRARY #IFDEF Remove dead code related to embedded server. Part 3: Remove NO_EMBEDDED_ACCESS_CHECKS ifdef [25]: commit 11d6946 Author: Dmitry Lenev <dmitry.lenev@oracle.com> Date: Fri Apr 21 09:52:09 2017 +0300 Removed old partitioning DDL implementation which is no longer in use. After WL#8971 "Deprecate and remove partition engine" and WL#9559 "InnoDB_New_DD: Support in-place ALTER PARTITION" code implementing "fast" partitioning DDL (fast_alter_partition_table() and related functions) became unused. This patch removes this dead code (including DDL_LOG implementation used by it) as a follow-up to the above WLs. [26]: commit 5fecdfa Author: Steinar H. Gunderson <steinar.gunderson@oracle.com> Date: Thu Sep 21 14:40:20 2017 +0200 Bug #26030132: REMOVE SQL_ALLOC Finally remove Sql_alloc, now that 8.0 has been branched off. This is a no-op change for all existing code; there is no code that does new Foo() inheriting from Sql_alloc, since that overload doesn't exist, and similarly no code that deletes such objects, since operator delete is private. After this, it is possible to write such code, and it will do the obvious thing (allocate on the regular heap, and destroy similarly). As usual, code that wants to allocate on a MEM_ROOT must say so explicitly by use of overloaded new. Change-Id: I2e98b39daa06d6dfd8b8c5b1d45a85b1ddc02b3b [27]: commit 4880f97 Author: Guilhem Bichot <guilhem.bichot@oracle.com> Date: Tue Nov 29 15:46:27 2016 +0100 WL#883 non-recursive CTE WL#3634 recursive CTE (...) [28]: commit 597a259 Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Tue Oct 6 14:04:03 2015 +0100 Bug#21944866: REFACTORING: REMOVE KILL_BAD_DATA Remove KILL_BAD_DATA from the THD::killed_state enum. KILL_BAD_DATA was only set by the connection itself when a warning was upgraded to error by STRICT mode. It therefore had nothing to do with the connection being killed by other connections (e.g. KILL QUERY / KILL CONNECTION). This is a refactoring with no changes in behavior. [29]: commit 2bf852b Author: Steinar H. Gunderson <steinar.gunderson@oracle.com> Date: Wed Apr 12 12:26:14 2017 +0200 Bug #25909342: USE C++11 THREAD-LOCAL STORAGE FOR CURRENT_THD Our current implementation of current_thd uses mysys' my_thread_* API, which maps onto pthreads (at least for Linux/UNIX). C++11 supports native TLS, which means we just need two simple MOV instructions to do a TLS fetch instead of a full shared library call; the interface is also much simpler. We probably don't have a lot of performance gain, since we don't pick out current_thd all that often, but it's a nice cleanup to do nevertheless. There are still some uses of the my_thread_* API, particularly in PFS; cleaning this up is for a later patch. Change-Id: Ia16e0c1ccfa52af815882bcddd1eb3be80f77c61 [30]: commit 162e016 Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Tue Mar 15 14:32:39 2016 +0100 Bug#22948828: REFACTOR USAGE OF MY_MICRO_TIME() Remove redundant calls to my_micro_time() to simplify code and reduce the usage of gettimeofdat(). Especially focused on code for handling new connections. sql/conn_handler/connection_handler_one_thread.cc - Removed my_micro_time() call. THD::start_utime has just been set by THD::THD() => THD::init(). THD::thr_create_utime is only needed by connection_handler_per_thread.cc sql/conn_handler/connection_handler_per_thread.cc - Removed my_micro_time() call. THD::start_utime has just been set by THD::THD() => THD::init(). Remove THD::thr_create_utime, we can use THD::start_utime instead. sql/event_queue.cc - Replaced usage of THD::set_current_time() with THD::set_time(). THD::set_time() does the same as long as THD::user_time is NULL which it is for events. Allows the removal of THD::set_current_time() sql/events.cc - Removed THD::set_time() call. THD::set_time() has just been called by THD::THD() => THD::init(). sql/log.cc - Removed default argument value for make_iso8601_timestamp() for clarity. - Reduced critical section of LOCK_logger. - Call my_micro_time() directly instead of the unnecessary THD::current_utime() wrapper. sql/log_event.cc - Use the existing THD::query_start() function instead of accessing THD::start_time directly. - Renamed THD::update_server_status() to THD::check_slow_query() since the old name was somewhat misleading. sql/sql_class.h - Removed THD::thr_create_utime, no longer needed. - Removed redundant inline keyword, added const where possible. - Removed THD::query_start_usec(), not used. - Removed THD::query_start_timeval(), no longer needed. - Removed THD::set_current_time(), no longer needed. - Removed THD::is_valid_time(), code moved to sql_parse.cc - Renamed THD::update_server_status() to THD::check_slow_query() since the old name was somewhat misleading. sql/sql_connect.cc - Use THD::start_utime instead of removed THD::thd_create_utime (had the same value) - Removed call to THD::set_time(). Called by THD::init_for_queries() right afterwards anyway. sql/sql_thd_internal_api.cc - Removed THD::set_time() call. THD::set_time() has just been called by THD::THD() => THD::init(). - Removed call to my_micro_time(). Already done by THD::set_time(). [31] commit 485545b Author: Mohit Joshi <mohit.joshi@oracle.com> Date: Tue Jan 17 11:56:43 2017 +0530 Bug#25166686::REMOVE ALL THE REDUNDANT .INC FILES FROM MTR Description: It is noticed that there are a lot of redundant inc files which are not being used by any of the MTR tests. Also, there are several inc files which are included in the test but are not needed anymore. Fix: Below inc files are not being used by any test and hence removed: 1. include/check_key_reads.inc 2. include/check_key_req.inc 3. include/commandline_option_test.inc 4. include/config_file_option_test.inc 5. include/explain.inc 6. include/finish_option_test.inc 7. include/have_binlog_checksum_off.inc 8. include/have_dynamic_loading.inc 9. include/have_shm.inc 10. include/init_option_test.inc 11. include/not_blackhole.inc 12. include/show_partition_checksums.inc 13. include/show_partition_contents.inc 14. include/test_outfile.inc Below are a few inc files which are used in MTR tests but are no more needed. Hence these inc files along with their references from all MTR tests have been removed. 1. have_symlink.inc 2. have_perfschema.inc 3. have_innodb.inc Note: include/not_valgrind_server.inc is not being used by any MTR test but still not removed as this was introduced as part of Bug#22447330 Reviewed by: Parveez Baig <parveez.baig@oracle.com> Deepa Dixit <deepa.dixit@oracle.com> [32] commit a0d0cab Author: Sivert Sorumgard <sivert.sorumgaard@oracle.com> Date: Thu Jun 30 16:00:31 2016 +0200 WL#8971: Deprecate and remove partitioning storage engine This worklog implements the following main changes: 1. Modify cmake files, do not build partitioning engine. 2. Remove files implementing the partitioning engine. 3. Remove some mtr tests using non-native partitioning. 4. Rewrite some mtr tests to use native partitioning. Various adjustments necessary, e.g. regarding EXPLAIN output. 5. Introduce error handling at the SQL layer to fail if using partitioning related DDL statements for engines not supporting it natively. 6. Remove reference to the DB_TYPE_PARTITIONING_DB internally, but keep the enum entry to avoid disturbing other values. 7. Add an mtr test testing the error handling. 8. Add an mtr test to check for regressions in support for partitioning in ndb. 9. Copy comments from storage/partition/ha_partition.* to sql/handler.*, include/my_base.h etc. [33] commit eda9201 Author: Annamalai Gurusami <annamalai.gurusami@oracle.com> Date: Thu Apr 28 12:29:00 2016 +0530 WL#9141 InnoDB: Refactor uncompressed BLOB code to facilitate partial fetch/update (...) [34] commit eda9201 Author: Annamalai Gurusami <annamalai.gurusami@oracle.com> Date: Thu Apr 28 12:29:00 2016 +0530 WL#9141 InnoDB: Refactor uncompressed BLOB code to facilitate partial fetch/update (...) [35] commit 2eec7f3 Author: Marko M<C3><A4>kel<C3><A4> <marko.makela@oracle.com> Date: Fri Apr 22 10:50:45 2016 +0200 Bug#23150562 REMOVE UNIV_MUST_NOT_INLINE AND UNIV_NONINL Problem: InnoDB used to support a compilation mode that allowed to choose whether the function definitions in .ic files are to be inlined or not. This stopped making sense when InnoDB moved to C++ in MySQL 5.6 (and ha_innodb.cc started to #include .ic files), and more so in MySQL 5.7 when inline methods and functions were introduced in .h files. Solution: Remove all references to UNIV_NONINL and UNIV_MUST_NOT_INLINE from all files, assuming that the symbols are never defined. Remove the files fut0fut.cc and ut0byte.cc which only mattered when UNIV_NONINL was defined. RB: 12470 Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com> [36]: commit 644aafd Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue May 24 19:02:55 2016 +0530 WL#9359 - Code cleanup [37]: commit 59b560e Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Feb 6 22:00:10 2018 +1100 Bug#27496251 REMOVE /*====*/ AND CHANGE /****/ /***/ TO /** .... */

Source tree changes - Remove include/my_global.h from .gitignore - it should be visible, especially when merging from 5.7 (where it's present) to 8.0 (where it's not). This partially fixes bug https://bugs.mysql.com/bug.php?id=91834 (Cleanup .gitignore, e.g. include/my_global.h) - Remove storage/innobase/.clang-format due to upstream reformatting CMake / preprocessor symbol changes - Make CMake option WITH_SYSTEM_LIBS configure -DWITH_READLINE=system instead of -DWITH_EDITLINE=system - Remove WITH_SSL CMake settings from build scripts, leaving the implicit default of -DWITH_SSL=system, because the alternative -DWITH_SSL=wolfssl requires more work and is broken by upstream in 8.0.11. - Use MYSQL_ADD_EXECUTABLE instead of ADD_EXECUTABLE in CMake scripts. - Remove HAVE_REPLICATION from diff to match upstream [14]. - Remove WITH_EMBEDDED_SERVER, EMBEDDED_LIBRARY, NO_EMBEDDED_ACCESS_CHECKS, mysql-test/include/not_embedded.inc mentions due to upstream [15], [16], [24] etc. - Remove WITH_PARTITION_STORAGE_ENGINE due to upstream [32]. - Replace MYSQL_CLIENT preprocessor macro uses with !MYSQL_SERVER due to [22]. - Add sql/binlog_crypt_data.cc and sql/event_crypt.cc as mysqlbinlog source files in CMake, to replace the former practice of including them from source files. - Remove setting C++98/03 and C++11 detection from CMake due to upstream converting to C++11 in WL#8896. This includes removing cmake/check_stdcxx11.cmake and its uses from TokuDB / MyRocks. - Make CMake define __STDC_WANT_LIB_EXT1__=1 on macOS to get memset_s. - Define LOG_COMPONENT_TAG for various plugins and unit tests, to accomodate WL#11150 [19]. - Remove uses of UNIV_NONINL due to [35], consequently fold buf0dblwr.ic to buf0dblwr.h. - Revmove UNIV_DEBUG_THREAD_CREATION due to WL#9359 [36] Compilation changes - Fix now-enabled-by-default unused parameter warnings, by removing the unused parameters, or by marking them unused. C to C++ conversion - Convert mysys/system_key.c to C++ mysys/system_key.cc to match upstream [1]. - Convert storage/heap/hp_record.c and storage/heap/hp_dspace.c to C++ hp_record.cc and hp_dspace.cc respectively to match upstream [2]. - Mechanical renames, changes, and partial conversions, incomplete list: - my_bool/FALSE/TRUE/Sys_var_mybool uses to bool/false/true/Sys_var_bool due to upstream my_bool removal at [6]. - #define to [static] const constexpr - Declare local variables at first use - Pointer arguments to const reference ones, where applicable - Convert typedef struct declaration to plain struct, removed redundant struct tags. - NULL/0 to nullptr - compile_time_assert to static_assert - various ad-hoc macros to std::min - various integer types/0/1 to bool/false/true where applicable - Convert mysqldump.cc patches to C++11, remove uses of HASH, make key_type_t an enum class instead of enum. At the same time replace uses of LIST with std::list and std::forward_list, although this was not strictly required. - Remove C_MODE_START/C_MODE_END and extern "C" blocks from C++-only headers. C++03 to C++11 conversion - Replace HASH with std::unordered_map in many patches, because upstream removed HASH in series of commits for "25997748". - Mechanical renames and changes, incomplete list: - ut_snprintf uses to snprintf due to upstream removal of the former at [12]. - my_snprintf to snprintf due to C++11 guaranteeing the presence of the latter. - Applied noexcept to matching functions and methods. - Applied final to matching classes. - boost::movelib::unique_ptr to std::unique_ptr. - insert to emplace for standard containers, where applicable - converted some of std::map uses to std::unordered_map, if ordering is not required - std::auto_ptr to std::unique_ptr - Assignments in constructors and constructor initializer lists to default member initializers. - UNIV_THREAD_LOCAL to thread_local - Replaced uses of compiler builtins with std::atomic for atomic variables. User statistics - Due to HASH replacement with modern C++ collections, cleanup userstat to store objects directly instead of heap-allocated pointers to them. For this, add constructors to TABLE_STATS and INDEX_STATS to replace init_user_stats and init_thread_stats functions. Binlog encryption - Adjust binlog encryption patches for MYSQL_BIN_LOG "append" to "write" rename in cass methods. Parser - Put Percona tokens at the end of the token list due to upstream attempt to ensure token digest value stability ([17], [18]). - Because of upstream bottom-up parser refactoring for CREATE TABLE [23], which includes column attributes, rewrite TokuDB clustered index and InnoDB compression dictionary support: - add apply_zip_dict method to PT_column_attr_base class; - add zip_dict_name arg to the constructor and m_zip_dict_name field to PT_column_format_column_attr class, implement apply_zip_dict; - add m_zip_dict field to PT_field_def_base, get its value in contextualize_attrs method, and pass it to Alter_info instance in PT_column_def::contextualize and PT_alter_table_change_column::contextualize - rename PT_unique_key_column_attr to PT_unique_combo_clustering_key_column_attr, make it handle both UNIQUE and CLUSTERING - move zip_dict_name from LEX to HA_CREATE_INFO in line with the upstream LEX reduction in parser refactoring. - Typedef new alter tablespace option PT_alter_tablespace_option_encryption for general tablespace encryption. Performance Schema - Adjust PFS object declarations for the new volatility and documentation fields, account for PSI_FLAG_GLOBAL rename to PSI_FLAG_SINGLETON. - Move server PSF memory key declarations to sql/psi_memory_key.h and sql/psi_memory_key.cc - Adjust inline_mysql_unix_connect PFS instrumentation in include/mysql/psi/mysql_file.h to modern PFS instrumentation pattern which was introduced back in [3] because the old one stopped compiling in 8.0. While at that, switch from HAVE_PSI_INTERFACE to HAVE_PSI_FILE_INTERFACE. Audit log plugin - Converted the plugin to C++. - Made a lot of declarations in audit_log.cc static and/or const. - Converted hash tables in filter.cc to collation_unordered_set<std::string>. At the same time convert the hash table value struct creator/initialization functions to constructors. Backup locks - Remove everything associated with LOCK BINLOG FOR BACKUP / UNLOCK BINLOG due to upstream performance_schema.log_status table. - Refer to Oracle backup lock as "instance backup lock" and Percona backup lock as "table backup lock" to try to reduce confusion between the two. Expanded fast index creation - Change the type of delayed_key_list container in Alter_info from List to Mem_root_array. Threadpool - Re-implement TP_STATISTICS::num_worker_threads, pool_timer_t::current_microtime and next_timeout_check using C++11 atomics. - Convert Worker_thread_context struct to class with constructor/destructor instead of save/restore methods, to be used as RAII. New data dictionary - Percona features that need to store information in the new data dictionary (CLUSTERED keys for TokuDB; compression dictionary information for InnoDB have not been re-implemented for the new DD yet). For now the legacy implementation is disabled, to be removed together with the new implementation, except for the reading bits, because they are needed for upgrades from 5.7. Specifically, leave SYSTEM_TABLE_NAME patch in, dict_process_sys_zip_dict function Changed page tracking - Move fil_name_process patch not to cause any tablespace in filesystem changes when called by redo log tracking to fil_tablespace_redo_delete and fil_tablespace_redo_create. - Introduce apply arg for fil_tablespace_redo_encryption for redo log parsing mode only. - log_get_tracked_lsn and log_set_tracked_lsn have been remvoed after the tracked LSN conversion to C++11 atomic variable. - Fold log_check_tracking_margin to log0write.cc:log_write_write_buffer. - Move redo log signalling from log_io_complete_checkpoint to log0chkp.cc:log_checkpoint. - Drop log_group_read_log_seg release_mutex arg (which would have been moved to recv_read_log_seg) because of log_sys->mutex removal. - innobase_logs_empty_and_mark_at_shutdown patches moved to srv_log_shutdown. - Due to os_file_dir_t and associated function removals in [13], rewrite their users in log0online.cc to use MY_DIR and my_dir from mysys. - Remove log_group_t mentions from log0online.cc - Add handling for MLOG_COMP_PAGE_CREATE_SDI, MLOG_PAGE_CREATE_SDI, and MLOG_TABLE_DYNAMIC_META MTR types. - Adjust redo log reading code for log_group_read_log_seg rename to recv_read_log_seg, convert the latter from static to public function. - Refactor out redo log tracker-waiting code in fil0fil.cc to a new private static method Fil_system::wait_for_changed_page_tracker. Other changes due to upstream - Remove YaSSL-related source code (mysys_ssl/yassl.cpp, HAVE_YASSL check, build-binary.sh --with-yassl option) - Replace my_global.h includes with the needed smaller ones (my_inttypes.h, mysql/service_mysql_alloc.h, my_macros.h) due to upstream patch series "Bug #24710065: SPLIT MY_GLOBAL.H", and move my_global.h patches to appropriate places: s6_addr32 define to violite.h. - Remove THD::thr_create_utime uses and replace THD::update_server_status calls with THD::update_slow_query_status due to [30]. - Remove Sql_alloc uses due to [26]. - Adjust for class Key rename to Key_spec with declaration moved to sql/key_spec.h in [4]. - Remove NORMAL_PART_NAME uses because of [25]. - Break up the dependency between sql_class.h and handler.h by moving opt_log_slow_admin_statements and opt_log_slow_sp_statements from mysqld.h/mysqld.cc to log.h/log.cc, and including log.h instead of mysqld.h (which then includes handler.h) in sql_class.h. Also move struct fragmentation_stats_t declaration to system_variables.h from sql_thd_internal_api.h, put a forward declaration there, and do not include the latter header from the former. This is done due to [21]. - Remove THD::KILL_BAD_DATA uses due to [28]. - Remove THR_THD uses due to [29]. - Convert uses of dict_table_is_temporary to dict_table_t::is_temporary() due to [7]. - Remove TABLE_SHARE::uniques uses from sql_base.cc Table_Node::Table_Node due to upstream removal at [27]. It is not clear whether the resulting code is correct though. - Replace uses of SRV_LOG_SPACE_FIRST_ID with dict_sys_t::log_space_first_id due to [8] and the n dict_sys_t::s_log_space_first_id due to [9]. - Move srv_parallel_doublewrite_path initialisation to innobase_init_params to reflect InnoDB startup refactoring - Adjust btr_copy_externally_stored_field callers to call lob::btr_copy_externally_stored_field (WL#9141, [34]), and pass index definition to it (WL#8960, [33]). - Move dict_index_get_nth_col_pos patches to dict_index_t::get_col_pos because of [11]. - Adjust get-space-by-space-id code for fil_system mutex removal, sharding, and fil_node_complete_io -> complete_io rename, introduced in WL#8619 [10]. - Remove innodb_use_global_flush_log_at_trx_commit and commit innodb_flush_log_at_trx_commit back to global variable, and remove associated srv_use_global_flush_log_at_trx_commit source code variable and thd_flush_log_at_trx_commit function - Replace struct st_mysql_sys_var occurences with SYS_VAR due to [20]. - Adjust warn_or_info class for InnoDB logging changes at [5], which remove the actual logging from logger::~logger, pass diagnostics level to its constructor, and conditionally disable actual logging using UNIV_NO_ERR_MSGS preprocessor define. - Removed /*=...==*/ comments and replace /**...**//**\n with /** in the InnoDB patch according to [37]. - Replace reserved_system_space_name with dict_sys_t::sys_space_name because of [8], and then to dict_sys_t::s_sys_space_name because of [9]. Misc cleanups - Split assert(condition1 && condition2) to assert(condition1); assert(condition2); - Create type aliases for all the added function pointers in the handlerton interface. - Remove hash table size information leftovers that should have been removed in the 5.7 porting branch - Replace some of the ulint uses with space_id_t where applicable - Remove our #define SRV_MAX_N_PURGE_THREADS in favor of upstream MAX_PURGE_THREADS. - Remove remnants of innodb_flush_method = ALL_O_DIRECT: defines DEFAULT_SRV_LOG_WRITE_AHEAD_SIZE and MAX_SRV_LOG_WRITE_AHEAD_SIZE, xb_simulate_all_o_direct_write_failure, xb_simulate_all_o_direct_read_failure debug points. - Introduce flush_changed_page_bitmaps_t and purge_changed_page_bitmaps_t types for the corresponding handlerton methods. - Remove .is_percona_server_bzr - Replace README.MySQL contents with upstream README - Update man pages from mysql-8.0.11.tar.gz Testsuite These are partial changes only, as there was no goal to run the testsuite at this stage. - Remove uses of have_dynamic_loading.inc, have_symlink.inc, have_innodb.inc, have_perfschema.inc include files due to [31]. TokuDB, MyRocks, PAM authentication These three plugins are not handled in this merge yet. [1]: commit c7a720b Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Thu Sep 24 12:46:44 2015 +0100 Bug#21881278: SWITCH MYSYS LIBRARY FROM C TO C++ This patch switches all mysys/ files from .c to .cc. By doing this we get stricter compile checks (e.g. type checks) and it makes it possible to use C++ features for future bug fixes and extensions inside mysys. [2]: commit 4133e44 Author: Steinar H. Gunderson <steinar.gunderson@oracle.com> Date: Thu Sep 7 15:18:39 2017 +0200 Bug #26781567: REMOVE C LEGACY [noclose] Convert the remaining C files to C++, except for external source code (e.g. memcached) and a few .c files that are there to verify that our client library still builds from C code. The only real exception is libservices/, where ODR violations (services are defined as structs in some files and as void * in others) mean that linking with C++ linkage would break linking on Windows. Change-Id: I61f6a281ffcd5c0fca9f4b9976aeb580dd1b0b65 [3]: commit bd8c99e Author: Marc Alff <marc.alff@oracle.com> Date: Fri Jul 1 10:46:20 2011 +0200 Bug#12346211 PERF IMPACT OF PERFORMANCE-SCHEMA WITH DISABLED INSTRUMENTS/CONSUMERS Performance improvement #1: instrumentation interface. In the instrumentation interface in include/mysql/psi/psi.h, implemented the following changes. 1) Removal of "if (PSI_server != NULL)". Instead of having PSI_server == NULL when the performance schema is disabled, a dummy implementation of the instrumentation interface is provided in mysys/psi_noop.cc. Replacing a test by calling an empty function is similar for the disabled code path. For the instrumented code path (the performance schema is enabled), removing this test completely improves performances. 2) Push down of "if (that->m_psi != NULL)". Testing if an object is instrumented has been pushed down to the implementation of the instrumentation interface. In case of psi_noop.cc, this test is not needed, which improves performances for the disabled case. 3) Removal of 1 "if (locker != NULL)" test. Before the fix, the coding pattern was: locker= ... if (locker != NULL) PSI_server->start_event() <instrumented code> if (locker != NULL) PSI_server->end_event() After the fix, the coding pattern is: locker= ... if (locker != NULL) { PSI_server->start_event() <instrumented code> PSI_server->end_event() } else { <instrumented code> } This saves an extra if() in both the instrumented and non instrumented code path. 4) Packing of helper structures. Unused attributes such as m_src_file and m_src_line have been removed. Attributes have been reordered to: - avoid holes caused by padding - put attributes used very frequently first 5) Use PSI_CALL() Calls to the performance schema implementation have been changed from: PSI_server->function(); to PSI_CALL(function)(); This is cosmetic only, and can help later to make static calls to pfs_function() instead of using the PSI_server pointer. [4]: commit 3f3c259 Author: Tor Didriksen <tor.didriksen@oracle.com> Date: Thu Mar 26 16:56:31 2015 +0100 Bug#20756282 MOVE CLASS KEY AND FOREIGN_KEY OUT OF SQL_CLASS.H Move Key_part_spec, Key_spec (new name for Key) and Foreign_key_spec (new name for Foreign_key) to new files key_spec.h/cc [5]: commit 29e3723 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Feb 16 14:30:31 2018 +1100 WL#11009: Logging services: log writers: add error ID to traditional error log (InnoDB changes) Add a unique ID to all log InnoDB messages. Remove InnoDB: prefix from the pattern matching in the tests. RB#18486 Reviewed by: Ingo, Annamalai and Praveen Hulakund. [6]: commit 6c174bd Author: Steinar H. Gunderson <steinar.gunderson@oracle.com> Date: Tue Feb 21 13:39:24 2017 +0100 Bug #25597667: REMOVE MY_BOOL Replace my_bool with bool everywhere. Change-Id: Ibfe9f9b4a05d8ee82d09ffdc4f1f038ec4555b89 [7]: commit 2238b41 Author: El<C5><BC>bieta Babij <elzbieta.babij@oracle.com> Date: Tue Jul 5 12:22:26 2016 +0200 Bug#23748128 - Add some accessor methods to dict_table_t. dict_table_t::get_n_user_cols(): Get the number of user-specified columns, possibly including a hidden FTS_DOC_ID column. dict_table_t::get_n_sys_cols(): Get the number of internal system columns. dict_table_t::get_n_cols(): Get the number of all columns. dict_table_t::get_col(): Get a column. dict_table_t::get_col_name(): Get a column name. dict_table_t::get_sys_col(): Get an internal system column. [8]: commit cd7649d Author: Allen Lai <zheng.lai@oracle.com> Date: Thu Apr 13 10:59:58 2017 +0800 wl#9525 InnoDB_New_DD: update InnoDB specific metadata into new DD for DDL wl#9534 InnoDB_New_DD: Instantiate InnoDB in-memory metadata with newDD objects wl#9530 InnoDB_New_DD: FTS index support for newDD wl#9531 InnoDB_New_DD: Enable table encryption and transparent compression wl#9532 InnoDB_New_DD: InnoDB_New_DD: Move data dictionary tables to a single tablespace wl#9559 InnoDB_New_DD: Support in-place ALTER PARTITION wl#9537 InnoDB_New_DD: Support InnoDB table import/export for new DD wl#9357<C2><A0>InnoDB: Upgrade steps for new data dictionary wl#9570<C2><A0>Extend SHOW statements to list hidden columns and index information wl#9461<C2><A0>Upgrade for Innodb Internal Dictionary Migration This is the InnoDB side of work to support an unified metadata system between MySQL server and Storage Engines. It removes the InnoDB dependency on InnoDB system tables, that is, all InnoDB metadata are now read from server objects, instead of reading from InnoDB system tables. In these wls, we replaced dict_table_open_* (on_id/on_name) with their newDD counterpart. It replaced metadata instantiation (dict_load_table etc.) with newDD counterpart too. And also, we migrated all related features, like partition, encryption, compression and import/export etc. Upgrade from mysql-5.7 is supported as well. [9]: 3452648 Author: Jimmy Yang <jimmy.yang@oracle.com> AuthorDate: Thu Aug 17 00:34:24 2017 +0200 Commit: Jimmy Yang <jimmy.yang@oracle.com> CommitDate: Thu Aug 17 00:46:42 2017 +0200 Parent: f8076b3 Bug#25727892: Refactor Item::const_item() as a non-virtual function Merged: 5.5 5.6 mysql-5.7.7 Containing: 8.0 mysql-8.0.11-percona-patches mysql-8.0.3-percona-patches mysql-8.0.4-percona-patches (1 more) Follows: mysql-8.0.2 (1261) Precedes: mysql-8.0.3 (144) InnoDB NewDD worklogs(WL#9535 and WL#9536) to support Atomic DDL and completely remove old InnoDB system tables: 1. WL#9535: InnoDB_New_DD: Remove InnoDB System Table and modify the view of their I_S counterparts This worklog obsoletes all InnoDB System tables, which are no longer used and could only be accessed by upgrade. All metadata is now persisted and read from DD system tables in "mysql tablespace" instead of "old" InnoDB System Tables. All corresponding information_schema.innodb_sys_* tables are replaced by new internal system views. These new views have the name similar to the names of old system tables, just removing the 'SYS_' from the middle of the names. 2. WL#9536: InnoDB_New_DD: Support crash-safe DDL This worklog starts to support crash-safe DDL for InnoDB. DDLs like CREATE TABLE, DROP TABLE, RENAME TABLE, TRUNCATE TABLE, ALTER TABLE, CREATE TABLESPACE and DROP TABLESPACE are covered by this worklog. DDLs operations including DD system table updates and corresponding file changes(create, delete, rename etc.) are atomic. Physical data files would be synchronized with the transaction for DDL, thus there should be no physical data files for temporary tables or FTS auxiliary tables, etc. left after recovery. [10]: commit cf63b24 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Dec 8 12:10:42 2017 +1100 WL#8619 - InnoDB: Provide offline database portability without ISL files Summary of changes. 1. Introduce --innodb-directories := "dir1;...;dirN". This is renamed from --innodb-scan-directories which was introduced in WL#9499. Currently this is not dynamic. 2. Revert to 5.6 behaviour where we scan directories and find .ibd files. If you move tablespaces to a new location, add the path to innodb-data-directories so that InnoDB can "discover" the files on startup. 3. Tablespaces can only be created under known directories. This eliminates the need for .isl files. 4. Upgrade redo log format, not backward compatible 5. Change redo log type IDs 6. Code cleanup 7. Shard the fil system data structures for performance 8. Remove WL#9499 files that were used to track open tablespaces. RB#16842 Approved by: Jimmy and Kevin. [11]: commit b3ba94c Author: El<C5><BC>bieta Babij <elzbieta.babij@oracle.com> Date: Tue Jul 26 15:00:04 2016 +0200 Bug #24361023 - Add accessors to indexes. dict_index_t::add_field(): Replaces dict_mem_index_add_field(). dict_index_t::get_field(): Replaces dict_index_get_nth_field(). dict_index_t::get_col(): Replaces dict_index_get_nth_col(). dict_index_t::get_col_no(): Replaces dict_index_get_nth_col_no(). dict_index_t::get_col_pos(): Replaces dict_index_get_nth_col_pos() and dict_index_get_nth_col_or_prefix_pos(). dict_index_t::get_sys_col_pos(): Replaces dict_index_get_sys_col_pos(). RB#13487 Reviewed by Kevin Lewis [12]: commit 98bba73 Author: El<C5><BC>bieta Babij <elzbieta.babij@oracle.com> Date: Mon May 23 10:21:20 2016 +0200 Bug #23329353 - Replace ut_snprint with the C++11 snprintf function. [13]: commit 31350e8 Author: Gopal Shankar <gopal.shankar@oracle.com> Date: Tue Nov 3 13:27:06 2015 +0530 WL#6378 New data dictionary (umbrella). (...) [14]: commit f98b7bc Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Wed Jan 18 10:39:10 2017 +0100 Bug#25413703: REMOVE #IFDEF HAVE_REPLICATION After the removal of embedded server, the HAVE_REPLICATION ifdef is always true and can therefore be removed. [15]: commit 18c6270 Author: Tor Didriksen <tor.didriksen@oracle.com> Date: Tue Nov 8 09:55:35 2016 +0100 WL#9816 Do not build/package/test embedded server - Remove/Cleanup all cmake related code pertaining to embedded server - Remove/Cleanup embedded server from RMP and Debian packages - Remove embedded tests from our mtr test collections - Remove all source code in the libmysqld directory - Remove embedded code from mysql-test-run Change-Id: Ib2432e50043f060f1da64328815cdb9d53981de4 [16]: commit 0a68d5d Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Mon Jan 16 12:35:47 2017 +0100 Bug#25304844: REMOVE EMBEDDED_LIBRARY #IFDEF Remove dead code related to embedded server. Part 5: Remove EMBEDDED_LIBRARY ifdef from client/ and storage/ This patch also removes the --server-arg option from 'mysql'. This option was embedded server specific and undocumented. --server-arg and --server-file are, for the same reason, removed from 'mysqltest'. [17]: commit db3fd68 Author: Gleb Shchepa <gleb.shchepa@oracle.com> Date: Thu Dec 7 16:21:28 2017 +0400 WL#8989: Create an internal API for the MySQL parser to enumerate reserved and non-reserved words Part I. Grammar preparation. * Replace the existent type tag <symbol> with <keyword> for a better readability. * Add the new type tag <keyword> to currently untyped terminal and non-terminal symbols. Part II. Create a simple compile-time program for transforming sql_yacc.yy into a .h file that contains an array of reserved and non-reserved words. * Find %token declarations with the <keyword> type tag by a regexp. * Separate %type declarations are ignored for the simplicity. Part III. Introduce a new system view: INFORMATION_SCHEMA.keywords. * Hardcode the statical list of keywords and reserved words into a view on top of a long JSON_TABLE(...) expression. * Use a trick in the .test file to trace changes in keywords: notify the developer to update the target I_S version in the mysqld binary where necessary -- this will force DD to refresh I_S view declarations on the next run. [18]: commit a2f3a60 Author: Marc Alff <marc.alff@oracle.com> Date: Thu Mar 10 10:10:25 2016 +0100 Bug#22906606 IMPROVE DIGEST STABILITY Before this fix, adding new tokens in the parser caused query digests to change. The root cause was that tokens are typically added in the middle of the list, causing renumbering of tokens. With this fix, the process is now changed and documented: new tokens should be added at the end of the list. The tool gen_lex_token, used during the build, is improved to enforce this process. [19]: commit f9d1c83 Author: Tatiana Azundris Nuernberg <tatjana.nuernberg@oracle.com> Date: Sat Mar 3 13:31:37 2018 +0000 WL#11150: Add <component> to error log messages by default Adds another key/value pair to log events. The new field "subsystem" will contain one of the values "InnoDB" (for errors thrown in the InnoDB storage engine proper), "Repl" (for those thrown by the replication subsystem), or "Server" (otherwise). This value is also added to the traditional log. This field will appear between '[' <error_number> ']' and '[' <error_text> ']'. [20]: commit 545b168 Author: Steinar H. Gunderson <steinar.gunderson@oracle.com> Date: Wed Sep 13 12:57:13 2017 +0200 Bug #26781567: REMOVE C LEGACY Replace the final typedefs to st_* and *_st structs. Note in particular that st_mysql_show_var was called sometimes SHOW_VAR and sometimes STATUS_VAR (by way of duplicated forward-declaring typedefs). This patch changes it so that it's universally SHOW_VAR, which seems to match better with the original type name. Change-Id: Ide483ceac31284d909c42adb46496308dd4d4f85 [21]: commit f58cb36 Author: Steinar H. Gunderson <steinar.gunderson@oracle.com> Date: Tue Oct 10 14:54:53 2017 +0200 Bug #26927386: REDUCE WEIGHT OF SQL_CLASS.H [noclose] Remove field.h, handler.h and sql_profile.h from sql_class.h, and add a simple test to make sure they don't get back. Takes sql_class.h down from approx 1.0 to 0.85 seconds (GCC 7.2.0, -O2 -g). Change-Id: I72618c977c7af330d1215899971e2f9e49ab7b88 [22]: commit be029db Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Thu Jan 19 15:14:58 2017 +0100 Bug#25428042: REPLACE MYSQL_CLIENT IFDEF WITH !MYSQL_SERVER Replace the MYSQL_CLIENT #ifdef with !MYSQL_SERVER to simplify conditional compilation. Also remove unnecessary usage of MYSQL_SERVER and move setting of MYSQL_SERVER from C++ to CMake. [23]: commit f5c32ea Author: Gleb Shchepa <gleb.shchepa@oracle.com> Date: Thu Jul 14 21:48:55 2016 +0400 WL#8067: True bottom-up server parser: refactoring of the CREATE TABLE statement WL#8434: True bottom-up server parser: refactoring of partitioning-related stuff WL#8435: True bottom-up server parser: cleanup and refactoring column definition stuff WL#8433: Separate DD commands from regular SQL queries in the parser grammar WL#7840: Allow parsing a single expression [24]: commit ed007c8 Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Thu Dec 22 11:42:43 2016 +0100 Bug#25304844: REMOVE EMBEDDED_LIBRARY #IFDEF Remove dead code related to embedded server. Part 3: Remove NO_EMBEDDED_ACCESS_CHECKS ifdef [25]: commit 11d6946 Author: Dmitry Lenev <dmitry.lenev@oracle.com> Date: Fri Apr 21 09:52:09 2017 +0300 Removed old partitioning DDL implementation which is no longer in use. After WL#8971 "Deprecate and remove partition engine" and WL#9559 "InnoDB_New_DD: Support in-place ALTER PARTITION" code implementing "fast" partitioning DDL (fast_alter_partition_table() and related functions) became unused. This patch removes this dead code (including DDL_LOG implementation used by it) as a follow-up to the above WLs. [26]: commit 5fecdfa Author: Steinar H. Gunderson <steinar.gunderson@oracle.com> Date: Thu Sep 21 14:40:20 2017 +0200 Bug #26030132: REMOVE SQL_ALLOC Finally remove Sql_alloc, now that 8.0 has been branched off. This is a no-op change for all existing code; there is no code that does new Foo() inheriting from Sql_alloc, since that overload doesn't exist, and similarly no code that deletes such objects, since operator delete is private. After this, it is possible to write such code, and it will do the obvious thing (allocate on the regular heap, and destroy similarly). As usual, code that wants to allocate on a MEM_ROOT must say so explicitly by use of overloaded new. Change-Id: I2e98b39daa06d6dfd8b8c5b1d45a85b1ddc02b3b [27]: commit 4880f97 Author: Guilhem Bichot <guilhem.bichot@oracle.com> Date: Tue Nov 29 15:46:27 2016 +0100 WL#883 non-recursive CTE WL#3634 recursive CTE (...) [28]: commit 597a259 Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Tue Oct 6 14:04:03 2015 +0100 Bug#21944866: REFACTORING: REMOVE KILL_BAD_DATA Remove KILL_BAD_DATA from the THD::killed_state enum. KILL_BAD_DATA was only set by the connection itself when a warning was upgraded to error by STRICT mode. It therefore had nothing to do with the connection being killed by other connections (e.g. KILL QUERY / KILL CONNECTION). This is a refactoring with no changes in behavior. [29]: commit 2bf852b Author: Steinar H. Gunderson <steinar.gunderson@oracle.com> Date: Wed Apr 12 12:26:14 2017 +0200 Bug #25909342: USE C++11 THREAD-LOCAL STORAGE FOR CURRENT_THD Our current implementation of current_thd uses mysys' my_thread_* API, which maps onto pthreads (at least for Linux/UNIX). C++11 supports native TLS, which means we just need two simple MOV instructions to do a TLS fetch instead of a full shared library call; the interface is also much simpler. We probably don't have a lot of performance gain, since we don't pick out current_thd all that often, but it's a nice cleanup to do nevertheless. There are still some uses of the my_thread_* API, particularly in PFS; cleaning this up is for a later patch. Change-Id: Ia16e0c1ccfa52af815882bcddd1eb3be80f77c61 [30]: commit 162e016 Author: Jon Olav Hauglid <jon.hauglid@oracle.com> Date: Tue Mar 15 14:32:39 2016 +0100 Bug#22948828: REFACTOR USAGE OF MY_MICRO_TIME() Remove redundant calls to my_micro_time() to simplify code and reduce the usage of gettimeofdat(). Especially focused on code for handling new connections. sql/conn_handler/connection_handler_one_thread.cc - Removed my_micro_time() call. THD::start_utime has just been set by THD::THD() => THD::init(). THD::thr_create_utime is only needed by connection_handler_per_thread.cc sql/conn_handler/connection_handler_per_thread.cc - Removed my_micro_time() call. THD::start_utime has just been set by THD::THD() => THD::init(). Remove THD::thr_create_utime, we can use THD::start_utime instead. sql/event_queue.cc - Replaced usage of THD::set_current_time() with THD::set_time(). THD::set_time() does the same as long as THD::user_time is NULL which it is for events. Allows the removal of THD::set_current_time() sql/events.cc - Removed THD::set_time() call. THD::set_time() has just been called by THD::THD() => THD::init(). sql/log.cc - Removed default argument value for make_iso8601_timestamp() for clarity. - Reduced critical section of LOCK_logger. - Call my_micro_time() directly instead of the unnecessary THD::current_utime() wrapper. sql/log_event.cc - Use the existing THD::query_start() function instead of accessing THD::start_time directly. - Renamed THD::update_server_status() to THD::check_slow_query() since the old name was somewhat misleading. sql/sql_class.h - Removed THD::thr_create_utime, no longer needed. - Removed redundant inline keyword, added const where possible. - Removed THD::query_start_usec(), not used. - Removed THD::query_start_timeval(), no longer needed. - Removed THD::set_current_time(), no longer needed. - Removed THD::is_valid_time(), code moved to sql_parse.cc - Renamed THD::update_server_status() to THD::check_slow_query() since the old name was somewhat misleading. sql/sql_connect.cc - Use THD::start_utime instead of removed THD::thd_create_utime (had the same value) - Removed call to THD::set_time(). Called by THD::init_for_queries() right afterwards anyway. sql/sql_thd_internal_api.cc - Removed THD::set_time() call. THD::set_time() has just been called by THD::THD() => THD::init(). - Removed call to my_micro_time(). Already done by THD::set_time(). [31] commit 485545b Author: Mohit Joshi <mohit.joshi@oracle.com> Date: Tue Jan 17 11:56:43 2017 +0530 Bug#25166686::REMOVE ALL THE REDUNDANT .INC FILES FROM MTR Description: It is noticed that there are a lot of redundant inc files which are not being used by any of the MTR tests. Also, there are several inc files which are included in the test but are not needed anymore. Fix: Below inc files are not being used by any test and hence removed: 1. include/check_key_reads.inc 2. include/check_key_req.inc 3. include/commandline_option_test.inc 4. include/config_file_option_test.inc 5. include/explain.inc 6. include/finish_option_test.inc 7. include/have_binlog_checksum_off.inc 8. include/have_dynamic_loading.inc 9. include/have_shm.inc 10. include/init_option_test.inc 11. include/not_blackhole.inc 12. include/show_partition_checksums.inc 13. include/show_partition_contents.inc 14. include/test_outfile.inc Below are a few inc files which are used in MTR tests but are no more needed. Hence these inc files along with their references from all MTR tests have been removed. 1. have_symlink.inc 2. have_perfschema.inc 3. have_innodb.inc Note: include/not_valgrind_server.inc is not being used by any MTR test but still not removed as this was introduced as part of Bug#22447330 Reviewed by: Parveez Baig <parveez.baig@oracle.com> Deepa Dixit <deepa.dixit@oracle.com> [32] commit a0d0cab Author: Sivert Sorumgard <sivert.sorumgaard@oracle.com> Date: Thu Jun 30 16:00:31 2016 +0200 WL#8971: Deprecate and remove partitioning storage engine This worklog implements the following main changes: 1. Modify cmake files, do not build partitioning engine. 2. Remove files implementing the partitioning engine. 3. Remove some mtr tests using non-native partitioning. 4. Rewrite some mtr tests to use native partitioning. Various adjustments necessary, e.g. regarding EXPLAIN output. 5. Introduce error handling at the SQL layer to fail if using partitioning related DDL statements for engines not supporting it natively. 6. Remove reference to the DB_TYPE_PARTITIONING_DB internally, but keep the enum entry to avoid disturbing other values. 7. Add an mtr test testing the error handling. 8. Add an mtr test to check for regressions in support for partitioning in ndb. 9. Copy comments from storage/partition/ha_partition.* to sql/handler.*, include/my_base.h etc. [33] commit eda9201 Author: Annamalai Gurusami <annamalai.gurusami@oracle.com> Date: Thu Apr 28 12:29:00 2016 +0530 WL#9141 InnoDB: Refactor uncompressed BLOB code to facilitate partial fetch/update (...) [34] commit eda9201 Author: Annamalai Gurusami <annamalai.gurusami@oracle.com> Date: Thu Apr 28 12:29:00 2016 +0530 WL#9141 InnoDB: Refactor uncompressed BLOB code to facilitate partial fetch/update (...) [35] commit 2eec7f3 Author: Marko M<C3><A4>kel<C3><A4> <marko.makela@oracle.com> Date: Fri Apr 22 10:50:45 2016 +0200 Bug#23150562 REMOVE UNIV_MUST_NOT_INLINE AND UNIV_NONINL Problem: InnoDB used to support a compilation mode that allowed to choose whether the function definitions in .ic files are to be inlined or not. This stopped making sense when InnoDB moved to C++ in MySQL 5.6 (and ha_innodb.cc started to #include .ic files), and more so in MySQL 5.7 when inline methods and functions were introduced in .h files. Solution: Remove all references to UNIV_NONINL and UNIV_MUST_NOT_INLINE from all files, assuming that the symbols are never defined. Remove the files fut0fut.cc and ut0byte.cc which only mattered when UNIV_NONINL was defined. RB: 12470 Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com> [36]: commit 644aafd Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue May 24 19:02:55 2016 +0530 WL#9359 - Code cleanup [37]: commit 59b560e Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Feb 6 22:00:10 2018 +1100 Bug#27496251 REMOVE /*====*/ AND CHANGE /****/ /***/ TO /** .... */

Use void* for function arguments, and cast Item_field* in function body. compare_fields_by_table_order(Item_field*, Item_field*, void*) through pointer to incorrect function type 'int (*)(void *, void *, void *)' sql/sql_optimizer.cc:3904: note: compare_fields_by_table_order(Item_field*, Item_field*, void*) defined here #0 0x34ac57d in base_list::sort(int (*)(void*, void*, void*), void*) sql/sql_list.h:278:13 #1 0x49ad303 in substitute_for_best_equal_field(Item*, COND_EQUAL*, Change-Id: I4f5417304a24201682f32fc7631034de7aa62589

…ENERATED_READ_FIELDS It's a SELECT with WHERE "(-1) minus 0x4d". this operation has a result type of "unsigned" (because 0x4d is unsigned integer) and the result (-78) doesn't fit int an unsigned type. This WHERE is evaluated by InnoDB in index condition pushdown: #0 my_error #1 Item_func::raise_numeric_overflow ... #7 Item_cond_and::val_int #8 innobase_index_cond ... #12 handler::index_read_map ... #15 handler::multi_range_read_next ... #20 rr_quick #21 join_init_read_record As val_int() has no "error" return code, the execution continues until frame #12; there we call update_generated_read_fields(), which has an assertion about thd->is_error() which fails. Fix: it would be nice to detect error as soon as it happens, i.e. in innodb code right after it calls val_bool(). But innodb's index condition pushdown functions only have found / not found return codes so they cannot signal "error" to the upper layers. Same is true for MyISAM. Moreover, "thd" isn't easily accessible there. Adding a detection a bit above in the stack (handler::* functions which do index reads) is also possible but would require fixing ~20 functions. The chosen fix here is to change update_generated_*_fields() to return error if thd->is_error() is true. Note that the removed assertion was already one cause of bug 27041382.

Group Replication does implement conflict detection on multi-primary to avoid write errors on parallel operations. The conflict detection is also engaged in single-primary mode on the particular case of primary change and the new primary still has a backlog to apply. Until the backlog is flushed, conflict detection is enabled to prevent write errors between the backlog and incoming transactions. The conflict detection data, which we name certification info, is also used to detected dependencies between accepted transactions, dependencies which will rule the transactions schedule on the parallel applier. In order to avoid that the certification info grows forever, periodically all members exchange their GTID_EXECUTED set, which full intersection will provide the set of transactions that are applied on all members. Future transactions cannot conflict with this set since all members are operating on top of it, so we can safely remove all write-sets from the certification info that do belong to those transactions. More details at WL#6833: Group Replication: Read-set free Certification Module (DBSM Snapshot Isolation). Though a corner case was found on which the garbage collection was purging more data than it should. The scenario is: 1) Group with 2 members; 2) Member1 executes: CREATE TABLE t1(a INT, b INT, PRIMARY KEY(a)); INSERT INTO t1 VALUE(1, 1); Both members have a GTID_EXECUTED= UUID:1-4 Both members certification info has: Hash of item in Writeset snapshot version (Gtid_set) #1 UUID1:1-4 3) member1 executes TA UPDATE t1 SET b=10 WHERE a=1; and blocks immediately before send the transaction to the group. This transaction has snapshot_version: UUID:1-4 4) member2 executes TB UPDATE t1 SET b=10 WHERE a=1; This transaction has snapshot_version: UUID:1-4 It goes through the complete patch and it is committed. This transaction has GTID: UUID:1000002 Both members have a GTID_EXECUTED= UUID:1-4:1000002 Both members certification info has: Hash of item in Writeset snapshot version (Gtid_set) #1 UUID1:1-4:1000002 5) member2 becomes extremely slow in processing transactions, we simulate that by holding the transaction queue to the GR pipeline. Transaction delivery is still working, but the transaction will be block before certification. 6) member1 is able to send its TA transaction, lets recall that this transaction has snapshot_version: UUID:1-4. On conflict detection on member1, it will conflict with #1, since this snapshot_version does not contain the snapshot_version of #1, that is TA was executed on a previous version than TB. On member2 the transaction will be delivered and will be put on hold before conflict detection. 7) meanwhile the certification info garbage collection kicks in. Both members have a GTID_EXECUTED= UUID:1-4:1000002 Its intersection is UUID:1-4:1000002 Both members certification info has: Hash of item in Writeset snapshot version (Gtid_set) #1 UUID1:1-4:1000002 The condition to purge write-sets is: snapshot_version.is_subset(intersection) We have "UUID:1-4:1000002".is_subset("UUID:1-4:1000002) which is true, so we remove #1. Both members certification info has: Hash of item in Writeset snapshot version (Gtid_set) <empty> 8) member2 gets back to normal, we release transaction TA, lets recall that this transaction has snapshot_version: UUID:1-4. On conflict detection, since the certification info is empty, the transaction will be allowed to proceed, which is incorrect, it must rollback (like on member1) since it conflicts with TB. The problem it is on certification garbage collection, more precisely on the condition used to purge data, we cannot leave the certification info empty otherwise this situation can happen. The condition must be changed to snapshot_version.is_subset_not_equals(intersection) which will always leave a placeholder to detect delayed conflicting transaction. So a trace of the solution is (starting on step 7): 7) meanwhile the certification info garbage collection kicks in. Both members have a GTID_EXECUTED= UUID:1-4:1000002 Its intersection is UUID:1-4:1000002 Both members certification info has: Hash of item in Writeset snapshot version (Gtid_set) #1 UUID1:1-4:1000002 The condition to purge write-sets is: snapshot_version.is_subset_not_equals(intersection) We have "UUID:1-4:1000002".is_subset_not_equals("UUID:1-4:1000002) which is false, so we do not remove #1. Both members certification info has: Hash of item in Writeset snapshot version (Gtid_set) #1 UUID1:1-4:1000002 8) member2 gets back to normal, we release transaction TA, lets recall that this transaction has snapshot_version: UUID:1-4. On conflict detection on member2, it will conflict with #1, since this snapshot_version does not contain the snapshot_version of #1, that is TA was executed on a previous version than TB. This is the same scenario that we see on this bug, though here the pipeline is being blocked by the distributed recovery procedure, that is, while the joining member is applying the missing data through the recovery channel, the incoming data is being queued. Meanwhile the certification info garbage collection kicks in and purges more data that it should, the result it is that conflicts are not being detected.

… work correctly for nes… Upstream commit ID : fb-mysql-5.6.35/6ddedd8f1e0ddcbc24e8f9a005636c5463799ab7 PS-4476 : Merge prod201801 Summary: …ted statements Variant #1: When the statement fails, we should roll back to the latest savepoint taken at the top level. Closes facebook/mysql-5.6#804 Differential Revision: D7509380 Pulled By: hermanlee fbshipit-source-id: 9a6f414

A subset of binlog encryption tests was crashing with: * thread percona#39, stop reason = signal SIGSTOP frame #0: 0x00007fff56063b66 libsystem_kernel.dylib`__pthread_kill + 10 frame #1: 0x00007fff5622e080 libsystem_pthread.dylib`pthread_kill + 333 frame #2: 0x000000010657442b mysqld-debug`my_write_core(sig=11) at stacktrace.cc:278 frame #3: 0x0000000104d84334 mysqld-debug`::handle_fatal_signal(sig=11) at signal_handler.cc:254 frame #4: 0x00007fff56221f5a libsystem_platform.dylib`_sigtramp + 26 frame #5: 0x00007fff5622934d libsystem_pthread.dylib`pthread_mutex_lock + 1 frame #6: 0x0000000106578d05 mysqld-debug`native_mutex_lock(mutex=0x0000000000000000) at thr_mutex.h:93 frame #7: 0x0000000106578a57 mysqld-debug`safe_mutex_lock(mp=0x0000000000000000, try_lock=false, file="/Users/laurynas/percona/mysql-server/mysys/mf_iocache2.cc", line=113) at thr_mutex.cc:70 frame #8: 0x000000010653cd3a mysqld-debug`my_mutex_lock(mp=0x00007ffb6b215038, file="/Users/laurynas/percona/mysql-server/mysys/mf_iocache2.cc", line=113) at thr_mutex.h:180 frame #9: 0x000000010653b2cc mysqld-debug`inline_mysql_mutex_lock(that=0x00007ffb6b215038, src_file="/Users/laurynas/percona/mysql-server/mysys/mf_iocache2.cc", src_line=113) at mysql_mutex.h:267 * frame #10: 0x000000010653b0d8 mysqld-debug`my_b_append_tell(info=0x00007ffb6b214fd8) at mf_iocache2.cc:113 frame #11: 0x0000000105ed6a96 mysqld-debug`MYSQL_BIN_LOG::write_buffer(this=0x00007ffb6b214cb8, buf="", len=47, mi=0x00007ffb6b1f6a00) at binlog.cc:7128 frame #12: 0x0000000105f4d54b mysqld-debug`queue_event(mi=0x00007ffb6b1f6a00, buf="", event_len=47, do_flush_mi=true) at rpl_slave.cc:7756 frame #13: 0x0000000105f3a243 mysqld-debug`::handle_slave_io(arg=0x00007ffb6b1f6a00) at rpl_slave.cc:5382 frame #14: 0x00000001065b87a5 mysqld-debug`pfs_spawn_thread(arg=0x00007ffb6a543af0) at pfs.cc:2836 frame #15: 0x00007fff5622b661 libsystem_pthread.dylib`_pthread_body + 340 frame #16: 0x00007fff5622b50d libsystem_pthread.dylib`_pthread_start + 377 frame #17: 0x00007fff5622abf9 libsystem_pthread.dylib`thread_start + 13 This was caused by my_b_append_tell trying to lock a nullptr IO_CACHE::append_buffer_lock. The lock was nullptr, because it's only initialized for SEQ_READ_APPEND IO_CACHEs, whereas we have WRITE_CACHE. This mismatch was introduced by WL#8599 [1] changing the IO_CACHE type from the former to the latter. Fix by using the correct API for the new IO_CACHE type: my_b_tell instead of my_b_append_tell. [1]: commit dbd2ca2 Author: Joao Gramacho <joao.gramacho@oracle.com> Date: Tue Nov 1 06:45:39 2016 +0000 WL#8599: Reduce contention in IO and SQL threads (...)

create_table_info_t::create_table_def leaked memory in the case enable_encryption(table) call failed: worker[5] Sanitizer report from /tmp/results/PS/mysql-test/var/5/log/mysqld.2.err after tests: binlog_encryption.binlog_encryption_without_keyring group_replication.gr_change_master_hidden group_replication.gr_server_uuid_matches_group_name group_replication.gr_perfschema_connect_status group_replication.gr_single_primary_and_leader_election_on_error group_replication.gr_without_perfschema rpl.rpl_key_rotation -------------------------------------------------------------------------- ==14131==ERROR: LeakSanitizer: detected memory leaks Direct leak of 1136 byte(s) in 1 object(s) allocated from: #0 0x7fe9233f1602 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602) #1 0xc692483 in ut_allocator<unsigned char>::allocate(unsigned long, unsigned char const*, unsigned int, bool, bool) storage/innobase/include/ut0new.h:608 #2 0xc692483 in mem_heap_create_block_func(mem_block_info_t*, unsigned long, unsigned long) storage/innobase/mem/memory.cc:281 #3 0xb99ff96 in mem_heap_create_func storage/innobase/include/mem0mem.ic:464 #4 0xbae8604 in create_table_info_t::create_table_def(dd::Table const*) storage/innobase/handler/ha_innodb.cc:10349 #5 0xbaee018 in create_table_info_t::create_table(dd::Table const*) storage/innobase/handler/ha_innodb.cc:12420 #6 0xbaf1aba in int innobase_basic_ddl::create_impl<dd::Table>(THD*, char const*, TABLE*, HA_CREATE_INFO*, dd::Table*, bool, bool, bool, unsigned long, unsigned long) storage/innobase/handler/ha_innodb.cc:12805 #7 0xbaf7e6a in ha_innobase::create(char const*, TABLE*, HA_CREATE_INFO*, dd::Table*) storage/innobase/handler/ha_innodb.cc:13756 #8 0x2857f7a in ha_create_table(THD*, char const*, char const*, char const*, HA_CREATE_INFO*, List<Create_field> const*, bool, bool, dd::Table*) sql/handler.cc:5156 #9 0x19d0d9f in rea_create_base_table sql/sql_table.cc:991 #10 0x19d0d9f in create_table_impl sql/sql_table.cc:7118 #11 0x19d37cf in mysql_create_table_no_lock(THD*, char const*, char const*, HA_CREATE_INFO*, Alter_info*, unsigned int, bool, bool*, handlerton**) sql/sql_table.cc:7200 #12 0x19dffb2 in mysql_create_table(THD*, TABLE_LIST*, HA_CREATE_INFO*, Alter_info*) sql/sql_table.cc:7950 #13 0x3b58b9b in Sql_cmd_create_table::execute(THD*) sql/sql_cmd_ddl_table.cc:319 #14 0x15917c1 in mysql_execute_command(THD*, bool) sql/sql_parse.cc:4417 #15 0x15b086e in mysql_parse(THD*, Parser_state*, bool) sql/sql_parse.cc:5139 #16 0x8efc7fd in Query_log_event::do_apply_event(Relay_log_info const*, char const*, unsigned long) sql/log_event.cc:5295 #17 0x8f7ea48 in Log_event::apply_event(Relay_log_info*) sql/log_event.cc:3882 #18 0x91cb682 in apply_event_and_update_pos sql/rpl_slave.cc:4352 #19 0x9215e69 in exec_relay_log_event sql/rpl_slave.cc:4812 #20 0x9254685 in handle_slave_sql sql/rpl_slave.cc:6912 #21 0xb1913a3 in pfs_spawn_thread storage/perfschema/pfs.cc:2836 #22 0x7fe9231436b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9) Fix by adding the missing mem_heap_free(heap) call.

Avoid undefined behavior in audit_log_update_thd_local by avoiding passing NULL as source pointer to memcpy, even with zero length. The UBSan report fixed is /usr/include/x86_64-linux-gnu/bits/string3.h:53:71: runtime error: null pointer passed as argument 2, which is declared to never be null #0 0x7fe5aad56fb1 in memcpy /usr/include/x86_64-linux-gnu/bits/string3.h:53 #1 0x7fe5aad56fb1 in audit_log_update_thd_local plugin/audit_log/audit_log.cc:987 #2 0x7fe5aad56fb1 in audit_log_notify plugin/audit_log/audit_log.cc:1105 #3 0x1ecac37 in plugins_dispatch sql/sql_audit.cc:1284 #4 0x1ecac37 in event_class_dispatch sql/sql_audit.cc:1322 #5 0x1ecb311 in event_class_dispatch_error sql/sql_audit.cc:1340 #6 0x1ed21b1 in mysql_audit_notify(THD*, mysql_event_connection_subclass_t, char const*, int) sql/sql_audit.cc:438 #7 0x1350071 in check_connection sql/sql_connect.cc:868 #8 0x1350071 in login_connection sql/sql_connect.cc:929 #9 0x1357881 in thd_prepare_connection(THD*, bool) sql/sql_connect.cc:1084 #10 0x1e66347 in handle_connection sql/conn_handler/connection_handler_per_thread.cc:313 #11 0xb1913a3 in pfs_spawn_thread storage/perfschema/pfs.cc:2836 #12 0x7fe5d352f6b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9) #13 0x7fe5d0bd741c in clone (/lib/x86_64-linux-gnu/libc.so.6+0x10741c)

… keyring Issue: when the server is started without a keyring, the redo log encryption checks were never performed, redo log encryption wasn't initialized, but the setting was left on, misleading the user. Later operations which triggered this check could possibly fail. Issue #2: redo log encryption didn't work properly when it was turned on using the command line: in some cases, the redo log remained unencrypted (until another operation triggered the redo log encryption setup methods to be called again) Both issues were caused by the refactoring of the periodic, once every second encryption checks in upstream, in PS-5189. This commit: * changes the code so redo log encryption routines are called at least twice during every startup with the encryption settings on. This fixes issue #2. * changes the code so redo log encryption routines are called at least once during every startup, even without a keyring, or a read-only server: this fixes issues #1 * adds a test ensuring that in an invalid configuration (without a keyring) the the server remains running, but the redo log encryption setting correctly displays the OFF status * adds a test ensuring that when the redo log is correctly configured (using the dynamic variable or a command line parameter) the log is encrypted, and no unencrypted data is present in it. * backports additional redo log related bugfixes/refactorings from 8.0 (92198f4). The original commit contains fixes for both the redo and the undo log, the backport only includes the redo log related parts refactored to match our changes. (cherry picked from commit c9f151e)

Upstream commit ID : fb-mysql-5.6.35/8cb1dc836b68f1f13e8b2655b2b8cb2d57f400b3 PS-5217 : Merge fb-prod201803 Summary: Original report: https://jira.mariadb.org/browse/MDEV-15816 To reproduce this bug just following below steps, client 1: USE test; CREATE TABLE t1 (i INT) ENGINE=MyISAM; HANDLER t1 OPEN h; CREATE TABLE t2 (i INT) ENGINE=RocksDB; LOCK TABLES t2 WRITE; client 2: FLUSH TABLES WITH READ LOCK; client 1: INSERT INTO t2 VALUES (1); So client 1 acquired the lock and set m_lock_rows = RDB_LOCK_WRITE. Then client 2 calls store_lock(TL_IGNORE) and m_lock_rows was wrongly set to RDB_LOCK_NONE, as below ``` #0 myrocks::ha_rocksdb::store_lock (this=0x7fffbc03c7c8, thd=0x7fffc0000ba0, to=0x7fffc0011220, lock_type=TL_IGNORE) #1 get_lock_data (thd=0x7fffc0000ba0, table_ptr=0x7fffe84b7d20, count=1, flags=2) #2 mysql_lock_abort_for_thread (thd=0x7fffc0000ba0, table=0x7fffbc03bbc0) #3 THD::notify_shared_lock (this=0x7fffc0000ba0, ctx_in_use=0x7fffbc000bd8, needs_thr_lock_abort=true) #4 MDL_lock::notify_conflicting_locks (this=0x555557a82380, ctx=0x7fffc0000cc8) #5 MDL_context::acquire_lock (this=0x7fffc0000cc8, mdl_request=0x7fffe84b8350, lock_wait_timeout=2) #6 Global_read_lock::lock_global_read_lock (this=0x7fffc0003fe0, thd=0x7fffc0000ba0) ``` Finally, client 1 "INSERT INTO..." hits the Assertion 'm_lock_rows == RDB_LOCK_WRITE' failed in myrocks::ha_rocksdb::write_row() Fix this bug by not setting m_locks_rows if lock_type == TL_IGNORE. Closes facebook/mysql-5.6#838 Pull Request resolved: facebook/mysql-5.6#871 Differential Revision: D9417382 Pulled By: lth fbshipit-source-id: c36c164e06c

…E TO A SERVER Problem ======================================================================== Running the GCS tests with ASAN seldomly reports a user-after-free of the server reference that the acceptor_learner_task uses. Here is an excerpt of ASAN's output: ==43936==ERROR: AddressSanitizer: heap-use-after-free on address 0x63100021c840 at pc 0x000000530ff8 bp 0x7fc0427e8530 sp 0x7fc0427e8520 WRITE of size 8 at 0x63100021c840 thread T3 #0 0x530ff7 in server_detected /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_transport.c:962 #1 0x533814 in buffered_read_bytes /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_transport.c:1249 #2 0x5481af in buffered_read_msg /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_transport.c:1399 #3 0x51e171 in acceptor_learner_task /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:4690 #4 0x562357 in task_loop /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/task.c:1140 #5 0x5003b2 in xcom_taskmain2 /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:1324 #6 0x6a278a in Gcs_xcom_proxy_impl::xcom_init(unsigned short, node_address*) /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/gcs_xcom_proxy.cc:164 #7 0x59b3c1 in xcom_taskmain_startup /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/gcs_xcom_control_interface.cc:107 #8 0x7fc04a2e4dd4 in start_thread (/lib64/libpthread.so.0+0x7dd4) #9 0x7fc047ff2bfc in __clone (/lib64/libc.so.6+0xfebfc) 0x63100021c840 is located 64 bytes inside of 65688-byte region [0x63100021c800,0x63100022c898) freed by thread T3 here: #0 0x7fc04a5d7508 in __interceptor_free (/lib64/libasan.so.4+0xde508) #1 0x52cf86 in freesrv /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_transport.c:836 #2 0x52ea78 in srv_unref /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_transport.c:868 #3 0x524c30 in reply_handler_task /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:4914 #4 0x562357 in task_loop /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/task.c:1140 #5 0x5003b2 in xcom_taskmain2 /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:1324 #6 0x6a278a in Gcs_xcom_proxy_impl::xcom_init(unsigned short, node_address*) /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/gcs_xcom_proxy.cc:164 #7 0x59b3c1 in xcom_taskmain_startup /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/gcs_xcom_control_interface.cc:107 #8 0x7fc04a2e4dd4 in start_thread (/lib64/libpthread.so.0+0x7dd4) previously allocated by thread T3 here: #0 0x7fc04a5d7a88 in __interceptor_calloc (/lib64/libasan.so.4+0xdea88) #1 0x543604 in mksrv /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_transport.c:721 #2 0x543b4c in addsrv /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_transport.c:755 #3 0x54af61 in update_servers /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_transport.c:1747 #4 0x501082 in site_install_action /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:1572 #5 0x55447c in import_config /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/site_def.c:486 #6 0x506dfc in handle_x_snapshot /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:5257 #7 0x50c444 in xcom_fsm /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:5325 #8 0x516c36 in dispatch_op /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:4510 #9 0x521997 in acceptor_learner_task /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:4772 #10 0x562357 in task_loop /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/task.c:1140 #11 0x5003b2 in xcom_taskmain2 /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:1324 #12 0x6a278a in Gcs_xcom_proxy_impl::xcom_init(unsigned short, node_address*) /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/gcs_xcom_proxy.cc:164 #13 0x59b3c1 in xcom_taskmain_startup /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/gcs_xcom_control_interface.cc:107 #14 0x7fc04a2e4dd4 in start_thread (/lib64/libpthread.so.0+0x7dd4) Analysis ======================================================================== The server structure is reference counted by the associated sender_task and reply_handler_task. When they finish, they unreference the server, which leads to its memory being freed. However, the acceptor_learner_task keeps a "naked" reference to the server structure. Under the right ordering of operations, i.e. the sender_task and reply_handler_task terminating after the acceptor_learner_task acquires, but before it uses, the reference to the server structure, leads to the acceptor_learner_task accessing the server structure after it has been freed. Solution ======================================================================== Let the acceptor_learner_task also reference count the server structure so it is not freed while still in use. Reviewed-by: André Negrão <andre.negrao@oracle.com> Reviewed-by: Venkatesh Venugopal <venkatesh.venugopal@oracle.com> RB: 21209

… keyring Issue: when the server is started without a keyring, the redo log encryption checks were never performed, redo log encryption wasn't initialized, but the setting was left on, misleading the user. Later operations which triggered this check could possibly fail. Issue #2: redo log encryption didn't work properly when it was turned on using the command line: in some cases, the redo log remained unencrypted (until another operation triggered the redo log encryption setup methods to be called again) Both issues were caused by the refactoring of the periodic, once every second encryption checks in upstream, in PS-5189. This commit: * changes the code so redo log encryption routines are called at least twice during every startup with the encryption settings on. This fixes issue #2. * changes the code so redo log encryption routines are called at least once during every startup, even without a keyring, or a read-only server: this fixes issues #1 * adds a test ensuring that in an invalid configuration (without a keyring) the the server remains running, but the redo log encryption setting correctly displays the OFF status * adds a test ensuring that when the redo log is correctly configured (using the dynamic variable or a command line parameter) the log is encrypted, and no unencrypted data is present in it. * backports additional redo log related bugfixes/refactorings from 8.0 (92198f4). The original commit contains fixes for both the redo and the undo log, the backport only includes the redo log related parts refactored to match our changes.

George O. Lorch III added 4 commits December 1, 2015 21:22

laurynas-biveinis reviewed Dec 3, 2015
View reviewed changes

laurynas-biveinis added a commit that referenced this pull request Dec 3, 2015

Merge pull request #1 from georgelorchpercona/tokudb-5.7

1a17bb9

Tokudb 5.7

laurynas-biveinis merged commit 1a17bb9 into laurynas-biveinis:percona-server-5.7.9-up-to-date-5.6 Dec 3, 2015

laurynas-biveinis added a commit that referenced this pull request Dec 4, 2015

Merge pull request #1 from georgelorchpercona/tokudb-5.7

5a4d76e

Tokudb 5.7

laurynas-biveinis pushed a commit that referenced this pull request Oct 25, 2016

Fix Issue #1: AUTO_INCREMENT value doesn't survive server shutdown

74a3ec1

Use ha_rocksdb::open() should check table->found_next_number_field, not table->next_number_field (like InnoDB does)

laurynas-biveinis pushed a commit that referenced this pull request Oct 25, 2016

Add remaining tests to mysql-test/suite/rocksdb. This fixes Issue #1

883b152

Two testcases are empty because RocksDB-SE doesn't support the feature: -autoincrement.test -autoinc_secondary.test

laurynas-biveinis pushed a commit that referenced this pull request Oct 25, 2016

Fix Issue #1: Enable Row-based binlogging and disable statement-based.

573b18e

laurynas-biveinis pushed a commit that referenced this pull request Oct 25, 2016

Fix Issue #1: Enable Row-based binlogging and disable statement-based.

410f5aa

laurynas-biveinis pushed a commit that referenced this pull request Jul 21, 2017

wl#9819 patch #1 : NDBD angel process reports its process id to kerne…

f51bd28

…l process

laurynas-biveinis pushed a commit that referenced this pull request Jul 21, 2017

wl#9819 Version 2. Patch #1: ndbinfo processes table definition

d1ae317

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tokudb 5.7 #1

Tokudb 5.7 #1

george-lorch commented Dec 2, 2015

laurynas-biveinis Dec 3, 2015

george-lorch Dec 3, 2015

Tokudb 5.7 #1

Tokudb 5.7 #1

Conversation

george-lorch commented Dec 2, 2015

laurynas-biveinis Dec 3, 2015

Choose a reason for hiding this comment

george-lorch Dec 3, 2015

Choose a reason for hiding this comment