Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PS 7798 [5.7] - Merge fb-prod202102, fb-prod202103, fb-prod202104 #4401

Merged
merged 73 commits into from
Jul 14, 2021

Conversation

ldonoso
Copy link
Contributor

@ldonoso ldonoso commented Jul 12, 2021

https://jira.percona.com/browse/PS-7798
https://github.com/facebook/mysql-5.6/tree/fb-prod202102
https://github.com/facebook/mysql-5.6/tree/fb-prod202103
https://github.com/facebook/mysql-5.6/tree/fb-prod202104
https://ps57.cd.percona.com/view/5.7/job/percona-server-5.7-param/488/


  1. Update rocksdb submodule to c7f8ae9f17743774ac48e2ffc6e150eafcfae416 SequenceIterWrapper should use internal comparator (#8328) (v6.21.0)

  2. Add new MyRocks variable rocksdb_manual_compaction_bottommost_level.

  3. Cherry-pick the following commits from fb-prod202104:

  1. Skipped upstream RocksDB commits:
  1. Null cherry-picked upstream commits:

Luis Donoso and others added 4 commits July 12, 2021 15:40
Upstream commit ID : fb-mysql-5.6.35/64ba7e29cdc2d444e21e100acdbe2351e1c88384
PS-7798 : Merge fb-prod202102

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/0fb83bb7569838026b912af23de52077244693b3
PS-7798 : Merge fb-prod202102

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/f896415fa0d31ca956fbe2e63ee0eed544879f96
PS-7798 : Merge fb-prod202102

Summary:
This re-introduces Luqun's get_rocksdb_files.sh (which was reverted due to RocksDB perf regression in 6.17) to support building RocksDB 6.17+ and updates RocksDB to 6.19. The regression in 6.17 was reverted in RocksDB 6.18 so we should be able to proceed with new version.

update-submodule: rocksdb

Reviewed By: Pushapgl

Differential Revision: D27378086

fbshipit-source-id: dd0e0eb7ba3
Upstream commit ID : fb-mysql-5.6.35/f12eea3370f5c147305afdf0727b355bcd2cf4ec
PS-7798 : Merge fb-prod202103

Summary: In `inplace_populate_sk`, `ha_index_init` can return an error if the query has been killed, and if that happens, it's not safe to continue with calling `index_first`/`index_next`. The fix is to check for errors and return early if needed.

Reviewed By: luqun

Differential Revision: D26960027

fbshipit-source-id: aaf7bd24f9d
@ldonoso ldonoso requested review from george-lorch and inikep July 12, 2021 15:26
--disable_warnings
--source include/have_rocksdb.inc
--source include/have_binlog_format_row.inc
--source include/master-slave.inc
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

master-slave should be the last include

Comment on lines 1 to 2
--rocksdb_max_subcompactions=1
--rocksdb_default_cf_options=target_file_size_base=100k;max_bytes_for_level_multiplier=1;max_bytes_for_level_base=1m;target_file_size_multiplier=1
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--rocksdb_ should be always --loose_rocksdb_ in case RocksDB is not compiled

Comment on lines 14 to 16
--disable_warnings
DROP TABLE IF EXISTS t1;
--enable_warnings
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be removed as it it no longer required.

select * from t1 order by id;
source include/sync_slave_sql_with_master.inc;

connection slave;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--source include/rpl_connection_slave.inc

select * from t1 order by id;
source include/sync_slave_sql_with_master.inc;

connection slave;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--source include/rpl_connection_slave.inc

@@ -0,0 +1,53 @@
connection master;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--source include/rpl_connection_master.inc

select * from t1 order by id;


connection master;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--source include/rpl_connection_master.inc

select * from t1 order by id;


connection master;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--source include/rpl_connection_master.inc

rahku and others added 23 commits July 13, 2021 16:08
… MyRocks Manual Compaction

Upstream commit ID : fb-mysql-5.6.35/e00196f49460de6eeb91ee6f1e761da576b93ceb
PS-7798 : Merge fb-prod202103

Summary:
Currently BottommostLevelCompaction is set to default kForceOptimized for
manual compaction. This is fine usually. However, for privacy triggered
compactions, it does not make much sense, because it compacts Lmax SST files
that do not have any overlapping key updates at all. This adds a session
variable to specify the option for bottommost level compaction before
triggering manual compaction.

Porting from 8.0.20 branch

Reviewed By: yoshinorim

Differential Revision: D27780545

fbshipit-source-id: e5b9e7c6012
Upstream commit ID : fb-mysql-5.6.35/997d25216c3f968fa0439bcba90d25ffd2bbb695
PS-7798 : Merge fb-prod202102

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/e4f83712bced1e13a4a25da545719677d20890ab
PS-7798 : Merge fb-prod202104

Summary:
Port of D27419569.

**Porting Notes**
Core Data Demand Efficiency team wants to track physical size of the assoc types and fbtypes, as well as existing logical size. Tracking physical size of the specific assoc/fb types is not available as of now.
***Change***
Added the new function GET_INDEX_SIZE_BY_PREFIX which returns the index size by prefix. Currently we support only numeric keys.

Reviewed By: yizhang82

Differential Revision: D27919708

fbshipit-source-id: c1ba5976cbe
Upstream commit ID : fb-mysql-5.6.35/ccfbfe798da49c4ea49483048647b9128850ef5d
PS-7798 : Merge fb-prod202104

Summary:
As title, stay on the latest rocksdb branch for each full feature as per Yoshi's recommendation.

update-submodule: rocksdb

Reviewed By: yoshinorim

Differential Revision: D28049296

fbshipit-source-id: b74b75b101f
Upstream commit ID : fb-mysql-5.6.35/dcf392a57223ff46cefe6bf07130a3bfd8c40cc9
PS-7798 : Merge fb-prod202104

Summary: see D28049296 (facebook/mysql-5.6@ccfbfe7) discussion, there is a rocksdb change which cause this memtable size difference

Reviewed By: yizhang82

Differential Revision: D28155396

fbshipit-source-id: 3267cd53d68
Upstream commit ID : fb-mysql-5.6.35/d45c30966752a04ca931c86552e00df3ef933bd7
PS-7798 : Merge fb-prod202104

Summary:
Previously the `mysql_ldb` commands could not operate on a
MyRocks instance due to wrong comparator provided. In particular,
MyRocks column families can use a mix of "RocksDB_SE_v3.10" and
"rev:RocksDB_SE_v3.10"; however, the provided comparator was always
"RocksDB_SE_v3.10". This diff registers both possible comparators, which
allows each column family's proper comparator to be deserialized from
the OPTIONS file. The user must pass `--try_load_options` to their `ldb`
command.

Reviewed By: yizhang82

Differential Revision: D28313731

fbshipit-source-id: e9219a1199d
Upstream commit ID : fb-mysql-5.6.35/ba9d330448aa82009df5303bef21f952e49d2547
PS-7798 : Merge fb-prod202104

Summary:
When hitting some I/O errors in RocksDB (e.g. out of space), current RocksDB may return error codes other than kIOError at engine commit phase, while not returning any error at prepare phase, depending on how it hits errors. Our expectation is any engine commit failure of the two phase commit protocol should abort the process, since binlog commit is already done.

This diff makes sure to call abort() to terminate the process on RDB_IO_ERROR_TX_COMMIT at commit phase, not just RDB_IO_ERROR_DICT_COMMIT. This prevents from a case that a transaction gets written to binlog (and replicated) but it is rolled back in MyRocks engine due to commit failure, which is an inconsistent state. With this diff, binlog and engine will be consistent based on binlog state (either rolled forward or rolled back, aligned to binlog state).

This diff makes another change that it no longer aborts on systems errors on MyRocks Prepare. Prior to this diff, kIOError at Prepare() terminated the process, but it was not necessary since transactions could be safely rolled back on Prepare() error.

Clients hitting errors at Prepare() will get ER_INTERNAL_ERROR with RocksDB error status. Default ER_ERROR_DURING_COMMIT is not returned. This was because ER_ERROR_DURING_COMMIT does not take string arguments so it is difficult for users to understand why it failed.

Reviewed By: hermanlee

Differential Revision: D28576723

fbshipit-source-id: 1b0b6893f30
Upstream commit ID : fb-mysql-5.6.35/6e53ec0ef23f6a16f966669bd58b72a7a9ced698
PS-7798 : Merge fb-prod202104

Summary:
pull in the latest rocksdb version for next full feature (prod202104)

update-submodule: rocksdb

Reviewed By: yoshinorim

Differential Revision: D28689774

fbshipit-source-id: 455d6d93e54
Upstream commit ID : fb-mysql-5.6.35/2451c64901e59e10be92843da298733710cb2bd7
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/623780af3ee8295a98ce407ab46a48e865b9e95e
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/c607a5a9104ee50b87efd2326f741bded737f70e
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/e40167380581697596b398e832d2d35d132f5bc6
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/0b215f8c7eb777e6e4d2202b621c40519c03b0c3
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/e4a49506500c0b8c46b2340b8c15afe3e64e8ed5
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/6b1db2cfa9c1c8fb60e01d09c6b7c2d8cdc2dcae
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/6ab1c7c1ac92693062ccfbcb1de1552cbf0cb162
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/d8d5fed47c5509ee5fab4eb596f3378e877f46e3
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/4cd2cff73fa4d821c1f7bd2ecfcd9f17f98e9fe1
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/9486d32143a49416e399cfa1526d7f9a7c738e31
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/50a8e298885ba5ddf0415d11eee528cd31ccf90b
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/c018587221e551162ac587eab2fcbbd0960fe372
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/77b9160bcaa58d5e2cb9ce262efc6064ca7642b3
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/9d01043b1df6de6537cf223a7b62987d8649de5f
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Luis Donoso added 26 commits July 13, 2021 16:08
Upstream commit ID : fb-mysql-5.6.35/acd99073327aaa7c4121696d1996a135caf75555
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/f9fd0ce3069b0a95bcdc3f6efc823b8fb56845a8
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
…covery

Upstream commit ID : fb-mysql-5.6.35/d4626033dd8a8e46bee8be0ec0c338994d1cea48
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/2901d9a9463c89e6bd24e918fe02844850f8d981
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/05007850d65b3a69cbaa3e9442aea33abe9250a8
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
…appened

Upstream commit ID : fb-mysql-5.6.35/354ea91e0d9ccf471ea08d8724eef7d44291db04
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/e19d09073012954be372d32ba93b38bd497f0685
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/7cdcea348f9ff0c0d3c3b0e8729476229f090819
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/077b8d254efd92cc7c055c8b915ba6262e18cb6e
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/2378c1e8c1e59d55c0659e97316cae5ac787e81b
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/ddda374d0a518c9e13e9fa4563a55ba67e321e24
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/01e535506ca5c4565a9f51cadec9b84d2a748e29
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/73b7ec9184c70acf7199b217778fffb23563e7b2
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/1b80de18f6b009aa75efcb128f26d3f89912f3dd
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/05a8c91dcb787bc3a423765ba05c1b2b858d33de
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/bae825b5b8f332b16ba9e8065d43e1fa7a364867
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/47fc3c7282e0726761c50431f9e046daa533e2aa
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/39ce50d9bffe13c8e89c279ef0810e9ecb16f2dc
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/baaf5dbe0ab754d87a9721e3cb9ecbc876efe464
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/c807ef482fe4e998a804622165264eedf18d62d6
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/0f1f275a0255b88b43b4e20b23b9498558fd9b60
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/124deb25ff19049b287d824273ed1a072ca3fccc
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
…mit ordering

Upstream commit ID : fb-mysql-5.6.35/ce8c4740a0c0dfed0d219b7f1d30d05da8ffb40c
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/f43c69d0e5b4bb9c655698933f14fc017306de53
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
Upstream commit ID : fb-mysql-5.6.35/4edc56e8b7f473ea35bc6629335221c51a5aac2d
PS-7798 : Merge fb-prod202104

This is a NULL cherry-pick to Percona Server to create the commit placeholder for the corresponding upstream commit.
Reason: Patch not taken into Percona Server
@ldonoso ldonoso requested a review from inikep July 13, 2021 14:12
Copy link
Collaborator

@inikep inikep left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ldonoso ldonoso merged commit 3983aa6 into percona:5.7 Jul 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants