ddl: Support `FLASHBACK DATABASE` #8424

JaySon-Huang · 2023-11-27T07:54:15Z

What problem does this PR solve?

Issue Number: close #8450

Problem Summary:

tidb support FLASHBACK DATABASE DBName [TO newDBName] since v6.4.0 https://docs.pingcap.com/tidb/stable/sql-statement-flashback-database

After executing FLASHBACK DATABASE ... TO ..., the storage instances in TiFlash are not recovered as non tombstone. After the gc safepoint advanced, the storage instances will be physically drop.

What is changed and how it works?

Recover all the tables on the database when FLASHBACK DATABASE is executed.

Note that the implementation is different from that on release-7.1, release-6.5

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Fix the issue that `FLASHBACK DATABASE` is not applied to TiFlash replica data

JaySon-Huang · 2023-12-01T07:32:13Z

/run-all-tests

JaySon-Huang · 2023-12-01T08:43:13Z

tests/fullstack-test2/ddl/alter_drop_database.test

Rewritten into flashback/flashback_database.test

JaySon-Huang · 2023-12-01T08:43:31Z

/run-all-tests

…_database

Lloyd-Pottiger

Others LGTM

dbms/src/TiDB/Schema/SchemaBuilder.cpp

JaySon-Huang · 2023-12-04T02:44:56Z

/run-all-tests

JaySon-Huang · 2023-12-04T03:34:56Z

/run-all-tests

hongyunyan · 2023-12-05T12:57:31Z

dbms/src/TiDB/Schema/SchemaBuilder.cpp

+    LOG_INFO(log, "Recover database begin, database_id={}", database_id);
+    auto db_name = name_mapper.mapDatabaseName(database_id, keyspace_id);
+    auto db = context.tryGetDatabase(db_name);
+    if (unlikely(!db))


If the db is not existed now, do we need to create the db and related tables based on it's db info?

I'm a little worried about this scenario: the customer is stuck in the gc time to do the flash back, and tiflash may receive and process it later, which may be just longer than tiflash's local gc, so the db and tables still don't exist.

the tombstone_ts for IDatabase and IStorage are allocated by tiflash when it receives the DROP DATABASE or DROP TABLE DDL. So the tombstone_ts are usually sliently larger than the user actually executed those DROP statements. When a user execute FLASHBACK DATABASE, those IDatabase and IStorage instance are possible not pass the gc safepoint.

If those IStorage and IDatabase instance are actually get physically dropped in TiFlash instance, it is dangerous to "create" them when applying "FLASHBACK DATABASE". Because there are some rows is permanently deleted from tiflash instance. If we silently "create" them, it means the error is hard to find out but data inconsistency happens. An explicit error is better than implicit data loss.

hongyunyan

LGTM

ti-chi-bot · 2023-12-07T02:47:36Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hongyunyan, Lloyd-Pottiger

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Lloyd-Pottiger,hongyunyan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2023-12-07T02:47:38Z

[LGTM Timeline notifier]

Timeline:

2023-12-01 13:52:31.48675992 +0000 UTC m=+1189980.151986100: ☑️ agreed by Lloyd-Pottiger.
2023-12-07 02:47:37.481343317 +0000 UTC m=+1668486.146569497: ☑️ agreed by hongyunyan.

ti-chi-bot · 2023-12-07T02:59:40Z

@JaySon-Huang: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

trigger some heavy tests which will not run always when PR updated.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-12-07T03:45:29Z

In response to a cherrypick label: new pull request created to branch release-7.5: #8477.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

close pingcap#8450

close #8450

ti-chi-bot bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 27, 2023

Support FLASHBACK DATABASE ...

21e59f2

JaySon-Huang force-pushed the support_flashback_database branch from 4b8bda6 to 21e59f2 Compare November 30, 2023 04:56

ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 30, 2023

JaySon-Huang changed the title ~~ddl: Support FLASHBACK DATABASE~~ [WIP] ddl: Support FLASHBACK DATABASE Nov 30, 2023

ti-chi-bot bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 30, 2023

Add more test case about partition table

e2ef50c

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 1, 2023

JaySon-Huang added 3 commits December 1, 2023 15:42

Remove useless changes

8be7640

Unify the naming

35608e8

Add test case

776fdf4

JaySon-Huang commented Dec 1, 2023

View reviewed changes

tests/fullstack-test2/ddl/alter_drop_database.test Outdated

Copy link

Contributor Author

JaySon-Huang Dec 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rewritten into flashback/flashback_database.test

JaySon-Huang changed the title ~~[WIP] ddl: Support FLASHBACK DATABASE~~ ddl: Support FLASHBACK DATABASE Dec 1, 2023

ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 1, 2023

JaySon-Huang added 2 commits December 1, 2023 17:36

Add hint

6383e4c

Merge remote-tracking branch 'upstream/master' into support_flashback…

1ae4f54

…_database

JaySon-Huang requested a review from Lloyd-Pottiger December 1, 2023 09:48

JaySon-Huang removed needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. labels Dec 1, 2023

Fix typo

481b236

ti-chi-bot bot added needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. labels Dec 1, 2023

Lloyd-Pottiger approved these changes Dec 1, 2023

View reviewed changes

dbms/src/TiDB/Schema/SchemaBuilder.cpp Outdated Show resolved Hide resolved

ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Dec 1, 2023

Refine log level

6ceeb01

Add tombstone timestamp to logging

221708d

hongyunyan reviewed Dec 5, 2023

View reviewed changes

hongyunyan approved these changes Dec 7, 2023

View reviewed changes

ti-chi-bot bot added the lgtm label Dec 7, 2023

ti-chi-bot bot removed the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Dec 7, 2023

Merge branch 'master' into support_flashback_database

48ea185

JaySon-Huang removed needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. labels Dec 7, 2023

ti-chi-bot bot merged commit 3085414 into pingcap:master Dec 7, 2023
6 checks passed

ti-chi-bot mentioned this pull request Dec 7, 2023

ddl: Support FLASHBACK DATABASE (release-7.5) (#8424) #8477

Merged

12 tasks

ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Dec 7, 2023

This is an automated cherry-pick of pingcap#8424

944c6b9

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

JaySon-Huang deleted the support_flashback_database branch December 7, 2023 03:52

JaySon-Huang added a commit to ti-chi-bot/tiflash that referenced this pull request Dec 7, 2023

ddl: Support FLASHBACK DATABASE (pingcap#8424)

35c2e8b

close pingcap#8450

ti-chi-bot bot pushed a commit that referenced this pull request Dec 7, 2023

ddl: Support FLASHBACK DATABASE (release-7.5) (#8424) (#8477)

91ec283

close #8450

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ddl: Support `FLASHBACK DATABASE` #8424

ddl: Support `FLASHBACK DATABASE` #8424

JaySon-Huang commented Nov 27, 2023 •

edited

Loading

JaySon-Huang commented Dec 1, 2023

JaySon-Huang Dec 1, 2023

JaySon-Huang commented Dec 1, 2023

Lloyd-Pottiger left a comment

JaySon-Huang commented Dec 4, 2023

JaySon-Huang commented Dec 4, 2023

hongyunyan Dec 5, 2023

hongyunyan Dec 5, 2023

JaySon-Huang Dec 6, 2023

hongyunyan left a comment

ti-chi-bot bot commented Dec 7, 2023

ti-chi-bot bot commented Dec 7, 2023

ti-chi-bot bot commented Dec 7, 2023

ti-chi-bot commented Dec 7, 2023

ddl: Support FLASHBACK DATABASE #8424

ddl: Support FLASHBACK DATABASE #8424

Conversation

JaySon-Huang commented Nov 27, 2023 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

JaySon-Huang commented Dec 1, 2023

JaySon-Huang Dec 1, 2023

Choose a reason for hiding this comment

JaySon-Huang commented Dec 1, 2023

Lloyd-Pottiger left a comment

Choose a reason for hiding this comment

JaySon-Huang commented Dec 4, 2023

JaySon-Huang commented Dec 4, 2023

hongyunyan Dec 5, 2023

Choose a reason for hiding this comment

hongyunyan Dec 5, 2023

Choose a reason for hiding this comment

JaySon-Huang Dec 6, 2023

Choose a reason for hiding this comment

hongyunyan left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Dec 7, 2023

ti-chi-bot bot commented Dec 7, 2023

[LGTM Timeline notifier]

ti-chi-bot bot commented Dec 7, 2023

ti-chi-bot commented Dec 7, 2023

ddl: Support `FLASHBACK DATABASE` #8424

ddl: Support `FLASHBACK DATABASE` #8424

JaySon-Huang commented Nov 27, 2023 •

edited

Loading