ddl: Fix failure on executing exchange partition(release-6.5) (#8374)

[ti-chi-bot]

This is an automated cherry-pick of #8374

What problem does this PR solve?

Issue Number: close #8372

Problem Summary:

For example:

1. Create a partition table, logical_table_id=100, partition p0 physical_table_id=101
2. The partition table schema is synced to TiFlash
3. A non-partition table (table_id=222) is created
4. Execute exchange partition immediately, exchange non-partition table_id=222 with physical_table_id=101
5. TiFlash starts its schema sync, try to get the TableInfo of table_id=222 from TiKV
6. TiKV return nothing because table_id=222 is a partition table of logical_table_id=100 now. TiFlash treat the table as dropped and failed to create IStorage for table_id=222.
7. TiFlash tries to execute the EXCHANGE PARTITION DDL, but failed to find the IStorage instance of table_id=222. However, the new partition list is persisted to .SQL file, which contains the partition_id of 222

8. tiflash/dbms/src/TiDB/Schema/SchemaBuilder.cpp

   Lines 812 to 830 in 8138e8a

   	auto orig_table_info = storage->getTableInfo();
   	orig_table_info.partition = table_info->partition;
   	{
   	auto alter_lock = storage->lockForAlter(getThreadName());
   	storage->alterFromTiDB(
   	alter_lock,
   	AlterCommands{},
   	name_mapper.mapDatabaseName(*pt_db_info),
   	orig_table_info,
   	name_mapper,
   	context);
   	}
   	FAIL_POINT_TRIGGER_EXCEPTION(FailPoints::exception_after_step_1_in_exchange_partition);
   	/// step 2 change non partition table to a partition of the partition table
   	storage = tmt_context.getStorages().get(npt_table_id);
   	if (storage == nullptr)
   	throw TiFlashException(fmt::format("miss table in TiFlash : {}", name_mapper.debugCanonicalName(*npt_db_info, *table_info)),
   	Errors::DDL::MissingTable);

9. When other DDL try to apply, it can not pass the following check because the IStorage instance of table_id=222 is not created at all

10. tiflash/dbms/src/TiDB/Schema/SchemaBuilder.cpp

    Lines 694 to 703 in 8138e8a

    	if (new_table_info->isLogicalPartitionTable())
    	{
    	auto & tmt_context = context.getTMTContext();
    	for (const auto & part_def : new_table_info->partition.definitions)
    	{
    	auto part_storage = tmt_context.getStorages().get(part_def.id);
    	if (part_storage == nullptr)
    	{
    	throw Exception(fmt::format("miss old table id in Flash {}", part_def.id));
    	}

What is changed and how it works?

• In the step 1 for executing exchange partition, create the new table which is new in the latest partition list. This ensure all the physical table in the partition list persisted in disk have been created.
• But do not drop the non-existing ids in the step 1, because the non-existing ids could be changed into non-partition table
• Suppress a verbose logging for rate limiter

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2023-11-20 02:55:53.781795415 +0000 UTC m=+200182.447021610: ☑️ agreed by JinheLin.
• 2023-11-20 02:56:07.869782987 +0000 UTC m=+200196.535009194: ☑️ agreed by JaySon-Huang.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JaySon-Huang, JinheLin, Lloyd-Pottiger

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [JaySon-Huang,JinheLin,Lloyd-Pottiger]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[JinheLin]

/run-all-tests

[JinheLin]

/run-all-tests

[JinheLin]

/run-integration-test tidb=v6.5.4

[purelind]

/run-integration-test tidb=release-6.5

[purelind]

/run-integration-test tidb=v6.5.4

wait for this https://ci.pingcap.net/blue/organizations/jenkins/tiflash-ghpr-integration-tests/detail/tiflash-ghpr-integration-tests/15125/pipeline

[purelind]

/run-integration-test tidb=release-6.5