sink(ticdc): change the directory of storage sink only when ddl event occurs

[CharlesCheung96]

What problem does this PR solve?

Issue Number: close #8890, close #8891

What is changed and how it works?

1. Change the directory of storage sink only when ddl event occurs.
2. Make storage sink support database level DDL.

• File path changes:

1. table schema file:
   from:
   {scheme}://{prefix}/{schema}/{table}/{table-version-separator}/schema.json
   to:
   {scheme}://{prefix}/{schema}/{table}/meta/schema_{tso}_{schema-crc32-hash}.json
2. database schema:
   {scheme}://{prefix}/{schema}/meta/schema_{tso}_{schema-crc32-hash}.json
3. index file:
   from:
   {scheme}://{prefix}/{schema}/{table}/{table-version-separator}/{partition-separator}/{date-separator}/CDC.index
   to:
   {scheme}://{prefix}/{schema}/{table}/{table-version-separator}/{partition-separator}/{date-separator}/meta/CDC.index

Check List

Tests

• Unit test
• Integration test

Questions

Will it cause performance regression or break compatibility?

• The schema-related storage path structure in storage sink has changed

Do you need to update user documentation, design documentation or monitoring documentation?

• need to update user documentation and design documentation

Release note

`Fixed the table version directory written by storage sink may change while cdc restart or table scheduling`.
`Support database level DDL in storage sink.`

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• hi-rustin
• nongfushanquan

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[ti-chi-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[CharlesCheung96]

/test all

[CharlesCheung96]

/test all

[CharlesCheung96]

/test cdc-integration-storage-test

[CharlesCheung96]

/test all

[CharlesCheung96]

/test cdc-integration-storage-test

[CharlesCheung96]

/test all

[CharlesCheung96]

/test cdc-integration-storage-test

[CharlesCheung96]

/test cdc-integration-storage-test

[nongfushanquan]

pls add comments to describe the layout of the data and schema

[CharlesCheung96]

/test all

[CharlesCheung96]

/test all

[CharlesCheung96]

/retest

[CharlesCheung96]

/test dm-compatibility-test

[ti-chi-bot]

@CharlesCheung96: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

• /test cdc-integration-kafka-test
• /test cdc-integration-mysql-test
• /test cdc-integration-storage-test
• /test dm-compatibility-test
• /test dm-integration-test
• /test engine-integration-test
• /test verify

Use /test all to run the following jobs that were automatically triggered:

• pingcap/tiflow/ghpr_verify

In response to this:

/test dm-compatibility-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[CharlesCheung96]

/test dm-compatibility-test

[ti-chi-bot]

@CharlesCheung96: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

• /test cdc-integration-kafka-test
• /test cdc-integration-mysql-test
• /test cdc-integration-storage-test
• /test dm-compatibility-test
• /test dm-integration-test
• /test engine-integration-test
• /test verify

Use /test all to run the following jobs that were automatically triggered:

• pingcap/tiflow/ghpr_verify

In response to this:

/test dm-compatibility-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[CharlesCheung96]

/retest

[CharlesCheung96]

/retest

[CharlesCheung96]

/retest

[ti-chi-bot]

@CharlesCheung96: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

trigger some heavy tests which will not run always when PR updated.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[CharlesCheung96]

/test dm-integration-test

[ti-chi-bot]

@CharlesCheung96: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
wip/pull-cdc-integration-storage-test	d002b27	link	true	/test cdc-integration-storage-test

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[CharlesCheung96]

/test dm-integration-test

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-7.1: #8920.

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-6.5: #8921.