Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

doc(ticdc): add design document for storage sink #8540

Merged
merged 5 commits into from
Mar 31, 2023
Merged

doc(ticdc): add design document for storage sink #8540

merged 5 commits into from
Mar 31, 2023

Conversation

CharlesCheung96
Copy link
Contributor

What problem does this PR solve?

Issue Number: ref #6797

What is changed and how it works?

  1. add design document for storage sink

Check List

Tests

  • No code

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note 

`None`.

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Mar 15, 2023 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • amyangfei
  • asddongmen

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 15, 2023
@CharlesCheung96 CharlesCheung96 force-pushed the add_storage_sink_design_doc branch 2 times, most recently from e78f1d8 to 3aa75fb Compare March 22, 2023 15:39
@ti-chi-bot ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 22, 2023
@CharlesCheung96 CharlesCheung96 marked this pull request as ready for review March 22, 2023 15:50
@ti-chi-bot ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 22, 2023
@CharlesCheung96 CharlesCheung96 added component/docs Document component. area/ticdc Issues or PRs related to TiCDC. labels Mar 23, 2023
@CharlesCheung96
Copy link
Contributor Author

/run-all-tests

@CharlesCheung96 CharlesCheung96 added the needs-cherry-pick-release-7.0 Should cherry pick this PR to release-7.0 branch. label Mar 23, 2023
amyangfei
amyangfei reviewed Mar 23, 2023
View reviewed changes
docs/design/2023-03-15-ticdc-storage-sink.md Outdated Show resolved Hide resolved
docs/design/2023-03-15-ticdc-storage-sink.md
- `year`: the separator is the year of the transaction commit date, for example, <code>s3://bucket/prefix1/prefix2/schema1/table1/9999/**2022**</code>.
- `month`: the separator is the year and month of the transaction commit date, for example, <code>s3://bucket/prefix1/prefix2/schema1/table1/9999/**2022-01**</code>.
- `day`: the separator is the year, month, and day of the transaction commit date, for example, <code>s3://bucket/prefix1/prefix2/schema1/table1/9999/**2022-01-02**</code>.
- `num`: saves the serial number of the file that records the data change, for example, <code>s3://bucket/prefix1/prefix2/schema1/table1/9999/2022-01-02/CDC**000005**.csv</code>.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When does the serial number increase, is there any restriction to the size of single csv file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Data change files are saved to the downstream when either flush-interval or file-size meets the requirements. Ref: https://docs.pingcap.com/tidb/stable/ticdc-sink-to-cloud-storage#configure-nfs

docs/design/2023-03-15-ticdc-storage-sink.md Outdated

##### Transactional constraints
- In a single CSV file, the commit-ts of a row is equal to or smaller than that of the subsequent row.
- The same transactions of a single table are stored in the same CSV file.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this suitable for large transaction?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good advice, the above conditions can be guaranteed only when transaction-atomicity is set to table level.

@CharlesCheung96
Copy link
Contributor Author

/run-all-tests

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 23, 2023
@CharlesCheung96
Copy link
Contributor Author

/run-verify

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Mar 30, 2023
@CharlesCheung96
Copy link
Contributor Author

/run-all-tests

@CharlesCheung96
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: ac0fc45

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 30, 2023
@CharlesCheung96
Copy link
Contributor Author

/test all

@ti-chi-bot ti-chi-bot merged commit 96ac281 into pingcap:master Mar 31, 2023
@ti-chi-bot ti-chi-bot mentioned this pull request Mar 31, 2023
Closed
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-7.0: #8708.

3AceShowHand pushed a commit to 3AceShowHand/tiflow that referenced this pull request Apr 3, 2023
@CharlesCheung96 @3AceShowHand
doc(ticdc): add design document for storage sink (pingcap#8540)
23127bf 
ref pingcap#6797
@CharlesCheung96 CharlesCheung96 added the needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. label Apr 3, 2023
@ti-chi-bot ti-chi-bot mentioned this pull request Apr 3, 2023
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-6.5: #8722.

ti-chi-bot added a commit that referenced this pull request Apr 4, 2023
@ti-chi-bot
doc(ticdc): add design document for storage sink (#8540) (#8722)
f0b6a25 
close #8746
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@asddongmen asddongmen asddongmen approved these changes

@amyangfei amyangfei amyangfei approved these changes

Assignees
No one assigned
Labels
area/ticdc Issues or PRs related to TiCDC. component/docs Document component. needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.0 Should cherry pick this PR to release-7.0 branch. release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@CharlesCheung96 @ti-chi-bot @amyangfei @asddongmen