Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ticdc/mq: accurately demonstrate txn_batch_size metric for MQ sink #3609

Merged
merged 16 commits into from
Dec 9, 2021
Merged

ticdc/mq: accurately demonstrate txn_batch_size metric for MQ sink #3609

merged 16 commits into from
Dec 9, 2021

Conversation

zhaoxinyu
Copy link
Contributor

What problem does this PR solve?

#3431

What is changed and how it works?

In order to accurately show the txn_batch_size metric, we use the total number of rows instead of messages when the sink target is MQ.

Check List

Tests

  • Unit test
  • Integration test

Release note 

None

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Nov 24, 2021 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • 3AceShowHand
  • liuzix

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added first-time-contributor Indicates that the PR was contributed by an external member and is a first-time contributor. release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Nov 24, 2021
@zier-one zier-one added the type/bugfix This PR fixes a bug. label Dec 1, 2021
3AceShowHand
3AceShowHand reviewed Dec 1, 2021
View reviewed changes
cdc/sink/codec/json.go Outdated
@@ -461,6 +461,7 @@ func (d *JSONEventBatchEncoder) AppendRowChangedEvent(e *model.RowChangedEvent)
message.Ts = e.CommitTs
message.Schema = &e.Table.Schema
message.Table = &e.Table.Table
message.RowsCount++
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please take a look at craft codec, it also batch multiple rows into the same message.

3AceShowHand
3AceShowHand reviewed Dec 1, 2021
View reviewed changes
cdc/sink/codec/interface.go Outdated
Table *string // table
Type model.MqMessageType // type
Protocol Protocol // protocol
RowsCount int // rows in one MQ Message
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer make RowsCount private and export Get and Inc method for it.

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Dec 2, 2021
@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Dec 3, 2021
@zhaoxinyu
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Member

@zhaoxinyu: /merge is only allowed for the committers, you can assign this pull request to the committer in list by filling /assign @committer in the comment to help merge this pull request.

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@Rustin170506
Copy link
Member

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 2f48a7a

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Dec 8, 2021
@zhaoxinyu
Copy link
Contributor Author

/run-integration-test

@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 8, 2021
@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 8, 2021
@ti-chi-bot ti-chi-bot merged commit b531f14 into pingcap:master Dec 9, 2021
@overvenus overvenus mentioned this pull request Dec 10, 2021
Closed
@overvenus overvenus added needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. needs-cherry-pick-release-5.0 Should cherry pick this PR to release-5.0 branch. needs-cherry-pick-release-5.1 Should cherry pick this PR to release-5.1 branch. needs-cherry-pick-release-5.2 Should cherry pick this PR to release-5.2 branch. needs-cherry-pick-release-5.3 Should cherry pick this PR to release-5.3 branch. labels Dec 10, 2021
@overvenus
Copy link
Member

/run-cherry-picker

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Dec 10, 2021
@zhaoxinyu @ti-chi-bot
This is an automated cherry-pick of pingcap#3609
bd96f9e 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
This was referenced Dec 10, 2021
Merged
Merged
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #3818.

@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #3817.

@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #3819.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Dec 10, 2021
@zhaoxinyu @ti-chi-bot
This is an automated cherry-pick of pingcap#3609
79d2351 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request Dec 10, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #3820.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Dec 10, 2021
@zhaoxinyu @ti-chi-bot
This is an automated cherry-pick of pingcap#3609
5cb5a17 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request Dec 10, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #3821.

ti-chi-bot added a commit that referenced this pull request Dec 13, 2021
@ti-chi-bot
ticdc/mq: accurately demonstrate txn_batch_size metric for MQ sink (#…
8b015fa 
…3609) (#3821)
ti-chi-bot added a commit that referenced this pull request Dec 20, 2021
@ti-chi-bot
ticdc/mq: accurately demonstrate txn_batch_size metric for MQ sink (#…
522a408 
…3609) (#3820)
zhaoxinyu added a commit to zhaoxinyu/ticdc that referenced this pull request Dec 29, 2021
@zhaoxinyu
ticdc/mq: accurately demonstrate txn_batch_size metric for MQ sink (p…
e6be662 
…ingcap#3609)
overvenus pushed a commit that referenced this pull request Jan 17, 2022
@ti-chi-bot @zhaoxinyu @3AceShowHand
ticdc/mq: accurately demonstrate txn_batch_size metric for MQ sink (#…
6c434c2 
…3609) (#3819)

* fix the txn_batch_size metric inaccuracy bug when the sink target is MQ

* address comments

* add comments for exported functions

* fix the compiling problem

* fix conflicts.

Co-authored-by: zhaoxinyu <zhaoxinyu512@gmail.com>
Co-authored-by: Ling Jin <7138436+3AceShowHand@users.noreply.github.com>
Co-authored-by: 3AceShowHand <jinl1037@hotmail.com>
ti-chi-bot added a commit that referenced this pull request Jan 17, 2022
@ti-chi-bot
ticdc/mq: accurately demonstrate txn_batch_size metric for MQ sink (#…
eaf4c5c 
…3609) (#3818)
overvenus pushed a commit that referenced this pull request Jan 17, 2022
@ti-chi-bot @zhaoxinyu @3AceShowHand
ticdc/mq: accurately demonstrate txn_batch_size metric for MQ sink (#…
87115bc 
…3609) (#3817)

* fix the txn_batch_size metric inaccuracy bug when the sink target is MQ

* address comments

* add comments for exported functions

* fix the compiling problem

* fix conflicts.

* fix comment.

Co-authored-by: zhaoxinyu <zhaoxinyu512@gmail.com>
Co-authored-by: 3AceShowHand <jinl1037@hotmail.com>
Co-authored-by: Ling Jin <7138436+3AceShowHand@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@liuzix liuzix liuzix approved these changes

@3AceShowHand 3AceShowHand 3AceShowHand approved these changes

Assignees
No one assigned
Labels
first-time-contributor Indicates that the PR was contributed by an external member and is a first-time contributor. needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. needs-cherry-pick-release-5.0 Should cherry pick this PR to release-5.0 branch. needs-cherry-pick-release-5.1 Should cherry pick this PR to release-5.1 branch. needs-cherry-pick-release-5.2 Should cherry pick this PR to release-5.2 branch. needs-cherry-pick-release-5.3 Should cherry pick this PR to release-5.3 branch. release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. type/bugfix This PR fixes a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@zhaoxinyu @ti-chi-bot @Rustin170506 @overvenus @3AceShowHand @liuzix @zier-one