Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VReplication: tracking rows_copied #7980

Merged
merged 9 commits into from
Apr 29, 2021

Conversation

shlomi-noach
Copy link
Contributor

Description

_vt.vreplication has new column, rows_copied, which is updated periodically during vreplication work, to reflect the actual number of rows written on the target table.

The value is written together with time_updated, so it does not get transactionally synced with each copy iteration, it follows closely though, and makes a good estimate.

This will be later used by Online DDL to track Vreplication-based migration progress & ETA.

Related Issue(s)

Checklist

  • Tests were added or are not required
  • Documentation was added or is not required

Deployment Notes

shlomi-noach and others added 8 commits April 27, 2021 15:44
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach shlomi-noach changed the title Vrepl rows copied2 VReplication: tracking rows_copied Apr 28, 2021
@shlomi-noach
Copy link
Contributor Author

Good to review!

Copy link
Contributor

@rohit-nayak-ps rohit-nayak-ps left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Just noting here that after the copy phase the rows_copied will never change.

@shlomi-noach
Copy link
Contributor Author

@rohit-nayak-ps currently - that's the objective. Just count the number of rows copied by vcopier. There is no adjustment by vplayer's work. That'll come slightly later, and I see some larger things at play, I'll write more tomorrow. So anyway, leaving this to a different PR.

While this PR is good, I do notice that on a very long running vreplication stream, and with near zero traffic, GenerateUpdatePos either doesn't get called or is not applied. I see 25 minutes go by without an update to time_updated, nor transaction_timestamp, and thereby obviously not to rows_copied.

That's an unexpected behavior. It also relates to what I'll write more about tomorrow. So this is another thing to iterate on: I'd like to see rows_copied on the fly and frequently (at least once per minute).

@shlomi-noach shlomi-noach merged commit d439f78 into vitessio:master Apr 29, 2021
@shlomi-noach shlomi-noach deleted the vrepl-rows-copied2 branch April 29, 2021 04:51
@askdba askdba added Component: VReplication Type: Enhancement Logical improvement (somewhere between a bug and feature) labels May 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Component: VReplication Type: Enhancement Logical improvement (somewhere between a bug and feature)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants