Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix VReplication logging to file and db #8521

Merged
merged 1 commit into from
Jul 26, 2021

Conversation

rohit-nayak-ps
Copy link
Contributor

@rohit-nayak-ps rohit-nayak-ps commented Jul 22, 2021

Description

  • Errors like duplicate key errors while bulk inserting during the copy phase can generate huge log lines. Since we also write to _vt.vreplication_log that table also gets filled up fast since we keep retrying within vreplication. This PR truncates the logs both in the file logs and in the _vt.vreplication_log table
  • The logic for updating count of recurring errors in _vt.vreplication_log was not working because the controller would set the state of the restarted workflow again, which was getting logged to _vt.vreplication_log alternating with the actual error. This has been fixed by ignoring this spurious state change log
  • Some unnecessary/repetitive log lines have been removed

Aside

One reason why log files were filling up is that we retry the workflow on error every 5 seconds, by default. The reason we retry is that transient errors like network failures or server restarts may occur and we don't want the vreplication workflow to error out permanently but keep retrying. We may want to increase this default since, in general, most such situations appear to be non-recoverable and the user needs to intervene to fix it. This needs more discussion though.

For now users can setup -vreplication_retry_delay to change this.

Checklist

  • Tests were added or are not required
  • Documentation was added or is not required

Deployment Notes

…mming the logs. Fix logic for incrementing count of recurring error messages.

Signed-off-by: Rohit Nayak <rohit@planetscale.com>
@rohit-nayak-ps rohit-nayak-ps requested review from shlomi-noach, deepthi and a team July 25, 2021 17:37
@rohit-nayak-ps rohit-nayak-ps marked this pull request as ready for review July 25, 2021 17:38
Copy link
Member

@deepthi deepthi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@deepthi deepthi merged commit 1053cff into vitessio:main Jul 26, 2021
@deepthi deepthi deleted the rn-vr-logs branch July 26, 2021 03:53
rohit-nayak-ps added a commit that referenced this pull request Jul 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants