Fix VReplication logging to file and db #8521
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
_vt.vreplication_log
that table also gets filled up fast since we keep retrying within vreplication. This PR truncates the logs both in the file logs and in the_vt.vreplication_log
table_vt.vreplication_log
was not working because the controller would set the state of the restarted workflow again, which was getting logged to_vt.vreplication_log
alternating with the actual error. This has been fixed by ignoring this spurious state change logAside
One reason why log files were filling up is that we retry the workflow on error every 5 seconds, by default. The reason we retry is that transient errors like network failures or server restarts may occur and we don't want the vreplication workflow to error out permanently but keep retrying. We may want to increase this default since, in general, most such situations appear to be non-recoverable and the user needs to intervene to fix it. This needs more discussion though.
For now users can setup
-vreplication_retry_delay
to change this.Checklist
Deployment Notes