Make the frequency at which heartbeats update the _vt.vreplication table configurable

[rohit-nayak-ps]

Description

Motivation

For an idle source shard, the source vstreamer sends a heartbeat. Currently that is once per second. On receiving the heartbeat the target vreplication module updates the time_updated column of the relevant row of _vt.vreplication. For some setups this is a problem, for example:

• if there are too many streams the extra write qps or cpu load due to these updates are unacceptable
• if there are too many streams and/or a large source field (lot of participating tables) which generates unacceptable increase in the binlog size
• even for a single stream, if the server is of a lower configuration, then the resultant qps/binlog increase may be significant as a percentage of resources

See related issues for more information.

Implementation

This PR provides a tablet level configuration parameter -vreplication_heartbeat_update_interval which determines how often the time_updated column is updated if there are no real events on the source and the source. Default is 1 second. Keep this low if you expect high QPS and are monitoring this column to alert about potential outages. Keep this high if you experience any of the issues mentioned above.

Notes

This is an initial light-weight implementation to address immediate concerns of affected users.

This also doesn't address the problem of long-running workflows (like materialize or reverse replication) for high qps systems.

To address this, in the long term, we will be refactoring the vreplication table so that the static part (including the source proto) will be separated from the dynamic part (position, timestamps) , so as to minimize the size of binlog entries.

The VReplicationHeartbeat metric will continue to be updated at 1 second, so for systems using this metric for monitoring, you will continue to get 1-second granularity for alerts.

Signed-off-by: Rohit Nayak rohit@planetscale.com

Related Issue(s)

• VReplication heartbeat updates can cause significant resource utilization on the target #6945
• Heartbeat on streaming vreplication can have high binary log space overhead #6805

Checklist

• Should this PR be backported?
• Tests were added or are not required
• Documentation was added or is not required

Impacted Areas in Vitess

Components that this PR will affect:

• Query Serving
• VReplication
• Cluster Management
• Build/CI
• VTAdmin

[shlomi-noach]

Let's keep an upper bound on that interval. The maximum interval should not exceed one minute. The reason is that we must have some tracking. Specifically, OnlineDDL looks at the update timestamp to say "this is healthy/active".

[rohit-nayak-ps]

Let's keep an upper bound on that interval. The maximum interval should not exceed one minute. The reason is that we must have some tracking. Specifically, OnlineDDL looks at the update timestamp to say "this is healthy/active".

Thanks that was a good point. I added an upper bound. I have enforced it in code at the moment. I couldn't see any validations of flags in vttablet: I would ideally like to fail the vttablet launch itself if a higher value is provided so that we are not silently fixing it.

[deepthi]

Thanks that was a good point. I added an upper bound. I have enforced it in code at the moment. I couldn't see any validations of flags in vttablet: I would ideally like to fail the vttablet launch itself if a higher value is provided so that we are not silently fixing it.

Mandatory flags are checked in vttablet.go. For example:

	if *tabletPath == "" {
		log.Exit("-tablet-path required")
	}

[rohit-nayak-ps]

Mandatory flags are checked in vttablet.go. For example:

This are for flags defined in that cmd itself. The ones in vreplicator.go are not in scope here or in tm_init.go. It seems like an overkill to validate this inside vreplication and fail a vreplication stream. Maybe I just add this to the flags documentation for now?

[deepthi]

Mandatory flags are checked in vttablet.go. For example:

This are for flags defined in that cmd itself. The ones in vreplicator.go are not in scope here or in tm_init.go. It seems like an overkill to validate this inside vreplication and fail a vreplication stream. Maybe I just add this to the flags documentation for now?

Let us log a warning if the user provided value is being overridden by the limit - either when starting the VR engine, or when a stream is created. If it turns out that people are being affected and not realizing it we can revisit whether to fail vttablet startup.

[rohit-nayak-ps]

Let us log a warning if the user provided value is being overridden by the limit - either when starting the VR engine, or when a stream is created. If it turns out that people are being affected and not realizing it we can revisit whether to fail vttablet startup.

added the log

[shlomi-noach]

🚀