Heartbeat on streaming vreplication can have high binary log space overhead

[aquarapid]

Scenario:

• You have two keyspaces, a source keyspace and a destination keyspace.
• As a simplifying assumption, each keyspace is single sharded; but this problem occurs (and is worse) if the source is multi-shard too.
• You have setup vreplication from the source to the destination keyspace for a set of tables. How does not really matter, could be via MoveTables, Materialize, Reshard or just native vreplication constructs.
• The copy phase for the tables has completed, and now vreplication is just applying changes from the source to the destination keyspace.

Problem:

• Each stream has a vreplication heartbeat, which will update the _vt.vreplication sidecar db for each stream (i.e. row) every second.
• Normally, this isn't an issue, but if the vreplication.source column is large (e.g. lots of tables in the stream), this update will generate large binlog entries every second.
• As an example, we saw a case where vreplication.source for a single stream was around 13 kb, which resulted in a binlog entry of 13kb + 13kb + change = 27kb each second; even though the upstream database was not being updated. This means around 100 MB of additional binlogs per hour, just for the binlogs. If you have multiple streams, this would be multiplied. If you have MySQL's expire_logs_days set to 3 (the Vitess default); this is an additional 7.2 GB of diskspace per stream in this case, just to maintain the heartbeat.
• For "real" environments, this is probably not a big deal; but for dev/test this is probably excessive.

Potential solutions:

• Heartbeat less frequently? Or less frequently when there is limited DB activity?
• Move the vreplication.source column to an FK table?

[mattlord]

@aquarapid and @shlomi is this still an issue today with on-demand heartbeats? I think we can close this but want to double check. Thanks!

[mattlord]

I'm going to close this as done for now. If I'm missing anything please let me know and we can re-open it.