Suggestion: always enable heartbeat on vttablet

[shlomi-noach]

vttablet has these flags:

• -heartbeat_enable
• -heartbeat_interval

The use of heartbeat is so important, that I suggest it must always be enabled. Plus, it should have a maximum interval that is subsecond.

Heartbeats are to be used by throttling services, by orchestrator, etc.

Is there any reason why we woudln't always enable heartbeat?

[sougou]

It was badly implemented and had bugs. But we've fixed them all over time. We should encourage using this feature now.

[deepthi]

We document that only one of -heartbeat_enable and -enable_replication_reporter should be used. Right now they both default to false. From that POV it is safe to default heartbeat to true. However, it should be documented in the release notes as a possibly breaking change.

[shlomi-noach]

We document that only one of -heartbeat_enable and -enable_replication_reporter should be used

I couldn't find where this is documented, searched in vitessio/vitess and in vitessio/website repos. The one related thing I found in vitessio/website is:

-heartbeat_enable and -heartbeat interval duration: cause vttablet to write heartbeats to the sidecar database. This information is also used by the replication reporter to assess replica lag.

[shlomi-noach]

#6665 was created to close this issue. However, #6668, the throttler PR, implicitly enables heartbeat without changing configuration. So #6665 was closed without merging.

We're still in experimental mode for the tablet throttler. It's debatable whether we can call this issue "implicitly implemented" or "won't fix"... Opting for "won't fix" for now and closing.