Support configurable/default heartbeat default/max intervals.

[cretz]

What was changed

Currently in Go the frequency pending heartbeats are relayed to the server are 80% of the given heartbeat timeout or if no heartbeat timeout, 8 minutes (80% of 10 minutes). This changes that to be 80% of given heartbeat timeout or 30 seconds and caps it at a configurable max, or if the max is not configured, 60 seconds.

This does mean users who previously had heartbeats way less frequent, e.g. every few minutes, will now have them more frequent (default no less frequent than 60s). We do not believe going down from the large default is particularly harmful to running users though it technically could cause more traffic.

Why?

It was decided all SDKs should behave the same with regards to heartbeat frequency. See #656 and its associated links.

Checklist

1. Closes Heartbeat throttling interval should be configurable and default interval should be less than 10 minutes #656
2. How was this tested: In the same way the 8 minute heartbeat was not tested in an integration setting, neither is this. Without a time-skipping server, timeout-based tests are not very feasible. But of course logic to derive the timeout based on configuration is tested.

[bergundy]

Overall LGTM.

[bergundy]

I find this a bit complicated to read.

It would have been easier to only have to consider ActivityOptions.heartbeatTimeout, WorkerOptions.{default,max}HeartbeatThrottleInterval and have defaults for those options provided outside of this method.

[cretz]

They are provided outside this method too. But there are a couple of ways this is called in test code, so I wanted to do it here just to be safe.

[bergundy]

So I'd split it into 2 methods, one that sets the default options and one that does the calculation to reduce the complexity.

[cretz]

Those methods would be a few lines and each called only once.

[bergundy]

Can we do something like this?

In func newActivityTaskHandlerWithCustomProvider
do:

defaultHeartbeatThrottleInterval:
  providedOrDefaultInterval(params.DefaultHeartbeatThrottleInterval, defaultDefaultHeartbeatThrottleInterval),
maxHeartbeatThrottleInterval:
  providedOrDefaultInterval(params.MaxHeartbeatThrottleInterval, defaultMaxHeartbeatThrottleInterval),

And make getHeartbeatThrottleInterval something like:

if heartbeatTimeout > 0 {
		heartbeatThrottleInterval = time.Duration(0.8 * float64(heartbeatTimeout))
} else {
		heartbeatThrottleInterval = ath.defaultHeartbeatThrottleInterval
}

if heartbeatThrottleInterval > ath.maxHeartbeatThrottleInterval {
		heartbeatThrottleInterval = ath.maxHeartbeatThrottleInterval
}
return heartbeatThrottleInterval

If we're missing the initialization in tests just make sure ActivityTaskHandler is properly initialized.

[cretz]

I have changed this to separate the obtaining of interval (from timeout or fall back to configured or fall back to system) and max (from configured or fall back to system) and then to the check after that. Hopefully this is clearer.