THRIFT-5240: Tweak the default go server connectivity check interval

[fishy]

Client: go

This is a follow up to 4db7a0a.

Because of the Go runtime bug 1, the previous default value of 1ms is
not a great default as it could cause excessive cpu usage. Use 5ms
instead as a balance between being useful and not causing too much cpu
overhead.

It's still configurable.

• Did you create an Apache Jira ticket? (not required for trivial changes)
• If a ticket exists: Does your pull request title follow the pattern "THRIFT-NNNN: describe my issue"?
• Did you squash your changes to a single commit? (not required, but preferred)
• Did you do your best to avoid breaking changes? If one was needed, did you label the Jira ticket with "Breaking-Change"?
• If your change does not involve any code, include [skip ci] anywhere in the commit message to free up build resources.

[fishy]

Actually dig deeper to the go runtime bug, it looks like the excessive cpu only happens when the ticker interval is short (1ms). using the example code given by that bug, using 1ms it causes ~20% cpu, but changing that to 5ms makes the cpu drop to ~6%, and 10ms makes it drop to ~3%.

So maybe 5ms is a better default than 0? @dcelasun what do you think?

[dcelasun]

I'm not so sure about a 5ms default. We have a fairly loaded Go 1.15 server (so with the ticker optimization) with 5ms/8ms p50/p75 latency respectively and a fixed 5ms increase seems a bit too much.

Would there be enough cpu load reduction with a 2-3ms ticker?

[fishy]

2ms it's ~13% and 3ms it's ~9%. Also all these tests are done on go1.15.2 on linux/amd64.

I think around 10% cpu might still be too high as the default value. I think if we decided to not disable it by default, then the default value should be something like 5ms or 10ms, so that it doesn't auto add too much overhead, and still being useful for the majority of cases. For services with very low latency endpoints (like yours), they just need to change the default value, either:

1. Disable it (set it to 0) if they don't want those excessive cpu usage

or

2. Set it to a smaller number (1ms/2ms/3ms) to balance between the extra cpu usage and able to abandon requests early when it's no longer needed.

I think even for a 5ms p50 service, the 5ms default value can still be helpful for the long tails?

[dcelasun]

I think even for a 5ms p50 service, the 5ms default value can still be helpful for the long tails?

Yeah that's probably true. Let's go with 5ms then :)