Refactor concurrency in scaler requestCounts function

[arschles]

The requestCounts function in the scaler has "funky" concurrency:

• It starts up one goroutine per interceptor endpoint inside an errgroup. Each of these tries to send that interceptor's counts on a channel called countsCh
• It starts up one final goroutine that ranges over the same countsCh channel
• countsCh is closed in a defer

This design means that, when requestCounts returns, none of the updates are processed because everything is waiting for the defer close(countsCh). The updates will happen after it returns. This behavior means that updates to an interceptor might not be viewable immediately after requestCounts completes, and that's confusing.

Use-Case

When code calls requestCounts, it should be able to observe updates to all counts within the queue pinger immediately after that function returns.

Specification

• Refactor requestCounts to update all counts before it returns
• Write tests that call requestCounts directly
• The queue pinger should not start the background goroutine in its constructor. A separate function call should be required for that.

Tell us in detail how this feature should work