fix: Create copy of backoff struct

The backoff is created in `interceptor/main.go` and `.Step()` is a
pointer receiver, so `.Step()` will mutate the underling number of
`.Steps` within the struct. When you have 0 replicas, the default 500ms
timeout of the Dial is hit which basically guarantees that you will end
up calling `.Step()`. The value inside the the struct will never be
reset, and so every single time you're scaling from 0, you're guaranteed
to reduce the number of `.Steps` by 1 and this will never reset until
the application itself has been restarted.

This change makes sure that every time we execute
`DialContextWithRetry`, we start with a fresh `backoff` which will start
the `.Steps` at 5 since it's a clone of the original `backoff`. This
`backoff` is available in the context of the function returned by
`DialContextWithRetry`, which will be the one that gets decremented, and
then will be garbage collected and we get a brand new 5 steps the next
time we execute `DialContextWithRetry`.

Closes #586

Signed-off-by: Aaron Batilo <AaronBatilo@gmail.com>

pkg/net/dial_context.go

@@ -29,6 +29,10 @@ type DialContextFunc func(ctx context.Context, network, addr string) (net.Conn,
 func DialContextWithRetry(coreDialer *net.Dialer, backoff wait.Backoff) DialContextFunc {
 	numDialTries := backoff.Steps
 	return func(ctx context.Context, network, addr string) (net.Conn, error) {
+		// We need to copy the backoff struct because Step() mutates it but the
+		// number of steps is never reset.
+		backoff := backoff
+
 		// note that we could test for backoff.Steps >= 0 here, but every call to backoff.Step()
 		// (below) decrements the backoff.Steps value. If you accidentally call that function
 		// more than once inside the loop, you will reduce the number of times the loop