queue proxy should distinguish shutting down from not ready yet

[mattmoor]

/area API
/area networking

Describe the feature

This came up digging into #8147

It seems like the intent of the high-frequency of probing was to speed shutdowns:

serving/pkg/reconciler/revision/resources/queue.go

Lines 151 to 155 in e0b4b30

	// We want to mark the service as not ready as soon as the
	// PreStop handler is called, so we need to check a little
	// bit more often than the default. It is a small
	// sacrifice for a low rate of 503s.
	PeriodSeconds: 1,

However, we don't distinguish between "not yet ready" and shutting down in the health-state management logic:

serving/pkg/queue/health/health_state.go

Lines 107 to 110 in e0b4b30

	case h.IsShuttingDown():
	sendNotAlive()
	case prober != nil && !prober():
	sendNotAlive()

I also don't see any matched logic here:

serving/cmd/queue/main.go

Lines 273 to 287 in 67f920c

	res, lastErr := httpClient.Do(req)
	if lastErr != nil {
	// Return nil error for retrying
	return false, nil
	}
	defer res.Body.Close()
	success := health.IsHTTPProbeReady(res)
	// The check for preferForScaledown() fails readiness faster
	// in the presence of the label
	if preferScaleDown, err := preferPodForScaledown(env.DownwardAPILabelsPath); err != nil {
	fmt.Fprintln(os.Stderr, err)
	} else if !success && preferScaleDown {
	return false, errors.New("failing probe deliberately for pod scaledown")
	}
	return success, nil

So if I'm reading things correctly, we will probe until K8s times out even after we've received the shutdown signal.

This seems like an easy way to save 10s during shutdown to potentially offset a change like: #8148

[mattmoor]

/kind good-first-issue

[tcnghia]

/assign @rafaeltello