[WIP] Use user-defined readiness probes through queue-proxy #4600

joshrider · 2019-07-02T15:28:12Z

Fixes #4014

TODO

check defaults and validations => lots of questions about desired behaviour here. Make a list.
lots of general cleanup
make sure zap errors don't produce lengthy errors that destroy readability of event log
does activator still keep using probeUserContainer?

Proposed Changes

move user-defined readinessProbes from kubelet to the queue-proxy
show default probe in Revision spec if no probe provided
allow the possibility of either our existing, aggressive probe retry semantics (which should speed up cold start), or standard k8s probe semantics

Release Note

knative-prow-robot

@joshrider: 3 warnings.

In response to this:

Fixes #4014

TODO

check defaults and validations => lots of questions about desired behaviour here. Make a list.

investigate flaky TestAutoScaleUpDownUp

lots of general cleanup

pkg/reconciler/revision/resources/deploy_test.go only remaining user of queueReadinessProbe

pkg/reconciler/revision/resources/queue_test.go probe stuff into table tests?

make sure zap errors don't produce lengthy errors that destroy readability of event log

does activator still keep using probeUserContainer?

Proposed Changes

move user-defined readinessProbes from kubelet to the queue-proxy

show default probe in Revision spec if no probe provided

allow the possibility of either our existing, aggressive probe retry semantics (which should speed up cold start), or standard k8s probe semantics

Release Note

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pkg/queue/readiness/probe.go

googlebot · 2019-07-02T15:28:35Z

So there's good news and bad news.

👍 The good news is that everyone that needs to sign a CLA (the pull request submitter and all commit authors) have done so. Everything is all good there.

😕 The bad news is that it appears that one or more commits were authored or co-authored by someone other than the pull request submitter. We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that here in the pull request.

Note to project maintainer: This is a terminal state, meaning the cla/google commit status will not change from this state. It's up to you to confirm consent of all the commit author(s), set the cla label to yes (if enabled on your project), and then merge this pull request when appropriate.

ℹ️ Googlers: Go here for more info.

joshrider · 2019-07-02T15:45:28Z

/assign @mattmoor @dgerd

knative-prow-robot · 2019-07-02T20:43:08Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: joshrider
To complete the pull request process, please assign mattmoor
You can assign the PR to them by writing /assign @mattmoor in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

dgerd · 2019-07-03T20:47:08Z

cmd/queue/main.go

 	mux.HandleFunc(queue.RequestQueueDrainPath, healthState.DrainHandler())

 	return mux
 }

-func probeQueueHealthPath(port int, timeout time.Duration) error {
+func probeQueueHealthPath(port int, timeout int) error {


Since we are now dealing with an int can be useful to make clear that we are dealing with seconds.

Suggested change

func probeQueueHealthPath(port int, timeout int) error {

func probeQueueHealthPath(port int, timeoutSeconds int) error {

dgerd · 2019-07-03T20:47:51Z

cmd/queue/main.go

 	url := fmt.Sprintf(healthURLTemplate, port)

+	// use aggressive retries


nit:

Suggested change

// use aggressive retries

// Use aggressive sub-second retries.

dgerd · 2019-07-03T20:49:04Z

cmd/queue/main.go

+	res, err := httpClient.Get(url)
+
+	if err != nil {
+		return errors.Wrap(err, "failed to probe")


If the original error does not return the URL this could be valuable for debugging.

I'm not sure I understand this comment. Is the suggestion to include the URL in the error message? We build the URL on the fly, but (at present) it will always be http://127.0.0.1:8022/health.

Maybe something better than "failed to probe" is called for?

Yep all I meant was fail to probe: http://127.0.0.1:8022/health provides a little bit more detail to someone looking at the error in the logs without having to trace through the code.

dgerd · 2019-07-03T20:50:55Z

cmd/queue/main.go

 	}

 	var lastErr error

 	// The 25 millisecond retry interval is an unscientific compromise between wanting to get
 	// started as early as possible while still wanting to give the container some breathing
 	// room to get up and running.
-	timeoutErr := wait.PollImmediate(25*time.Millisecond, timeout, func() (bool, error) {
+	timeoutErr := wait.PollImmediate(25*time.Millisecond, readiness.PollTimeout, func() (bool, error) {


nit: Make the 25ms a constant rather than a magic value here

dgerd · 2019-07-03T20:53:34Z

cmd/queue/main.go

@@ -353,9 +395,16 @@ func main() {
 		StatChan:   statChan,
 	}, time.Now())

+	coreProbe, err := parseProbe(*ucProbe)
+	if err != nil {
+		logger.Fatalf("Queue container failed to parse readiness probe %#v", err)


Suggested change

logger.Fatalf("Queue container failed to parse readiness probe %#v", err)

logger.Fatalw("Queue container failed to parse readiness probe", zap.Error(err))

dgerd · 2019-07-03T20:54:22Z

cmd/queue/main.go

@@ -293,11 +325,21 @@ func probeQueueHealthPath(port int, timeout time.Duration) error {
 	return nil
 }

+// parseProbe takes a json serialised *corev1.Probe and returns a Probe or an error


nit: comments for declarations should be sentences and end in periods.

Suggested change

// parseProbe takes a json serialised *corev1.Probe and returns a Probe or an error

// parseProbe takes a json serialised *corev1.Probe and returns a Probe or an error.

dgerd · 2019-07-03T20:57:06Z

pkg/queue/readiness/probe.go

+}
+
+func (p *Probe) timeout() time.Duration {
+	if p.IsStandardProbe() {


Is this supposed to be inversed?

I think it's correct, but defaultProbeTimeout is poorly named.

This represents the maximum length of time waited for a TCP connection/HTTP response. For a user-supplied probe, we are using the value given to us for our single attempt. For our aggressive retries, we timeout much more quickly and just try again.

dgerd · 2019-07-03T20:59:10Z

cmd/queue/main.go

 	mux := http.NewServeMux()
-	mux.HandleFunc(requestQueueHealthPath, healthState.HealthHandler(probeUserContainer))
+
+	if p.IsStandardProbe() {


There seems to be lots of commonality between HealthHandler and HealthHandlerKProbe. Why the switch here rather than just having a HealthHandler function that configures based on StandardProbe flag?

Made a change. LMK what you think.

dgerd · 2019-07-03T21:02:32Z

pkg/apis/serving/k8s_validation.go

+	}
+
+	errs := validateProbe(p)
+


Looks like we have previously not validated the bounds of periodSeconds and let the K8s API server handle the validation. Since we are using the periodSeconds in our controllers now we probably should add the validation that this is a positive number.

I'd probably also split the changes to pkg/apis into it's own change, since it's fairly independent.

Added a check to make sure periodSeconds is non-negative (allowing 0 to signal the aggressive probe) and checks to make sure timeoutSeconds and failureThreshold are non-negative in the case where we're not using the aggressive probe.

dgerd · 2019-07-03T21:05:15Z

pkg/queue/readiness/probe.go

+)
+
+const (
+	defaultProbeTimeout = 100 * time.Millisecond


This is defined here and in main.go. Is this supposed to stay in sync? If so we should make this public and share it between the two places.

The probe timeout in probe.go and main.go are used for different things that don't necessary need to sync.

defaultProbeTimeout in main.go is used in probeUserContainer which is used as a signal to the activator that the pod is up and ready to accept traffic. If we want to keep using the pre-existing, hard-coded TCP connect as the activator's test for readiness, it seems like the constants are two similar, but different, things.

Should the activator's test be the same as the one that's set off from kubelet?

@vagababov @mattmoor Any thoughts here?

pkg/reconciler/revision/resources/queue.go

mattmoor

Thanks for the change @joshrider

Left a few comments throughout, but mainly this is a big change. I think it'll be clearer to tackle in pieces now that you seem to have things working. I made a couple suggestions on places this might split nicely.

pkg/reconciler/revision/resources/queue.go

mattmoor · 2019-07-05T04:23:50Z

pkg/reconciler/revision/resources/deploy.go

@@ -134,8 +134,13 @@ func makePodSpec(rev *v1alpha1.Revision, loggingConfig *logging.Config, observab
 		userContainer.TerminationMessagePolicy = corev1.TerminationMessageFallbackToLogsOnError
 	}

+	if userContainer.ReadinessProbe != nil {
+		if userContainer.ReadinessProbe.HTTPGet != nil || userContainer.ReadinessProbe.TCPSocket != nil {
+			userContainer.ReadinessProbe = nil


This is worth a comment for those without the context of this code review.

mattmoor · 2019-07-05T04:25:49Z

pkg/queue/readiness/probe.go

+}
+
+// IsKProbe indicates whether the default Knative probe with aggressive retries should be used.
+func (p *Probe) IsKProbe() bool {


Perhaps use a more descriptive name like IsAggressive or HasUnrestrictedInterval?

mattmoor · 2019-07-05T04:27:04Z

pkg/queue/readiness/probe.go

+	} else if p.TCPSocket != nil {
+		err = p.tcpProbe()
+	} else if p.Exec != nil {
+		// Assumes the true readinessProbe is being executed directly on user container. See (#4086).


I wonder if we should still TCP probe in this case, since we otherwise have no verification of the networking path from the queue to the user-container.

mattmoor · 2019-07-05T04:28:40Z

pkg/queue/readiness/probe.go

+		return true
+	} else {
+		// using Fprintf for a concise error message in the event log
+		fmt.Fprintf(os.Stderr, "unimplemented probe type.")


Is there any state we can add to this to surface what type we did get?

We can print the whole corev1.Probe here, but all this really means at this point is that all 3 of the extant probe types are nil. Is it worth printing the whole object to catch future cases where there may be an additional probe type?

While leads me to the possibility that the error message may be less precise than it could be...

Also fmt.Fprint, since no formatting.

mattmoor · 2019-07-05T04:36:08Z

pkg/queue/health/probe.go

 // TCPProbe checks that a TCP socket to the address can be opened.
 // Did not reuse k8s.io/kubernetes/pkg/probe/tcp to not create a dependency
 // on klog.
-func TCPProbe(addr string, socketTimeout time.Duration) error {
-	conn, err := net.DialTimeout("tcp", addr, socketTimeout)
+func TCPProbe(config TCPProbeConfigOptions) error {


I would also consider splitting just the changes needed for this package into its own PR.

mattmoor · 2019-07-05T04:38:08Z

pkg/apis/serving/v1beta1/revision_defaults.go

+	if container.ReadinessProbe == nil {
+		container.ReadinessProbe = &corev1.Probe{}
+	}
+	if container.ReadinessProbe.TCPSocket == nil && container.ReadinessProbe.HTTPGet == nil {


What if it is an exec probe?

good catch 👍

mattmoor · 2019-07-05T04:39:40Z

pkg/apis/serving/k8s_validation.go

+	}
+
+	errs := validateProbe(p)
+


I'd probably also split the changes to pkg/apis into it's own change, since it's fairly independent.

mattmoor · 2019-07-05T04:42:35Z

pkg/reconciler/revision/resources/queue.go

+
+	probeJSON, err := json.Marshal(rp)
+	if err != nil {
+		probeJSON = []byte{}


🤔 I don't like swallowing errors :(

mattmoor · 2019-07-05T04:50:01Z

cmd/queue/main.go

+		logger.Fatalw("Queue container failed to parse readiness probe", zap.Error(err))
+	}
+
+	rp := readiness.NewProbe(coreProbe, logger.With(zap.String(logkey.Key, "readinessProbe")))


Consider exposing readiness.EncodeProbe() and readiness.DecodeProbe to [de]serializing the probe.

vagababov · 2019-07-05T05:59:35Z

cmd/queue/main.go

+	// The 25 millisecond retry interval is an unscientific compromise between wanting to get
+	// started as early as possible while still wanting to give the container some breathing
+	// room to get up and running.
+	kProbePollInterval = 25 * time.Millisecond


the naming here seems non-unified.

vagababov · 2019-07-05T06:05:24Z

cmd/queue/main.go

+			// Do not use the cached connection
+			DisableKeepAlives: true,
+		},
+		Timeout: time.Duration(timeoutSeconds) * time.Second,


Do you really need this cast?

I think so?

Timeout: timeoutSeconds * time.Second,

cmd/queue/main.go:278:27: invalid operation: timeoutSeconds * time.Second (mismatched types int and time.Duration)

vagababov · 2019-07-05T06:40:13Z

cmd/queue/main.go

+// parseProbe takes a json serialised *corev1.Probe and returns a Probe or an error.
+func parseProbe(ucProbe string) (*corev1.Probe, error) {
+	p := &corev1.Probe{}
+	err := json.Unmarshal([]byte(ucProbe), p)


if err := ...; err != nil {...}
is nicer.

- Refactor http probe into health package & add tests - Refactor probe function for simplicity

- reduced the timings on the unit test from 39s-> 27s

- Add support of http probe scheme - Add support for parsing json probe in queue

Co-authored-by: Shash Reddy <shashwathireddy@gmail.com>

knative-metrics-robot · 2019-07-08T20:54:56Z

The following is the coverage report on pkg/.
Say /test pull-knative-serving-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/apis/serving/k8s_validation.go	98.9%	99.0%	0.1
pkg/apis/serving/v1beta1/revision_defaults.go	87.5%	89.5%	2.0
pkg/queue/health/probe.go	100.0%	90.9%	-9.1
pkg/queue/readiness/probe.go	Do not exist	100.0%
pkg/queue/readiness/probe_encoding.go	Do not exist	87.5%

knative-prow-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 2, 2019

knative-prow-robot requested review from dgerd, markusthoemmes and mdemirhan July 2, 2019 15:28

knative-prow-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Jul 2, 2019

knative-prow-robot reviewed Jul 2, 2019

View reviewed changes

pkg/queue/readiness/probe.go Show resolved Hide resolved

pkg/queue/readiness/probe.go Show resolved Hide resolved

pkg/queue/readiness/probe.go Show resolved Hide resolved

knative-prow-robot added area/API API objects and controllers area/networking labels Jul 2, 2019

googlebot added the cla: no Indicates the PR's author has not signed the CLA. label Jul 2, 2019

joshrider mentioned this pull request Jul 2, 2019

Apply default livenessProbe and readinessProbe to the user container #4014

Closed

knative-prow-robot assigned dgerd and mattmoor Jul 2, 2019

dgerd reviewed Jul 3, 2019

View reviewed changes

joshrider force-pushed the custom-probes branch from 4a5f26a to 2e3e5e5 Compare July 4, 2019 17:50

mattmoor-sockpuppet reviewed Jul 4, 2019

View reviewed changes

joshrider commented Jul 4, 2019

View reviewed changes

pkg/reconciler/revision/resources/queue.go Show resolved Hide resolved

mattmoor reviewed Jul 5, 2019

View reviewed changes

vagababov reviewed Jul 5, 2019

View reviewed changes

joshrider and others added 9 commits July 8, 2019 10:00

add readiness probe to revision defaults

13b2059

don't export probe factory

64b8e20

fix tcp default host

770842a

add dynamic probe builders

448b04e

- Implment http headers

2cde986

- Refactor http probe into health package & add tests - Refactor probe function for simplicity

rename functions, use dynamic timeout periods

8ac735f

- http probe config and tcp probe config options

b2ee48c

- reduced the timings on the unit test from 39s-> 27s

- Add support of http probe path

b56eddc

- Add support of http probe scheme - Add support for parsing json probe in queue

begin to add test for args

703bbfd

joshrider and others added 22 commits July 8, 2019 10:01

handle exec probe, set defaults

5183eef

add security context to queue probe test

7f22ea0

add validation for special probe behaviour

777ce8c

change arg name to prevent name collision

3e2263c

accept probe timeout as arg in queue binary

99731ca

dynamically build queue probe

3f20af2

directly create empty probe args containerOption

9e3d09f

always probe user-container when using standard probe

d50fb6d

use descriptive name for success count, respect initial delay in kprobe

0e4908a

improve test name readability

74b4a2d

move readinessprobe out

faed876

add kublet probe header to user's http probes

81bc123

clean up names, add comments

7cdce14

add comments to readines probe

7ac5d0b

remove data race

ec43e6f

clean up

9b3f3ef

Co-authored-by: Shash Reddy <shashwathireddy@gmail.com>

move TCP probe generation tests to table tests

13cbf7a

Co-authored-by: Shash Reddy <shashwathireddy@gmail.com>

remove unused variable

738ced3

Co-authored-by: Shash Reddy <shashwathireddy@gmail.com>

consolidate HealthHandlers, fix nits

3092a3f

fix typo

5baa6aa

clean up naming, handle exec probes

af4cc3e

validate readinessProbe values in revisions

4a959a1

joshrider force-pushed the custom-probes branch from 2e582df to 4a959a1 Compare July 8, 2019 15:21

add readiness probe validations

adc701c

This was referenced Jul 8, 2019

Prepare queue healthHandler for optional aggressive retries #4649

Merged

Add queue readiness probe package #4668

Merged

joshrider closed this Jul 16, 2019

joshrider deleted the custom-probes branch August 6, 2019 15:17

	func probeQueueHealthPath(port int, timeout int) error {
	func probeQueueHealthPath(port int, timeoutSeconds int) error {

		url := fmt.Sprintf(healthURLTemplate, port)

		// use aggressive retries

	// use aggressive retries
	// Use aggressive sub-second retries.

	logger.Fatalf("Queue container failed to parse readiness probe %#v", err)
	logger.Fatalw("Queue container failed to parse readiness probe", zap.Error(err))

	// parseProbe takes a json serialised *corev1.Probe and returns a Probe or an error
	// parseProbe takes a json serialised *corev1.Probe and returns a Probe or an error.

[WIP] Use user-defined readiness probes through queue-proxy #4600

[WIP] Use user-defined readiness probes through queue-proxy #4600

Conversation

joshrider commented Jul 2, 2019 • edited Loading

TODO

Proposed Changes

knative-prow-robot left a comment

Choose a reason for hiding this comment

TODO

Proposed Changes

googlebot commented Jul 2, 2019

joshrider commented Jul 2, 2019

knative-prow-robot commented Jul 2, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattmoor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

knative-metrics-robot commented Jul 8, 2019

joshrider commented Jul 2, 2019 •

edited

Loading