Ingress Readiness and Liveness check stop working after an upstream change

[grac3gao-zz]

Describe the bug
Ingress readiness and liveness (using kncloudevents library from eventing) check stop working after an upstream change: knative/eventing#3917

Expected behavior
A clear and concise description of what you expected to happen.

To Reproduce
change the following line to return response.WriteHeader(nethttp.StatusMethodNotAllowed)

knative-gcp/pkg/broker/ingress/handler.go

Lines 123 to 125 in 544ec2d

	if request.URL.Path == heathCheckPath {
	response.WriteHeader(nethttp.StatusOK)
	return

It should fail the readiness and liveness check for the ingress pod, however, ingress pod will continue work.

Revert the upstream change, then return response.WriteHeader(nethttp.StatusMethodNotAllowed) , readiness and liveness check will work.

Knative-GCP release version

Additional context
Add any other context about the problem here such as proposed priority

[grac3gao-zz]

kncloudevents are using drainer in pkg to run liveness and readiness check now (https://github.com/knative/pkg/blob/d5c09d2aef18eaabbb9ce3ac71a487288ca542aa/network/handlers/drain.go#L78), all inbound request will firstly send through drainer then to the handler we configured. Drainer will take every request from k8s probe and truncated it instead of continuing sending it to our handler, which makes all our customized configuration for liveness and readiness useless as no request from k8s probe will be send into handler.
There are two ways to fix it:

1. In the upstream drainer library, adding an interface for a customizedCheck function. Drainer will need to run customizedCheck for probe requests instead of directly returning 200. Then, change kncloudevents to also have a field for customizedCheck to initialize drainer.
2. Have our own message_receive which is mostly same as kncloudevents except wrapping the drainer to run some customized check.

[grac3gao-zz]

knative-pkg PR merged: knative/pkg#1977

[grac3gao-zz]

Knative-eventing PR merged: knative/eventing#4707

Now we are OK to add customized liveness and readiness check