pingsource-mt-adapter lease renewal failed, causing pod to restart

[yuchunyu97]

Describe the bug

$ kubectl get pod -n knative-eventing 
NAME                                    READY   STATUS      RESTARTS   AGE
eventing-controller-7fb8b5fb96-xxcml    1/1     Running     0          4d13h
eventing-webhook-94c94f66b-wp8pf        1/1     Running     0          4d13h
imc-controller-7749965797-xrl2f         1/1     Running     0          4d13h
imc-dispatcher-6cbb8f596b-r85gx         1/1     Running     0          4d13h
mt-broker-controller-59b67b9fb9-hsd4x   1/1     Running     0          4d13h
mt-broker-filter-5975fc7dd7-ppjgx       1/1     Running     0          4d13h
mt-broker-ingress-5996958b6d-xvwhp      1/1     Running     0          4d13h
pingsource-mt-adapter-645668955-drvs9   1/1     Running     18         2d18h
sugar-controller-7c7fbc4944-ng5xl       1/1     Running     0          4d13h
v0.16.0-broker-cleanup-k7kz6            0/1     Completed   0          6d16h

$ kubectl logs -n knative-eventing pingsource-mt-adapter-645668955-drvs9 --previous --tail=10
{"level":"error","ts":"2020-09-02T02:20:03.132Z","logger":"pingsource-mt-adapter","caller":"mtping/runner.go:127","msg":"failed to send cloudevent{result 26 0  Post \"http://kong-proxy.knative-kong/async/74403444-6310-4cb2-b637-f8e3c76ab7eb/4bpnhkmsqk/todo-list.%3Fping\": http: ContentLength=11 with Body length 0}","commit":"b9e0b09","stacktrace":"knative.dev/eventing/pkg/adapter/mtping.(*cronJobsRunner).cronTick.func1\n\tknative.dev/eventing/pkg/adapter/mtping/runner.go:127\ngithub.com/robfig/cron/v3.FuncJob.Run\n\tgithub.com/robfig/cron/v3@v3.0.1/cron.go:136\ngithub.com/robfig/cron/v3.(*Cron).startJob.func1\n\tgithub.com/robfig/cron/v3@v3.0.1/cron.go:312"}
{"level":"error","ts":"2020-09-02T02:20:03.133Z","logger":"pingsource-mt-adapter","caller":"mtping/runner.go:127","msg":"failed to send cloudevent{result 26 0  Post \"http://kong-proxy.knative-kong/async/74403444-6310-4cb2-b637-f8e3c76ab7eb/4bpnhkmsqk/yanshi-func1.%3Fping\": http: ContentLength=11 with Body length 0}","commit":"b9e0b09","stacktrace":"knative.dev/eventing/pkg/adapter/mtping.(*cronJobsRunner).cronTick.func1\n\tknative.dev/eventing/pkg/adapter/mtping/runner.go:127\ngithub.com/robfig/cron/v3.FuncJob.Run\n\tgithub.com/robfig/cron/v3@v3.0.1/cron.go:136\ngithub.com/robfig/cron/v3.(*Cron).startJob.func1\n\tgithub.com/robfig/cron/v3@v3.0.1/cron.go:312"}
{"level":"error","ts":"2020-09-02T02:20:03.133Z","logger":"pingsource-mt-adapter","caller":"mtping/runner.go:127","msg":"failed to send cloudevent{result 26 0  Post \"http://kong-proxy.knative-kong/async/c4dc81c6-da86-427e-b9f9-60e495062a70/6lsk9nlxr5/yanshi-func1.LATEST%3Fping\": http: ContentLength=11 with Body length 0}","commit":"b9e0b09","stacktrace":"knative.dev/eventing/pkg/adapter/mtping.(*cronJobsRunner).cronTick.func1\n\tknative.dev/eventing/pkg/adapter/mtping/runner.go:127\ngithub.com/robfig/cron/v3.FuncJob.Run\n\tgithub.com/robfig/cron/v3@v3.0.1/cron.go:136\ngithub.com/robfig/cron/v3.(*Cron).startJob.func1\n\tgithub.com/robfig/cron/v3@v3.0.1/cron.go:312"}
{"level":"error","ts":"2020-09-02T02:20:03.133Z","logger":"pingsource-mt-adapter","caller":"mtping/runner.go:127","msg":"failed to send cloudevent{result 26 0  Post \"http://kong-proxy.knative-kong/async/74403444-6310-4cb2-b637-f8e3c76ab7eb/4bpnhkmsqk/todo-create.%3Fping\": http: ContentLength=11 with Body length 0}","commit":"b9e0b09","stacktrace":"knative.dev/eventing/pkg/adapter/mtping.(*cronJobsRunner).cronTick.func1\n\tknative.dev/eventing/pkg/adapter/mtping/runner.go:127\ngithub.com/robfig/cron/v3.FuncJob.Run\n\tgithub.com/robfig/cron/v3@v3.0.1/cron.go:136\ngithub.com/robfig/cron/v3.(*Cron).startJob.func1\n\tgithub.com/robfig/cron/v3@v3.0.1/cron.go:312"}
I0902 02:20:12.851482       1 leaderelection.go:288] failed to renew lease knative-eventing/pingsource-mt-adapter: failed to tryAcquireOrRenew context deadline exceeded
{"level":"info","ts":"2020-09-02T02:20:12.968Z","logger":"pingsource-mt-adapter.event-broadcaster","caller":"record/event.go:281","msg":"Event(v1.ObjectReference{Kind:\"Lease\", Namespace:\"knative-eventing\", Name:\"pingsource-mt-adapter\", UID:\"11b6039f-3332-4bc3-a147-3195102d8de8\", APIVersion:\"coordination.k8s.io/v1\", ResourceVersion:\"17590110\", FieldPath:\"\"}): type: 'Normal' reason: 'LeaderElection' pingsource-mt-adapter-645668955-drvs9_392a5732-d3ef-4e2e-b0ba-c7ba1c4ade7a stopped leading","commit":"b9e0b09"}
{"level":"info","ts":"2020-09-02T02:20:18.929Z","logger":"pingsource-mt-adapter","caller":"leaderelection/context.go:138","msg":"\"pingsource-mt-adapter-645668955-drvs9_392a5732-d3ef-4e2e-b0ba-c7ba1c4ade7a\" has stopped leading \"pingsource-mt-adapter\"","commit":"b9e0b09"}
E0902 02:20:18.925845       1 leaderelection.go:368] Failed to update lock: rpc error: code = Unavailable desc = transport is closing
E0902 02:20:18.929340       1 leaderelection.go:308] Failed to release lock: rpc error: code = Unavailable desc = transport is closing
{"level":"info","ts":"2020-09-02T02:20:18.977Z","logger":"pingsource-mt-adapter","caller":"mtping/adapter.go:82","msg":"runner stopped","commit":"b9e0b09"}

From #3957
@grantr

Expected behavior
Even if the lease renewal fails, the pod should not restart.

To Reproduce

1. Enable HA
2. Reduce the timeout period.
3. Make the response speed of the cluster slower.

In this way, the lease renewal process may time out and cause the renewal to fail.

Knative release version
v0.17.2

[slinkydeveloper]

ping @lionelvillard

[lberk]

@yuchunyu97 are you able to reproduce this with master? #4009 may have fixed this.

[lberk]

@yuchunyu97 please feel free to reopen this issue if you're still hitting it. Otherwise we're assuming #4009 fixed the issue, thanks!