Skip to content

Commit

Permalink
fix crash when run sonobuoy in serverless kubernetes cluster
Browse files Browse the repository at this point in the history
We found a crash when ran sonobuoy in serverless kubernetes cluster, see below error log.

In serverless kubernetes cluster, pod may need 10s to start after creation, before container start, pod.Status.StartTime is nil.


# kubectl -n sonobuoy get pod
NAME                                READY   STATUS    RESTARTS   AGE
sonobuoy                            0/1     Error     0          4h57m
sonobuoy-e2e-job-61031ae7baac48e1   2/2     Running   0          4h57m

# kubectl -n sonobuoy logs sonobuoy
time="2020-04-20T02:49:23Z" level=info msg="Starting aggregation server" address=0.0.0.0 port=8080
time="2020-04-20T02:49:23Z" level=info msg="Starting annotation update routine"
time="2020-04-20T02:49:23Z" level=info msg="Running plugin" plugin=e2e
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x1010651]
goroutine 121 [running]:
github.com/vmware-tanzu/sonobuoy/pkg/plugin/driver/utils.IsPodFailing(0xc0004e8a80, 0x16a5e60, 0xc000514500, 0xc0004e8a80)
	/go/src/github.com/vmware-tanzu/sonobuoy/pkg/plugin/driver/utils/utils.go:71 +0x3b1
github.com/vmware-tanzu/sonobuoy/pkg/plugin/driver/job.(*Plugin).monitorOnce(0xc0000fa000, 0x16a5e60, 0xc000514500, 0x0, 0x0, 0x0, 0xc0003e8600, 0x0)
	/go/src/github.com/vmware-tanzu/sonobuoy/pkg/plugin/driver/job/job.go:219 +0x8d
github.com/vmware-tanzu/sonobuoy/pkg/plugin/driver/job.(*Plugin).Monitor(0xc0000fa000, 0x16744e0, 0xc000129ec0, 0x16a5e60, 0xc000514500, 0xc000284840, 0x1, 0x1, 0xc000129e00)
	/go/src/github.com/vmware-tanzu/sonobuoy/pkg/plugin/driver/job/job.go:195 +0x190
created by github.com/vmware-tanzu/sonobuoy/pkg/plugin/aggregation.(*Aggregator).RunAndMonitorPlugin
	/go/src/github.com/vmware-tanzu/sonobuoy/pkg/plugin/aggregation/run.go:233 +0x227
  • Loading branch information
jovizhangwei authored Apr 20, 2020
1 parent 7d97bde commit 064dea1
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion pkg/plugin/driver/utils/utils.go
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ func IsPodFailing(pod *v1.Pod) (bool, string) {
}

// Check if it can't fetch its image within the maximum wait time
if waiting := cstatus.State.Waiting; waiting != nil {
if waiting := cstatus.State.Waiting; waiting != nil && pod.Status.StartTime != nil {
elapsedPodTime := time.Now().Sub(pod.Status.StartTime.Time)
if elapsedPodTime > maxWaitForImageTime && (waiting.Reason == "ImagePullBackOff" || waiting.Reason == "ErrImagePull") {
errstr := fmt.Sprintf("Failed to pull image for container %v within %v. Container is in state %v", cstatus.Name, maxWaitForImageTime, waiting.Reason)
Expand Down

0 comments on commit 064dea1

Please sign in to comment.