Events/Uploads related SIGSEGV

[deusxanima]

Description

What happened:

Customer started observing consistent segfaults on 4.3.7. After upgrading to 4.4.9 customer observed no segfaults for 24hours, but as soon as their team started working on the cluster again the segfaults reappeared, now even more aggressively than before.

Server Details

• Teleport version (run teleport version): 4.4.9

Debug Logs

Apr 09 14:21:16 pylondal1201a.softlayer.local teleport[53234]: panic: runtime error: invalid memory address or nil pointer dereference
Apr 09 14:21:16 pylondal1201a.softlayer.local teleport[53234]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xdc3762]
Apr 09 14:21:16 pylondal1201a.softlayer.local teleport[53234]: goroutine 1718 [running]:
Apr 09 14:21:16 pylondal1201a.softlayer.local teleport[53234]: github.com/gravitational/teleport/lib/events.(*UploadCompleter).CheckUploads(0xc001354500, 0x2da8f80, 0xc000f2be40, 0x2, 0x1)
Apr 09 14:21:16 pylondal1201a.softlayer.local teleport[53234]: /go/src/github.com/gravitational/teleport/lib/events/complete.go:154 +0x6f2
Apr 09 14:21:16 pylondal1201a.softlayer.local teleport[53234]: github.com/gravitational/teleport/lib/events/filesessions.(*Uploader).Serve(0xc00024fe00, 0x120, 0x110)
Apr 09 14:21:16 pylondal1201a.softlayer.local teleport[53234]: /go/src/github.com/gravitational/teleport/lib/events/filesessions/fileasync.go:221 +0x9de
Apr 09 14:21:16 pylondal1201a.softlayer.local teleport[53234]: github.com/gravitational/teleport/lib/service.(*TeleportProcess).initUploaderService.func3(0x0, 0xffffffffffffffff)
Apr 09 14:21:16 pylondal1201a.softlayer.local teleport[53234]: /go/src/github.com/gravitational/teleport/lib/service/service.go:1851 +0x48
Apr 09 14:21:16 pylondal1201a.softlayer.local teleport[53234]: github.com/gravitational/teleport/lib/service.(*LocalService).Serve(0xc000935f60, 0x284b18c, 0x14)
Apr 09 14:21:16 pylondal1201a.softlayer.local teleport[53234]: /go/src/github.com/gravitational/teleport/lib/service/supervisor.go:450 +0x2a
Apr 09 14:21:16 pylondal1201a.softlayer.local teleport[53234]: github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1(0xc00064a300, 0x2da96c0, 0xc000935f60)
Apr 09 14:21:16 pylondal1201a.softlayer.local teleport[53234]: /go/src/github.com/gravitational/teleport/lib/service/supervisor.go:242 +0x30b
Apr 09 14:21:16 pylondal1201a.softlayer.local teleport[53234]: created by github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve
Apr 09 14:21:16 pylondal1201a.softlayer.local teleport[53234]: /go/src/github.com/gravitational/teleport/lib/service/supervisor.go:237 +0x6e
Apr 09 14:21:16 pylondal1201a.softlayer.local systemd[1]: teleport.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Apr 09 14:21:16 pylondal1201a.softlayer.local systemd[1]: Unit teleport.service entered failed state.
Apr 09 14:21:16 pylondal1201a.softlayer.local systemd[1]: teleport.service failed.

[xacrimon]

Fixed via #6338 in master and should be backported to v4

[xacrimon]

Fixed in 4.4 via #6433

[deusxanima]

Confirmed fix and backport with impacted customer. Closing issue.