k8s exec: Perform liveness check of clients

[DrJosh9000]

Description

Add a mechanism for terminating if a client stops communicating.

Context

Sometimes, a container will just die (e.g. it is OOM-killed). If that happens, the agent can sit around happily until the heat death of the universe waiting for Exit to be called by something that isn't running any more.

Changes

• Some code rearrangement
• Store clients in a slice instead of a map, since they're keyed by sequential int
• Finer-grained locking
  • Locking around channel close inside a sync.Once.Do is pointless, given nothing else writes to the channels. The existing locking only prevented data races on the clients, and the clients map/slice never changes, so the mutexes can be made client-specific.
• Add a goroutine that inspects client states and terminates the job if a client that should be connected goes silent for too long.

Testing

• Tests have run locally (with go test ./...). Buildkite employees may check this if the pipeline has run automatically.
• Code is formatted (with go fmt ./...)

[wolfeidau]

seem solid, supervisor is 👍🏻 🔍