K8s runner retry exec

[joshrwolf]

this fixes a class of flakes that appear when trying to run a remote command (k exec) in a pod that isn't ready to receive requests yet.

the most common example we've seen is on gke autopilot, where nodes can scale up all the way to pod Running, but not have a healthy networking backplane yet (anetd is still spinning), resulting in a flake that looks like:

2023/07/16 05:58:08 error during command execution: failed to build package: unable to run pipeline: running remote command: failed to run remote command: error dialing backend: No agent available

this ensures we retry everything that isn't an actual build error, up to 4 times