Prevent Cancel from running when a k8s job is cancelled already

[CerealBoy]

Description

When a job in k8s completes, the job output will often contain the line # Received cancellation signal, interrupting even though the job itself has already completed. This appears to be a problem with the way the Cancel() method is called during the k8s job execution, and isn't validating that there's any error generated to warrant its execution.

For those looking at logs, this makes the job appear to have failed, when in all other circumstances it has succeeded.

Context

https://coda.io/d/Escalations-Feedback_dHnUHNps1YO/Pipelines-Escalations_su7FT#Pipelines-Escalations-Board_tu__K/r634&view=modal

Changes

A small logic tweak to when Cancel() is triggered based on the k8sAgentSocket.Await() call. Currently Cancel() will always be called regardless of what happens, where it should only be called if there's an error from the Await(). There's additional logic here that will generate an error log line for the job if the error is specifically not a cancellation.

Testing

• Tests have run locally (with go test ./...). Buildkite employees may check this if the pipeline has run automatically. Tests have passed locally and via the pipeline.
• Code is formatted (with go fmt ./...)

[DrJosh9000]

👍