Host has no proc named

[jaraco]

We recently observed a condition where a proc would not be on a host as expected during a Swarm.

We were dispatching a swarm that had previously been deployed to this squad (rolling back from a bad release). When we did so, this error occurred for only one proc in a swarm of a dozen or so procs, but it blocked the completion.

[jaraco]

I suspect the issue might have been a race condition with the process that cleans up unused procs. Is that possible?

[lbolla]

A bit earlier in the logs it says:

[velociraptor.tasks] Error while deleting Panoptic-28.6.1-EMEA-63706837-web-5413 on yt2.ldc.yougov.local: SoftTimeLimitExceeded(). Will retry.

So, I think that VR tried to teardown 28.6.1 on yt2, but supervisor timed out. When this happens, we schedule a retry of the teardown task after a minute or so. This retry is what caused the error above, because it looks like supervisord eventually became responsive and managed to teardown the proc.
I am surprised that it blocked the swarm, being an async task, though.