Job processing failing after booting new worker

[bagedevimo]

We're seeing situations where a job process (or multiple) is no longer processing jobs after being booted during a deploy.

Our platform runs on ECS in containers, with multiple containers booted to run jobs with some configuration on which queues each set of workers runs from and which priority, via CLI flags. Sporadically, we see situations where the good_job process is booted but does not participate in running any jobs. We've only had a few attempts to try and debug this, and so far discovered very little. We're seeing nothing of interest in the logs or in our error monitoring.

What I'm hoping to figure out is: what tips do y'all have for debugging this?

[bensheldon]

That's strange!

The good_job CLI runs your application (by loading config/environment.rb), so if you adjust your application's logging level (and turn on things like Active Record verbose query logs) you should be able to get a lot of logs out of it.

Are you seeing the process in the good_job_processes table and the queue name looks correct?

[bagedevimo]

Yes, and exec'ing into the container and running the ps auxf shows the expected processes. gdb attaching the process shows a bunch of threads.

[bagedevimo]

Hmm a bit more (very slow, sorry) debugging has revealed it might be due to a connection being dropped by the server and good job not noticing somehow. Maybe related to RDS? We're trying some more things now.