Jobs stuck in the unfinished state

[MatUrbanski]

Hi, I'm using good_job in the Ruby on Rails 6.0.4.1 app that uses Ruby 2.7.2. Application is deployed to the AWS using Fargate and uses AWS RDS PostgreSQL. After the deployment application is working as expected but at some point, all new jobs are stuck with unfinished state and good_job is not processing them. Here is my configuration:

config/database.yml:

default: &default
  adapter: postgresql
  host: localhost
  pool: <%= ENV.fetch('RAILS_MAX_THREADS') { 4 } %>
  encoding: unicode

development:
  <<: *default
  database: test_app_development

test:
  <<: *default
  database: test_app_test

non_production:
  <<: *default
  database: test_app_non_production
  host: <%= ENV['TEST_APP_NONPROD_DATABASE_HOST'] %>
  username: <%= ENV['TEST_APP_NONPROD_DATABASE_USER'] %>
  password: <%= ENV['TEST_APP_NONPROD_DATABASE_PASSWORD'] %>

production:
  <<: *default
  database: test_app_production
  host: <%= ENV['TEST_APP_DATABASE_HOST'] %>
  username: <%= ENV['TEST_APP_DATABASE_USER'] %>
  password: <%= ENV['TEST_APP_DATABASE_PASSWORD'] %>

/config/initializers/good_job.rb

# Keeps job records in your database even after jobs are completed.
GoodJob.preserve_job_records = true

GoodJob::Engine.middleware.use(Rack::Auth::Basic) do |username, password|
  ActiveSupport::SecurityUtils.secure_compare(ENV['GOOD_JOB_USERNAME'], username) &&
    ActiveSupport::SecurityUtils.secure_compare(ENV['GOOD_JOB_PASSWORD'], password)
end

Any idea what can cause issue like that?

[bensheldon]

@MatUrbanski that's not good! Some questions to help me think through what might be happening:

What execution mode are you running in Production? e.g. setting config.good_job.execution_mode = (or $GOOD_JOB_EXECUTION_MODE). The default is :external which necessitates running a separate good_job process. The other likely option is :async which runs the jobs within your web process.

Do you have any logs? The telltale info logs are Notifier subscribed with LISTEN and Notifier unsubscribed with UNLISTEN that happen when GoodJob connects and disconnects from the database.

(linking to #421 which would be a feature that might help diagnose this easier)

[MatUrbanski]

@bensheldon We are running good_job using the :external execution mode in production. I don't have any logs unfortunately, I've deployed to the environment I'm not responsible for and it's not catching logs from the worker process. Right now we changed the execution mode in production to :async and at this moment everything is working as expected.

[bensheldon]

@MatUrbanski I'm glad we found a workaround by changing the execution mode. I'm not sure why the good_job worker processes would stop executing. A shot in the dark is that the container/process supervisor is shutting them down for some reason (maybe #403 would help?)