Handle SIGTERM in run_single_worker.rb

[NickLaMuro]

Termination of workers via the MiqServer::WorkerManager currently happens by sending a exit message over DRB to the worker that it will process in between doing jobs (so nothing is halfway in flight). The MiqServer will wait for a specific amount of time (timeout defined in the settings, but defaults to 5.minutes or so usually) before killing it (at this point, it is assumed the worker is stuck).

Currently, SIGTERMS are trapped and handled by the worker runner class to run the do_exit method, but they don't allow the work currently in progress to finish. The addition of the #setup_sigterm_trap method will introduce Kernel traps that properly inform the worker that once they have finished with their next round of work, they should exit.

Steps for Testing/QA

• Apply the following diff (needed becuase the VimBroker won't be present, and adds some handy debugging output):

  diff --git a/app/models/miq_generic_worker/runner.rb b/app/models/miq_generic_worker/runner.rb
  index 022db94..15c0175 100644
  --- a/app/models/miq_generic_worker/runner.rb
  +++ b/app/models/miq_generic_worker/runner.rb
  @@ -1,3 +1,3 @@
   class MiqGenericWorker::Runner < MiqQueueWorkerBase::Runner
  -  self.delay_startup_for_vim_broker = true # NOTE: For ems_operations and smartstate roles
  +  self.delay_startup_for_vim_broker = false # NOTE: For ems_operations and smartstate roles
   end
  diff --git a/app/models/miq_queue_worker_base/runner.rb b/app/models/miq_queue_worker_base/runner.rb
  index 206066b..0179d9d 100644
  --- a/app/models/miq_queue_worker_base/runner.rb
  +++ b/app/models/miq_queue_worker_base/runner.rb
  @@ -96,6 +96,7 @@ class MiqQueueWorkerBase::Runner < MiqWorker::Runner
     end
  
     def deliver_queue_message(msg)
  +    print "delivering queue message #{msg.class_name}.#{msg.method_name}..."
       reset_poll_escalate if poll_method == :sleep_poll_escalate
  
       begin
  @@ -128,6 +129,7 @@ class MiqQueueWorkerBase::Runner < MiqWorker::Runner
         #
         clean_broker_connection
       end
  +    puts "done!"
     end
  
     def deliver_message(msg)

• Create some long running work for your worker to act on (the time it sleeps in :args can be adjusted as needed):

  $ bin/rails runner 'MiqQueue.put :class_name => "Kernel", :method_name => "sleep", :args => 200, :zone => MiqServer.my_server.zone.name, :queue_name => "generic"'

• Start a worker using the run_single_worker.rb script:

  $ ruby lib/workers/bin/run_single_worker.rb MiqGenericWorker

• Wait for the worker to start processing the job, then interrupt it (Crtl-C).

The worker should output done! before exiting.

[NickLaMuro]

@miq-bot assign @gtanzillo

[miq-bot]

Checked commits NickLaMuro/manageiq@514af2a~...7776e35 with ruby 2.2.6, rubocop 0.47.1, and haml-lint 0.20.0
2 files checked, 0 offenses detected
Everything looks fine. ⭐

[jrafanie]

LGTM, merging. @gtanzillo you might want to take a look.

[jrafanie]

LOL, received... I noticed the spelling mistake after merging 😢

Does it make sense to also initialize @sigterm_received = false somewhere?

[NickLaMuro]

@jrafanie good catch on both of those. Accessing a instance var will cause warnings when in verbose mode (so these probably exist in in the test output from travis), so yeah, those should be fixed.

Will open up a follow up PR shortly.

[jrafanie]

YOLO MERGE FTW