Set the monitor_running event to unblock the main thread

[agrare]

The do_before_work_loop starts the event monitor.

This spins up the thread and waits for it to be started. The amazon event catcher only sets the started event to true once an event is raised.

This means if you try to stop the event catcher before an event occurs you have to SIGKILL it as it will never shutdown cleanly.

[jrafanie]

This fixes the issues I saw in ManageIQ/manageiq#20290 where we'd do event_catcher#kill on an amazon catcher and it would stick around for 10 minutes, the stopping timeout in pods. @carbonin suggested I try SIGTERM locally and indeed SIGTERM was seemingly being ignored on my amazon event catcher.

@agrare tracked this down to where we trap interrupts and later react to them once do_work completes. Since do_work calls start_event_monitor in the base event catcher and start_event_monitor won't start until the first event is received, it never gets the exit in the event of a tiny setup with very few events... it couldn't be more obvious now...

[djberg96]

@agrare Don't suppose there's a way to spec this, is there?

[agrare]

@djberg96 the existing tests for Worker Runners are pretty hacky and have to stub a bunch of stuff to not actually start up a worker, so there isn't an easy way to spec this

[agrare]

For reference this is what the VMware event catcher does: https://github.com/ManageIQ/manageiq-providers-vmware/blob/master/app/models/manageiq/providers/vmware/infra_manager/event_catcher/runner.rb#L27-L29

[miq-bot]

Checked commit agrare@d8308e7 with ruby 2.5.7, rubocop 0.69.0, haml-lint 0.28.0, and yamllint
1 file checked, 0 offenses detected
Everything looks fine. 👍

[djberg96]

@agrare Could there be a connection between this and #620 ?

[agrare]

Hm that's a good question, my first reaction was no because once an event is caught and yielded to that block it would set the event that the monitor thread was started. There could be some weirdness though in the fact that the main runner thread isn't in the do_work loop where it actually processes events until that event is set.

[simaishi]

Jansa backport details:

$ git log -1
commit dd5052771d9ab30b215be05dbc92dac6c3ef60fa
Author: Oleg Barenboim <chessbyte@gmail.com>
Date:   Thu Jun 25 12:09:17 2020 -0400

    Merge pull request #633 from agrare/event_monitor_never_sets_monitor_running

    Set the monitor_running event to unblock the main thread

    (cherry picked from commit 3d9f0905fbb5561ccf452be25496759385a71fd8)