Allow use of systemd for worker management

[agrare]

Allow for workers to be run and managed by systemd. This follows exactly the containerized worker model so it doesn't remove a lot of code, but I see this as a way to get it in and start testing.

A worker consists of the following component:

A target which groups multiple logical units together so they can be managed as a group

cat /etc/systemd/system/generic.target
[Unit]
PartOf=cfme.target

A service template which defines the options for the service, this is the basis for multiple unit services like most queue workers

cat /etc/systemd/system/generic@.service 
[Unit]
PartOf=generic.target
[Install]
WantedBy=generic.target
[Service]
Environment=HOME=/root
WorkingDirectory=/var/www/miq/vmdb
ExecStart=/bin/bash -lc 'exec ruby lib/workers/bin/run_single_worker.rb MiqGenericWorker --heartbeat --guid=%i'
Restart=always
Slice=cfme-generic.slice

A settings directory where the per worker settings are stored:

cat /etc/systemd/system/generic@.d/settings.conf 
[Service]

MemoryHigh=524288000
TimeoutStartSec=600
TimeoutStopSec=600

When running it looks like this:

systemctl status cfme.slice
● cfme.slice
   Loaded: loaded
   Active: active since Wed 2019-04-10 11:43:32 EDT; 4h 48min ago
    Tasks: 16
   Memory: 846.1M
   CGroup: /cfme.slice
           ├─cfme-event_handler.slice
           │ └─event_handler@0e8dbcc4-5ee2-4c66-9704-9c89b5edfcf8.service
           │   └─25044 MIQ: MiqEventHandler id: 88, queue: ems
           ├─cfme-generic.slice
           │ └─generic@5a6638a6-c3fa-4287-92c4-b574d176548f.service
           │   └─25222 MIQ: MiqGenericWorker id: 89, queue: generic
           ├─cfme-priority.slice
           │ └─priority@899c9287-7d7b-4dad-981a-b67001d06c9a.service
           │   └─25052 MIQ: MiqPriorityWorker id: 90, queue: generic
           ├─cfme-reporting.slice
           │ └─reporting@1aa08cef-7579-40fb-ae09-194b3a4252d9.service
           │   └─25056 MIQ: MiqReportingWorker id: 91, queue: reporting
           └─cfme-schedule.slice
             └─schedule@a51d1818-776d-4c21-a9f2-537dedfad128.service
               └─25060 MIQ: MiqScheduleWorker id: 92

Apr 10 11:43:32 agrare-thinkpad systemd[1]: Created slice cfme.slice.

We can get logs by slice:

journalctl --unit=cfme-generic.slice
-- Logs begin at Wed 2019-04-10 11:41:48 EDT, end at Wed 2019-04-10 16:33:02 EDT. --
Apr 10 11:43:32 agrare-thinkpad systemd[1]: Created slice cfme-generic.slice.
Apr 10 11:43:39 agrare-thinkpad ruby[2261]:  INFO -- : MIQ(Vmdb::Loggers.apply_config) Log level for journald has been changed to [INFO]
Apr 10 11:43:39 agrare-thinkpad ruby[2261]:  INFO -- : MIQ(Vmdb::Loggers.apply_config) Log level for azure.log has been changed to [WARN]
Apr 10 11:43:42 agrare-thinkpad bash[2261]: ** Using session_store: ActionDispatch::Session::MemCacheStore
Apr 10 11:43:42 agrare-thinkpad bash[2261]: ** ManageIQ master, codename: Ivanchuk
Apr 10 11:43:43 agrare-thinkpad MIQ: MiqGenericWorker id: 87, queue: generic[2261]:  INFO -- : MIQ(MiqGenericWorker::Runner#sync_config) ID [87], PID [2261], GUID [327d1248-7257-41c9-8d18-ae82b5e913d5], Zone [d
Apr 10 11:43:43 agrare-thinkpad MIQ: MiqGenericWorker id: 87, queue: generic[2261]:  INFO -- :

And also only for example show log levels warn or above, journalctl --unit=cfme-generic.slice --priority=warning

TODO:

• Define how to handle provider workers, do we use the unit instance for the ems_id (can't handle ems_id and count then) or do we write out unit files with the ems_id builtin to the unit file env vars, or can we use the env vars in the unit settings
• Remove dependence on temporary worker instance column

Depends on: ManageIQ/manageiq-loggers#5

[miq-bot]

Some comments on commits agrare/manageiq@ff0febc~...9ba6c3a

lib/workers/bin/run_single_worker.rb

• ⚠️ - 69 - Detected puts. Remove all debugging statements.

[miq-bot]

Checked commits agrare/manageiq@ff0febc~...9ba6c3a with ruby 2.3.3, rubocop 0.69.0, haml-lint 0.20.0, and yamllint 1.10.0
10 files checked, 9 offenses detected

app/models/miq_worker/systemd_common.rb

• ❗ - Line 7, Col 9 - Layout/EmptyLineAfterGuardClause - Add empty line after guard clause.

lib/miq_environment.rb

• ❗ - Line 18, Col 7 - Layout/EmptyLineAfterGuardClause - Add empty line after guard clause.

lib/vmdb/loggers.rb

• ❗ - Line 24, Col 34 - Style/GlobalVars - Do not introduce global variables.
• ❗ - Line 24, Col 66 - Style/GlobalVars - Do not introduce global variables.
• ❗ - Line 51, Col 7 - Style/GlobalVars - Do not introduce global variables.
• ❗ - Line 80, Col 55 - Style/GlobalVars - Do not introduce global variables.
• ❗ - Line 80, Col 75 - Style/GlobalVars - Do not introduce global variables.
• ❗ - Line 90, Col 5 - Layout/RescueEnsureAlignment - rescue at 90, 4 is not aligned with def self.create_journald_logger at 85, 25.

lib/workers/bin/run_single_worker.rb

• ❗ - Line 69, Col 3 - Style/StderrPuts - Use warn instead of STDERR.puts to allow such output to be disabled.

[carbonin]

@Fryguy @agrare is this ready to go?

[agrare]

@carbonin it is ready from my side

[carbonin]

So this is still off by default. Do we know who from QE we should rope in to see if they can run some tests with this enabled?

[agrare]

@dmetzger57 can we get QE to setup an appliance with this feature turned on?

[dmetzger57]

I'll work with QE to get a resource(s) allocated for initial testing, @agrare should I have then contact you directly with any specific questions?

[agrare]

@agrare should I have then contact you directly with any specific questions?

Definitely not 😆 of course