CC-1206: Add a check for zombie DJ workers

[radamanthus]

Description of your patch

Properly recover when a DelayedJob worker was terminated and is lingering as a zombie process.

There is a bug here:

ey-cookbooks-stable-v5/cookbooks/delayed_job4/templates/default/dj.erb

Lines 51 to 62 in 7310e1b

	if [ -e $LOCK_FILE ]; then
	LAST_LOCK_PID=`cat $LOCK_FILE`
	if [ -n $LAST_LOCK_PID -a -z "`ps axo pid|grep $LAST_LOCK_PID`" -a -f $LOCK_FILE ];then
	sleep 1
	logger -t "monit-dj:$WORKER[$$]" "Removing stale lock file for $WORKER ($LAST_LOCK_PID)"
	rm $LOCK_FILE 2>&1
	else
	logger -t "monit-dj:$WORKER[$$]" "Monit already messing with $WORKER ($LAST_LOCK_PID)"
	RESULT=1
	exit_cleanly
	fi
	fi

The test

if [ -n $LAST_LOCK_PID -a -z " ps axo pid|grep $LAST_LOCK_PID" -a -f $LOCK_FILE ];then 

tests if $LAST_LOCK_PID is defined and there’s no running process with that pid, but there is a lock file. It goes to the "Monit already messing with..." block if there is a running process, even if it's a zombie process.

This PR adds an additional test to check if the PID matches a running process but the process is a zombie.

Recommended Release Notes

Updates the delayed_job4 recipe to properly handle zombie workers

Estimated risk

Low

Components involved

DelayedJob custom chef recipe

Description of testing done

See QA instructions

QA Instructions

NOTE: These are the same as the QA instructions for PR #224. This PR can be tested with #224.

Test on configuration A_dj

Configuration A
rails_activejob_example (delayed_job branch) App
Unicorn
Ruby 2.3
RubyGems 2.6.5
Postgres 9.5
US East Virginia
Solo

Boot the test environment under the QA stack
Enable the delayed_job recipe.
Modify the recipe to install DelayedJob on the solo instance
Modify the recipe and set a very low worker memory limit (e.g. low enough to always trigger the memory limit even with zero workload, e.g. 10MB)
Run chef
Observe the delayed_job processes by running ps -ef | grep elay
Make sure:

• monit is terminating the DelayedJob process
• new workers are being started
• no orphan processes are being left behind

[crigor]

LGTM