Horizon jobs pauses and doesn't get processed

[imphus]

Hi everyone,

I have issue with horizon in one of my projects. So I have supervisor running to check if horizon is working and if not start horizon. Horizon is running ok for couple of days or so and then after that I get call from clients saying that their emails is not sending out again, once I check horizon dashboard I can see jobs paused there.

Here is snippet of my horizon log before I have to restart it manually for jobs to be processed:

Could not open input file: artisan
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
Could not open input file: artisan
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
Could not open input file: artisan
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
PHP Fatal error: Uncaught Error: Class 'Symfony\Component\Debug\Exception\FatalThrowableError' not found in /home/quantilliveerp/releases/20180912144025/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php:77
Stack trace:
#0 [internal function]: Illuminate\Foundation\Bootstrap\HandleExceptions->handleException(Object(Error))
#1 {main}
thrown in /home/quantilliveerp/releases/20180912144025/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php on line 77
PHP Fatal error: Uncaught Error: Class 'Symfony\Component\Debug\Exception\FatalErrorException' not found in /home/quantilliveerp/releases/20180912144025/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php:136
Stack trace:
#0 /home/quantilliveerp/releases/20180912144025/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php(123): Illuminate\Foundation\Bootstrap\HandleExceptions->fatalExceptionFromError(Array, 0)
#1 [internal function]: Illuminate\Foundation\Bootstrap\HandleExceptions->handleShutdown()
#2 {main}
thrown in /home/quantilliveerp/releases/20180912144025/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php on line 136
Horizon started successfully.
[2018-09-17 08:45:11] Processing: ERP\Core\Jobs\DocDesigner\SendDocumentNotification

Not really sure where the issue is or how to solve it.

Thanks for all your help!

[mfn]

Could not open input file: artisan
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory

This is an indicator of a problem with the underlying file system.

Is it a network mounted filesystem? It may have had temporarily problems causing it files not being accessible?

[kylestev]

It looks like you might be using something similar to envoyer.io for managing your releases. Is it possible that you had a Horizon worker that was not terminated and the releases folder for it was deleted?

[koenvu]

For me, during the deploy process I would call artisan horizon:terminate and let supervisor restart the queue to make sure it was running with newer code. What happened was that I was running 1.3.1 on one release, then updated to 1.4.0 and deployed. When calling horizon:terminate it could not find the hostname in the masters list because the hostname was determined in a new way since PR #359 .

This prevented horizon:terminate from sending the kill signal to the process, and effectively had it keep running the old instance, since supervisor was (correctly) still aware that horizon was running. After a few more deploys, the old release folder was cleaned up so workers could no longer be started, however the master supervisor had everything it needed in memory and was happy to keep spawning invalid supervisors and/or workers that were never able to process any queue jobs.

Manually restarting horizon solved the issue.

[mfn]

Ah, I remember this.

I think I was hit with the same issue but weren't able to connect to dots here. Great detective work there 🔍

I would seem that stopping horizon, making deploy, starting horizon would fix this.

But not really with supervisor because it expects to always run the service. So even if you send the terminate before deploying the new code, the instance will likely be instantly up again. 🤷‍♀️

[driesvints]

@imphus please check the responses above to see if something can help you.

@kylestev @koenvu @mfn thanks for helping out 👍