Queue worker stops suddenly

[remyhonig]

What were you trying to do?

My app needs to import thousands of pdf files and extract their text and generate thumbnails. This is done using the queue because I do not want to block the UI for an hour. This is done by dispatching a job that reads all the files in a directory and make that job dispatch a job for each file it finds.

What happened?

I expect the queue worker to process all the dispatched jobs. But after about 200 jobs it stops. If I restart using artisan native:serve the job starts processing about another 200 jobs and then stops again. However, if I run the artisan native:queue now it processes all jobs as expected without a problem.

How to reproduce the bug

Insert thousands of jobs on the queue and run artisan native:serve, it will stop after a short while.

Package Versions

{ "installed": [ { "name": "nativephp/electron", "direct-dependency": true, "homepage": "https://github.com/nativephp/electron", "source": "https://github.com/NativePHP/electron/tree/0.6.5", "version": "0.6.5", "description": "Electron wrapper for the NativePHP framework.", "abandoned": false }, { "name": "nativephp/laravel", "direct-dependency": false, "homepage": "https://github.com/nativephp/laravel", "source": "https://github.com/NativePHP/laravel/tree/0.5.3", "version": "0.5.3", "description": "Laravel wrapper for the NativePHP framework.", "abandoned": false }, { "name": "nativephp/php-bin", "direct-dependency": true, "homepage": "https://nativephp.com", "source": "https://github.com/NativePHP/php-bin/tree/0.4.0", "version": "0.4.0", "description": "PHP binaries used by the NativePHP framework", "abandoned": false } ] }

PHP Version

8.3

Laravel Version

11.0

Node Version

21.7.0

Which operating systems have you seen this occur on?

macOS

OS version

macos 14.5

Notes

No response

[simonhamp]

@remyhonig it would be really helpful if you're able to provide a repo that reproduces this issue

[remyhonig]

Hi @simonhamp . This is the code that reproduces it remyhonig/nativephp-queue-halting@96523ae

To run:

1. clone repo
2. composer install
3. php artisan native:migrate:fresh
4. php artisan native:serve
5. click on the button "Dispatch Many Jobs"

Expected:

• Seeing the 1000 times an info log line "SmallerJob: "

Actual:

• Consumer halts at 154 or so

[remyhonig]

I simplified the example further. The button now dispatches 1000 jobs from the Main Livewire component directly and removed any database access operations from the job. It is really just a log line that is generated now. It seems the database just locks up after an x amount of jobs having been processed.

[remyhonig]

Maybe it's not a solution that fixes the root cause but automatically restarting the worker if it crashes might be an acceptable solution?

[simonhamp]

It seems the database just locks up after an x amount of jobs having been processed.

Is it the database locking? Can you hit the database from elsewhere (e.g. your main application) and see if it hangs?

automatically restarting the worker if it crashes

It feels like we should definitely handle this - maybe this won't be the only way that the queue worker could crash... but what I'm not sure of yet is how we detect that it's crashed to restart it.

heads up: I haven't as yet had a chance to review your repo

[remyhonig]

I have not been able to reproduce the database lock error message I got once in the console log this evening.

With regards to accessing the database from another process I do know that running a separate process artisan native:queue will process all jobs as expected.

The worker does not seem to really crash either now I'm inspecting the running processes with htop and then filter on processes with queue in the name. The process is still alive even though it does not process anymore messages.

I have also tried this in vendor/nativephp/electron/src/Commands/QueueWorkerCommand.php:

public function handle(): void
    {
        $phpBinary = __DIR__.'/../../resources/js/resources/php/php';

        while (true) {
            intro('Starting NativePHP queue worker…');
            Log::info('Starting NativePHP queue worker…');

            try {
                $result = Process::path(base_path())
                    ->env([
                        ...
                    ])
                    ->forever()
                    # CHANGED: added --max-jobs=100 and place inside while loop
                    ->run($phpBinary . ' artisan queue:work --max-jobs=100', function (string $type, string $output) {
                        echo $output;
                    });
                Log::info('queue:work exit code: ' . $result->exitCode());
            } catch (\Throwable $e) {
                Log::error('queue:work exception: ' . $e->getMessage());
            }
        }
    }

and changing this in vendor/nativephp/electron/resources/js/node_modules/@nativephp/electron-plugin/dist/server/php.js:

function startQueueWorker(secret, apiPort, phpIniSettings = {}) {
    const env = {
        APP_ENV: process.env.NODE_ENV === 'development' ? 'local' : 'production',
        APP_DEBUG: process.env.NODE_ENV === 'development' ? 'true' : 'false',
        NATIVEPHP_STORAGE_PATH: storagePath,
        NATIVEPHP_DATABASE_PATH: databaseFile,
        NATIVEPHP_API_URL: `http://localhost:${apiPort}/api/`,
        NATIVEPHP_RUNNING: true,
        NATIVEPHP_SECRET: secret
    };
    const phpOptions = {
        cwd: appPath,
        env
    };
    //CHANGED: native:queue instead of queue:work
    return callPhp(['artisan', 'native:queue'], phpOptions, phpIniSettings);
}

This resulted in the worker being restarted once (but not always and never more than once).

I understand you have a lot on your plate. So please don't feel rushed, I am not expecting quick replies. I just hope this research might contribute to a solution at some point. Thank you for you work!!

[simonhamp]

@remyhonig I finally got around to this. I made a couple of really important discoveries here! So thank you for preparing your repo 🙏🏼

1. Database locking: You are most likely experiencing database locking because we don't use WAL mode in SQLite... yet. I believe this is a no-brainer change that would be a sensible default for everyone, so expect this change to come in a very-near-future release 🚀
2. Queue failure: I found that the queue worker just silently fails without any output or logs if its STDOUT isn't being listened to, which then led me to just make it not output anything by passing the -q|--quiet option. I'm not entirely sure why this would make a difference, but in general it feels like it's getting clogged up with no outlet... This "fix" will also be released shortly.

Another note: we will likely be removing the native:queue command in an upcoming release (see #402 (comment)). If you have any thoughts about this, please chime in on that conversation.

I really do appreciate you taking the time to put this together and try stuff out here, it has helped immensely! I'll cycle back here once these fixes are out

[simonhamp]

All done

[simonhamp]

Released:

• nativephp/laravel v0.6.2
• nativephp/electron v0.8.5