Feature request: Working with long running forked processes

[gwleuverink]

Hi all,

I've got this idea for NativePHP I'd love to get some feedback on. It looks like a fun challenge and could open up NativePHP to cater to a whole new set of new use-cases.

The problem

While working on Phost (the app I use as a nativephp proving ground) I had the need to have a long running php process in the background. In this case I use that to spawn a SMTP server with ReactPHP.

In a Electron app this is done using a forked process. In NativePHP there is no recommended way to do this right now. (as far as I know)

I’ve tried a bunch of approaches, in the end I landed on a scheduled job that keeps the smtp server alive, then continually poll the back-end to check if it’s running.

This works but lacks on a couple of fronts.

1. I have very little control over the process. Like stopping/restarting is hard, I have to manage the PID myself and lacks a cross-platform api
2. It falls short when we have multiple long running processes started by the scheduler
3. Behaviour can be somewhat unexpected. For example in production builds I sometimes have stray processes that stay alive even though the app is terminated. Which results in the new process not being able to start due to the port already being in use
4. There is a significant delay in starting the process using the scheduler in production builds

It would be very powerful if NativePHP provided a first party method to manage parallel php processes (with ReactPHP for example). The recent addition of the IPC stuff has gotten me thinking this might be the key to building a nice api for this.

A solution?

I'm still workshopping the idea, but I envision something like this:

// From a controller, Livewire action or even in a Service Provider 
// you can start a parallel php script on a forked node process

ForkedProcess(App\Native\SmtpServer::class, alias: 'smtp-server')->run()

Then using the new Native helper I imagine we can have a api to perform some basic operations on the process.

// Managing the process
Native.process('smtp-server').status() // true: running, false: stopped
Native.process('smtp-server').restart()
Native.process('smtp-server').stop()

Additionally I can see some benefit to dispatch process events internally we can react to in our webview

Native.process('smtp-server').errored(error => {
    //
})

Native.process('smtp-server').started(() => {
    //
})

Native.process('smtp-server').stopped(() => {
    //
})

As to the process communicating messages back to the UI, It could be enough to use the existing Events over the nativephp channel for inter process communication. Something more sophisticated might also be possible, to pass messages from and to the process, but this will do for now not to complicate it too early.

// You could just dispatch events from the process
Native.on('App\\Events\\MessageReceived', payload => {
    //
})

---

With this concept I could (hopefully) achieve something like this; a way to dynamically update a status indicator & restart the process from the UI. For example using Alpine:

<!-- Status indicator -->
<span 
  x-data="{
    online: Native.process('smtp-server').status()
  }"
  x-init="
    
    Native.process('smtp-server').started(() => {
      this.online = true
    })

    Native.process('smtp-server').stopped(() => {
      this.online = false
    })
  "
  class="relative flex h-3 w-3"
>

  <span 
    :class="{ 
      'bg-sky-400': online,
      'animate-ping bg-red-400': !online,
    }" 
    class="absolute inline-flex h-full w-full rounded-full  opacity-75"
  ></span>

  <span 
     :class="{ 
       'bg-sky-500': online,
       'bg-red-500': !online,
    }" 
    class="relative inline-flex rounded-full h-3 w-3"
  ></span>


</span>

A big asterisk

Right now I have no idea if this is even possible. Just riffing on a possible API.

Love to get your feedback on this idea!

[simonhamp]

Yep, I love this. Quite a few folks are asking about spinning up processes when their app runs.

On the surface, this feels simply like PHP could handle this all... but if you want decent performance and "proper" control and management of the processes that are spawned, you'd need pcntl, but that will never work on Windows.

Bummer!

Luckily, both Electron and Tauri have pretty good first-party support for cross-platform process management via Node and Rust, respectively.

I like your ForkedProcess concept, but perhaps it can just be called Process? With an appropriate facade, this could have a really nice API, maybe quite similar to the Laravel Process façade - except, where Laravel's Process uses "signals" (POSIX signals which are supported by the underlying Symfony Process component), we would think in terms of "messages":

use Native\Laravel\Facades\Process;

/**
 * @param string $name A unique name for this process
 * @param array $cmd The command to run
 * @param string $cwd The working directory to run the command from
 * @param array $env Override
 */
Process::start('internal-server', ['php', 'artisan', 'serve'], env: ['CUSTOM_ENV' => 'yes please!']);

// Then other methods can just receive the process name. The Laravel app doesn't necessarily need any concept
// of 'remembering' processes, they can just be referenced by name.

Process::stop('internal-server');
Process::sendMessage('internal-server', 'some message string');

// Maybe a convenience for handling events...
Process::onMessage('internal-server', function ($message) {});
Process::onExit('internal-server', function ($output) {});

// These Could potentially be chained before or after the `start` call too - the order shouldn't matter
Process::create('internal-server', ['php', 'artisan', 'serve'], env: ['CUSTOM_ENV' => 'yes please!'])
    ->onMessage(function ($message) {})
    ->onExit(fn ...)
    ->start();

Then on the Electron side, there would need to be an endpoint that can handle this request and asynchronously turn it into a managed spawned command (I recommend using Electron's utilityProcess module). Async so that it can 'return' to PHP quickly, but internally keeps tabs on the process.

I guess we should use a processes state (similar to how we manage windows) that keeps a register of all of the spawned processes.

Events

We should register standardised event handlers for each of the supported events on each process we spawn. These events should fire new NativePHP events - again, very similar to what we do for windows.

In terms of where we make event handling available, I'd try to keep things pretty simple to start with: i.e. let's just use the event mechanics we already have rather than inventing new ones in Javascript... i.e. avoid Native.process for now as it's not apparent yet that we'd need that.

[simonhamp]

Maybe this is overkill tho...

I've realised that proc_open in PHP is available cross-platform and should provide the same primitives that the above proposed approach provides without having to offload the work to each runtime (Electron/Tauri).

So maybe we don't need to go to all of this trouble?

proc_open isn't quite as nice as pcntl as it's not allowing us to 'fork' a process from within our PHP process, so some extra work would be required to make sure processes shut down cleanly and I guess there's always a chance that they won't if the app crashes and that cleanup can't execute...

One to ponder... 🤔

[gwleuverink]

I did a little bit of tinkering but have not had success yet. It's kind of hard to trace what exactly is happening, but I have some failing tests that could be a good starting point.

This StackOverflow answer suggests processes called by proc_open close whenever the main process exits. The solution involves some questionable stuff to only push the process to the background after the pid was returned.

It feels a bit gross to use it this way, but my brain is not big enough to make a good judgement on the sustainability of this workaround.

This old library achieves something similar. Dropping this here for reference.

[gwleuverink]

Hmm never mind my previous comment. It seems I have something working.

Dealing with some zombie processes though. Needs more work

[pkristian]

Hello there.
Recently I had chat with Simon and I expesed same desire: spin up process on background, and waiting for scheduller 1 to 59 seconds is not cool.

My use case:
Managment game. main process in frontend when all icons and clicking happens.
In background engine which is accesing SQLite database and doing proper ticking and whole magic.

What i need:
simpler approach:

• spin up engine on app start
• kill engine when app exist
• restart engine if crashes

more complex aproach:

• start engine on frontend action (continue/new game/load game/save game)
• stop engine on main menu
• when stopping - gracefully stop it, with time limit when had kill will be applied
• notify parent app when process exits (get exit code/exception)
• when spinning up process, make possible to pass params
• common interface would be nice - i imagine child proces could implemnt certrain class and inherit it.

api You guys proveded looks good, would love to see it happen <3

[simonhamp]

This is really useful context @pkristian, thank you 🙏

I think what we've proposed here would definitely work for you - in either approach. The main thing I think we'll need to figure out that we haven't already discussed here is how to spin up this secondary process when the app starts and make sure it gets killed when the app closes (your "simpler approach").

There could be a couple of ways to do that:

1. Bootup: Create the process in the NativeAppServiceProvider - this only gets loaded once at the beginning of the app lifecycle... or more idiomatic: listen for the ApplicationBooted event.
2. Shutdown: If Electron doesn't handle shutting down child processes during app shutdown, we may need a new event to fire when we become aware that the app will quit (e.g. ApplicationQuitting)

For your "more complex approach", I believe what we've proposed would already cover most of your needs. The only ones I'm not currently sure about are:

• notify parent app when process exits (get exit code/exception)

I know we'll have a ProcessExited event that will fire when your process exits, but I'm not sure how Electron will handle this on some exception within the child process... will need to test.

As for getting the exception/error that comes out, that will most likely come via STDOUT or STDERR, so you'd likely be able to listen for it using the onMessage hook.

• common interface would be nice - i imagine child proces could implemnt certrain class and inherit it.

I'm not super clear what you mean by this, could you elaborate?

[pkristian]

I'm not super clear what you mean by this, could you elaborate?

I only can express things i would love to have, but i am not able to implemnet it myself, so my vision could be somehow optimistic, but in practise not possible. i would share anyway :)

imagine this as child process

class Foo extends Fork
{
    //main loop
    public function main()
    {

        try {
            while (true) {
                //do something
                $this->updateStatus($someGreatNews);
            }
        } catch (\Exception $e) {
            // tell parent we have boo boo
            $this->reportException($e);
            exit;
        }
    }


    private function updateStatus(mixed $data)
    {
        $parentResponse = $this->notifyParent($data);

        if ($parentResponse->shutdownGracefully === true) {
            // do cleanup work
            exit;
        }
    }


    public function __destruct()
    {
        // tell parent i died
        $this->reportShutdown($exitCode);
    }

}

and from parent imagine something like:

$foo = new ForkHandler(Foo::class);
$foo->onUpdate(function ($data){
    //callback
});
$foo->onException(function ($e){
    //callback
});
$foo->onShutdown(function ($exitCode){
    //callback
});
$foo->run();//spin up child process

# wait, do something

$foo->shutdown( wait: 5);//wait 5 seconds, then kill forcefully

I think, or assume, what in future it would be nice to have it implemented in Laravel like fashion, so i would say, object approach to this issue could be neat

Well, assuming, we would only want to run other PHPs, for which this api could work somehow

[simonhamp]

For anyone who's following this thread, progress is being made on this in the following PRs:

• Child processes #389
• Child processes electron#115

Current status is - it's working 🎉 but there is more to do to make it robust enough for general consumption

[simonhamp]

This got released yesterday!

@pkristian let me know when if you want to go through this together