STOMP: "Was not possible to write frame!" causing zombie connections

[atrauzzi]

I'm able to somewhat reliably reproduce a situation where my queue worker in Laravel ends up no longer able to successfully connect to a broker. It takes a few consecutive jobs before it happens, but eventually my one queue gets the exception "Was not possible to write frame!".

Thereafter, any attempt to send messages to the same broker end up reporting the same exception. It seems like the connection becomes corrupted here and no recovery is attempted by enqueue or the stomp-php library itself.

Is there any general advice on how to handle this or is this something that enqueue can catch and recreate its connection?

[atrauzzi]

@makasim - Can totally do a screen share with you on this one to get a better understanding if necessary. If the fix is simple enough, I'm always happy to contribute more. 😄

[makasim]

I suspect that the server closes connection due to lack of activity. Do the jobs arrive one after another or is there any pause between them ?

[makasim]

There is a message to keep the connection alive https://github.com/stomp-php/stomp-php/blob/master/src/Network/Connection.php#L681

[atrauzzi]

I believe the keep alive / heartbeating is working as I'm able to sustain a connection for prolonged periods of time. Even overnight! 😆

This situation seems to be intermittent. Almost like if I trigger a message, wait a bit and then trigger another, I get the error. Around 8-10 seconds.

For fun, I tried this in StompProducer and enqueue is able to "recover":

        try {
            $this->stomp->send($destination->getQueueName(), $stompMessage);
        }
        catch (ConnectionException $ex) {
        
            dump("IT BORKED!!!!");
        
            $this->stomp->disconnect(true);
            $this->stomp->connect();
        
            $this->stomp->send($destination->getQueueName(), $stompMessage);
        }

[makasim]

You can add a consumption extension that calls keepAlive method. Also, set lower receive timeout than connection timeout so the socket gives control back to PHP and consumption extensions are executed. That would do the trick.

The solution does not solve all the possible cases. If you have a long-running job you might still run into the issue. More on it here

[atrauzzi]

That's for AMQP, does that still apply to STOMP? Is there any way to apply an extension from Laravel? I'm only seeing Symfony instructions in the docs.

Also, it's probably worth noting that this is for dispatching, not for the queue consumer itself. So it's almost like keeping a connection open that's only used for dispatch is the scenario here...

Additionally, I'm not sure I understand how your suggestion can cover what I'm encountering. It seems more like there's an issue with connections for sending messages.