[JENKINS-70531] Apply timeout on WebSocket write operations (and simplify AbstractByteBufferCommandTransport)

[jglick]

A user reported a WebSocket agent hanging indefinitely after a reload of nginx configuration

"pool-1-thread-… for … id=… / waiting for … id=…" #… daemon prio=5 os_prio=0 cpu=89.18ms elapsed=300.35s tid=… nid=… waiting on condition  […]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@11.0.17/Native Method)
        - parking to wait for  <…> (a java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.park(java.base@11.0.17/LockSupport.java:194)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.17/AbstractQueuedSynchronizer.java:885)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.17/AbstractQueuedSynchronizer.java:1039)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.17/AbstractQueuedSynchronizer.java:1345)
        at java.util.concurrent.CountDownLatch.await(java.base@11.0.17/CountDownLatch.java:232)
        at io.jenkins.remoting.shaded.org.glassfish.tyrus.core.TyrusFuture.get(TyrusFuture.java:53)
        at io.jenkins.remoting.shaded.org.glassfish.tyrus.core.TyrusRemoteEndpoint$Basic.processFuture(TyrusRemoteEndpoint.java:149)
        at io.jenkins.remoting.shaded.org.glassfish.tyrus.core.TyrusRemoteEndpoint$Basic.sendBinary(TyrusRemoteEndpoint.java:131)
        at hudson.remoting.Engine$1AgentEndpoint$Transport.write(Engine.java:652)
        at hudson.remoting.AbstractByteBufferCommandTransport.write(AbstractByteBufferCommandTransport.java:303)
        at hudson.remoting.Channel.send(Channel.java:765)
        - locked <…> (a hudson.remoting.Channel)
        at hudson.remoting.Request.call(Request.java:167)
        - locked <…> (a hudson.remoting.RemoteInvocationHandler$RPCRequest)
        - locked <…> (a hudson.remoting.Channel)
        at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:288)

despite PingThread being explicitly enabled on the agent side (contra #85, pending jenkinsci/jenkins#7580). Hypothesis: Tyrus is failing to detect the loss of the connection cleanly, and Future.get() without timeout never returns, and PingThread does not receive a TimeoutException or any other response. If true, we can try to apply a timeout (currently hard-coded to 5m), though this is a bit tricky since the asynch variant of the endpoint does not appear to support transmission of a sequence of buffers as part of a single binary frame.

Corresponding jenkinsci/jenkins#7596 as well, though in this case it is the agent-side write that appears to be the culprit.

Iterative testing via WebSocketAgentsTest:

mvnd -Pquick-build install
mvnd -f ../jenkins -pl core,war -Pquick-build install
mvnd -f ../jenkins -pl test -Dtest=WebSocketAgentsTest

Not tested in any realistic context.

[jglick]

despite PingThread being explicitly enabled on the agent side

Actually in this case I cannot see PingThread running, yet it may be irrelevant. The thread dump also shows

"pool-1-thread-…" #… daemon prio=5 os_prio=0 cpu=11261.88ms elapsed=697254.89s tid=… nid=… waiting for monitor entry  […]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at hudson.remoting.Channel.terminate(Channel.java:1068)
        - waiting to lock <…> (a hudson.remoting.Channel)
        at hudson.remoting.Channel$1.terminate(Channel.java:620)
        at hudson.remoting.AbstractByteBufferCommandTransport.terminate(AbstractByteBufferCommandTransport.java:314)
        at hudson.remoting.Engine$1AgentEndpoint.lambda$onClose$1(Engine.java:632)
        at hudson.remoting.Engine$1AgentEndpoint$$Lambda$171/….run(Unknown Source)
        at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)

where the agent did receive a WebSocket close event but failed to close in response. (For over a week, apparently!) #595 might have fixed this as well; unclear.

[jglick]

Seeing some flakes, but I get some in trunk too, so I am not sure they are related.

[jglick]

💡 but we retry those since #142 and I think the actual problem was that prior to 6f43114 one branch actually failed, then other branches got confused about test reporting.

[jglick]

Just realized that https://docs.oracle.com/en/java/javase/11/docs/api/java.net.http/java/net/http/WebSocket.html could potentially be used to implement the WebSocket client without needing the Tyrus dep. Worth experimenting with.

[amuniz]

FYI this patch has probed to be effective in a production controller.

[jglick]

Specifically 2d296a8; I do not think f2ead41 (+ 6f43114) was tested in the field.

[jglick]

Filing this as https://issues.jenkins.io/browse/JENKINS-70531.

[olamy]

not expert on the field of remoting but changes looks straightforward

[Vlatombe]

LGTM

[jglick]

Downstream passed; releasing.