Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jetty patch to avoid hanging threads #5

Conversation

xiaochen-db
Copy link

@xiaochen-db xiaochen-db commented Apr 27, 2023

The investigation of https://databricks.atlassian.net/browse/PROD-35631 suggested a jetty websocket's sendString() can lead to the calling thread hanging on blockingWrite (See jetty#2061 for the OSS discussion).

This PR implemented the workaround that's suggested by the OSS community (jetty#2061 (comment)) to avoid hanging threads like

"WebappServiceThreadPool-197" #197 prio=5 os_prio=0 cpu=40130.40ms elapsed=88215.54s tid=0x00007fcc2f4f1800 nid=0xe5 waiting on condition  [0x00007fca389c9000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native Method)
	- parking to wait for  <0x00000003a16421b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.7/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.7/AbstractQueuedSynchronizer.java:2081)
	at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:203)
	at org.eclipse.jetty.websocket.common.BlockingWriteCallback$WriteBlocker.block(BlockingWriteCallback.java:82)
	at org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.blockingWrite(WebSocketRemoteEndpoint.java:107)
	at org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendString(WebSocketRemoteEndpoint.java:393)
	at com.databricks.webapp.comm.DriverWebSocket.$anonfun$sendDNCMessage$4(DriverConnection.scala:618)
	at com.databricks.webapp.comm.DriverWebSocket.$anonfun$sendDNCMessage$4$adapted(DriverConnection.scala:618)
	at com.databricks.webapp.comm.DriverWebSocket$$Lambda$7464/0x0000000842ba3040.apply(Unknown Source)
	at scala.Option.foreach(Option.scala:407)
	at com.databricks.webapp.comm.DriverWebSocket.sendDNCMessage(DriverConnection.scala:618)
	at com.databricks.webapp.comm.DriverConnection.send(DriverConnection.scala:295)
	- locked <0x00000003d1fbdc38> (a com.databricks.webapp.comm.DriverConnection)
	at com.databricks.webapp.comm.DirectNotebookChannelManagerImpl.$anonfun$handleMsgToDriver$1(DirectNotebookChannelManager.scala:403)
	at com.databricks.webapp.comm.DirectNotebookChannelManagerImpl$$Lambda$6641/0x000000084285fc40.apply$mcV$sp(Unknown Source)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at com.databricks.logging.UsageLogging.$anonfun$recordOperation$1(UsageLogging.scala:562)
	at com.databricks.logging.UsageLogging$$Lambda$627/0x0000000840bd7c40.apply(Unknown Source)

Please note that branch origin/jetty-patch-to-avoid-hanging-threads is directly checked out from tag jetty-9.3.30.v20211001 (used by webapp):

* 148ea77050 - (HEAD -> jetty-patch-to-avoid-hanging-threads, tag: jetty-9.3.30.v20211001, origin/jetty-patch-to-avoid-hanging-threads) Updating to version 9.3.30.v20211001 (1 year, 7 months ago) <Joakim Erdfelt>

We plan to publish this change to origin/jetty-patch-to-avoid-hanging-threads as a new version, and then apply it to webapp repo.

@xiaochen-db xiaochen-db changed the title getIdleTimeout returns 60 sec Jetty patch to avoid hanging threads Apr 27, 2023
@xiaochen-db xiaochen-db requested a review from chadward-db April 27, 2023 20:34
@xiaochen-db xiaochen-db merged commit 6ac61f5 into databricks:jetty-patch-to-avoid-hanging-threads May 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants