Executor may hung when using multiple devices(GPU)

[heliumsun]

I've encountered executor hung issue for several times. It randomly happens when I submit several CaffeOnSpark tasks and each of the task is using 2 devices(GPU).

Call stack:

2017-03-30 10:40:51
Full thread dump OpenJDK 64-Bit Server VM (25.65-b01 mixed mode):

"ForkJoinPool-1-worker-5" #128 daemon prio=5 os_prio=0 tid=0x0000120011960800 nid=0x1637b waiting on condition [0x00001000bbdfd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000fcba6838> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at com.yahoo.ml.caffe.ImageDataSource.nextBatch(ImageDataSource.scala:108)
at com.yahoo.ml.caffe.ImageDataSource.nextBatch(ImageDataSource.scala:27)
at com.yahoo.ml.caffe.CaffeProcessor.com$yahoo$ml$caffe$CaffeProcessor$$doTransform(CaffeProcessor.scala:334)
at com.yahoo.ml.caffe.CaffeProcessor$$anonfun$startThreads$1$$anonfun$apply$mcVI$sp$2$$anonfun$apply$mcZI$sp$1.apply$mcV$sp(CaffeProcessor.scala:151)
at com.yahoo.ml.caffe.CaffeProcessor$$anonfun$startThreads$1$$anonfun$apply$mcVI$sp$2$$anonfun$apply$mcZI$sp$1.apply(CaffeProcessor.scala:151)
at com.yahoo.ml.caffe.CaffeProcessor$$anonfun$startThreads$1$$anonfun$apply$mcVI$sp$2$$anonfun$apply$mcZI$sp$1.apply(CaffeProcessor.scala:151)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

"ForkJoinPool-1-worker-4" #127 daemon prio=5 os_prio=0 tid=0x000010013004a000 nid=0x1637a runnable [0x000010013920d000]
java.lang.Thread.State: RUNNABLE
at com.yahoo.ml.jcaffe.CaffeNet.train(Native Method)
at com.yahoo.ml.caffe.CaffeProcessor$$anonfun$com$yahoo$ml$caffe$CaffeProcessor$$doTrain$2.apply$mcVI$sp(CaffeProcessor.scala:447)
at com.yahoo.ml.caffe.CaffeProcessor$$anonfun$com$yahoo$ml$caffe$CaffeProcessor$$doTrain$2.apply(CaffeProcessor.scala:428)
at com.yahoo.ml.caffe.CaffeProcessor$$anonfun$com$yahoo$ml$caffe$CaffeProcessor$$doTrain$2.apply(CaffeProcessor.scala:428)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
at scala.collection.immutable.Range.foreach(Range.scala:141)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
at com.yahoo.ml.caffe.CaffeProcessor.com$yahoo$ml$caffe$CaffeProcessor$$doTrain(CaffeProcessor.scala:428)
at com.yahoo.ml.caffe.CaffeProcessor$$anonfun$startThreads$1$$anonfun$apply$mcVI$sp$1.apply$mcV$sp(CaffeProcessor.scala:145)
at com.yahoo.ml.caffe.CaffeProcessor$$anonfun$startThreads$1$$anonfun$apply$mcVI$sp$1.apply(CaffeProcessor.scala:145)
at com.yahoo.ml.caffe.CaffeProcessor$$anonfun$startThreads$1$$anonfun$apply$mcVI$sp$1.apply(CaffeProcessor.scala:145)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

"ForkJoinPool-1-worker-3" #126 daemon prio=5 os_prio=0 tid=0x0000120011960000 nid=0x16379 runnable [0x000010013940d000]
java.lang.Thread.State: RUNNABLE
at com.yahoo.ml.caffe.CaffeProcessor.takeFromQueue(CaffeProcessor.scala:232)
at com.yahoo.ml.caffe.CaffeProcessor.com$yahoo$ml$caffe$CaffeProcessor$$doTransform(CaffeProcessor.scala:336)
at com.yahoo.ml.caffe.CaffeProcessor$$anonfun$startThreads$1$$anonfun$apply$mcVI$sp$3$$anonfun$apply$mcZI$sp$2.apply$mcV$sp(CaffeProcessor.scala:159)
at com.yahoo.ml.caffe.CaffeProcessor$$anonfun$startThreads$1$$anonfun$apply$mcVI$sp$3$$anonfun$apply$mcZI$sp$2.apply(CaffeProcessor.scala:159)
at com.yahoo.ml.caffe.CaffeProcessor$$anonfun$startThreads$1$$anonfun$apply$mcVI$sp$3$$anonfun$apply$mcZI$sp$2.apply(CaffeProcessor.scala:159)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

"ForkJoinPool-1-worker-2" #125 daemon prio=5 os_prio=0 tid=0x000012001195f000 nid=0x16378 waiting on condition [0x0000100139a0d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000fcba6838> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at com.yahoo.ml.caffe.ImageDataSource.nextBatch(ImageDataSource.scala:108)
at com.yahoo.ml.caffe.ImageDataSource.nextBatch(ImageDataSource.scala:27)
at com.yahoo.ml.caffe.CaffeProcessor.com$yahoo$ml$caffe$CaffeProcessor$$doTransform(CaffeProcessor.scala:334)
at com.yahoo.ml.caffe.CaffeProcessor$$anonfun$startThreads$1$$anonfun$apply$mcVI$sp$2$$anonfun$apply$mcZI$sp$1.apply$mcV$sp(CaffeProcessor.scala:151)
at com.yahoo.ml.caffe.CaffeProcessor$$anonfun$startThreads$1$$anonfun$apply$mcVI$sp$2$$anonfun$apply$mcZI$sp$1.apply(CaffeProcessor.scala:151)
at com.yahoo.ml.caffe.CaffeProcessor$$anonfun$startThreads$1$$anonfun$apply$mcVI$sp$2$$anonfun$apply$mcZI$sp$1.apply(CaffeProcessor.scala:151)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

"ForkJoinPool-1-worker-1" #124 daemon prio=5 os_prio=0 tid=0x000012001195e800 nid=0x16377 waiting on condition [0x00001000bb9fd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000fcb2e398> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at com.yahoo.ml.caffe.CaffeProcessor$$anonfun$com$yahoo$ml$caffe$CaffeProcessor$$doTrain$2.apply$mcVI$sp(CaffeProcessor.scala:442)
at com.yahoo.ml.caffe.CaffeProcessor$$anonfun$com$yahoo$ml$caffe$CaffeProcessor$$doTrain$2.apply(CaffeProcessor.scala:428)
at com.yahoo.ml.caffe.CaffeProcessor$$anonfun$com$yahoo$ml$caffe$CaffeProcessor$$doTrain$2.apply(CaffeProcessor.scala:428)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
at scala.collection.immutable.Range.foreach(Range.scala:141)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
at com.yahoo.ml.caffe.CaffeProcessor.com$yahoo$ml$caffe$CaffeProcessor$$doTrain(CaffeProcessor.scala:428)
at com.yahoo.ml.caffe.CaffeProcessor$$anonfun$startThreads$1$$anonfun$apply$mcVI$sp$1.apply$mcV$sp(CaffeProcessor.scala:145)
at com.yahoo.ml.caffe.CaffeProcessor$$anonfun$startThreads$1$$anonfun$apply$mcVI$sp$1.apply(CaffeProcessor.scala:145)
at com.yahoo.ml.caffe.CaffeProcessor$$anonfun$startThreads$1$$anonfun$apply$mcVI$sp$1.apply(CaffeProcessor.scala:145)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

"shuffle-client-0" #114 daemon prio=5 os_prio=0 tid=0x00001001a8095000 nid=0x160ed runnable [0x000010018f61d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000c2a6eee0> (a io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000000c2a6ef00> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000c2a6ee98> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)

"Executor task launch worker-0" #118 daemon prio=5 os_prio=0 tid=0x000010011c07e000 nid=0x160e0 runnable [0x000010018f01c000]
java.lang.Thread.State: RUNNABLE
at com.yahoo.ml.caffe.CaffeProcessor.feedQueue(CaffeProcessor.scala:195)
at com.yahoo.ml.caffe.CaffeOnSpark$$anonfun$10$$anonfun$11.apply(CaffeOnSpark.scala:330)
at com.yahoo.ml.caffe.CaffeOnSpark$$anonfun$10$$anonfun$11.apply(CaffeOnSpark.scala:330)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.reduceLeft(TraversableOnce.scala:172)
at scala.collection.AbstractIterator.reduceLeft(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.reduce(TraversableOnce.scala:195)
at scala.collection.AbstractIterator.reduce(Iterator.scala:1157)
at com.yahoo.ml.caffe.CaffeOnSpark$$anonfun$10.apply(CaffeOnSpark.scala:330)
at com.yahoo.ml.caffe.CaffeOnSpark$$anonfun$10.apply(CaffeOnSpark.scala:325)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$21.apply(RDD.scala:730)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$21.apply(RDD.scala:730)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:309)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:273)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:239)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"driver-heartbeater" #117 daemon prio=5 os_prio=0 tid=0x000010011c07b000 nid=0x160d8 waiting on condition [0x000010018ee1d000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c2a76fd8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"shuffle-client-0" #116 daemon prio=5 os_prio=0 tid=0x000010011c075800 nid=0x160d2 runnable [0x000010018dc0d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000c2a772c8> (a io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000000c2a772e8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000c2a77280> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)

"shuffle-server-0" #115 daemon prio=5 os_prio=0 tid=0x000010011c051800 nid=0x160d1 runnable [0x000010018da0d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000c2aad3e8> (a io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000000c2aad408> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000c2aad3a0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)

"metrics-logger-reporter-1-thread-1" #113 daemon prio=5 os_prio=0 tid=0x0000100004c83000 nid=0x160cf waiting on condition [0x000010018d80d000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c2abcb50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"BROADCAST_VARS cleanup timer" #112 daemon prio=5 os_prio=0 tid=0x0000100004bac000 nid=0x160ce in Object.wait() [0x000010018d40d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c2a85cb0> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x00000000c2a85cb0> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"BLOCK_MANAGER cleanup timer" #111 daemon prio=5 os_prio=0 tid=0x0000100004baa800 nid=0x160cd in Object.wait() [0x000010018d20d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c2abcdb8> (a java.util.TaskQueue)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
- locked <0x00000000c2abcdb8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"shuffle-client-0" #87 daemon prio=5 os_prio=0 tid=0x0000100180001000 nid=0x160cc runnable [0x000010013be0d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000c1281418> (a io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x00000000c1283498> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000c1281380> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)

"netty-rpc-env-timeout" #109 daemon prio=5 os_prio=0 tid=0x0000100004b52800 nid=0x160cb waiting on condition [0x000010013bc0d000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c12a8118> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O server boss #6" #107 daemon prio=5 os_prio=0 tid=0x0000100160123800 nid=0x160c9 runnable [0x000010013b60d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000c2a99c18> (a sun.nio.ch.Util$2)
- locked <0x00000000c2a99c28> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000c2a99bd0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O worker #5" #106 daemon prio=5 os_prio=0 tid=0x000010016003c800 nid=0x160c8 runnable [0x000010013b40d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000c2a6f198> (a sun.nio.ch.Util$2)
- locked <0x00000000c2a6f1a8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000c2a6f150> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O worker #4" #105 daemon prio=5 os_prio=0 tid=0x00001001600a2800 nid=0x160c7 runnable [0x000010013b20d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000c2a86180> (a sun.nio.ch.Util$2)
- locked <0x00000000c2a86190> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000c2a86138> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O boss #3" #104 daemon prio=5 os_prio=0 tid=0x000010016003d800 nid=0x160c6 runnable [0x000010013b00d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000c2abcfd8> (a sun.nio.ch.Util$2)
- locked <0x00000000c2abcfe8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000c2abcf90> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O worker #2" #102 daemon prio=5 os_prio=0 tid=0x0000100160032000 nid=0x160c5 runnable [0x000010013ae0d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000c2ac7d28> (a sun.nio.ch.Util$2)
- locked <0x00000000c2ac7d38> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000c2ac7ce0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"New I/O worker #1" #101 daemon prio=5 os_prio=0 tid=0x0000100160030800 nid=0x160c4 runnable [0x000010013ac0d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000c2aad6a0> (a sun.nio.ch.Util$2)
- locked <0x00000000c2aad6b0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000c2aad658> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"sparkExecutorActorSystem-akka.remote.default-remote-dispatcher-6" #100 daemon prio=5 os_prio=0 tid=0x0000100004b35800 nid=0x160c3 waiting on condition [0x000010013aa0d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c2aef8c0> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"sparkExecutorActorSystem-akka.remote.default-remote-dispatcher-5" #99 daemon prio=5 os_prio=0 tid=0x000010015800d000 nid=0x160c2 waiting on condition [0x000010013a80d000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c2aef8c0> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"sparkExecutorActorSystem-akka.actor.default-dispatcher-4" #98 daemon prio=5 os_prio=0 tid=0x0000100158001000 nid=0x160c1 waiting on condition [0x000010013a40d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c3b1be20> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"sparkExecutorActorSystem-scheduler-1" #95 daemon prio=5 os_prio=0 tid=0x0000100004a83000 nid=0x160be waiting on condition [0x0000100139e0d000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-31" #86 daemon prio=5 os_prio=0 tid=0x000010000491c800 nid=0x160bd waiting on condition [0x00001000b8dfd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-30" #85 daemon prio=5 os_prio=0 tid=0x000010000491b800 nid=0x160bc waiting on condition [0x00001000b8bfd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-29" #84 daemon prio=5 os_prio=0 tid=0x000010000491a800 nid=0x160bb waiting on condition [0x00001000b89fd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-28" #83 daemon prio=5 os_prio=0 tid=0x0000100004919000 nid=0x160ba waiting on condition [0x00001000b87fd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-27" #82 daemon prio=5 os_prio=0 tid=0x0000100004918000 nid=0x160b9 waiting on condition [0x00001000b85fd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-26" #81 daemon prio=5 os_prio=0 tid=0x0000100004916800 nid=0x160b8 waiting on condition [0x00001000b83fd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-25" #80 daemon prio=5 os_prio=0 tid=0x0000100004910800 nid=0x160b7 waiting on condition [0x00001000b81fd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-24" #79 daemon prio=5 os_prio=0 tid=0x000010000490f800 nid=0x160b6 waiting on condition [0x0000100077fbd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-23" #78 daemon prio=5 os_prio=0 tid=0x000010000474f000 nid=0x160b5 waiting on condition [0x0000100077dbd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-22" #77 daemon prio=5 os_prio=0 tid=0x0000100004760800 nid=0x160b4 waiting on condition [0x0000100077bbd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-21" #76 daemon prio=5 os_prio=0 tid=0x0000100004913800 nid=0x160b3 waiting on condition [0x00001000779bd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-20" #75 daemon prio=5 os_prio=0 tid=0x000010000475d000 nid=0x160b2 waiting on condition [0x0000100076f6d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-19" #74 daemon prio=5 os_prio=0 tid=0x000010000475e800 nid=0x160b1 waiting on condition [0x00001000b8ffd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-18" #73 daemon prio=5 os_prio=0 tid=0x000010000475b000 nid=0x160b0 waiting on condition [0x00001000b91fd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-17" #72 daemon prio=5 os_prio=0 tid=0x0000100004759000 nid=0x160af waiting on condition [0x00001000b93fd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-16" #71 daemon prio=5 os_prio=0 tid=0x0000100004766000 nid=0x160ae waiting on condition [0x00001000b95fd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-15" #70 daemon prio=5 os_prio=0 tid=0x0000100004762000 nid=0x160ad waiting on condition [0x00001000b97fd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-14" #69 daemon prio=5 os_prio=0 tid=0x0000100004764000 nid=0x160ac waiting on condition [0x00001000b99fd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-13" #68 daemon prio=5 os_prio=0 tid=0x0000100004751000 nid=0x160ab waiting on condition [0x00001000b9bfd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-12" #67 daemon prio=5 os_prio=0 tid=0x000010000478d000 nid=0x160aa waiting on condition [0x00001000b9dfd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-11" #66 daemon prio=5 os_prio=0 tid=0x0000100004785000 nid=0x160a9 waiting on condition [0x00001000b9ffd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-10" #65 daemon prio=5 os_prio=0 tid=0x000010000478b000 nid=0x160a8 waiting on condition [0x00001000ba1fd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-9" #64 daemon prio=5 os_prio=0 tid=0x000010000476a800 nid=0x160a7 waiting on condition [0x00001000ba3fd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-8" #63 daemon prio=5 os_prio=0 tid=0x000010000476c800 nid=0x160a6 waiting on condition [0x00001000ba5fd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-7" #62 daemon prio=5 os_prio=0 tid=0x0000100004770800 nid=0x160a5 waiting on condition [0x00001000ba7fd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-6" #61 daemon prio=5 os_prio=0 tid=0x0000100004772800 nid=0x160a4 waiting on condition [0x00001000ba9fd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-5" #60 daemon prio=5 os_prio=0 tid=0x0000100004774800 nid=0x160a3 waiting on condition [0x00001000babfd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-4" #59 daemon prio=5 os_prio=0 tid=0x0000100004778800 nid=0x160a2 waiting on condition [0x00001000badfd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-3" #58 daemon prio=5 os_prio=0 tid=0x000010000477a800 nid=0x160a1 waiting on condition [0x00001000baffd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-2" #57 daemon prio=5 os_prio=0 tid=0x000010000477f000 nid=0x160a0 waiting on condition [0x00001000bb1fd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-1" #56 daemon prio=5 os_prio=0 tid=0x0000100004781000 nid=0x1609f waiting on condition [0x00001000bb3fd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"dispatcher-event-loop-0" #55 daemon prio=5 os_prio=0 tid=0x0000100004783000 nid=0x1609e waiting on condition [0x00001000bb5fd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00110f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"threadDeathWatcher-2-1" #54 daemon prio=1 os_prio=0 tid=0x0000100130011800 nid=0x16096 waiting on condition [0x00001000bbffd000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:137)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)

"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x00001000040fc000 nid=0x1602e runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00001000040f1800 nid=0x16028 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00001000040e4000 nid=0x16027 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00001000040e2000 nid=0x16025 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00001000040b4800 nid=0x16022 in Object.wait() [0x0000100074d6d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c10f2328> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000000c10f2328> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00001000040b2800 nid=0x16021 in Object.wait() [0x0000100074b6d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c10f2540> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
- locked <0x00000000c10f2540> (a java.lang.ref.Reference$Lock)

"main" #1 prio=5 os_prio=0 tid=0x0000100004009800 nid=0x16008 waiting on condition [0x000010000135c000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c12a8e40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1465)
at org.apache.spark.rpc.netty.Dispatcher.awaitTermination(Dispatcher.scala:182)
at org.apache.spark.rpc.netty.NettyRpcEnv.awaitTermination(NettyRpcEnv.scala:275)
at org.apache.spark.executor.EGOExecutorBackend$$anonfun$run$1.apply$mcV$sp(EGOExecutorBackend.scala:140)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:69)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:68)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:68)
at org.apache.spark.executor.EGOExecutorBackend$.run(EGOExecutorBackend.scala:91)
at org.apache.spark.executor.EGOExecutorBackend$.main(EGOExecutorBackend.scala:198)
at org.apache.spark.executor.EGOExecutorBackend.main(EGOExecutorBackend.scala)

"VM Thread" os_prio=0 tid=0x00001000040ad000 nid=0x16020 runnable

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x000010000401e800 nid=0x16009 runnable

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x0000100004020000 nid=0x1600a runnable

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x0000100004022000 nid=0x1600b runnable

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x0000100004024000 nid=0x1600c runnable

"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x0000100004025800 nid=0x1600d runnable

"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x0000100004027800 nid=0x1600e runnable

"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x0000100004029800 nid=0x1600f runnable

"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x000010000402b000 nid=0x16010 runnable

"GC task thread#8 (ParallelGC)" os_prio=0 tid=0x000010000402d000 nid=0x16011 runnable

"GC task thread#9 (ParallelGC)" os_prio=0 tid=0x000010000402e800 nid=0x16012 runnable

"GC task thread#10 (ParallelGC)" os_prio=0 tid=0x0000100004030800 nid=0x16013 runnable

"GC task thread#11 (ParallelGC)" os_prio=0 tid=0x0000100004032800 nid=0x16014 runnable

"GC task thread#12 (ParallelGC)" os_prio=0 tid=0x0000100004034000 nid=0x16015 runnable

"GC task thread#13 (ParallelGC)" os_prio=0 tid=0x0000100004036000 nid=0x16016 runnable

"GC task thread#14 (ParallelGC)" os_prio=0 tid=0x0000100004038000 nid=0x16017 runnable

"GC task thread#15 (ParallelGC)" os_prio=0 tid=0x0000100004039800 nid=0x16018 runnable

"GC task thread#16 (ParallelGC)" os_prio=0 tid=0x000010000403b800 nid=0x16019 runnable

"GC task thread#17 (ParallelGC)" os_prio=0 tid=0x000010000403d000 nid=0x1601a runnable

"GC task thread#18 (ParallelGC)" os_prio=0 tid=0x000010000403f000 nid=0x1601b runnable

"GC task thread#19 (ParallelGC)" os_prio=0 tid=0x0000100004041000 nid=0x1601c runnable

"GC task thread#20 (ParallelGC)" os_prio=0 tid=0x0000100004042800 nid=0x1601d runnable

"GC task thread#21 (ParallelGC)" os_prio=0 tid=0x0000100004044800 nid=0x1601e runnable

"GC task thread#22 (ParallelGC)" os_prio=0 tid=0x0000100004046800 nid=0x1601f runnable

"VM Periodic Task Thread" os_prio=0 tid=0x0000100004111000 nid=0x16030 waiting on condition

JNI global references: 441

Heap
PSYoungGen total 320512K, used 109661K [0x00000000eab00000, 0x0000000100000000, 0x0000000100000000)
eden space 294400K, 35% used [0x00000000eab00000,0x00000000f1077a90,0x00000000fca80000)
from space 26112K, 22% used [0x00000000fca80000,0x00000000fd01fd48,0x00000000fe400000)
to space 24576K, 0% used [0x00000000fe800000,0x00000000fe800000,0x0000000100000000)
ParOldGen total 699392K, used 86025K [0x00000000c0000000, 0x00000000eab00000, 0x00000000eab00000)
object space 699392K, 12% used [0x00000000c0000000,0x00000000c54027d8,0x00000000eab00000)
Metaspace used 42265K, capacity 42928K, committed 43136K, reserved 1085440K
class space used 5945K, capacity 6154K, committed 6272K, reserved 1048576K

[junshi15]

Try to disable validation by setting the following in your solver.prototxt file:

test_iter: 0
test_interval: 0