Connection refused errors when transforming BAM file with BQSR #516
Comments
|
Are you running ADAM 0.15.0? |
|
No, I am running ADAM 0.14.0. I wasn't aware of the 0.15.0 release. I can try using this version, perhaps this issue may have been resolved in the latest release. |
|
Let us know what you find either way. If I were to guess, the issues you're seeing with disconnecting services has to do with them being overwhelmed by GC. You should find that 0.15.0 uses much less memory. https://github.com/bigdatagenomics/adam/releases/tag/adam-parent-0.15.0 |
|
Following up on this issue, I still get the same connection refused errors even with the 0.15.0 release. I've attached a screen shot of the Spark Web UI right before it crashes.
|
|
From the errors shown above, it should be a problem of your spark cluster setting. Have you tried a wordcount test on a large file on your spark cluster? If it failed, it may not be related to adam. |
|
Would it be ok to close this issue as unable to reproduce? |
|
Yes. |
I am trying to transform a NA12878 BAM file (~300gb) with base quality score recalibration. I am running this on standalone mode (through a cluster with 1 master and 4 workers). After ~30min into this process, I get "java.net.ConnectException: Connection refused" errors.
It is important to note that I can successfully transform the BAM file to ADAM format as long as I don't turn on the recalibration parameter. I am also able to transform a shortened version of a NA12878 SAM (~250kb) file with base quality score recalibration.
Any ideas on why this error persists?
I've provided my spark cluster configurations and most of the stack trace message below.
------- cluster specs -------
1 master (174 g of memory)
4 workers (each with 240g of memory, 31 cores)
------- ~/spark-1.1.0-bin-hadoop2.3/conf/spark-env.sh -------
export SPARK_DAEMON_MEMORY=100g
export SPARK_WORKER_CORES=30
export SPARK_WORKER_MEMORY=220g
------ stack trace -----
-bash-4.1$ adam transform /pod/pstore/projects/BigDataGenomics/NA12878.hiseq.wgs.bwa.raw.bam /pod/pstore/projects/BigDataGenomics/NA12878.recal.adam -known_snps /pod/pstore/projects/BigDataGenomics/All.suspectedremoved.vcf -recalibrate_base_qualities
Spark assembly has been built with Hive, including Datanucleus jars on classpath
2014-12-01 11:30:44 WARN SparkConf:71 - In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN).
2014-12-01 11:30:46 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-12-01 11:46:45 WARN BlockManagerMasterActor:71 - Removing BlockManager BlockManagerId(2, podk-1-2.local, 41993, 0) with no recent heart beats: 60946ms exceeds 45000ms
2014-12-01 11:46:47 ERROR ConnectionManager:75 - Corresponding SendingConnection to ConnectionManagerId(podk-1-1.local,48288) not found
2014-12-01 11:46:47 ERROR ConnectionManager:75 - Corresponding SendingConnection to ConnectionManagerId(podk-1-2.local,41993) not found
2014-12-01 11:46:47 ERROR ConnectionManager:75 - Corresponding SendingConnection to ConnectionManagerId(podk-1-3.local,50288) not found
2014-12-01 11:46:48 ERROR ConnectionManager:75 - Corresponding SendingConnection to ConnectionManagerId(podk-1-4.local,49200) not found
2014-12-01 11:46:48 ERROR ConnectionManager:75 - Corresponding SendingConnection to ConnectionManagerId(podk-1-4.local,49200) not found
2014-12-01 11:46:49 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:46:57 WARN TaskSetManager:71 - Lost task 13.0 in stage 2.0 (TID 15, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:04 WARN SendingConnection:92 - Error finishing connection to podk-1-4.local/10.50.1.104:49200
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 108.0 in stage 2.0 (TID 110, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 118.0 in stage 2.0 (TID 120, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-1.local/10.50.1.101:48288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 78.0 in stage 2.0 (TID 80, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 90.0 in stage 2.0 (TID 92, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-4.local/10.50.1.104:49200
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 71.0 in stage 2.0 (TID 73, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-1.local/10.50.1.101:48288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 60.0 in stage 2.0 (TID 62, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 27.0 in stage 2.0 (TID 29, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 41.0 in stage 2.0 (TID 43, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-4.local/10.50.1.104:49200
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 96.0 in stage 2.0 (TID 98, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-1.local/10.50.1.101:48288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-4.local/10.50.1.104:49200
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 107.0 in stage 2.0 (TID 109, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 36.0 in stage 2.0 (TID 38, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 45.0 in stage 2.0 (TID 47, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 61.0 in stage 2.0 (TID 63, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 65.0 in stage 2.0 (TID 67, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-1.local/10.50.1.101:48288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 103.0 in stage 2.0 (TID 105, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-1.local/10.50.1.101:48288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 67.0 in stage 2.0 (TID 69, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-4.local/10.50.1.104:49200
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 88.0 in stage 2.0 (TID 90, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-1.local/10.50.1.101:48288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 77.0 in stage 2.0 (TID 79, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 119.0 in stage 2.0 (TID 121, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-1.local/10.50.1.101:48288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 99.0 in stage 2.0 (TID 101, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-1.local/10.50.1.101:48288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-4.local/10.50.1.104:49200
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 39.0 in stage 2.0 (TID 41, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 79.0 in stage 2.0 (TID 81, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 80.0 in stage 2.0 (TID 82, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 42.0 in stage 2.0 (TID 44, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-1.local/10.50.1.101:48288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 58.0 in stage 2.0 (TID 60, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 115.0 in stage 2.0 (TID 117, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-4.local/10.50.1.104:49200
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 56.0 in stage 2.0 (TID 58, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 112.0 in stage 2.0 (TID 114, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-4.local/10.50.1.104:49200
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 72.0 in stage 2.0 (TID 74, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-1.local/10.50.1.101:48288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 46.0 in stage 2.0 (TID 48, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 43.0 in stage 2.0 (TID 45, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-4.local/10.50.1.104:49200
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 84.0 in stage 2.0 (TID 86, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 68.0 in stage 2.0 (TID 70, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-4.local/10.50.1.104:49200
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-1.local/10.50.1.101:48288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 28.0 in stage 2.0 (TID 30, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 55.0 in stage 2.0 (TID 57, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 75.0 in stage 2.0 (TID 77, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-1.local/10.50.1.101:48288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-4.local/10.50.1.104:49200
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 111.0 in stage 2.0 (TID 113, podk-1-1.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 116.0 in stage 2.0 (TID 118, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 105.0 in stage 2.0 (TID 107, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 86.0 in stage 2.0 (TID 88, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 93.0 in stage 2.0 (TID 95, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-4.local/10.50.1.104:49200
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 26.0 in stage 2.0 (TID 28, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 104.0 in stage 2.0 (TID 106, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 89.0 in stage 2.0 (TID 91, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-4.local/10.50.1.104:49200
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 100.0 in stage 2.0 (TID 102, podk-1-4.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 66.0 in stage 2.0 (TID 68, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 57.0 in stage 2.0 (TID 59, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 82.0 in stage 2.0 (TID 84, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 30.0 in stage 2.0 (TID 32, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 70.0 in stage 2.0 (TID 72, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 69.0 in stage 2.0 (TID 71, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 38.0 in stage 2.0 (TID 40, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 74.0 in stage 2.0 (TID 76, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 110.0 in stage 2.0 (TID 112, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 114.0 in stage 2.0 (TID 116, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 54.0 in stage 2.0 (TID 56, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 50.0 in stage 2.0 (TID 52, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 102.0 in stage 2.0 (TID 104, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 98.0 in stage 2.0 (TID 100, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 34.0 in stage 2.0 (TID 36, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 106.0 in stage 2.0 (TID 108, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 33.0 in stage 2.0 (TID 35, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 94.0 in stage 2.0 (TID 96, podk-1-2.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 37.0 in stage 2.0 (TID 39, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 109.0 in stage 2.0 (TID 111, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 9.0 in stage 2.0 (TID 11, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 101.0 in stage 2.0 (TID 103, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 85.0 in stage 2.0 (TID 87, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 73.0 in stage 2.0 (TID 75, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 81.0 in stage 2.0 (TID 83, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 97.0 in stage 2.0 (TID 99, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 117.0 in stage 2.0 (TID 119, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 113.0 in stage 2.0 (TID 115, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 29.0 in stage 2.0 (TID 31, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 1.0 in stage 2.0 (TID 3, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN SendingConnection:92 - Error finishing connection to podk-1-3.local/10.50.1.103:50288
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.spark.network.SendingConnection.finishConnect(Connection.scala:313)
at org.apache.spark.network.ConnectionManager$$anon$8.run(ConnectionManager.scala:226)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 49.0 in stage 2.0 (TID 51, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 53.0 in stage 2.0 (TID 55, podk-1-3.local): TaskResultLost (result lost from block manager)
2014-12-01 11:47:12 ERROR TaskSchedulerImpl:75 - Lost executor 1 on podk-1-1.local: remote Akka client disassociated
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 135.0 in stage 2.0 (TID 137, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 39.1 in stage 2.0 (TID 200, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 153.0 in stage 2.0 (TID 155, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 126.0 in stage 2.0 (TID 128, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 56.1 in stage 2.0 (TID 194, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 77.1 in stage 2.0 (TID 185, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 138.0 in stage 2.0 (TID 140, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 147.0 in stage 2.0 (TID 149, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 132.0 in stage 2.0 (TID 134, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 119.1 in stage 2.0 (TID 184, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 149.0 in stage 2.0 (TID 151, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 158.0 in stage 2.0 (TID 160, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 161.0 in stage 2.0 (TID 163, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 125.0 in stage 2.0 (TID 127, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 84.1 in stage 2.0 (TID 199, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 134.0 in stage 2.0 (TID 136, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 137.0 in stage 2.0 (TID 139, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 42.1 in stage 2.0 (TID 189, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 103.1 in stage 2.0 (TID 180, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 96.1 in stage 2.0 (TID 174, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 88.1 in stage 2.0 (TID 183, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 75.1 in stage 2.0 (TID 201, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 99.1 in stage 2.0 (TID 186, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 166.0 in stage 2.0 (TID 168, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 148.0 in stage 2.0 (TID 150, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 121.0 in stage 2.0 (TID 123, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 90.1 in stage 2.0 (TID 171, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 151.0 in stage 2.0 (TID 153, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 160.0 in stage 2.0 (TID 162, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 65.1 in stage 2.0 (TID 179, podk-1-1.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 ERROR TaskSchedulerImpl:75 - Lost executor 2 on podk-1-2.local: remote Akka client disassociated
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 74.1 in stage 2.0 (TID 218, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 129.0 in stage 2.0 (TID 131, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 34.1 in stage 2.0 (TID 226, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 100.1 in stage 2.0 (TID 211, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 164.0 in stage 2.0 (TID 166, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 114.1 in stage 2.0 (TID 220, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 140.0 in stage 2.0 (TID 142, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 102.1 in stage 2.0 (TID 223, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 105.1 in stage 2.0 (TID 205, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 30.1 in stage 2.0 (TID 214, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 38.1 in stage 2.0 (TID 217, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 79.1 in stage 2.0 (TID 190, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 69.1 in stage 2.0 (TID 225, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 86.1 in stage 2.0 (TID 207, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 66.1 in stage 2.0 (TID 216, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 145.0 in stage 2.0 (TID 147, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 106.1 in stage 2.0 (TID 228, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 127.0 in stage 2.0 (TID 129, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 110.1 in stage 2.0 (TID 219, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 163.0 in stage 2.0 (TID 165, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 82.1 in stage 2.0 (TID 213, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 50.1 in stage 2.0 (TID 222, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 72.1 in stage 2.0 (TID 195, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 142.0 in stage 2.0 (TID 144, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 124.0 in stage 2.0 (TID 126, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 54.1 in stage 2.0 (TID 221, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 57.1 in stage 2.0 (TID 212, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 80.1 in stage 2.0 (TID 188, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 141.0 in stage 2.0 (TID 143, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 WARN TaskSetManager:71 - Lost task 168.0 in stage 2.0 (TID 170, podk-1-2.local): ExecutorLostFailure (executor lost)
2014-12-01 11:47:12 ERROR TaskSchedulerImpl:75 - Lost executor 3 on podk-1-3.local: remote Akka client disassociated
The text was updated successfully, but these errors were encountered: