Skip to content

Commit cd86533

Browse files
committed
Try to use jgo for classpath instead of shaded fat jar
Needs - mvn on classpath - jgo from [resolve-only](scijava/jgo#37) pull request (Python >= 3.7) Currently fails with: ``` Unable to convert into Paintera dataset: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 736, 10.36.110.15, executor 69): java.lang.NoSuchMethodError: org.apache.commons.compress.compressors.gzip.GzipCompressorOutputStream.<init>(Ljava/io/Ou at org.janelia.saalfeldlab.n5.GzipCompression.getOutputStream(GzipCompression.java:89) at org.janelia.saalfeldlab.n5.DefaultBlockWriter.write(DefaultBlockWriter.java:49) at org.janelia.saalfeldlab.n5.DefaultBlockWriter.writeBlock(DefaultBlockWriter.java:83) at org.janelia.saalfeldlab.n5.N5FSWriter.writeBlock(N5FSWriter.java:133) at org.janelia.saalfeldlab.n5.imglib2.N5LabelMultisets.saveLabelMultisetBlock(N5LabelMultisets.java:308) at org.janelia.saalfeldlab.n5.imglib2.N5LabelMultisets.saveLabelMultisetBlock(N5LabelMultisets.java:337) at org.janelia.saalfeldlab.label.spark.convert.ConvertToLabelMultisetType.lambda$convertToLabelMultisetType$dce05d2$1(ConvertToLabelMultisetType.java:226) at org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction$1.apply(JavaPairRDD.scala:1040) at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) at scala.collection.Iterator$class.foreach(Iterator.scala:893) at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at scala.collection.TraversableOnce$class.reduceLeft(TraversableOnce.scala:185) at scala.collection.AbstractIterator.reduceLeft(Iterator.scala:1336) at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$14.apply(RDD.scala:1015) at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$14.apply(RDD.scala:1013) at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130) at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:109) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ``` Probably dependency issue between spark and N5 (or other libraries) that is not an issue with shaded fat jar.
1 parent 4718964 commit cd86533

File tree

1 file changed

+62
-0
lines changed

1 file changed

+62
-0
lines changed

flintstone-jgo.sh

+62
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
#!/bin/bash
2+
3+
if [ "$#" -lt "3" ]; then
4+
echo -e "Not enough arguments!" 1>&2
5+
exit 1
6+
fi
7+
8+
CONTAINING_DIRECTORY="$( dirname "${BASH_SOURCE[0]}" )"
9+
SPARK_JANELIA="${SPARK_JANELIA:-${CONTAINING_DIRECTORY}/spark-janelia/spark-janelia}"
10+
# MVN="${MVN:-/misc/local/maven-3.2.2/bin/mvn}"
11+
12+
RUNTIME="${RUNTIME:-8:00}"
13+
SPARK_VERSION="${SPARK_VERSION:-2.3.1}"
14+
TERMINATE="${TERMINATE:-1}"
15+
MIN_WORKERS="${MIN_WORKERS:-1}"
16+
17+
N_EXECUTORS_PER_NODE="${N_EXECUTORS_PER_NODE:-6}"
18+
N_CORES_PER_EXECUTOR="${N_CORES_PER_EXECUTOR:-5}"
19+
MEMORY_PER_NODE="${MEMORY_PER_NODE:-300}"
20+
SPARK_OPTIONS="${SPARK_OPTIONS:-}"
21+
22+
N_DRIVER_THREADS="${N_DRIVER_THREADS:-16}"
23+
24+
N_NODES=$1; shift
25+
COORDINATE="$1"; shift
26+
CLASS=$1; shift
27+
ARGV="$@"
28+
29+
ARTIFACT="$(echo $COORDINATE | tr ':' '\n' | head -n2 | tail -n1)"
30+
WORKSPACE="$(jgo --repository scijava.public=https://maven.scijava.org/content/groups/public --resolve-only $COORDINATE)"
31+
MAIN_JAR="$(ls ${WORKSPACE}/*jar -1 | grep -E "${ARTIFACT}-[0-9]+")"
32+
ALL_JARS="$(ls ${WORKSPACE}/*jar -1 | tr '\n' ',' | sed 's/,$//')"
33+
34+
# echo "$MEMORY_PER_NODE / $N_EXECUTORS_PER_NODE"
35+
# echo "$N_NODES * $N_EXECUTORS_PER_NODE"
36+
export MEMORY_PER_EXECUTOR="$(($MEMORY_PER_NODE / $N_EXECUTORS_PER_NODE))"
37+
export N_EXECUTORS="$(($N_NODES * $N_EXECUTORS_PER_NODE))"
38+
export PARALLELISM="$(($N_EXECUTORS * $N_CORES_PER_EXECUTOR * 3))"
39+
40+
SUBMIT_ARGS="${SUBMIT_ARGS} --verbose"
41+
SUBMIT_ARGS="${SUBMIT_ARGS} --conf spark.default.parallelism=$PARALLELISM"
42+
SUBMIT_ARGS="${SUBMIT_ARGS} --conf spark.executor.instances=$N_EXECUTORS_PER_NODE"
43+
SUBMIT_ARGS="${SUBMIT_ARGS} --conf spark.executor.cores=$N_CORES_PER_EXECUTOR"
44+
SUBMIT_ARGS="${SUBMIT_ARGS} --conf spark.executor.memory=${MEMORY_PER_EXECUTOR}g"
45+
SUBMIT_ARGS="${SUBMIT_ARGS} ${SPARK_OPTIONS}"
46+
SUBMIT_ARGS="${SUBMIT_ARGS} --jars $ALL_JARS"
47+
SUBMIT_ARGS="${SUBMIT_ARGS} --class $CLASS"
48+
SUBMIT_ARGS="${SUBMIT_ARGS} ${MAIN_JAR}"
49+
SUBMIT_ARGS="${SUBMIT_ARGS} ${ARGV}"
50+
51+
LOG_FILE="${HOME}/.sparklogs/${CLASS}.o%J"
52+
53+
"${SPARK_JANELIA}" \
54+
--nnodes="${N_NODES}" \
55+
--no_check \
56+
--driveronspark \
57+
--silentlaunch \
58+
--minworkers="${MIN_WORKERS}" \
59+
--hard_runtime=${RUNTIME} \
60+
--submitargs="${SUBMIT_ARGS}" \
61+
--driveroutfile=${LOG_FILE} \
62+
lsd

0 commit comments

Comments
 (0)