ADAM on Slurm/LSF

[jpdna]

I understand some people run Spark on their local Slurm (or LSF?) cluster like:
https://www.princeton.edu/researchcomputing/faq/spark-via-slurm/

It would be useful to provide instructions for this in our user guide, as Slurm/LSF is the cluster infrastructure that most bioinformatics users have access to.

Is there a slurm/LSF cluster at Berkeley I could try this on?

[fnothaft]

Is there a slurm/LSF cluster at Berkeley I could try this on?

Let me ask around...

[heuermh]

I'm not sure what the infrastructure actually is at the link above. The examples show submitting jobs to a Spark cluster using Slurm, not that Spark is actually running on the Slurm cluster.

There's another link on that page describing "Spark framework and the submission guidelines using YARN", but it doesn't say whether Spark via YARN is installed on the Slurm cluster or separately.

[jpdna]

The examples show submitting jobs to a Spark cluster using Slurm

Good point @heuermh
This may be more relevant and what I was thinking of - actually running executors as jobs on LSF/Slurm:

https://github.com/LLNL/magpie

In general, my intuition that when running Spark on HPC in this way all you would really lose is data locality, but otherwise an application like ADAM would run the same as it does on an HDFS cluster.

[devin-petersohn]

I have some experience running Spark on Slurm from the University of Missouri. They have a large cluster managed by Slurm that runs Spark.

In that case, we dynamically created Spark clusters using Slurm, so the entire environment was torn down at the end of the allocation. HDFS works the same way. For Adam on Slurm, I don't think there would be too many steps, aside from perhaps changing the SPARK_HOME (which we set dynamically). Since we are starting a collaboration, there may be an opportunity to use their cluster as a test case for this.

[fnothaft]

Since we are starting a collaboration, there may be an opportunity to use their cluster as a test case for this.

+1!

[heuermh]

Fixed by #1571