Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

A fix for reading GS:// references from a Spark cluster #6070

Merged
merged 1 commit into from
Aug 1, 2019
Merged

A fix for reading GS:// references from a Spark cluster #6070

merged 1 commit into from
Aug 1, 2019

Conversation

SHuang-Broad
Copy link
Contributor

a bug fix suggested by Louis.
@lbergelson I don't know how to add a test.

Without this you'd see

org.apache.spark.SparkException: Job aborted due to stage failure: Task 45 in stage 9.0 failed 8 times, most recent failure: Lost task 45.7 in stage 9.0 (TID 734, shuang-svdps-ceu-w-1.c.broad-dsde-methods.internal, executor 2): java.nio.file.FileSystemNotFoundException: Provider "gs" not installed
	at java.nio.file.Paths.get(Paths.java:147)
	at org.broadinstitute.hellbender.engine.spark.datasources.ReferenceFileSparkSource.getReferencePath(ReferenceFileSparkSource.java:53)
	at org.broadinstitute.hellbender.engine.spark.datasources.ReferenceFileSparkSource.getReferenceBases(ReferenceFileSparkSource.java:60)
	at org.broadinstitute.hellbender.engine.spark.datasources.ReferenceMultiSparkSource.getReferenceBases(ReferenceMultiSparkSource.java:89)
	at org.broadinstitute.hellbender.tools.spark.sv.discovery.BreakEndVariantType.getRefBaseString(BreakEndVariantType.java:89)
	at org.broadinstitute.hellbender.tools.spark.sv.discovery.BreakEndVariantType.access$200(BreakEndVariantType.java:20)
	at org.broadinstitute.hellbender.tools.spark.sv.discovery.BreakEndVariantType$InterChromosomeBreakend.<init>(BreakEndVariantType.java:253)
	at org.broadinstitute.hellbender.tools.spark.sv.discovery.BreakEndVariantType$InterChromosomeBreakend.getOrderedMates(BreakEndVariantType.java:261)
	at org.broadinstitute.hellbender.tools.spark.sv.discovery.inference.NovelAdjacencyAndAltHaplotype.toSimpleOrBNDTypes(NovelAdjacencyAndAltHaplotype.java:246)
	at org.broadinstitute.hellbender.tools.spark.sv.discovery.inference.SimpleNovelAdjacencyInterpreter.inferType(SimpleNovelAdjacencyInterpreter.java:129)
	at org.broadinstitute.hellbender.tools.spark.sv.discovery.inference.SimpleNovelAdjacencyInterpreter.lambda$inferTypeFromSingleContigSimpleChimera$24ddc343$1(SimpleNovelAdjacencyInterpreter.java:107)
	at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043)
	at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
	at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:217)
	at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1094)
	at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1085)
	at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1020)
	at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1085)
	at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:811)
	at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:49)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:49)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:109)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1661)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1649)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1648)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1648)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
	at scala.Option.foreach(Option.scala:257)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:831)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1882)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1831)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1820)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2034)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2055)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2074)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2099)
	at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:945)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
	at org.apache.spark.rdd.RDD.collect(RDD.scala:944)
	at org.apache.spark.api.java.JavaRDDLike$class.collect(JavaRDDLike.scala:361)
	at org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:45)
	at org.broadinstitute.hellbender.tools.spark.sv.discovery.inference.SimpleNovelAdjacencyInterpreter.makeInterpretation(SimpleNovelAdjacencyInterpreter.java:48)
	at org.broadinstitute.hellbender.tools.spark.sv.discovery.SvDiscoverFromLocalAssemblyContigAlignmentsSpark.extractSimpleVariants(SvDiscoverFromLocalAssemblyContigAlignmentsSpark.java:328)
	at org.broadinstitute.hellbender.tools.spark.sv.discovery.SvDiscoverFromLocalAssemblyContigAlignmentsSpark.dispatchJobs(SvDiscoverFromLocalAssemblyContigAlignmentsSpark.java:303)
	at org.broadinstitute.hellbender.tools.spark.sv.discovery.SvDiscoverFromLocalAssemblyContigAlignmentsSpark.runTool(SvDiscoverFromLocalAssemblyContigAlignmentsSpark.java:170)
	at org.broadinstitute.hellbender.engine.spark.GATKSparkTool.runPipeline(GATKSparkTool.java:534)
	at org.broadinstitute.hellbender.engine.spark.SparkCommandLineProgram.doWork(SparkCommandLineProgram.java:31)
	at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:139)
	at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:191)
	at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:210)
	at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:162)
	at org.broadinstitute.hellbender.Main.mainEntry(Main.java:205)
	at org.broadinstitute.hellbender.Main.main(Main.java:291)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.nio.file.FileSystemNotFoundException: Provider "gs" not installed
	at java.nio.file.Paths.get(Paths.java:147)
	at org.broadinstitute.hellbender.engine.spark.datasources.ReferenceFileSparkSource.getReferencePath(ReferenceFileSparkSource.java:53)
	at org.broadinstitute.hellbender.engine.spark.datasources.ReferenceFileSparkSource.getReferenceBases(ReferenceFileSparkSource.java:60)
	at org.broadinstitute.hellbender.engine.spark.datasources.ReferenceMultiSparkSource.getReferenceBases(ReferenceMultiSparkSource.java:89)
	at org.broadinstitute.hellbender.tools.spark.sv.discovery.BreakEndVariantType.getRefBaseString(BreakEndVariantType.java:89)
	at org.broadinstitute.hellbender.tools.spark.sv.discovery.BreakEndVariantType.access$200(BreakEndVariantType.java:20)
	at org.broadinstitute.hellbender.tools.spark.sv.discovery.BreakEndVariantType$InterChromosomeBreakend.<init>(BreakEndVariantType.java:253)
	at org.broadinstitute.hellbender.tools.spark.sv.discovery.BreakEndVariantType$InterChromosomeBreakend.getOrderedMates(BreakEndVariantType.java:261)
	at org.broadinstitute.hellbender.tools.spark.sv.discovery.inference.NovelAdjacencyAndAltHaplotype.toSimpleOrBNDTypes(NovelAdjacencyAndAltHaplotype.java:246)
	at org.broadinstitute.hellbender.tools.spark.sv.discovery.inference.SimpleNovelAdjacencyInterpreter.inferType(SimpleNovelAdjacencyInterpreter.java:129)
	at org.broadinstitute.hellbender.tools.spark.sv.discovery.inference.SimpleNovelAdjacencyInterpreter.lambda$inferTypeFromSingleContigSimpleChimera$24ddc343$1(SimpleNovelAdjacencyInterpreter.java:107)
	at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043)
	at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
	at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:217)
	at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1094)
	at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1085)
	at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1020)
	at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1085)
	at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:811)
	at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:49)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:49)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:109)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Full stacktrace here

@SHuang-Broad SHuang-Broad requested a review from lbergelson August 1, 2019 14:01
lbergelson
lbergelson approved these changes Aug 1, 2019
View reviewed changes
Copy link
Member

@lbergelson lbergelson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a reasonable way to add a test for this other than just adding a full pipeline test on dataproc. 👍 to merging this without a matching unit test. I would suggest we add a full pipeline test on multiple nodes to catch things like this in the future.

@droazen droazen merged commit 20190cb into master Aug 1, 2019
@droazen droazen deleted the sh-spark-read-ref-from-bucket branch August 1, 2019 16:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lbergelson lbergelson lbergelson approved these changes

Assignees
No one assigned
Labels
Spark
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@SHuang-Broad @lbergelson @droazen