Added check for bam index files in loadIndexedBam #1831

akmorrow13 · 2017-12-13T20:21:28Z

No description provided.

AmplabJenkins · 2017-12-13T20:29:17Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2512/

Build result: FAILURE

[...truncated 15 lines...] > /home/jenkins/git2/bin/git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ # timeout=15 > /home/jenkins/git2/bin/git rev-parse origin/pr/1831/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a -v --no-abbrev --contains 665e131 # timeout=10Checking out Revision 665e131 (origin/pr/1831/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 665e131385df3b695eb244735b2242cd385a4948First time build. Skipping changelog.Triggering ADAM-prb ? 2.6.2,2.11,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,1.6.3,centosTriggering ADAM-prb ? 2.7.3,2.10,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.10,2.2.0,centosTriggering ADAM-prb ? 2.6.2,2.10,1.6.3,centosTriggering ADAM-prb ? 2.6.2,2.11,2.2.0,centosTriggering ADAM-prb ? 2.7.3,2.11,2.2.0,centosADAM-prb ? 2.6.2,2.11,1.6.3,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.7.3,2.10,2.2.0,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.10,2.2.0,centos completed with result FAILUREADAM-prb ? 2.6.2,2.10,1.6.3,centos completed with result FAILUREADAM-prb ? 2.6.2,2.11,2.2.0,centos completed with result FAILUREADAM-prb ? 2.7.3,2.11,2.2.0,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

AmplabJenkins · 2017-12-13T20:53:51Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2513/
Test PASSed.

akmorrow13 · 2017-12-13T21:39:44Z

Fixes #1830

heuermh · 2017-12-13T21:55:57Z

We hadn't checked in binary files as test resources before, as then diffs in git aren't meaningful. If we're going to break that convention, and this is as good a time as any, then we should add an index for small.bam and unit test(s) for the happy paths.

Do you need this for the 0.23.0 release?

AmplabJenkins · 2017-12-13T22:05:14Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2514/
Test PASSed.

AmplabJenkins · 2017-12-14T19:09:24Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2515/
Test PASSed.

fnothaft

Looks good! A few small nits.

fnothaft · 2017-12-14T20:12:20Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

    // todo: can this method handle SAM and CRAM, or just BAM?
    val bamFiles = getFsAndFiles(path).filter(p => p.toString.endsWith(".bam"))

+    val indexFiles = getFsAndFiles(indexPath).filter(p => p.toString.endsWith(".bai"))


Use getFsAndFilesWithFilter.

fnothaft · 2017-12-14T20:15:16Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

+      new Path(pathName + ".bai")
+    else
+      path
+
    // todo: can this method handle SAM and CRAM, or just BAM?


Can you update this comment to // currently only supports BAM and cross reference #1833.

fnothaft · 2017-12-14T20:16:05Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

+
+    // If pathName is a directory of bam files, scan all files to
+    // select index files.
+    val indexPath = if (pathName.endsWith(".bam"))


Use {}'s.

Also, its not clear that this code and the comment match up. Can you add more to the comment to clarify how this bit of code is necessary for directory support?

fnothaft · 2017-12-14T20:18:29Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

    require(bamFiles.nonEmpty,
      "Did not find any BAM files at %s.".format(path))
+
+    // Make sure each bam file has an index
+    bamFiles.foreach(file => {


Nit: if multiple files are missing an index, this will just throw for the first one. Instead, I'd suggest:

val missingIndices = bamFiles.filter(f => indexFiles.contains(f.toString + ".bai")) require(missingIndices.isEmpty, "Missing indices for BAMs:\n%s".format(missingIndices.mkString("\n"))

fnothaft · 2017-12-14T20:19:05Z

adam-core/src/test/scala/org/bdgenomics/adam/rdd/ADAMContextSuite.scala

@@ -435,6 +435,14 @@ class ADAMContextSuite extends ADAMFunSuite {
    assert(reads.rdd.count == 4)
  }

+  sparkTest("loadIndexedBam should throw exception without an index file") {


Can you add a test case for the directory/glob case?

Ping! Also, add a test case for the myFile.bai case.

akmorrow13 · 2017-12-15T19:28:12Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

    val bamFiles = getFsAndFiles(path).filter(p => p.toString.endsWith(".bam"))

+    val indexFiles = getFsAndFiles(indexPath).filter(p => p.toString.endsWith(".bai"))


@fnothaft getFsAndFilesWith filter wasn't able to find .bai files unless just the directory was specified

Yeah makes sense.

AmplabJenkins · 2017-12-15T19:49:55Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2516/
Test PASSed.

fnothaft · 2017-12-23T06:06:37Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

+
+    // If pathName is a single file or *.bam, append .bai to find all bam indices.
+    // Otherwise, pathName is a directory and the entire path must be searched
+    // for indices.


Can you note:

// TODO: support BAMs where the index is located at s/.bam/.bai/g

These aren't common (myFile.bam + myFile.bai vs myFile.bam.bai) but you do see them in the wild occasionally.

Maybe create an issue for that after this merges.

Actually, if you do new Path(pathName.toString.replace(".bam", "*.bai")) here and make a change below (will drop a comment), I think you can handle both cases.

fnothaft · 2017-12-23T06:12:13Z

adam-core/src/main/scala/org/bdgenomics/adam/rdd/ADAMContext.scala

    require(bamFiles.nonEmpty,
      "Did not find any BAM files at %s.".format(path))
+
+    val missingIndices = bamFiles.filterNot(f => {
+      indexFiles.contains(f.toString + ".bai")


If you do indexFiles.contains(f.toString + ".bai") || indexFiles.contains(f.toString.dropRight(3) + "bai")) you should handle both the myFile.bam.bai and myFile.bai cases.

fnothaft · 2017-12-23T06:12:33Z

adam-core/src/test/scala/org/bdgenomics/adam/rdd/ADAMContextSuite.scala

@@ -435,6 +435,14 @@ class ADAMContextSuite extends ADAMFunSuite {
    assert(reads.rdd.count == 4)
  }

+  sparkTest("loadIndexedBam should throw exception without an index file") {


Ping! Also, add a test case for the myFile.bai case.

fnothaft

Thanks @akmorrow13! LGTM.

AmplabJenkins · 2017-12-24T03:43:47Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2527/
Test PASSed.

AmplabJenkins · 2017-12-24T04:48:42Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/2528/
Test PASSed.

akmorrow13 · 2018-01-03T16:04:27Z

@heuermh and @fnothaft can we merge this?

heuermh · 2018-01-03T16:10:05Z

I'll give it one last review check after the 0.23.0 release is cut

fnothaft · 2018-01-03T16:50:00Z

Yeah, this is good to go from my side but I have a strong preference for it being in 0.24.0 instead of 0.23.0.

heuermh · 2018-01-09T19:43:32Z

Thank you, @akmorrow13!

akmorrow13 force-pushed the indexedbam-index branch from 786643d to 6a4859f Compare December 13, 2017 21:38

akmorrow13 force-pushed the indexedbam-index branch from 6a4859f to 3380265 Compare December 14, 2017 18:46

fnothaft requested changes Dec 14, 2017

View reviewed changes

akmorrow13 commented Dec 15, 2017

View reviewed changes

fnothaft requested changes Dec 23, 2017

View reviewed changes

fnothaft requested a review from heuermh December 23, 2017 06:12

fnothaft approved these changes Dec 24, 2017

View reviewed changes

akmorrow13 added 2 commits December 23, 2017 22:25

Added check for bam index files in loadIndexedBam

bbc4782

added check for.bam.bai and .bai index files

ee06eda

akmorrow13 force-pushed the indexedbam-index branch from 5cfe378 to ee06eda Compare December 24, 2017 04:25

akmorrow13 added this to the 0.24.0 milestone Jan 6, 2018

heuermh approved these changes Jan 9, 2018

View reviewed changes

heuermh merged commit 87f4667 into bigdatagenomics:master Jan 9, 2018

heuermh mentioned this pull request Jan 9, 2018

loadIndexed bam lazily throws Exception if index does not exist #1830

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added check for bam index files in loadIndexedBam #1831

Added check for bam index files in loadIndexedBam #1831

akmorrow13 commented Dec 13, 2017

AmplabJenkins commented Dec 13, 2017

AmplabJenkins commented Dec 13, 2017

akmorrow13 commented Dec 13, 2017

heuermh commented Dec 13, 2017

AmplabJenkins commented Dec 13, 2017

AmplabJenkins commented Dec 14, 2017

fnothaft left a comment

fnothaft Dec 14, 2017

fnothaft Dec 14, 2017

fnothaft Dec 14, 2017

fnothaft Dec 14, 2017

fnothaft Dec 14, 2017

fnothaft Dec 23, 2017

akmorrow13 Dec 15, 2017

fnothaft Dec 23, 2017

AmplabJenkins commented Dec 15, 2017

fnothaft Dec 23, 2017

fnothaft Dec 23, 2017

fnothaft Dec 23, 2017

fnothaft Dec 23, 2017

fnothaft Dec 23, 2017

fnothaft left a comment

AmplabJenkins commented Dec 24, 2017

AmplabJenkins commented Dec 24, 2017

akmorrow13 commented Jan 3, 2018

heuermh commented Jan 3, 2018

fnothaft commented Jan 3, 2018

heuermh commented Jan 9, 2018

		val bamFiles = getFsAndFiles(path).filter(p => p.toString.endsWith(".bam"))

		val indexFiles = getFsAndFiles(indexPath).filter(p => p.toString.endsWith(".bai"))

Added check for bam index files in loadIndexedBam #1831

Added check for bam index files in loadIndexedBam #1831

Conversation

akmorrow13 commented Dec 13, 2017

AmplabJenkins commented Dec 13, 2017

Build result: FAILURE

AmplabJenkins commented Dec 13, 2017

akmorrow13 commented Dec 13, 2017

heuermh commented Dec 13, 2017

AmplabJenkins commented Dec 13, 2017

AmplabJenkins commented Dec 14, 2017

fnothaft left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Dec 15, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fnothaft left a comment

Choose a reason for hiding this comment

AmplabJenkins commented Dec 24, 2017

AmplabJenkins commented Dec 24, 2017

akmorrow13 commented Jan 3, 2018

heuermh commented Jan 3, 2018

fnothaft commented Jan 3, 2018

heuermh commented Jan 9, 2018