refactor RDD loading; explicitly load alignments

[ryan-williams]

There’s really not a case where we are loading a generic Parquet file of
SpecificRecords whose type we don’t have a strong requirement about;
on the other hand, there is a case where we know what type we want
(AlignmentRecord) but don’t know which code path to read it in via (sam,
bam, ifq, fq, parquet, …).

Here I’ve made the caller explicitly specify if it wants
AlignmentRecords (which it was previously doing by labeling the return
value), but wants to benefit from the smarts around file-extension
inference.

Parquet reads with a known type can still go through one place so that
we can continue to benefit from that code not needing to be duplicated,
by only relying on SpecificRecord.

[fnothaft]

+1, I think this is a good idea. We should extend it to variants as well, at a later time.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/382/

Build result: FAILURE

GitHub pull request #468 of commit b9b4718 automatically merged.[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-slave-01 (centos) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/468/merge^{commit} # timeout=10Checking out Revision 81f2b1f (detached) > git config core.sparsecheckout # timeout=10 > git checkout -f 81f2b1f > git rev-list a34074082d54668273cacb5528d8aaed1b9c6ea8 # timeout=10Triggering ADAM-prb » 1.0.4,centosTriggering ADAM-prb » 2.2.0,centosTriggering ADAM-prb » 2.3.0,centosADAM-prb » 1.0.4,centos completed with result FAILUREADAM-prb » 2.2.0,centos completed with result FAILUREADAM-prb » 2.3.0,centos completed with result FAILURE
Test FAILed.

[massie]

Jenkins retest this.

[ryan-williams]

@massie do you know why Jenkins failed? or why the retest? (I thought tests were passing for me locally; trying again now as well)

[fnothaft]

@ryan-williams it looks like the SAM loading tests are failing:

Error Message

Could not read footer: java.lang.RuntimeException: file:/tmp/reads124698313342189278134.sam is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [58, 55, 53, 10]
Stacktrace

      java.io.IOException: Could not read footer: java.lang.RuntimeException: file:/tmp/reads124698313342189278134.sam is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [58, 55, 53, 10]
      at parquet.hadoop.ParquetFileReader.readAllFootersInParallel(ParquetFileReader.java:190)
      at parquet.hadoop.ParquetFileReader.readAllFootersInParallelUsingSummaryFiles(ParquetFileReader.java:146)
      at parquet.hadoop.ParquetInputFormat.getFooters(ParquetInputFormat.java:458)
      at parquet.hadoop.ParquetInputFormat.getFooters(ParquetInputFormat.java:443)
      at parquet.hadoop.ParquetInputFormat.getSplits(ParquetInputFormat.java:344)
      at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:94)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
      at scala.Option.getOrElse(Option.scala:120)
      at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
      at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)
      at scala.Option.getOrElse(Option.scala:120)
      at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)
      at org.apache.spark.rdd.RDD.take(RDD.scala:1060)
      at org.bdgenomics.adam.plugins.Take10Plugin.run(Take10Plugin.scala:30)
      at org.bdgenomics.adam.cli.PluginExecutor.run(PluginExecutor.scala:121)
      at 

[ryan-williams]

yea seems real, let me look, I swear they were passing very shortly before I pushed

[ryan-williams]

cool, that should have fixed it, just needed to bring PluginExecutor into the future

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/383/
Test PASSed.

[ryan-williams]

bump! this good to go?

[fnothaft]

Ah, yes! Thanks for the ping. Can you rebase? I'll merge once rebased.

[ryan-williams]

I believe it is now rebased

[fnothaft]

Merged! Thanks @ryan-williams!

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/399/
Test PASSed.