Release ADAM version 0.20.0

[heuermh]

Issue triage for ADAM version 0.20.0 release

Mango needs changes to formats, utils, and adam in order to land their changes from feature branches into master.

• Move Interval trait to utils, added in #75 #1041
• overlap query reflects new formats #1043
• Added IntervalRDD submodule and its interval tree dependency. utils#75

Major refactoring around GenomicRDD and related

• Load/store sequence dictionaries alongside Genotype RDDs #909
• Port single file BAM and header code to VCF #1009
• Factor out *RDDFunctions classes #1011
• Remove implicit GenomicRDD to RDD conversion #1040

Updates to Hadoop-BAM to add gzipped/BGZF VCF support, among other things

• BAM output in ADAM appears to be corrupt #962
• Roll Jenkins JDK 8 changes into ./scripts/jenkins-test #1008
• Remove workaround for gzip/BGZF compressed VCF headers #1057
• Update hadoop-bam dependency version to 7.5.0 #989
• [ADAM-909] Refactoring variation RDDs. #1015

Improved feature support, including GFF3

• Support writing RDD[Feature] to various file formats #710
• Support GFF3 format #1007
• Port single file BAM and header code to VCF #1009
• Map IntervalList format column four to feature name or attributes? #1152
• TransformFormats write to .gff3 file path incorrectly writes as parquet #1168
• Update feature and related records to support GFF3 bdg-formats#82

BAM header bugs

• BAMHeader not set when running on a cluster #676
• BAM output in ADAM appears to be corrupt #962

Support CRAM input/output

• Support CRAM input/output #1141
• [ADAM-1141] Add support for saving/loading AlignmentRecords to/from CRAM. #1145

Improved VCF support, for Mayo VariantDB Challenge

• How to create a rich(er) VariantContext RDD? Reconstruct VCF INFO fields.  #878
• VCF sample metadata - proposal for a GenotypedSampleMetadata object #1039
• Support VCF annotation ANN field in vcf2adam and adam2vcf #1044
• VariantCallingAnnotations is getting populated with INFO fields #1063
• Variant dbsnp rs id tracking in vcf2adam and ADAM2Vcf #1103
• Improve test coverage of VariantContextConverter #1107
• VCF format tag SB field parse error in loading #1199
• [ADAM-194] Use separate filtersFailed and filtersPassed arrays for variant quality filters #1075
• Upgrade to bdg-formats-0.10.0 #1135
• Use separate filtersFailed and filtersPassed arrays for variant quality filters bdg-formats#88
• Refactor variant and genotype annotations bdg-formats#90
• Merge VariantAnnotation and DatabaseVariantAnnotation records bdg-formats#105

Support Sample schema

• VCF sample metadata - proposal for a GenotypedSampleMetadata object #1039

Support Spark version 2.0

• ADAM compatibility with Spark 2.0 #1021
• ADAM version 0.19.0 will not run on Spark version 2.0.0 #1093
• [ADAM-1093] Move to support Spark 2.0.0. #1123

Pipes API

• Add Pipe API #1112
• Add pipes API #1114

Region joins

• Add outer region join implementations #1098
• Make region joins public again #1143
• Add outer joins #1109

Documentation, code cleanup

• Write quick start guide for Cloudera Manager/CDH5 #42
• Update usage docs running for EC2 and CDH  #493
• Clean up if clauses in Transform #1053
• Clean up packages #1083
• Document deploying ADAM on Toil #1084
• Document core ADAM transform methods #1085
• AlignmentRecordRDD does not extend GenomicRDD per javac #1092
• Remove Gene and related models and parsing code #1129
• Users should be able to set size of buffer used for merging files #1160
• Users should be able to save files as -single without merging them #1161
• Should be able to merge shards across two different file systems #1165
• Documentation cleanup and minor refactor on the consensus package. #1055
• Cleanup on org.bdgenomics.adam.algorithms.smithwaterman package. #1056
• Cleanup in org.bdgenomics.adam.converters package. #1062
• Refactoring org.bdgenomics.adam.io package. #1064
• Remove code workarounds necessary for Spark 1.2.1/Hadoop 1.0.x support #1076

Bugs

• MDTagging fails on GRCh38 #1192

Packaging cleanup

• Invalid POM for cli on 0.19.0 #1066 <-- verify once release cut

[fnothaft]

@jpdna as per our discussion, I've assigned #42 and #493 to you.

[fnothaft]

Moving conversation with @heuermh from #1192 over to this ticket:

I'd bump everything but this to 0.21. I'm not sure I have bandwidth to fully test the pipe API stuff by Monday, since SnpEff is perhaps not the best first thing to try, given the variant annotation refactor is still incomplete. Have other uses for the pipe API in mind?

For dates I'd propose Oct 10th for 0.20.0, Dec 10th for 0.21, and sometime in 2017 for 0.22.

As I commented over there, I'd need a release with the pipes API (#1114) before December. Is the main goal of pulling the release in to 10/10 to get @ryan-williams off of his #1139 fork? I'm OK with that, but we should really plan to cut a follow-on release (0.20.1?) sooner, because I need #1114, and getting #1114 merged has really dragged out. As of next Wednesday, #1114 will have been open for two months...

My primary concern is probably less about the specific release dates, and more about the fact that they keep on slipping. I'm reasonably OK with putting an arbitrary line in the sand and cutting a release followed up with another release shortly thereafter, but there's a lot of issues here that have dragged out without resolution or action. Specifically, the VCF issues and documentation issues need to close before we can push out the ADAM paper. They're not really negotiable; they need to be done by November. I own two of the documentation issues, and I'll close on them ASAP (#1084, #1085). @heuermh can you commit to a timeline for closing the VCF issues? If not—I hate to say this—but I will take them on and close them. @jpdna can you commit to a timeline for #493?

Of the other open issues, I am OK with triaging #1092 to the "indefinite" pile.

[jpdna]

@jpdna can you commit to a timeline for #493?

Yes. Let's talk Monday about what the timeline should be, but I'm looking at it right now.

[heuermh]

#1114 is nearly ready to go as I see it. I'll work up another external tool example over the weekend to test. Is it ok to block the 0.20.0 release on it being merged?

Would then a Dec 10th target for 0.21 (including #1144 and the follow-on GenotypeAnnotation refactor) be reasonable, with another round of triage to focus on what is necessary for the paper?

#1092 is fairly significant, is it not? No code using the Java APIs will compile without using the non-obvious cast as a workaround.

[heuermh]

From today's discussion:

New release date for 0.20 of Friday, 14 October 2016 (@heuermh will cut the release)
Primary focus for 0.21 will be documentation and performance fixes relevant to the paper
Re-triage 0.20, 0.21, and 1.0 issues and milestones (@heuermh will do the triage)
More point releases, bi-weekly, automated, less overhead (@fnothaft to discuss on #1139)

[heuermh]

I've created a new 0.22 milestone for 15 Dec, moved 0.21 to 15 Nov, and moved 1.0 to 15 Feb. I then triaged the doc and variant annotation stuff to 0.21 and everything else to 0.22. Let me know what you think.

[fnothaft]

@heuermh this is good to go, now that #1203/#1209 is merged.

[heuermh]

Fixed by commits:

• 5981475
• 875fe18
• 434c510
• a662001

on the version 0.20.0 Scala 2.10, Scala 2.11, Spark 2 + Scala 2.10, and Spark 2 + Scala 2.11 branches, respectively.

[ryan-williams]

Was 0.20.0 released? I don't see it here.

[fnothaft]

The release was staged but failed Sonatype validation (see #1212). @heuermh is working with Sonatype to fix this.

[heuermh]

I'm still waiting to hear from Sonatype to help determine the least painful option. Meanwhile I'll have some pull requests that should fix things for the next release.

[ryan-williams]

Makes sense; should we re-open this?

[heuermh]

I don't think so, the release has been cut and tagged. The hangup with Sonatype Nexus is something else