VariantRDD union creates multiple records for the same SNP ID

[devin-petersohn]

VariantRDD.union() does not combine by SNP ID. This leads to the potential for multiple records with the same SNP ID.

If we were to write out the VariantRDD as a VCF, it would not be correct.

[devin-petersohn]

Link

[heuermh]

We already split alternate alleles at the same position into multiple Variant records, so I would not expect VariantRDD.union to combine by Variant.name. I would in fact expect to see many records at the same position with the same value(s) for Variant.name.

  /**
   Zero or more unique names or identifiers for this variant. If this is a dbSNP
   variant it is encouraged to use the rs number(s). VCF column 3 "ID" shared across
   all alleles in the same VCF record.
   */
  array<string> names = [];

[fnothaft]

+1 @heuermh, the union method specifies no contract WRT "duplicate" records.

Logically, in variant space, "duplicate" is not clearly defined, as two records with the same chr/pos/ref/alt could have conflicting annotations. I don't believe that the VCF spec says that dupe records are illegal, as long as both records are well formed and the data is properly sorted.

I suggest we close as won't fix.

[devin-petersohn]

I was thinking along the lines of the VCFTools implementation here, such that the output is a merged VCF.

If we don't care about this use case, feel free to close the issue.

[fnothaft]

union on GenotypeRDD/VariantRDD/VariantContextRDD implements both VCFTools merge and concat, albiet with possibly different semantics in the merge case.

[devin-petersohn]

Closing as won't fix.