Adding loadPairedFastqAsFragments method #1828
Conversation
|
Can one of the admins verify this patch? |
|
Jenkins, test this please. |
|
Test PASSed. |
|
This should be done for now, could someone look at this? |
|
I'm +1 on adding the new method, but would like some time to benchmark the implementation here against a few other possible ones. Will take a look after cutting the 0.23.0 release. |
| val rdd = recordsR1.zip(recordsR2) | ||
| .map { | ||
| case (r1, r2) => | ||
| val pairText = new Text(r1._2.toString + r2._2.toString) |
There was a problem hiding this comment.
Note zipping the records together assumes that they are sorted in the same order in each file.
There was a problem hiding this comment.
Yes indeed, this should always the case when a pair of fastq files. When this is not the case the fastq files are corrupt, at least as paired reads.
Adding a check add more computation, seems like a waste
During some talk o gitter I missing this method and looks simple enough to add it myself ;).
Some open issue here are how to do validation if the 2 files are indeed in sync. I can use
stringency: ValidationStringencylike inloadPairedFastq. Will follow up this on gitter also.