Predicate to filter conversion #234
Conversation
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/ADAM-prb/310/ |
| @@ -48,7 +48,7 @@ class PileupAggregator(protected val args: PileupAggregatorArgs) | |||
| val companion = PileupAggregator | |||
|
|
|||
| def run(sc: SparkContext, job: Job) { | |||
| val pileups: RDD[ADAMPileup] = sc.adamLoad(args.readInput, predicate = Some(classOf[LocusPredicate])) | |||
There was a problem hiding this comment.
Not sure why this was using LocusPredicate before?
There was a problem hiding this comment.
That predicate is necessary. Pileups are created from mapped reads only.
There was a problem hiding this comment.
Maybe we should put a comment to keep others from tripping up on this too?
There was a problem hiding this comment.
Sure - but for this is after pileup creation right? Also the fields that LocusPredicate will check against are not defined for an ADAMPileup. I was going to substitute the MappedReadPredicate, but wasn't able to actually because of the typing as that is only applicable on ADAMRecord
There was a problem hiding this comment.
You're right, Arun. This is after pileup creation so the predicate isn't needed.
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/ADAM-prb/311/ |
|
Please run scalariform, e.g. |
…ff RDDs on and after load
|
All automated tests passed. |
|
Thanks, Arun! I really liked seeing all the tests you put in this pull request. 👍 |
This PR is for issue #62
ADAMPredicate derives from UnboundRecordFilter and can be used to set ParquetInputFormat.setUnboundRecordFilter. It also has an apply method to filter an existing RDD. This will allow to use predicates on parquet files for predicate pushdown but also on an already loaded RDD (if we load from BAM/SAM file, or use the same filters after some processing (removing duplicates after mark_duplicates before proceeding to the other read-prep stages))
I added a few examples - HighQualityReadPredicate, UniqueMappedRead and GenotypeRecordPASSPredicate.
Also, in ADAMRecordConditions and ADAMGenotypeConditions are utility predicates which can be composed using AND and OR to create a new predicate. We can also specify non-equality predicates easily as well.