Adding several simple normalizations. #311
Conversation
| * | ||
| * @see pkn | ||
| */ | ||
| def apply[I <: Interval, T](rdd: RDD[(Double, I, T)]): RDD[(Double, I, T)] = { |
There was a problem hiding this comment.
Just out of curiosity, why the (Double, I, T) ordering here? I would have expected most of these RDDs to have a form keyed off the interval (i.e. (I, Double)) and if you wanted to carry along extra information, maybe a key-value where the value was a pair (i.e. (I, (Double, T))).
|
I'm still reviewing this @fnothaft, sorry for the delay... |
|
Ping @tdanford. |
|
Ping @fnothaft -- see my question, above, about the |
|
Ping @fnothaft :-) |
|
@tdanford just addressed your comment, and rebased the change on master. |
|
Jenkins, retest this please. |
| * @tparam T Type of data passed along. | ||
| */ | ||
| def apply[T](rdd: RDD[(Double, T)]): RDD[(Double, T)] = { | ||
| val cachedRdd = rdd.cache |
There was a problem hiding this comment.
Out of curiosity, Frank, why (here, and above as well) the explicit call to cache and unpersist? Given that these RDDs appear to only be used once...?
There was a problem hiding this comment.
@tdanford cachedRdd is used in 1 count and 3 separate map calls.
There was a problem hiding this comment.
Indeed; all your base are belong to me now.
|
Jenkins, retest this please. |
|
I'm going to merge this anyway, since it builds in my hands -- and the tests that are failing aren't the ones you've added here. |
Adding several simple normalizations.
|
Thanks, @fnothaft! |
Added two normalizations:
@tdanford @carlyeks and I discussed the best place for this, and thought that it made the most sense to have it inside of ADAM, as downstream tools like RNAdam will depend on it, and the normalizations are useful primitives across many omics algorithms. Also, I would entertain any comments about whether org.bdgenomics.adam.rdd.normalization is the best package for this.