Add a cumulative sum to KeyedList

[snoble]

Not sure where the best place to put this is (or where to add tests) but this is a method for taking cumulative sums that doesn't require all the entries to go through a single scanLeft. It accomplishes this by

• taking an ordered partition of the entries
• summing up the entries in each partition
• scanning through the sums (which does require going through a single scanLeft so there can't be too many partitions)
• then scanning through each partition starting from the cumulative sum of the prior partition (so each partition can't be too big either)

This has been compiled against master.

[johnynek]

this should be Option[(U, V)]

[snoble]

I seem to recall flattenValues was giving me grief when I was using an Option... I'll confirm though

[johnynek]

These look like useful functions, but the placement does not seem quite right. The first thing you do is call .toTypedPipe on these things. This is because KeyedListLike does not have any notion of sorting because (sadly) Cascading does not support sorting during a join.

So, I think this might be a case for enrichment:

package com.twitter.scalding.typed

/**
 * Extension for TypedPipe to add a cumulativeSum method
 */
object CumulativeSum {
  implicit class Extension[K, V](val pipe: TypedPipe[(K, V)]) extends AnyVal {
    def cumulativeSum[......
  }
}

Then a user accesses it with: import CumulativeSum.Extension.

It's tough to draw the line between what the core methods are and what should be on the edge. This feels like the latter to me.

[snoble]

This rings true to me that this doesn't feel like a core method. I'll move this around.

[snoble]

Moved the files around. Note this breaks building the 2.9 target.

Also, flattenValues really doesn't want to work with Options

[snoble]

bump

[ianoc]

This is failing the tests, we still have a 2.9 build that isn't working here.

[snoble]

@ianoc that's from using an implicit class. Is there a 2.9 way to add methods like this?

[ianoc]

implicit def toMyType(a: BaseType) = new MyType(a)

[ianoc]

implicit def toMyType(a: BaseType) = new MyType(a)

On Mon, Nov 3, 2014 at 2:50 PM, snoble notifications@github.com wrote:

@ianoc https://github.com/ianoc that's from using an implicit class. Is
there a 2.9 way to add methods like this?

—
Reply to this email directly or view it on GitHub
#1085 (comment).

[snoble]

tests are passing now

[snoble]

Added a new test for this method

[ianoc]

don't need field names when using the typed api

[ianoc]

you have added scalding-hadoop-test/NOTICE also, looks by mistake...

[johnynek]

if we remove this, you could return an SortedGrouped, I think, which can still do reduces on, if people want that. I think returning the TypedPipe is only a loss of power. Consider if you want to follow up with a fold or another scan (or a filter followed by a fold or scan).

[johnynek]

Make the simple case return type SortedGrouped: https://github.com/twitter/scalding/blob/develop/scalding-core/src/main/scala/com/twitter/scalding/typed/Grouped.scala#L55 and I'll click merge.

[snoble]

Done. (as an aside I'm still confused why Options don't seem to be working with flattenValues... /shrug)

[johnynek]

@snoble because Option is not TraversableOnce, there is an implicit to that. You could have done: .mapValues(_.iterator).flattenValues

[snoble]

oops, fixed

[johnynek]

Thank you, sir!