Skip to content

Conversation

@imarios
Copy link
Contributor

@imarios imarios commented May 30, 2017

After some gitter discussion I realized that we missed lit() for aggr.

Here is what you can do now with litAggr. Before, lit() would not work during aggregations.

case class Foo(a: String, b: Int)
val t = TypedDataset.create(Seq(Foo("a",1),Foo("b",2)))

t.groupBy(t('a)).agg(sum(t('b)), litAggr(2), litAggr(Vector(1,2))).show().run
+---+---+---+------+
| _1| _2| _3|    _4|
+---+---+---+------+
|  b|  2|  2|[1, 2]|
|  a|  1|  2|[1, 2]|
+---+---+---+------+

* sure the injection instance is in scope.
*
* apache/spark
*/
Copy link
Contributor Author

@imarios imarios May 30, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As you see, lit() is returning TypedColumn instead of TypedAggregate even though it's defined inside AggregateFunctions (so you would expect to operate during aggregation). Changing to lit() there was an ambiguity with lit() defined inside functions. So if both were in scope it was confusing the compiler. That's why I rename to litAggr.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder returning a TypedColumn[T, A] with TypedAggregate[T, A] would work

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, let me try this

Copy link
Contributor Author

@imarios imarios May 30, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@OlivierBlanvillain You suggest we change lit to: def lit[A: TypedEncoder, T](value: A): TypedAggregate[T,A] with TypedColumn[T, A]?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doing that gives me:

java.lang.ClassCastException: frameless.TypedColumn cannot be cast to frameless.TypedAggregate
  at frameless.functions.package$.lit(package.scala:14)
  ... 42 elided

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's the right hand side of your lit def?

A => B and A => C

Yeah that won't work. There is a hack which is to define the second one as def (a: A)(implicit d: DummyImplicit): C, but I guess this wouldn't apply here...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, that's the rhs of lit(). I am open to making this more elegant ... I did try to have the same name and make them both work, but I wasn't able to get there. That's when I resorted to changing the name and calling it litAggr.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know @kanterov worked on the TypedAggretate migration. Gleb, any suggestion here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm... one of the ways to fix would be to revert and get back to a hierarchy with TypedAggregateAndColumn.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kanterov this is precluded in #153. Let's move the discussion there.

@codecov-io
Copy link

codecov-io commented May 30, 2017

Codecov Report

Merging #143 into master will decrease coverage by 0.15%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #143      +/-   ##
==========================================
- Coverage   93.02%   92.86%   -0.16%     
==========================================
  Files          28       28              
  Lines         602      603       +1     
  Branches       12       11       -1     
==========================================
  Hits          560      560              
- Misses         42       43       +1
Impacted Files Coverage Δ
...scala/frameless/functions/AggregateFunctions.scala 100% <100%> (ø) ⬆️
...ataset/src/main/scala/frameless/TypedDataset.scala 92.43% <0%> (-0.85%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5b0f36b...6f6b838. Read the comment docs.

@imarios imarios force-pushed the adding_back_lit_for_agg branch from 9b2cfe3 to 6f6b838 Compare May 30, 2017 14:56
@imarios
Copy link
Contributor Author

imarios commented Jun 1, 2017

Hi @kanterov, any input on this one?

@imarios
Copy link
Contributor Author

imarios commented Jun 3, 2017

@OlivierBlanvillain @kanterov anything else we need to address here?

@OlivierBlanvillain
Copy link
Contributor

Let's be patient and wait for @kanterov feedback on this one 😄

@kanterov
Copy link
Contributor

kanterov commented Jun 7, 2017

Sorry, just noticed this, I don't know, need to consider :)

@imarios
Copy link
Contributor Author

imarios commented Jan 29, 2018

covered by #153. closing.

@imarios imarios closed this Jan 29, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants