[Final] More examples to highlight new 0.5 features. #241
Conversation
Codecov Report
@@ Coverage Diff @@
## master #241 +/- ##
=========================================
Coverage ? 96.56%
=========================================
Files ? 51
Lines ? 874
Branches ? 11
=========================================
Hits ? 844
Misses ? 30
Partials ? 0
Continue to review full report at Codecov.
|
imarios
force-pushed
the
update_docs_for_0.5
branch
2 times, most recently
from
9daa4d1 to
7668fbd
Compare
imarios
changed the title
|
Hi @OlivierBlanvillain this is ready for review. Thanks! |
imarios
force-pushed
the
update_docs_for_0.5
branch
from
7668fbd to
d037624
Compare
| Only column types that can be sorted are allowed to be selected for sorting. | ||
|
|
||
| ```tut:book | ||
| aptTypedDs.orderBy(aptTypedDs('city).asc).show(2).run() |
There was a problem hiding this comment.
default ordering as asc is also supported with #236. so this should be the same as
aptTypedDs.orderBy(aptTypedDs('city)).show(2).run()
There was a problem hiding this comment.
I had it like that but implicit with poly didn’t work in tut. Not sure why ... I will give it another shot
There was a problem hiding this comment.
Hmm you're right. It seems that explicitly calling the apply with the column type works but otherwise it fails to infer the sort. E.g.:
aptTypedDs.orderBy(aptTypedDs[String]('city)).show(2).run()That's fairly annoying.
What's interesting is that orderByMany works without that.
aptTypedDs.orderByMany(aptTypedDs('city)).show(2).run()It's not immediately obvious to me why the implicit isn't getting picked up. I'll try to look into it shortly.
OlivierBlanvillain
left a comment
OlivierBlanvillain
left a comment
There was a problem hiding this comment.
Thanks @imarios, this is very nicely done!
docs/src/main/tut/FeatureOverview.md
Outdated
|
|
||
| The union of `aptTypedDs2` with `aptTypedDs` uses all the fields of the caller (`aptTypedDs2`) | ||
| and expects the other (`aptTypedDs`) dataset to include all those fields. | ||
| If field names/types do not much you get a compilation error. |
docs/src/main/tut/FeatureOverview.md
Outdated
| ``` | ||
|
|
||
| The union of `aptTypedDs2` with `aptTypedDs` uses all the fields of the caller (`aptTypedDs2`) | ||
| and expects the other (`aptTypedDs`) dataset to include all those fields. |
There was a problem hiding this comment.
the other dataset (aptTypedDs) to ...
docs/src/main/tut/FeatureOverview.md
Outdated
|
|
||
| Frameless supports many of Spark's functions and transformations. | ||
| However, whenever a Spark function does not exist in Frameless, | ||
| calling the `.dataset` will take you back to vanilla `Dataset` where |
There was a problem hiding this comment.
will expose the underlying Dataset (from org.apache.spark.sql, the original Spark APIs), where you can use anything that would be missing from the Frameless API.
docs/src/main/tut/FeatureOverview.md
Outdated
| ``` | ||
|
|
||
| A simple way to add a column without loosing important schema information is | ||
| to project the entire source schema into a single column. |
There was a problem hiding this comment.
We should maybe note that his is something new that is not part of the vanilla APIs
|
|
||
| ## Aggregate vs Projected columns | ||
|
|
||
| Vanilla `Datasets` do not distinguish between columns created from aggregate operations, |
There was a problem hiding this comment.
I would all them "Spark's Datasets" in the docs
imarios
force-pushed
the
update_docs_for_0.5
branch
from
a524664 to
b8e343d
Compare
|
Thanks for all the help guys! @OlivierBlanvillain @frosforever ... don't forget the squash, I have a ton of useless commits here :) |
|
LGTM 👍 |
4 participants
@OlivierBlanvillain, this is ready for review, but I might be adding few more examples here (hence the WIP).