[SPARK-19041][SS] Fix code snippet compilation issues in Structured Streaming Programming Guide #16442
Conversation
There was a problem hiding this comment.
type won't compile in scala since it is a keyword, hence here change it to deviceType
There was a problem hiding this comment.
file format requires checkpointLocation and path to be explicitly specified
There was a problem hiding this comment.
currently we don't have parquet(...)/csv(...)/text(...)/etc. methods
lw-lin
force-pushed
the
ss-pro-guide-
branch
from
c029e25 to
96532ae
Compare
| query.recentProgress // an array of the most recent process updates for this query | ||
|
|
||
| query.sinkStatus() // progress information about data written to the output sink | ||
| query.lastProgress // the most recent process update of this streaming query |
There was a problem hiding this comment.
we've wrapped source/sink statuses into recentProcesses/lastProcess
| } | ||
| @Overrides void onQueryProgress(QueryProgressEvent queryProgress) { | ||
| @Override | ||
| public void onQueryProgress(QueryProgressEvent queryProgress) { |
There was a problem hiding this comment.
It's @Override in java, and these methods must be public
lw-lin
changed the title
|
@srowen would you take a look, thanks! |
| // Running average signal for each device type | ||
| import org.apache.spark.sql.expressions.scalalang.typed._ | ||
| ds.groupByKey(_.type).agg(typed.avg(_.signal)) // using typed API | ||
| import org.apache.spark.sql.expressions.scalalang.typed |
There was a problem hiding this comment.
this import should be ….scalalang.typed rather than ….scalalang.typed._
|
Test build #70763 has finished for PR 16442 at commit
|
|
Test build #70764 has finished for PR 16442 at commit
|
srowen
left a comment
srowen
left a comment
There was a problem hiding this comment.
I take your word for it on the recentProgress()/lastProgress() change but looks OK except for one comment
| noAggDF | ||
| .writeStream() | ||
| .parquet("path/to/destination/directory") | ||
| .writeStream |
There was a problem hiding this comment.
This is a method call? it needs parens then.
|
Test build #70769 has finished for PR 16442 at commit
|
|
Merged to master |
|
Thanks! LGTM. I'm going to also cherry-pick to branch 2.1 in case we may want to push new docs to the website.. |
…treaming Programming Guide ## What changes were proposed in this pull request? Currently some code snippets in the programming guide just do not compile. We should fix them. ## How was this patch tested? ``` SKIP_API=1 jekyll build ``` ## Screenshot from part of the change:  Author: Liwei Lin <lwlin7@gmail.com> Closes #16442 from lw-lin/ss-pro-guide-.
|
@lw-lin thanks for catching these! :) |
…treaming Programming Guide ## What changes were proposed in this pull request? Currently some code snippets in the programming guide just do not compile. We should fix them. ## How was this patch tested? ``` SKIP_API=1 jekyll build ``` ## Screenshot from part of the change:  Author: Liwei Lin <lwlin7@gmail.com> Closes apache#16442 from lw-lin/ss-pro-guide-.
…treaming Programming Guide ## What changes were proposed in this pull request? Currently some code snippets in the programming guide just do not compile. We should fix them. ## How was this patch tested? ``` SKIP_API=1 jekyll build ``` ## Screenshot from part of the change:  Author: Liwei Lin <lwlin7@gmail.com> Closes apache#16442 from lw-lin/ss-pro-guide-.
5 participants
What changes were proposed in this pull request?
Currently some code snippets in the programming guide just do not compile. We should fix them.
How was this patch tested?
Screenshot from part of the change: