[SPARK-3325][Streaming] Add a parameter to the method print in class DStream. #3237

watermen · 2014-11-13T02:52:35Z

I have deleted the repository with mistake, so I create a new PR. old PR:#2216

def print(num: Int = 10)

User can control the number of elements which to print.

AmplabJenkins · 2014-11-13T02:57:09Z

Can one of the admins verify this patch?

tdas · 2014-11-14T21:03:52Z

Jenkins, this is ok to test.

tdas · 2014-11-14T21:05:50Z

LGTM! As soon as the Jenkins test passes, I am going to merge this.

SparkQA · 2014-11-14T21:10:10Z

Test build #23386 has started for PR 3237 at commit 2279800.

This patch merges cleanly.

SparkQA · 2014-11-14T22:32:04Z

Test build #23386 has finished for PR 3237 at commit 2279800.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-14T22:32:07Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23386/
Test FAILed.

tdas · 2014-11-15T11:32:10Z

Aaah, this is getting into the binary compatibility issue. Adding the print(num) method to the interface JavaDStreamLike (Scala traits boil down to Java interfaces) is going to break existing implementations of the abstract class. But that is probably okay because I doubt any one has subclasses JavaDStreamLike. You will have to add the mima binary check exception in this file.
https://github.com/apache/spark/blob/master/project/MimaExcludes.scala

And you have to add exclusions for
ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.streaming.api.java.JavaDStreamLike.print") for Spark 1.2

Let me know if you need more information.

tdas · 2014-11-15T11:33:00Z

Also it would be cool if you can also extend the Python DStream's print :)
Should be pretty easy to do with the Java interface implemented.

SparkQA · 2014-11-17T07:07:30Z

Test build #23466 has started for PR 3237 at commit 21e8892.

This patch merges cleanly.

watermen · 2014-11-17T07:07:48Z

@tdas I'm not familiar with python, can you give me more information?

SparkQA · 2014-11-17T07:08:11Z

Test build #23466 has finished for PR 3237 at commit 21e8892.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-17T07:08:12Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23466/
Test FAILed.

AmplabJenkins · 2014-11-17T07:17:23Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23465/
Test FAILed.

tdas · 2014-11-19T22:14:08Z

@watermen For python you can take a look at the print in this file
https://github.com/apache/spark/blob/master/python/pyspark/streaming/dstream.py#L160 , and update it.
Also please fix the scala style issue, see the next Jenkins logs for more details.

tdas · 2014-11-19T22:15:13Z

Jenkins, test this please.

SparkQA · 2014-11-19T22:17:42Z

Test build #23634 has started for PR 3237 at commit 21e8892.

This patch merges cleanly.

SparkQA · 2014-11-19T22:18:22Z

Test build #23634 has finished for PR 3237 at commit 21e8892.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-19T22:18:23Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23634/
Test FAILed.

SparkQA · 2014-11-20T01:52:37Z

Test build #23651 has started for PR 3237 at commit 7be507f.

This patch does not merge cleanly.

SparkQA · 2014-11-20T03:43:24Z

Test build #23651 has finished for PR 3237 at commit 7be507f.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-20T03:43:28Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23651/
Test PASSed.

SparkQA · 2014-11-20T12:55:07Z

Test build #23668 has started for PR 3237 at commit 5fd7afd.

This patch does not merge cleanly.

SparkQA · 2014-11-20T13:40:06Z

Test build #23669 has started for PR 3237 at commit 3abdb1b.

This patch does not merge cleanly.

SparkQA · 2014-11-20T13:55:03Z

Test build #23670 has started for PR 3237 at commit b589a4b.

This patch merges cleanly.

SparkQA · 2014-11-20T14:44:56Z

Test build #23668 has finished for PR 3237 at commit 5fd7afd.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-20T14:45:00Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23668/
Test PASSed.

tdas · 2014-12-24T02:00:06Z

Hey @watermen This patch has conflicts. Could you update? Sorry for delaying the merging of this PR.

SparkQA · 2014-12-31T02:32:29Z

Test build #24927 has started for PR 3237 at commit 3b9d5cf.

This patch merges cleanly.

SparkQA · 2014-12-31T02:37:36Z

Test build #24929 has started for PR 3237 at commit f6ac3cb.

This patch merges cleanly.

watermen · 2014-12-31T02:48:44Z

@tdas I had fixed the conflicts, please merge this PR.

SparkQA · 2014-12-31T03:54:55Z

Test build #24927 has finished for PR 3237 at commit 3b9d5cf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-31T03:54:59Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24927/
Test PASSed.

SparkQA · 2014-12-31T04:01:27Z

Test build #24929 has finished for PR 3237 at commit f6ac3cb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-31T04:01:31Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24929/
Test PASSed.

SparkQA · 2014-12-31T07:22:30Z

Test build #24951 has started for PR 3237 at commit bb35d1a.

This patch merges cleanly.

SparkQA · 2014-12-31T08:30:39Z

Test build #24951 has finished for PR 3237 at commit bb35d1a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-31T08:30:43Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24951/
Test FAILed.

tdas · 2014-12-31T23:05:12Z

Jenkins, test this again.

JoshRosen · 2014-12-31T23:44:53Z

Jenkins, retest this please.

SparkQA · 2014-12-31T23:47:33Z

Test build #24978 has started for PR 3237 at commit bb35d1a.

This patch merges cleanly.

tdas · 2015-01-01T01:05:34Z

Oops sorry, my bad. I created another merge conflict by merging another PR. Let me fix that for you.

SparkQA · 2015-01-01T01:10:11Z

Test build #24978 has finished for PR 3237 at commit bb35d1a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-01T01:10:15Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24978/
Test PASSed.

tdas · 2015-01-01T01:16:03Z

I opened a new PR with your branch
#3865

You will get the main authorship :)

tdas · 2015-01-02T23:10:56Z

The other PR #3865 has been merged. Mind closing this PR? Thanks for all the effort!

…DStream This PR is a fixed version of the original PR #3237 by watermen and scwf. This adds the ability to specify how many elements to print in `DStream.print`. Author: Yadong Qi <qiyadong2010@gmail.com> Author: q00251598 <qiyadong@huawei.com> Author: Tathagata Das <tathagata.das1565@gmail.com> Author: wangfei <wangfei1@huawei.com> Closes #3865 from tdas/print-num and squashes the following commits: cd34e9e [Tathagata Das] Fix bug 7c09f16 [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into HEAD bb35d1a [Yadong Qi] Update MimaExcludes.scala f8098ca [Yadong Qi] Update MimaExcludes.scala f6ac3cb [Yadong Qi] Update MimaExcludes.scala e4ed897 [Yadong Qi] Update MimaExcludes.scala 3b9d5cf [wangfei] fix conflicts ec8a3af [q00251598] move to Spark 1.3 26a70c0 [q00251598] extend the Python DStream's print b589a4b [q00251598] add another print function

tdas · 2015-01-05T19:49:22Z

Mind closing this PR?

watermen mentioned this pull request Nov 13, 2014

[SPARK-3325] Add a parameter to the method print in class DStream. #2216

Closed

watermen changed the title ~~[SPARK-3325] Add a parameter to the method print in class DStream.~~ [SPARK-3325][Streaming] Add a parameter to the method print in class DStream. Nov 13, 2014

watermen force-pushed the fix-print branch from 5fd7afd to 3abdb1b Compare November 20, 2014 13:33

add another print function

b589a4b

scwf and others added 2 commits December 31, 2014 10:21

fix conflicts

3b9d5cf

Update MimaExcludes.scala

e4ed897

Update MimaExcludes.scala

f6ac3cb

watermen added 2 commits December 31, 2014 15:19

Update MimaExcludes.scala

f8098ca

Update MimaExcludes.scala

bb35d1a

tdas mentioned this pull request Jan 1, 2015

[SPARK-3325][Streaming] Add a parameter to the method print in class DStream #3865

Closed

watermen closed this Jan 6, 2015

[SPARK-3325][Streaming] Add a parameter to the method print in class DStream. #3237

[SPARK-3325][Streaming] Add a parameter to the method print in class DStream. #3237

Uh oh!

Conversation

watermen commented Nov 13, 2014

Uh oh!

AmplabJenkins commented Nov 13, 2014

Uh oh!

tdas commented Nov 14, 2014

Uh oh!

tdas commented Nov 14, 2014

Uh oh!

SparkQA commented Nov 14, 2014

Uh oh!

SparkQA commented Nov 14, 2014

Uh oh!

AmplabJenkins commented Nov 14, 2014

Uh oh!

tdas commented Nov 15, 2014

Uh oh!

tdas commented Nov 15, 2014

Uh oh!

SparkQA commented Nov 17, 2014

Uh oh!

watermen commented Nov 17, 2014

Uh oh!

SparkQA commented Nov 17, 2014

Uh oh!

AmplabJenkins commented Nov 17, 2014

Uh oh!

AmplabJenkins commented Nov 17, 2014

Uh oh!

tdas commented Nov 19, 2014

Uh oh!

tdas commented Nov 19, 2014

Uh oh!

SparkQA commented Nov 19, 2014

Uh oh!

SparkQA commented Nov 19, 2014

Uh oh!

AmplabJenkins commented Nov 19, 2014

Uh oh!

SparkQA commented Nov 20, 2014

Uh oh!

SparkQA commented Nov 20, 2014

Uh oh!

AmplabJenkins commented Nov 20, 2014

Uh oh!

SparkQA commented Nov 20, 2014

Uh oh!

SparkQA commented Nov 20, 2014

Uh oh!

SparkQA commented Nov 20, 2014

Uh oh!

SparkQA commented Nov 20, 2014

Uh oh!

AmplabJenkins commented Nov 20, 2014

Uh oh!

tdas commented Dec 24, 2014

Uh oh!

SparkQA commented Dec 31, 2014

Uh oh!

SparkQA commented Dec 31, 2014

Uh oh!

watermen commented Dec 31, 2014

Uh oh!

SparkQA commented Dec 31, 2014

Uh oh!

AmplabJenkins commented Dec 31, 2014

Uh oh!

SparkQA commented Dec 31, 2014

Uh oh!

AmplabJenkins commented Dec 31, 2014

Uh oh!

SparkQA commented Dec 31, 2014

Uh oh!

SparkQA commented Dec 31, 2014

Uh oh!

AmplabJenkins commented Dec 31, 2014

Uh oh!