Skip to content

[SPARK-4196][SPARK-4602][Streaming] Fix serialization issue in PairDStreamFunctions.saveAsNewAPIHadoopFiles #3457

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
tdas wants to merge 2 commits into from
Closed

[SPARK-4196][SPARK-4602][Streaming] Fix serialization issue in PairDStreamFunctions.saveAsNewAPIHadoopFiles #3457

tdas wants to merge 2 commits into from

Conversation

@tdas
Copy link
Contributor

@tdas tdas commented Nov 25, 2014

Solves two JIRAs in one shot

  • Makes the ForechDStream created by saveAsNewAPIHadoopFiles serializable for checkpoints
  • Makes the default configuration object used saveAsNewAPIHadoopFiles be the Spark's hadoop configuration

@tdas tdas changed the title [SPARK-4196][SPARK-4602] Fix serialization issue in PairDStreamFunctions.saveAsNewAPIHadoopFiles [SPARK-4196][SPARK-4602][Streaming] Fix serialization issue in PairDStreamFunctions.saveAsNewAPIHadoopFiles Nov 25, 2014
srowen
srowen reviewed Nov 25, 2014
View reviewed changes
streaming/src/main/scala/org/apache/spark/streaming/dstream/PairDStreamFunctions.scala Outdated
Copy link
Member

@srowen srowen Nov 25, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah you're already on it, yep, that looks good.

@srowen
Copy link
Member

srowen commented Nov 25, 2014

Just had a thought -- will saveAsHadoopFiles and JobConf need the same treatment?

@tdas
Copy link
Contributor Author

tdas commented Nov 25, 2014

That's exactly I am testing now. Adding an unit test at the very least.

@SparkQA
Copy link

SparkQA commented Nov 25, 2014

Test build #23842 has started for PR 3457 at commit b382ea9.

  • This patch merges cleanly.

@tdas
Copy link
Contributor Author

tdas commented Nov 25, 2014

@srowen Yep, saveAsHadoopFiles had the same issue. Fixed it. Thanks for catching this bug!

@AmplabJenkins
Copy link

AmplabJenkins commented Nov 25, 2014

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23842/
Test FAILed.

@tdas
Copy link
Contributor Author

tdas commented Nov 25, 2014

Jenkins, test this please.

@SparkQA
Copy link

SparkQA commented Nov 25, 2014

Test build #23843 has started for PR 3457 at commit bb4729a.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Nov 25, 2014

Test build #23843 has finished for PR 3457 at commit bb4729a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Nov 25, 2014

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23843/
Test PASSed.

mengxr pushed a commit to mengxr/spark that referenced this pull request Nov 26, 2014
@tdas
[SPARK-4196][SPARK-4602][Streaming] Fix serialization issue in PairDS…
a9944c8 
…treamFunctions.saveAsNewAPIHadoopFiles

Solves two JIRAs in one shot
- Makes the ForechDStream created by saveAsNewAPIHadoopFiles serializable for checkpoints
- Makes the default configuration object used saveAsNewAPIHadoopFiles be the Spark's hadoop configuration

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes apache#3457 from tdas/savefiles-fix and squashes the following commits:

bb4729a [Tathagata Das] Same treatment for saveAsHadoopFiles
b382ea9 [Tathagata Das] Fix serialization issue in PairDStreamFunctions.saveAsNewAPIHadoopFiles.

(cherry picked from commit 8838ad7)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
@asfgit asfgit closed this in 8838ad7 Nov 26, 2014
asfgit pushed a commit that referenced this pull request Nov 26, 2014
@tdas
[SPARK-4196][SPARK-4602][Streaming] Fix serialization issue in PairDS…
7aa592c 
…treamFunctions.saveAsNewAPIHadoopFiles

Solves two JIRAs in one shot
- Makes the ForechDStream created by saveAsNewAPIHadoopFiles serializable for checkpoints
- Makes the default configuration object used saveAsNewAPIHadoopFiles be the Spark's hadoop configuration

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes #3457 from tdas/savefiles-fix and squashes the following commits:

bb4729a [Tathagata Das] Same treatment for saveAsHadoopFiles
b382ea9 [Tathagata Das] Fix serialization issue in PairDStreamFunctions.saveAsNewAPIHadoopFiles.

(cherry picked from commit 8838ad7)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@tdas @srowen @SparkQA @AmplabJenkins