Skip to content

Conversation

@scwf
Copy link
Contributor

@scwf scwf commented Dec 15, 2014

WriteAheadLogBackedBlockRDDSuite failed due to the same random file name(https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24456/consoleFull)

/tmp/1418645532799-0/??????????
[info] - Read data available only in write ahead log, and test storing in block manager *** FAILED *** (2 milliseconds)

Using another way to produce random file name(http://alvinalexander.com/scala/creating-random-strings-in-scala).

@scwf
Copy link
Contributor Author

scwf commented Dec 15, 2014

link to #3687

@SparkQA
Copy link

SparkQA commented Dec 15, 2014

Test build #24458 has started for PR 3701 at commit b7f8955.

  • This patch merges cleanly.

@srowen
Copy link
Member

srowen commented Dec 15, 2014

This is for SPARK-4826 right? can you explain the fix? is it that reserved characters like / or non-ASCII chars were part of the file name?

@scwf
Copy link
Contributor Author

scwf commented Dec 15, 2014

Yes, it is. I have not noticed that PR. Based on my test, this failure is caused by Random.nextString(10), it produce the same string each time when test(a debug link on jenkins: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24456/consoleFull) for unkonw reason. I am not sure this is caused by bugs of Random.nextString or race condition

@scwf scwf changed the title [Hot Fix] Fix WriteAheadLogBackedBlockRDDSuite [Hot Fix][Streaming] Fix WriteAheadLogBackedBlockRDDSuite Dec 15, 2014
@SparkQA
Copy link

SparkQA commented Dec 15, 2014

Test build #24458 has finished for PR 3701 at commit b7f8955.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24458/
Test PASSed.

@JoshRosen
Copy link
Contributor

I've opened a new PR with what I think is is a more direct approach to fixing this issue: #3704. Please take a look and let me know what you think.

asfgit pushed a commit that referenced this pull request Dec 15, 2014
This PR should fix SPARK-4826, an issue where a bug in how we generate temp. file names was causing spurious test failures in the write ahead log suites.

Closes #3695.
Closes #3701.

Author: Josh Rosen <joshrosen@databricks.com>

Closes #3704 from JoshRosen/SPARK-4826 and squashes the following commits:

f2307f5 [Josh Rosen] Use Spark Utils class for directory creation/deletion
a693ddb [Josh Rosen] remove unused Random import
b275e41 [Josh Rosen] Move creation of temp. dir to beforeEach/afterEach.
9362919 [Josh Rosen] [SPARK-4826] Fix bug in generation of temp file names. in WAL suites.
86c1944 [Josh Rosen] Revert "HOTFIX: Disabling failing block manager test"

(cherry picked from commit f6b8591)
Signed-off-by: Josh Rosen <joshrosen@databricks.com>
@asfgit asfgit closed this in f6b8591 Dec 15, 2014
@scwf scwf deleted the patch-10 branch December 17, 2014 15:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants