[SPARK-19617][SS]Fix the race condition when starting and stopping a query quickly (branch-2.1) #16979
Conversation
There was a problem hiding this comment.
This file is almost same as #16947 except this comment.
There was a problem hiding this comment.
Also run mkdirs into runUninterruptiblyIfLocal because it calls Shell.runCommand.
There was a problem hiding this comment.
Fixed the comment. I added it in 88c43f4 but it was wrong. We don't need to use runUninterruptibly to workaround HADOOP-14084. The root cause is HADOOP-10622.
|
Test build #73091 has finished for PR 16979 at commit
|
…query quickly (branch-2.1)
|
Test build #73101 has started for PR 16979 at commit |
|
retest this please |
|
Test build #73117 has finished for PR 16979 at commit
|
| } | ||
|
|
||
| private def runUninterruptiblyIfLocal[T](body: => T): T = { | ||
| if (fileManager.isLocalFileSystem && Thread.currentThread.isInstanceOf[UninterruptibleThread]) { |
There was a problem hiding this comment.
Have to change the condition here because StreamExecution will create a HDFSMetadata in a non UninterruptibleThread. (mkdirs)
There was a problem hiding this comment.
So we are changing this to a best-effort attempt, rather than the try-and-explicitly-fail attempt, in the case of a local file system... right?
|
|
||
| private def runUninterruptiblyIfLocal[T](body: => T): T = { | ||
| if (fileManager.isLocalFileSystem && Thread.currentThread.isInstanceOf[UninterruptibleThread]) { | ||
| // When using a local file system, some file system APIs like "create" or "mkdirs" must be |
There was a problem hiding this comment.
I fixed the comments to point to the root cause: HADOOP-10622.
|
LGTM. |
|
Thanks. Merging to 2.1. |
… query quickly (branch-2.1) ## What changes were proposed in this pull request? Backport #16947 to branch 2.1. Note: we still need to support old Hadoop versions in 2.1.*. ## How was this patch tested? Jenkins Author: Shixiong Zhu <shixiong@databricks.com> Closes #16979 from zsxwing/SPARK-19617-branch-2.1.
3 participants
What changes were proposed in this pull request?
Backport #16947 to branch 2.1. Note: we still need to support old Hadoop versions in 2.1.*.
How was this patch tested?
Jenkins