[SPARK-26895][CORE] prepareSubmitEnvironment should be called within doAs for proxy users #23806

abellina · 2019-02-15T23:32:17Z

What changes were proposed in this pull request?

prepareSubmitEnvironment performs globbing that will fail in the case where a proxy user (--proxy-user) doesn't have permission to the file. This is a bug also with 2.3, so we should backport, as currently you can't launch an application that for instance is passing a file under --archives, and that file is owned by the target user.

The solution is to call prepareSubmitEnvironment within a doAs context if proxying.

How was this patch tested?

Manual tests running with --proxy-user and --archives, before and after, showing that the globbing is successful when the resource is owned by the target user.

I've looked at writing unit tests, but I am not sure I can do that cleanly (perhaps with a custom FileSystem). Open to ideas.

Please review http://spark.apache.org/contributing.html before opening a pull request.

…r proxy users

abellina · 2019-02-16T20:21:09Z

@jerryshao pinging you as the feature was introduced here: #18235.

Also pinging reviewers @vanzin, @cloud-fan, @jiangxb1987. Thanks in advance.

abellina · 2019-02-18T15:24:03Z

ok to test

vanzin · 2019-02-19T21:44:45Z

ok to test

vanzin · 2019-02-19T21:47:23Z

BTW your previous PR title was better. Describe the fix, not the problem.

abellina · 2019-02-19T22:39:17Z

@vanzin will update it. Thanks

SparkQA · 2019-02-20T02:48:42Z

Test build #102518 has finished for PR 23806 at commit 224d489.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

abellina · 2019-02-20T14:55:08Z

Hmm, python tests failed. Not sure if related though. The tests point at the /home/jenkins/workspace/SparkPullRequestBuilder/python/unit-tests.log file, but I can't get to that from jenkins UI it seems (are all prs writing to the same file?). Is this a known issue? Can we move it to the artifacts? (created https://issues.apache.org/jira/browse/SPARK-26944 for this)

From the error, it seems like the shmutil.rmtree function is trying to delete a directory, but there's likely another thread adding entries to a directory, so when it gets to os.rmdir(path) it blows up. I think the test should call q.awaitTermination after q.stop, before going on. (filed this one for the streaming issue https://issues.apache.org/jira/browse/SPARK-26945)

ERROR: test_query_manager_await_termination (pyspark.sql.tests.test_streaming.StreamingTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/sql/tests/test_streaming.py", line 259, in test_query_manager_await_termination
    shutil.rmtree(tmpPath)
  File "/home/anaconda/lib/python2.7/shutil.py", line 256, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/home/anaconda/lib/python2.7/shutil.py", line 254, in rmtree
    os.rmdir(path)
OSError: [Errno 39] Directory not empty: '/home/jenkins/workspace/SparkPullRequestBuilder/python/target/072153bd-f981-47be-bda2-e2b657a16f65/tmp4WGp7n'

abellina · 2019-02-21T14:28:00Z

ok to test

vanzin · 2019-02-21T17:20:46Z

retest this please

SparkQA · 2019-02-21T23:42:19Z

Test build #102590 has finished for PR 23806 at commit 224d489.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2019-02-22T19:14:54Z

Merging to master.

…tEnvironment` in SparkSubmit ## What changes were proposed in this pull request? Currently, if I run `spark-shell` in my local, it started to show the logs as below: ``` $ ./bin/spark-shell ... 19/02/28 04:42:43 INFO SecurityManager: Changing view acls to: hkwon 19/02/28 04:42:43 INFO SecurityManager: Changing modify acls to: hkwon 19/02/28 04:42:43 INFO SecurityManager: Changing view acls groups to: 19/02/28 04:42:43 INFO SecurityManager: Changing modify acls groups to: 19/02/28 04:42:43 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hkwon); groups with view permissions: Set(); users with modify permissions: Set(hkwon); groups with modify permissions: Set() 19/02/28 04:42:43 INFO SignalUtils: Registered signal handler for INT 19/02/28 04:42:48 INFO SparkContext: Running Spark version 3.0.0-SNAPSHOT 19/02/28 04:42:48 INFO SparkContext: Submitted application: Spark shell 19/02/28 04:42:48 INFO SecurityManager: Changing view acls to: hkwon ``` Seems to be the cause is #23806 and `prepareSubmitEnvironment` looks actually reinitializing the logging again. This PR proposes to uninitializing log later after `prepareSubmitEnvironment`. ## How was this patch tested? Manually tested. Closes #23911 from HyukjinKwon/SPARK-26895. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>

…lled within doAs for proxy users `prepareSubmitEnvironment` performs globbing that will fail in the case where a proxy user (`--proxy-user`) doesn't have permission to the file. This is a bug also with 2.3, so we should backport, as currently you can't launch an application that for instance is passing a file under `--archives`, and that file is owned by the target user. The solution is to call `prepareSubmitEnvironment` within a doAs context if proxying. Manual tests running with `--proxy-user` and `--archives`, before and after, showing that the globbing is successful when the resource is owned by the target user. I've looked at writing unit tests, but I am not sure I can do that cleanly (perhaps with a custom FileSystem). Open to ideas. Please review http://spark.apache.org/contributing.html before opening a pull request. Closes apache#23806 from abellina/SPARK-26895_prepareSubmitEnvironment_from_doAs. Lead-authored-by: Alessandro Bellina <abellina@gmail.com> Co-authored-by: Alessandro Bellina <abellina@yahoo-inc.com> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com> (cherry picked from commit 79a6504) Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>

…tEnvironment` in SparkSubmit Currently, if I run `spark-shell` in my local, it started to show the logs as below: ``` $ ./bin/spark-shell ... 19/02/28 04:42:43 INFO SecurityManager: Changing view acls to: hkwon 19/02/28 04:42:43 INFO SecurityManager: Changing modify acls to: hkwon 19/02/28 04:42:43 INFO SecurityManager: Changing view acls groups to: 19/02/28 04:42:43 INFO SecurityManager: Changing modify acls groups to: 19/02/28 04:42:43 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hkwon); groups with view permissions: Set(); users with modify permissions: Set(hkwon); groups with modify permissions: Set() 19/02/28 04:42:43 INFO SignalUtils: Registered signal handler for INT 19/02/28 04:42:48 INFO SparkContext: Running Spark version 3.0.0-SNAPSHOT 19/02/28 04:42:48 INFO SparkContext: Submitted application: Spark shell 19/02/28 04:42:48 INFO SecurityManager: Changing view acls to: hkwon ``` Seems to be the cause is apache#23806 and `prepareSubmitEnvironment` looks actually reinitializing the logging again. This PR proposes to uninitializing log later after `prepareSubmitEnvironment`. Manually tested. Closes apache#23911 from HyukjinKwon/SPARK-26895. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com> (cherry picked from commit 6e31ccf)

SPARK-26895: prepareSubmitEnvironment should be called within doAs fo…

f082fc6

…r proxy users

abellina force-pushed the SPARK-26895_prepareSubmitEnvironment_from_doAs branch from 71f8f26 to f082fc6 Compare February 16, 2019 03:54

SPARK-26895: update comments

224d489

abellina changed the title ~~SPARK-26895: prepareSubmitEnvironment should be called within doAs fo…~~ [SPARK-26895][CORE]: prepareSubmitEnvironment should be called within doAs fo… Feb 16, 2019

abellina changed the title ~~[SPARK-26895][CORE]: prepareSubmitEnvironment should be called within doAs fo…~~ [SPARK-26895][CORE] prepareSubmitEnvironment should be called within doAs fo… Feb 16, 2019

abellina marked this pull request as ready for review February 16, 2019 20:18

abellina changed the title ~~[SPARK-26895][CORE] prepareSubmitEnvironment should be called within doAs fo…~~ [SPARK-26895][CORE] proxy-users issue submitting applications with archives and other resources Feb 19, 2019

abellina changed the title ~~[SPARK-26895][CORE] proxy-users issue submitting applications with archives and other resources~~ [SPARK-26895][CORE] prepareSubmitEnvironment should be called within doAs for proxy users Feb 19, 2019

vanzin closed this in 79a6504 Feb 22, 2019

HyukjinKwon mentioned this pull request Feb 27, 2019

[SPARK-26895][CORE][FOLLOW-UP] Uninitializing log after prepareSubmitEnvironment in SparkSubmit #23911

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-26895][CORE] prepareSubmitEnvironment should be called within doAs for proxy users #23806

[SPARK-26895][CORE] prepareSubmitEnvironment should be called within doAs for proxy users #23806

Uh oh!

abellina commented Feb 15, 2019 •

edited

Loading

Uh oh!

abellina commented Feb 16, 2019 •

edited

Loading

Uh oh!

abellina commented Feb 18, 2019

Uh oh!

vanzin commented Feb 19, 2019

Uh oh!

vanzin commented Feb 19, 2019

Uh oh!

abellina commented Feb 19, 2019

Uh oh!

SparkQA commented Feb 20, 2019

Uh oh!

abellina commented Feb 20, 2019 •

edited

Loading

Uh oh!

abellina commented Feb 21, 2019

Uh oh!

vanzin commented Feb 21, 2019

Uh oh!

SparkQA commented Feb 21, 2019

Uh oh!

vanzin commented Feb 22, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[SPARK-26895][CORE] prepareSubmitEnvironment should be called within doAs for proxy users #23806

[SPARK-26895][CORE] prepareSubmitEnvironment should be called within doAs for proxy users #23806

Uh oh!

Conversation

abellina commented Feb 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

abellina commented Feb 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

abellina commented Feb 18, 2019

Uh oh!

vanzin commented Feb 19, 2019

Uh oh!

vanzin commented Feb 19, 2019

Uh oh!

abellina commented Feb 19, 2019

Uh oh!

SparkQA commented Feb 20, 2019

Uh oh!

abellina commented Feb 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

abellina commented Feb 21, 2019

Uh oh!

vanzin commented Feb 21, 2019

Uh oh!

SparkQA commented Feb 21, 2019

Uh oh!

vanzin commented Feb 22, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

abellina commented Feb 15, 2019 •

edited

Loading

abellina commented Feb 16, 2019 •

edited

Loading

abellina commented Feb 20, 2019 •

edited

Loading