This repository has been archived by the owner on Jan 9, 2020. It is now read-only.
Allow specifying non-local files to spark-submit (python files, and R files) #530
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… files) when isKubernetes is set (apache-spark-on-k8s#527)
|
Good catch, thank you for this. I seem to have missed this in my PRs. This LGTM seeing as CLI is passing. |
…uster mode for spark-submit
|
@felixcheung not sure, tbh. The intent seems to be that |
|
Thanks, I guess we should test this. Is there a way to call out what should be tested?
|
|
rerun integration tests please |
ifilonenko
reviewed
Oct 19, 2017
| PythonRunner.formatPaths(resolvedPyFiles).mkString(",") | ||
| } else { | ||
| // Ignoring formatting python path in yarn and mesos cluster mode, these two modes | ||
| // Ignoring formatting python path in yarn, mesos, and kubernetes cluster mode, these two modes |
There was a problem hiding this comment.
line is too long. beyond 100 characters so it will fail scalastyle
|
This looks like a CI/build system error, unrelated to the changes, but I am not able to fully interpret it. |
|
rerun integration test please |
|
Any more comments on this and objection merging this? |
|
ok to merge when tests pass. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refers to issue #527, this allows the use of python files, R and also using
--py-files, when using spark-submit. Previously, the client would deny any non-local URI types when submitting a python job, even though the kubernetes spark initcontainer would be able to fulfill them (for example,gs://URIs when the GCS connector is present in the initcontainer image).Changing the validation to support this when isKubernetes is set, allows python jobs to use non-local URIs successfully. Only the client (
spark-submit) requires this change, existing initcontainer images work fine.What changes were proposed in this pull request?
adding
&& !isKubernetesClustertocore/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L328And also for the R files check.
As suggested by @liyinan926 in #527 (comment)
How was this patch tested?
Command:
Before the change, the error was, and the job did not start:
After the change, I ran
./dev/make-distribution.sh --pip --tgz -Pmesos -Pyarn -Pkinesis-asl -Phive -Phive-thriftserver -Pkubernetes -Phadoop-2.7 -Dhadoop.version=2.7.3locally on my macOS dev machine, and then ran it'sspark-submit, and I was able to submit my python job successfully and obtain results via the logs.