From 9a832da21917091ab4e7b7e8de1fd049f84fd9d0 Mon Sep 17 00:00:00 2001 From: James Lohse Date: Sun, 24 Jan 2016 12:07:53 -0700 Subject: [PATCH 1/3] Provide same info as in spark-submit --help this is stated for --packages and --repositories. Without stating it for --jars, people expect a standard java classpath to work, with expansion and using a different delimiter than a comma. Currently this is only state in the --help for spark-submit "Comma-separated list of local jars to include on the driver and executor classpaths." --- docs/submitting-applications.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/submitting-applications.md b/docs/submitting-applications.md index acbb0f298fe47..cb02f5706a8fa 100644 --- a/docs/submitting-applications.md +++ b/docs/submitting-applications.md @@ -177,8 +177,9 @@ debugging information by running `spark-submit` with the `--verbose` option. # Advanced Dependency Management When using `spark-submit`, the application jar along with any jars included with the `--jars` option -will be automatically transferred to the cluster. Spark uses the following URL scheme to allow -different strategies for disseminating jars: +will be automatically transferred to the cluster. URLs supplied after --jars must be separated by commas. Each entry points to a specific jar file, resulting in a comma-separated list of local jars. That list is included on the driver and executor classpaths. Directory expansion does not work with --jars. + +Spark uses the following URL scheme to allow different strategies for disseminating jars: - **file:** - Absolute paths and `file:/` URIs are served by the driver's HTTP file server, and every executor pulls the file from the driver HTTP server. From d00f5c6ec45ffe7aa521faff0d52766095f48d44 Mon Sep 17 00:00:00 2001 From: James Lohse Date: Sun, 24 Jan 2016 12:35:24 -0700 Subject: [PATCH 2/3] removed sentence that was repetitious thanks for the feedback, good point. Re: verifying this, I would be interested to see what you come up with, in my implementation, sitting with my boss who has years of Java experience, he kept trying to use a standard classpath, which didn't work. We also discovered directory expansion doesn't work. --- docs/submitting-applications.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/submitting-applications.md b/docs/submitting-applications.md index cb02f5706a8fa..69b29d3d19575 100644 --- a/docs/submitting-applications.md +++ b/docs/submitting-applications.md @@ -177,7 +177,7 @@ debugging information by running `spark-submit` with the `--verbose` option. # Advanced Dependency Management When using `spark-submit`, the application jar along with any jars included with the `--jars` option -will be automatically transferred to the cluster. URLs supplied after --jars must be separated by commas. Each entry points to a specific jar file, resulting in a comma-separated list of local jars. That list is included on the driver and executor classpaths. Directory expansion does not work with --jars. +will be automatically transferred to the cluster. URLs supplied after `--jars` must be separated by commas. That list is included on the driver and executor classpaths. Directory expansion does not work with --jars. Spark uses the following URL scheme to allow different strategies for disseminating jars: From 73bf7e965a4f60be282f1a6e97c62114f28846d3 Mon Sep 17 00:00:00 2001 From: James Lohse Date: Sun, 24 Jan 2016 12:36:21 -0700 Subject: [PATCH 3/3] missed backticking one --jars --- docs/submitting-applications.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/submitting-applications.md b/docs/submitting-applications.md index 69b29d3d19575..413532f2f6cfa 100644 --- a/docs/submitting-applications.md +++ b/docs/submitting-applications.md @@ -177,7 +177,7 @@ debugging information by running `spark-submit` with the `--verbose` option. # Advanced Dependency Management When using `spark-submit`, the application jar along with any jars included with the `--jars` option -will be automatically transferred to the cluster. URLs supplied after `--jars` must be separated by commas. That list is included on the driver and executor classpaths. Directory expansion does not work with --jars. +will be automatically transferred to the cluster. URLs supplied after `--jars` must be separated by commas. That list is included on the driver and executor classpaths. Directory expansion does not work with `--jars`. Spark uses the following URL scheme to allow different strategies for disseminating jars: