Skip to content

Commit e8db144

Browse files
committed
Note that yarn-{client,cluster} are discouraged and fix one more instance of yarn-client
1 parent 4edcaff commit e8db144

File tree

2 files changed

+5
-3
lines changed

2 files changed

+5
-3
lines changed

docs/running-on-yarn.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -390,6 +390,6 @@ If you need a reference to the proper location to put log files in the YARN so t
390390
# Important notes
391391

392392
- Whether core requests are honored in scheduling decisions depends on which scheduler is in use and how it is configured.
393-
- In `cluster` mode, the local directories used by the Spark executors and the Spark driver will be the local directories configured for YARN (Hadoop YARN config `yarn.nodemanager.local-dirs`). If the user specifies `spark.local.dir`, it will be ignored. In `yarn-client` mode, the Spark executors will use the local directories configured for YARN while the Spark driver will use those defined in `spark.local.dir`. This is because the Spark driver does not run on the YARN cluster in `yarn-client` mode, only the Spark executors do.
393+
- In `cluster` mode, the local directories used by the Spark executors and the Spark driver will be the local directories configured for YARN (Hadoop YARN config `yarn.nodemanager.local-dirs`). If the user specifies `spark.local.dir`, it will be ignored. In `client` mode, the Spark executors will use the local directories configured for YARN while the Spark driver will use those defined in `spark.local.dir`. This is because the Spark driver does not run on the YARN cluster in `client` mode, only the Spark executors do.
394394
- The `--files` and `--archives` options support specifying file names with the # similar to Hadoop. For example you can specify: `--files localtest.txt#appSees.txt` and this will upload the file you have locally named `localtest.txt` into HDFS but this will be linked to by the name `appSees.txt`, and your application should use the name as `appSees.txt` to reference it when running on YARN.
395395
- The `--jars` option allows the `SparkContext.addJar` function to work if you are using it with local files and running in `cluster` mode. It does not need to be used if you are using it with HDFS, HTTP, HTTPS, or FTP files.

docs/submitting-applications.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -137,9 +137,11 @@ The master URL passed to Spark can be in one of the following formats:
137137
<code>client</code> or <code>cluster</code> mode depending on the value of <code>--deploy-mode</code>.
138138
The cluster location will be found based on the <code>HADOOP_CONF_DIR</code> or <code>YARN_CONF_DIR</code> variable.
139139
</td></tr>
140-
<tr><td> <code>yarn-client</code> </td><td> Equivalent to <code>yarn</code> with <code>--deploy-mode client</code>
140+
<tr><td> <code>yarn-client</code> </td><td> Equivalent to <code>yarn</code> with <code>--deploy-mode client</code>,
141+
which is preferred to `yarn-client`
141142
</td></tr>
142-
<tr><td> <code>yarn-cluster</code> </td><td> Equivalent to <code>yarn</code> with <code>--deploy-mode cluster</code>
143+
<tr><td> <code>yarn-cluster</code> </td><td> Equivalent to <code>yarn</code> with <code>--deploy-mode cluster</code>,
144+
which is preferred to `yarn-cluster`
143145
</td></tr>
144146
</table>
145147

0 commit comments

Comments
 (0)