We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
There was an error while loading. Please reload this page.
1 parent bb8fecb commit 0b275cfCopy full SHA for 0b275cf
docs/configuration.md
@@ -185,7 +185,10 @@ of the most common options to set are:
185
<td>
186
The amount of memory to be allocated to PySpark in each executor, in MiB
187
unless otherwise specified. If set, PySpark memory for an executor will be
188
- limited to this amount. If not set, Spark will not limit Python's memory use.
+ limited to this amount. If not set, Spark will not limit Python's memory use
189
+ and it is up to the application to avoid exceeding the overhead memory space
190
+ shared with other non-JVM processes. When PySpark is run in YARN, this memory
191
+ is added to executor resource requests.
192
</td>
193
</tr>
194
<tr>
0 commit comments