Skip to content

add some shuffle configurations in doc #2031

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
CrazyJvm wants to merge 5 commits into from
Closed

add some shuffle configurations in doc #2031

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
49b47f0
add configuration `spark.shuffle.spill.batchSize` and fix the value o…
CrazyJvm Aug 19, 2014
8ad9955
fix typo
CrazyJvm Aug 19, 2014
876256d
add spark.shuffle.safetyFraction in doc
CrazyJvm Aug 19, 2014
181c512
the config value of is case irrelevant
CrazyJvm Aug 20, 2014
117e9bf
style fix
CrazyJvm Aug 21, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -256,6 +256,14 @@ Apart from these, the following properties are also available, and may be useful
<code>spark.storage.memoryFraction</code>.
</td>
</tr>
<tr>
<td><code>spark.shuffle.safetyFraction</code></td>
<td>0.8</td>
<td>
An additional margin of safety fraction of Java heap to use for aggregation and cogroups during
shuffles, in case the size estimation of maps used for shuffle is not sufficiently accurate.
</td>
</tr>
<tr>
<td><code>spark.shuffle.compress</code></td>
<td>true</td>
Expand Down Expand Up @@ -286,11 +294,18 @@ Apart from these, the following properties are also available, and may be useful
<td>HASH</td>
<td>
Implementation to use for shuffling data. A hash-based shuffle manager is the default, but
starting in Spark 1.1 there is an experimental sort-based shuffle manager that is more
starting in Spark 1.1 there is an experimental sort-based shuffle manager that is more
memory-efficient in environments with small executors, such as YARN. To use that, change
this value to <code>SORT</code>.
</td>
</tr>
<tr>
<td><code>spark.shuffle.spill.batchSize</code></td>
<td>10000</td>
<td>
Size of object batches when reading/writing from serializers.
</td>
</tr>
<tr>
<td><code>spark.shuffle.sort.bypassMergeThreshold</code></td>
<td>200</td>
Expand Down