diff --git a/docs/configuration.md b/docs/configuration.md index f292bfbb7dcd6..984365748219f 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -864,6 +864,47 @@ Apart from these, the following properties are also available, and may be useful between nodes leading to flooding the network with those. +
spark.shuffle.io.modeNIO, pure java socket is used for the channel.
+ Else if you set EPOLL, higher performance native socket is used through JNI.
+ spark.shuffle.io.backLog200 is set for Windows, and 128 is set for others.
+ Especially, your platform has /proc/sys/net/core/somaxconn,
+ the value recorded in this file is preferred.
+ spark.shuffle.io.receiveBufferspark.shuffle.io.mode is used.
+ The optimal size for receive buffer should be
+ latency * network_bandwidth. Assuming latency = 1ms, network_bandwidth = 10Gbps,
+ buffer size should be ~ 1.25MB.
+ spark.shuffle.io.sendBufferspark.shuffle.io.receiveBuffer.
+ As well as receive buffer, the optimal size for send buffer should be
+ considered latency and network bandwidth.
+ spark.shuffle.io.preferDirectBufs