diff --git a/docs/configuration.md b/docs/configuration.md index f292bfbb7dcd6..984365748219f 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -864,6 +864,47 @@ Apart from these, the following properties are also available, and may be useful between nodes leading to flooding the network with those. + + spark.shuffle.io.mode + NIO + + (Netty only) IO mode for channel implementation. If you set NIO, pure java socket is used for the channel. + Else if you set EPOLL, higher performance native socket is used through JNI. + + + + spark.shuffle.io.backLog + -1 + + (Netty only) Maximum length of the queue incomming connections. + If this value <= 0, the default length depending on your platform is used. + 200 is set for Windows, and 128 is set for others. + Especially, your platform has /proc/sys/net/core/somaxconn, + the value recorded in this file is preferred. + + + + spark.shuffle.io.receiveBuffer + -1 + + (Netty only) Receive buffer size for transportation. + If this value <= 0, the default value depending on your platform and + IO mode represented by spark.shuffle.io.mode is used. + The optimal size for receive buffer should be + latency * network_bandwidth. Assuming latency = 1ms, network_bandwidth = 10Gbps, + buffer size should be ~ 1.25MB. + + + + spark.shuffle.io.sendBuffer + -1 + + (Netty only) Send buffer size for transportation. + If this value <= 0, the default value is set as with spark.shuffle.io.receiveBuffer. + As well as receive buffer, the optimal size for send buffer should be + considered latency and network bandwidth. + + spark.shuffle.io.preferDirectBufs true