sshfs is much slower than rclone mount over high-latency connections due to lack of SFTP pipelining

[nh2]

I know that sshfs is not currently actively developed, but writing this down here because so many people have had problems with this topic and it is not properly documented here in the upstream project.

Observation

cating a file over sshfs can be 100x slower over rclone mount with rclone's SFTP backend over high-latency connectinons (servers far away).

For example, on a 200 ms connection, pv < ssh-mount/myfile > /dev/null only prouces 1 MB/s when the connection shows 100 MB/s over TCP (measured with iperf3).

Reason

• The SSH protocol specifies a minimum packet size that all implementations must support.
• OpenSSH implements a maximum packet size of 256 KiB.
  • See link above and #define PACKET_MAX_SIZE (256 * 1024)
  • This is quite annoying because it effectively disables TCP Window Scaling, which was designed to provide high throughput over high-latency TCP links, by allowing more data to be in flight:
    • While the TCP would allow more data to be in flight, the OpenSSH implementation will just not send more data.
• The SFTP protocol has a mitigation for this:

  In general, this protocol follows a simple request-response model.
  Each request and response contains a sequence number and multiple requests may be pending simultaneously.
  spec source

  • This concept is often called pipelining.
• rclone implements pipelining with its SFTP concurrency setting, defaulting to 64.
  • Implemented by the Go library pkg/sftp in this code:

    // Split the read into multiple maxPacket-sized concurrent reads bounded by maxConcurrentRequests.
    // This allows writes with a suitably large buffer to transfer data at a much faster rate
    // by overlapping round trip times.

sshfs does not seem to implement request pipelining, thus is is much slower.