Git Bash SCP command is slow

[YngveNPettersen]

• [ x] I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

git version 2.17.1.windows.2
cpu: x86_64
built from commit: a60968cf435951d9411fc0f980a2e362d5cccea2
sizeof-long: 4

Note: I am still using 2.17 due to #2218 , which is still not fixed, either by Git for Windows, or the core Git team.

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

Windows 10; some installations are from this week, fully up to date.

• What options did you set as part of the installation? Or did you choose the
  defaults?

Editor Option: VIM
Path Option: BashOnly
Plink Path: C:\Program Files\PuTTY\plink.exe
SSH Option: Plink
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Performance Tweaks FSCache: Enabled
Use Credential Manager: Disabled
Enable Symlinks: Disabled

NOTE: Originally Putty was used, but due to even worse performance than Git Bash SCP, it was changed to use OpenSSH instead; some of the machines used no longer have Putty installed.

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

Much of recent testing was done on clean Win10 installs

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

Git Bash

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

scp <remote-location> <local-location>

• What did you expect to occur after running these commands?

When downloading from a Linux host on the same local network, often same switch, and definitely in the same room, download speed should be at least 1 Gigabit per second (100 MB/s), and only limited by network speed and/or distance

• What actually happened instead?

For same-network downloads max download speed was at most 7-8 MB/s (>65 Mbit/s), some tests might reach 9 MB/s. Some of the machines have 10 Gbps network connections.

Upload speeds were even slower (5-6 MB/s (~50 Mbit/s).

Using Windows 10 OpenSSH SCP achieved 100 MB/s (>800 Mbit/s) download speed on the same machine, same remote host, same file.

• If the problem was occurring with a specific repository, can you provide the
  URL to that repository to help us with testing?

N/A

NOTE: I suspect this also affects Git cloning speeds with OpenSSH.

It is possible that this is an upstream issue in OpenSSH, and that Microsoft patched the Win10 version themselves.

I have previously worked on Windows network code (in the Opera (Presto) browser, v12 and earlier), and as I recall, to get max network performance on Windows the code need to do the following (assumes async sockets):

• Receive and buffer all pending data from the socket, read non-blocking until 0 bytes is received . DO NOT start processing the data, just stash them into a buffer, and read again (tips: Never reallocate buffers to make more space, just create a new buffer in a chain of buffers; also read data directly to the target buffer, save memcpys)
• After all pending data are received, processing can start; but be ready to handle arriving data immediately.
• When sending data, continue sending data to the socket until it is no longer accepting data, then wait until the socket signals it is ready to send more data
• Buffers in either direction should be at least 128 KB (default in Opera from 2012), but maybe 1 MB is better today; I recommend testing with a machine with a direct 10+ Gbps network connection.

The reason for this, as I recall, is that Windows sockets does not send an ACK before the data have been read by the application (this may be the case on Linux as well, but it seems apps there are behaving differently wrt reading data). As data continues to be read quickly, the sender will push more and more data into the connection pipeline, increasing the transfer speed up to the limit of the connection. The same applies for sending data.

For reference, we are planning to start using the Windows 10 SCP application to work around this issue.

[YngveNPettersen]

We just deployed Git for Windows 2.30 on these machines: Download and upload of files now take at least twice the time v2.17 used

[dscho]

It is possible that this is an upstream issue in OpenSSH

We're not patching OpenSSH beyond what you see in https://github.com/git-for-windows/MSYS2-packages/tree/HEAD/openssh.

[fredkotze]

Same problem. It seems to max out at just over 4MB/s. It seems to definitely be related to MSYS2.
msys2/MSYS2-packages#2032

[dscho]

The puzzle might be solved by the difference between the OpenSSH that is in C:\WINDOWS\System32\OpenSSH and the MSYS2 version used by Git for Windows.

The source code for the former is hosted here (build instructions here: https://github.com/PowerShell/Win32-OpenSSH/wiki/Building-OpenSSH-for-Windows-(using-LibreSSL-crypto)), the latter's source code is constructed from the official openssh-portable source archives, plus a couple custom patches: https://github.com/git-for-windows/MSYS2-packages/tree/main/openssh

[Okeanos]

Once a Git build was released that contains git-for-windows/build-extra/pull/367 you can make Git used Window OpenSSH and related binaries including Windows scp. Has implications on existing setups, though, because SSH setups that rely on the Git unix-like sockets won't work anymore.

[dscho]

Since Git for Windows v2.33.0 was released with the option to prefer the system OpenSSH, I'll close this ticket.