Unable to get scalar clone to use test cache-server

[deshaw-ms]

• I was not able to find an open
  or closed issue matching
  what I'm seeing, including in the git-for-windows/git tracker.

Setup

• Which version of microsoft/git are you using? Is it 32-bit or 64-bit?

$ git --version --build-options

git version 2.36.1.vfs.0.0
cpu: x86_64
built from commit: c6886cdb2dd3bc5a365e74d2c5fa15fb93ff8342
sizeof-long: 4
sizeof-size_t: 8
shell-path: /bin/sh
feature: fsmonitor--daemon

Are you using Scalar or VFS for Git?

Yes, Scalar.

$ scalar version
git version 2.36.1.vfs.0.0

If VFS for Git, then what version?

$ gvfs version
GVFS 1.0.21014.1

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

Microsoft Windows [Version 10.0.22000.739]

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

GVFS Proxy setup as a VM. Windows Server 2019. ADO Server. Response from cache server url/gvfs/info

{
  "version": "18.181.32404.7",
  "serverState": "Ready",
  "maintenanceServiceStatus": "Running",
  "currentRequests": 1,
  "repos": [
    {
      "repoId": "49b0c9f4-555f-4624-8157-a57e6df513b3",
      "proxyUser": "ProxyService\\4030e0bc-a40a-4855-9915-712eb44836d6",
      "state": "None",
      "fetchFrequency": "00:05:00",
      "fetchProgress": []
    }
  ],
  "totalPackResponseHits": 2,
  "totalPackResponseMisses": 0,
  "totalLooseObjectMemoryCacheHits": 0,
  "totalLooseObjectMemoryCacheMisses": 7,
  "totalLooseObjectMemoryCacheBytes": 1579329,
  "totalObjectSizeMemoryCacheHits": 0,
  "totalObjectSizeMemoryCacheMisses": 0,
  "totalObjectSizeMemoryCacheBytes": 0,
  "totalLooseObjectDiskHits": 0,
  "totalLooseObjectDiskMisses": 7,
  "totalLooseObjectRemoteMisses": 0,
  "totalLooseObjectDiskCacheBytes": 14413824,
  "currentPrefetchRequests": 0,
  "currentSingleObjectRequests": 0,
  "currentMultiObjectRequests": 0,
  "currentObjectSizesRequests": 0
}

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

Git Bash, or CMD.

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

scalar clone --local-cache-path=/d/other_repos/lrt/scalar_cache/1 --cache-server-url=https://test-1js-proxy/49b0c9f4-555f-4624-8157-a57e6df513b3 https://dev.azure.com/office/office/_git/1JS

• What did you expect to occur after running these commands?

Expected the cache server to be used, to make the perf reasonable.

• What actually happened instead?

Instead its falling back on the origin url. There are some really long delays as well between outputting the cache-server url and starting the prefetch ( which also takes a long time, but thats another issue ).

I noticed the ssl cert error in the trace2 log (attached). The cert for the cache server is registered on the machine I am trying to clone from as a Trusted Root Cert. I can browse the server with Edge and hit the gvfs/info point without getting any cert errors. Name resolution only works because I added it to my hosts.ini file locally.

You can also see the long delays in the perf trace attached as well.

Full command response:

$ time scalar clone --local-cache-path=/d/other_repos/lrt/scalar_cache/1 --cache-server-url=https://test-1js-proxy/49b0c9f4-555f-4624-8157-a57e6df513b3 https://dev.azure.com/office/office/_git/1JS
Initialized empty Git repository in D:/other_repos/lrt/sc_proxy/1JS/src/.git/
Receiving vsts/info (bytes received):   0% (0/89555), done.
Receiving vsts/info (bytes received): 100% (824/824), done.
Cache server URL: https://test-1js-proxy/49b0c9f4-555f-4624-8157-a57e6df513b3
Prefetch 0 (1970-01-01 00:00:00 +0000) (bytes received): 9486622286, done.
Indexing objects: 100% (2572567/2572567), done.
Resolving deltas: 100% (1543167/1543167), done.
warning: fetch normally indicates which branches had a forced update,
but that check has been disabled; to re-enable, use '--show-forced-updates'
flag or run 'git config fetch.showForcedUpdates true'
Receiving packfile 1/1 with 14 objects (bytes received): 177073, done.
Indexing objects: 100% (14/14), done.
Switched to a new branch 'main'
branch 'main' set up to track 'origin/main'.
Your branch is up to date with 'origin/main'.

real    18m31.981s
user    0m0.000s
sys     0m0.046s

• If the problem was occurring with a specific repository, can you specify
  the repository?

  • Public repo: insert URL here
  • Windows monorepo
  • Office monorepo
  • Other Microsoft-internal repo: 1JS
  • Other internal repo.

git_perf.log

[derrickstolee]

@jeffhostetler If you get a chance, do you know if there is a special way to pass a flag to curl to allow self-signed certs? Do you know how hard it would be to add such a flag in the gvfs-helper layer (behind a config option)?

[deshaw-ms]

This actually gave me a clue. From git config --list

http.sslbackend=openssl
http.sslcainfo=C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt

Found this: How to trust self-signed certificate in cURL command line?

Will give the second answer a try.

[dscho]

A better alternative might be to switch to http.sslbackend=schannel, which will use Secure Channel, and when you tell Windows to trust the self-signed certificate in your browser, Git (via Secure Channel) should respect that.

[jeffhostetler]

Yeah, I always use schannel.

[deshaw-ms]

Ill give that a shot. Tried to get the self signed cert to be happy by creating a CA, but was not having any luck. If that does not work might try passing -k to curl for a test locally. If that works maybe put it behind an option for scalar. Like --test or something?

[dscho]

Tried to get the self signed cert to be happy by creating a CA, but was not having any luck.

Even by editing /mingw64/ssl/certs/ca-bundle.crt directly (this is assuming that you're still using http.sslBackend=openssl)?

f that does not work might try passing -k to curl for a test locally. If that works maybe put it behind an option for scalar.

For testing, that might be fine, and there is even a way to tell Git to not verify the SSL certificate, but it is a highly-discouraged practice.

[deshaw-ms]

As it turns out. Setting the backend to schannel works great if your under a windows based shell. Turns out I was running under mingw64/bash. schannel looks in the ca-budle.crt, and not the build in windows repository for certs. I did not add it to the /mingw64/ssl cert as that was tricky and yes, a hack. Instead just running from the windows shell worked fine as schannel used the windows cert store, that I had copied the cert from the server to for this test.

The issue is not with GIT actually, but with the use of CURL inside gvfs-helper. I debugged this a little. But once I figured out how to work around it all is good. It was basically my lack of knowledge around the differences in the impls of CURL from linux vs windows I think. Thanks @dscho for the pointer to using schannel and the fact that it auths correctly on windows.

Personally it might be nice to have the CURL usage in gvfs-helper look at the git setting to NOT verify the cert and add the -k option to the CURL call? Or some param equivalent? If folks think that might be worth doing happy to look at making a change and floating a PR.

[derrickstolee]

@deshaw-ms: wow, thanks for digging in so deeply here. I'd love to see that contribution if you have everything fresh in your mind.

The one thing I will say is that Git 2.37.0 is about to come out and we will switch our default branch from vfs-2.36.1 to vfs-2.37.0 soon. You might want to wait for the next release out of this repo before starting that contribution.

[dscho]

I find it strange that there is a difference of behavior between using Git Bash or Git CMD. The http.schannelUseSSLCAInfo config setting is supposed to tell Git to ask cURL not to look at that file and instead at the Windows Certificate Store. No matter from where you call git.exe.

So I think that we will need to find out more before we can think about ways to solve the problem, as we seem not to have gotten quite to the root cause yet.