New "info: detecting host provider for 'https://example.com'..." message after upgrading to git version 2.33.1.windows.1

[gpakosz]

Hello,

After upgrading to Git For Windows version 2.33.1, we started seeing new messages on stderr as well as slow downs.

gpakosz@DES-00703 MINGW64 /d/Projects/xxx (master)
$ git --version
git version 2.33.1.windows.1

gpakosz@DES-00703 MINGW64 /d/Projects/xxx (master)
$ git fetch -p
info: detecting host provider for 'https://scm.corp.example.com/'...
info: detecting host provider for 'https://scm.corp.example.com/'...
info: detecting host provider for 'https://scm.corp.example.com/'...

Even though I landed on https://github.com/microsoft/Git-Credential-Manager-Core/blob/main/docs/autodetect.md, I still don't understand what this auto-detection of host provider is for.

Can you please elaborate on why the core credential manager needs to be aware of the application running on the remote server? What's the link with the Git protocol and authentication?

We also noticed the portable version of Git which uses git-credential-wincred.exe doesn't exhibit this behavior while working perfectly.

Using

git config --global credential.scm.corp.example.com.provider generic

or

git config --global credential.scm.corp.example.com.provider bitbucket

doesn't seem to end up with other observable side effect than the suppression of the messages.

Finally note that our remote repositories are handled by an on premise Bitbucket Server instance which is behind a reverse proxy. A HEAD request exhibits the X-AREQUESTID header you're looking for:

curl -I https://scm.corp.example.com
HTTP/1.1 302
Server: nginx/1.14.2
Date: Thu, 14 Oct 2021 10:14:14 GMT
Connection: keep-alive
X-AREQUESTID: @1WFLNF3x734x12390371x1
x-xss-protection: 1; mode=block
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
Pragma: no-cache
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: no-cache
Cache-Control: no-store
Location: https://scm.corp.example.com/repos?visibility=public
Content-Language: en-US
Strict-Transport-Security: max-age=63072000; includeSubdomains; preload
X-Frame-Options: DENY
X-Content-Type-Options: nosniff

My understanding is that providers.TryGetFirst(x => x.IsSupported(probeResponse), out match) calls into BitbucketHostProvider.IsSupported() after other providers.

All in all, it seems too bad this auto detection is needed at all and done in sequence.

Thanks.

[lnu]

I have the same issue with git 2.25.1 on WSL and 2.33.1 on windows.
For WSL I had to change the config to(since 2.33.1 is delivered with core instead of the old one): credential.helper=/mnt/c/Program\ Files/Git/mingw64/libexec/git-core/git-credential-manager-core.exe instead of credential.helper=/mnt/c/Program\ Files/Git/mingw64/libexec/git-core/git-credential-manager.exe

The "detecting" message could be silent. Is it really an error?:https://github.com/microsoft/Git-Credential-Manager-Core/blob/1f926e42c40db15e3fab0c68a7a5c3aabd416ce7/src/shared/Microsoft.Git.CredentialManager/HostProviderRegistry.cs#L176

[mjcheetham]

@gpakosz:

I still don't understand what this auto-detection of host provider is for.

Can you please elaborate on why the core credential manager needs to be aware of the application running on the remote server? What's the link with the Git protocol and authentication?

In order to provide the best authentication experience, GCM needs to know what sort of Git hosting provider you're using. For example, is this GitHub.com, Azure DevOps, a GitHub Enterprise Server or Bitbucket Server instance?

For the "cloud" providers like GitHub.com or Azure DevOps we can simply look at the remote URL to see "github.com" or "dev.azure.com" and route to their preferred (or even required) authentication protocols (for example OAuth2, or OpenID Connect).

However, for the self-hosted ("on-prem") versions we don't know! The remote URL isn't any use to us since this can be named anything or even a local IP address. Also due to the limitation in the design, Git doesn't surface any HTTP response headers to us, only the host name!

GCM first checks all the supported host providers by matching the host name to the known cloud name:

--> Is the host name "bitbucket.org"?
--> Is the host name "github.com"?
--> Is the host name "dev.azure.com"?

If we reach this point, we then perform a HTTP HEAD request against the remote URL to see what comes back.
Each host provider is asked again if it recognises the response headers.

--> Does the Bitbucket provider recognise <headers>?
--> Does the GitHub provider recognise <headers>?
--> Does the Azure Repos provider recognise <headers>?

If we get all the way to this point, we don't know anything about the host provider and so fall to a "generic" solution.
The generic provider just prompts for a username and password and doesn't do anything special.

We also noticed the portable version of Git which uses git-credential-wincred.exe doesn't exhibit this behavior while working perfectly.

The wincred credential helper doesn't do any of the above. It just checks if theres a store credential based on the hostname, and if not, prompts for a simple username/password.

[Setting the credential.provider option] doesn't seem to end up with other observable side effect than the suppression of the messages.

When setting the provider option GCM does not do any of the above logic to determine the provider; you have told us to go directly to the "generic" or "bitbucket" provider.

All in all, it seems too bad this auto detection is needed at all and done in sequence.

I agree! The problem is trying to strike the right balance. Git doesn't tell the credential helpers nearly enough information to be able to "know" what sort of authentication protocol to use beyond "I need a password for example.com".

In an ideal world, Git would provide helpers like GCM with the HTTP authentication challenge (the WWW-Authenticate headers for example) and we could go directly to the correct credential authority.

If we didn't perform any of the auto-detection logic, then many people using on-prem servers like GitHub Enterprise Server or Bitbucket Server would need to configure GCM (with the credential.provider setting) to tell us "git.mycompany.com is actually a GHES instance" ahead of time.

@lnu:

The "detecting" message could be silent. Is it really an error?:

Writing optional/informational messages to the Standard Error stream is a common practice in the Git ecosystem. The Standard Output stream is often parsed for information, which we don't want to break.

In credential helper case, the stdout stream is being used to communicate back to the Git process.

Given extra noise it generates (for not much value), I think we can consider removing the info: detecting host provider for 'host'... messages.

[gpakosz]

In order to provide the best authentication experience, GCM needs to know what sort of Git hosting provider you're using. For example, is this GitHub.com, Azure DevOps, a GitHub Enterprise Server or Bitbucket Server instance?

From there I was still wondering: what will GCM do differently or better compared to merely asking for a username and password?

So tried the following

• Open Windows' Credentials Manager and remove the entry for scm.corp.example.com
• $ git config --global credential.scm.corp.example.com.provider generic
• $ git fetch -p
• Then a popup entitled "Windows Security" opened and I entered my username and password
  

Then I tried again with git config --global credential.scm.corp.example.com.provider bitbucket and this time the popup was different

Is the "Atlassian" authentication experience the best one though? Here are my observations

• In the "Atlassian experience", I have not hint about which remote server I'm entering credentials for
• In the "Atlassian experience", clicking "Can't login" or "Signup for an account" brings me to bitbucket.org while I'm interacting with an on premise instance (note that entering an email can't possibly work either)

Maybe I'm missing something but I'm sorry, the "best experience" looks more confusing 🤷‍♂️

Given extra noise it generates (for not much value), I think we can consider removing the info: detecting host provider for 'host'... messages.

Yes please. Maybe reserve info: detecting host provider for 'https://scm.corp.example.com/'... for when GIT_TRACE=1

[mjcheetham]

Looking at the Bitbucket provider, seems like it says that only password-based authentication is available for Bitbucket DC (there is no OAuth support).

https://github.com/microsoft/Git-Credential-Manager-Core/blob/30c3f61a2d3e12fc7f4fab78bccd219873f7d0d3/src/shared/Atlassian.Bitbucket/BitbucketHostProvider.cs#L225-L229

@mminns is this true? Does DC not also support OAuth?

In the "Atlassian experience", I have not hint about which remote server I'm entering credentials for

I know that the GitHub provider shows the URL when connecting to a GHES instance. This is something the Bitbucket UI should probably also surface.

In the "Atlassian experience", clicking "Can't login" or "Signup for an account" brings me to bitbucket.org while I'm interacting with an on premise instance (note that entering an email can't possibly work either)

This sounds like a bug. We should either hide those links, or direct to the relevant documentation for DC/the correct forgot password link.

[mminns]

No, Bitbucket DC does not currently support OAuth 2.

[mminns]

I thought it looked familiar :)

I did look at adding OAuth 1 support, included in Bitbucket DC, to GCMW this included showing the host mminns/Git-Credential-Manager-for-Windows@854fde0#diff-7b2d00fce1e8c79e3e929f1921bf8f2f78cb3d4fab0e586fa5889f718fcea3fe

It worked but the OAuth1 configuration was too painful to be practical.

[gpakosz]

While I'm at it, I have yet another question: why do we pay this auto-detection tax every time we git fetch even though the Windows' credentials manager has proper authentication information available?

[mminns]

FWIW OAuth2 is coming to Bitbucket DC, I believe by the end of this year.

I have a PoC branch of GCMC that implements OAuth2 support for DC https://github.com/mminns/Git-Credential-Manager-Core/tree/SHPLII-74.

[gpakosz]

@mjcheetham Does it make sense that I open another issue for the question above?

why do we pay this auto-detection tax every time we git fetch even though the Windows' credentials manager has proper authentication information available?

Thanks

[RPitt]

I know this issue is closed but I'm unsure where else to put this...

The auto-detection strategy here is really anoying when used with git servers that are not auto-detected.
In that case there are several roundtrips before it all fails and falls back to generic, costing precious seconds.

[rm-code]

This problem is even worse when you are behind a proxy:

git push
info: detecting host provider for '[snip]'...
warning: failed to probe '[snip]' to detect provider
warning: ServicePointManager unterstützt keine Proxys mit dem [snip]-Schema.
warning: see https://aka.ms/gcmcore-autodetect for more information.
info: detecting host provider for '[snip]'...
warning: failed to probe '[snip]' to detect provider
warning: ServicePointManager unterstützt keine Proxys mit dem [snip]-Schema.
warning: see https://aka.ms/gcmcore-autodetect for more information.
Everything up-to-date

@mjcheetham We use Azure Devops Server (On Premise) exclusively.

The documentation mentions that it can be configured in a way to avoid auto-detection completely.

Could you give a concrete example how we would do this for the "On Premise" variant?

EDIT:
Looks like this worked:

 git config --global credential.azure-devops.[my.domain].provider generic