Gitlab/OAuth2 cloning private projects fails with autopopulated tokens - Access denied

[nils-mosbach]

Describe the bug

We're currently have issues setting up a proper OAuth flow for cloning private GitLab projects. My intention was to allow cloning private projects for all of our developers using GitLab OAuth without the need of defining SSH-Tokens or Kubernetes secrets on user side. So basically add a button to all GitLab Project-Pages (like "Open in Che") which will does the trick (Login, clone, etc.) without user interaction.

GitLab-OAuth Login to Che and reading the devfile from a private GitLab-Repository (using a factory link) works great, but project sources can't be cloned in theia.

It throws "Permission denied" while cloning sources.

Git clone in the terminal (theia or dev-container) outputs the following.

Cloning into 'playground'...
fatal: unable to get credential storage lock: Permission denied
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://git.company.dev/MosbachN/playground.git/'

Are we missing something?

Che version

7.39@latest

Steps to reproduce

We've setup GitLab/Keycloak/Che as described in the documentation.

• Custom Che properties in Operator-Config:
  • CHE_INTEGRATION_GITLAB_SERVER__ENDPOINTS: "https://git.company.dev/"
  • CHE_INTEGRATION_GITLAB_OAUTH__ENDPOINT: "https://git.company.dev/"
• Keycloak realm and client configured as described in the documentation
  • Created a realm "git-dev"
  • Added a identity provider "GitLab"
    • Authorization URL: https://git.company.dev/oauth/authorize
    • Token URL: https://git.company.dev/oauth/token
    • Default scopes: api read_user read_api write_repository openid profile email

Expected behavior

Login to Che using GitLab Credentials (OAuth) works great.
What also works great is reading devfile.yamls from private repositories using (https://che-dev.company.dev/dashboard/f?url=http://git.company.dev/...).
So that's all good and theia pops up using all plugins and extensions defined by the devfile.

The Problem we have is that project sources cannot be cloned. See error above.

Runtime

Kubernetes (vanilla)

Screenshots

No response

Installation method

chectl/latest

Environment

Linux

Eclipse Che Logs

No response

Additional context

Git config seems to get injected properly by Che.

$ cat /etc/gitconfig 
[user]
        name = mosbachn
        email = nm@company.com

[credential]
        helper = store --file /home/theia/.git-credentials/credentials

Even credentials get mounted by a secred git-credentials-secret-6xk49 to /home/theia/.git-credentials/credentials.

$ cat /home/theia/.git-credentials/credentials
https://MosbachN:8f************e7@git.company.dev

[l0rd]

cc @skabashnyuk @xbaran4

[skabashnyuk]

@nils-mosbach Thank you for this report. Could you please share the way you've installed che and the devfile you've used?

[mshaposhnik]

For me it looks like git is unable to create .git-credentials.lock in the home folder

[nils-mosbach]

Permission denied - Get credential storage lock

I double checked that by copying the files (credential-store and git-config) as deployed by che to a fresh ubuntu container locally with git installed. Cloning still fails (Authentication failed), lock file error is gone. So, I'm not sure if the error message that belongs to the lock file will cause the authentication issue.

Installation of Che

I installed Che using chectl. Deployment: Operator.

$ chectl server:deploy \
  --installer=operator \
  --chenamespace=studio-next \
  --domain=company.dev \
  --multiuser \
  --platform=k8s \
  --deployment-name=studio-next \
  --skip-kubernetes-health-check \
  --che-operator-cr-patch-yaml=./operator-config.yaml \
  --batch

operator-config.yaml:

spec:
  devWorkspace:
    enable: false
  server:
    tlsSupport: true
    customCheProperties:
      CHE_SYSTEM_SUPER__PRIVILEGED__MODE: "true"
      CHE_LIMITS_USER_WORKSPACES_RUN_COUNT: "-1"
      CHE_LIMITS_WORKSPACE_IDLE_TIMEOUT: "9000000" 
      CHE_SYSTEM_ADMIN__NAME: mosbachn
      CHE_INTEGRATION_GITLAB_SERVER__ENDPOINTS: "https://git.company.dev/"
      CHE_INTEGRATION_GITLAB_OAUTH__ENDPOINT: "https://git.company.dev/"
    cheImagePullPolicy: Always
    devfileRegistryPullPolicy: Always
    pluginRegistryPullPolicy: Always
    
  database:
    externalDb: false
    postgresImagePullPolicy: Always

  storage:
    pvcStrategy: per-workspace
    pvcClaimSize: '5Gi'

  auth:
      externalIdentityProvider: true
      identityProviderURL: 'https://auth.company.dev/auth'
      identityProviderRealm: 'git-dev'
      identityProviderClientId: 'che'
      openShiftoAuth: false

  k8s:
      ingressDomain: 'company.dev'
      ingressStrategy: 'multi-host'
      tlsSecretName: 'tls-company-dev'

We're running a custom plugin and devfile registry, but the issue also exists, if I use the java/gradle image provided by the che-registry devfile.yaml. I tried cloning the private repository in the gradle container provided by the devfile using git clone https://git.company.dev/MosbachN/playground.git/. Still the same error.

It seems that the credential-store injected by che isn't able to authenticate.

@skabashnyuk I'm happy to dig a little deeper if you need additional information like logs, etc.

Just in case: Tested against GitLab Community Edition. I'm not sure if that's a limitation since reading the devfile from the project works.

[skabashnyuk]

might be related to #20642

[nils-mosbach]

might be related to #20642

I'm not sure, from what I can see that belongs to git lfs and doesn't affect authentication. A closer look at the merged request eclipse-che/che-theia#1236 (comment) doesn't show changes in code that's used by the authentication or credential storage.

Before enabling the GitLab login flow, we also hadn't had any issues working with private projects by entering a username and password after each git operation. That works fine by using the theia-UI or using all of the container terminals (theia, devcontainer, ...)

It actually seems that the provided token is not trusted by GitLab.

[skabashnyuk]

@mshaposhnik is investigating if the eclipse-che/che-theia#1236 (comment) overrides the necessary changes in .gitconfig that enable credential store.

[nils-mosbach]

While digging a little depper, I think I actually found the issue, or at least a possible solution to the problem.

Credential storage file contains the repository url and token by scheme https://{{username}}:{{oauth-token}}@{{git-url}} which doesn't work in my case. I found a stack-overflow topic (Using GitLab token to clone without authentication) which recommends using oauth2 instead of the username. e.g. https://oauth2:{{oauth-token}}@{{git-url}}.

I quickly gave it a go and it works. :)

# Doesn't work
$ cat /home/theia/.git-credentials/credentials
https://MosbachN:8f************e7@git.company.dev

# Works
$ cat /home/theia/.git-credentials/credentials
https://oauth2:8f************e7@git.company.dev

That might explain the storage lock error as well, since git tries to remove failed credentials from the credential store. That fails since that file is a read only mounted kubernetes secret.

I wasn't able to find anyting specific in the official GitLab documentation, bit at least I found a part of CI-Pipeline tokens which is kind of the same where you have something like https://gitlab-ci-token:{{token}}@git.company.dev. It looks like GitLab requires the origin of the token instead of the username.

Hope that helps.

[mshaposhnik]

Great catch. That's because we process both personal access token and OAuth tokens same way.
So, for PAT-s correct pattern is user:pass@.... while for OAuth it must be oauth2:pass@.... And, for GitHub, user:pass@... works just well with OAuth tokens too. Now we need some time to try to find a correct solution :)

[nils-mosbach]

Nice, thanks a lot.

[nils-mosbach]

@mshaposhnik any chance the fix (eclipse-che/che-server#185) will make it to the next release (7.40)?