Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[JENKINS-73471] Restore passing credentialsId to the GitSCM #867

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

Dohbedoh
Copy link
Contributor

@Dohbedoh Dohbedoh commented Aug 1, 2024

Proposal to fix #862. Reinstore the credentialsId in the GitSCM configuration.
It would also guarantee that credentials usage is still tracked. Checking down the line, GitClient still uses the authenticator credentials reference.

@yaroslavafenkin Per my understanding, the issue that SECURITY-3363 fixes was the clone link of the OAuth Authenticator at https://github.com/jenkinsci/bitbucket-branch-source-plugin/blob/886.v44cf5e4ecec5/src/main/java/com/cloudbees/jenkins/plugins/bitbucket/api/credentials/BitbucketOAuthAuthenticator.java#L48-L57 ? In which case instantiating the GitSCM with the credentialsId is fine ? I am not sure what is the scenario to validate that this does not bring back this security problem ?

Your checklist for this pull request

  • Make sure you are requesting to pull a topic/feature/bugfix branch (right side) and not your master branch!
  • Ensure that the pull request title represents the desired changelog entry
  • Please describe what you did
  • Link to relevant issues in GitHub or in Jenkins JIRA
  • Link to relevant pull requests, esp. upstream and downstream changes
  • Did you provide a test-case? That demonstrates feature works or fixes the issue.

@dwnusbaum
Copy link
Member

I don't know if Yaroslav is still active in the Jenkins project. It is unfortunate that ad359b3 did not add any tests. Here are the reproduction steps from the security ticket though to check if it regresses the fix manually:

  • Create a Bitbucket repository with a Jenkinsfile and a PR editing this Jenkinsfile (You need at least 2 branches)
  • Create an OAuth credentials following this documentation
  • Create a MultiBranch Pipeline
  • As a Branch Sources, select "Bitbucket"
  • Select your previously created OAuth credentials
  • As Owner, put the username of your Bitbucket repository
    (With a valid OAuth credentials and owner, it should automatically fill the repository name dropdown)
  • Save
  • Navigate to the status page MultiBranch pipeline ([JENKINS_INSTANCE]/job/[JOB_NAME])
  • Click on "Pull request"
  • Navigate to the related job
  • Open the build console output [and check whether the access token is displayed in the clone URLs]

@dwnusbaum
Copy link
Member

dwnusbaum commented Aug 1, 2024

@Dohbedoh Yaroslav asked me to pass on some info from when he worked on the security ticket initially. The reason he made the one-line change that you are proposing to revert in this PR has to do with unusual behavior he saw while testing:

There's a bit of weird behaviour that I've just noticed when testing. On a fresh instance I've setup a project, pointed it to a repo that I've set up for my testing purposes in BitBucket Cloud. I've added Username and Password credentials that correspond to OAuth key and secret. Scanning is successful, build on the main branch too. PR build kept failing with the following:

Log
Started by user [Admin](http://localhost:8080/user/admin)
Checking out git https://yafenkin-org@bitbucket.org/yafenkin-org/test.git into /Users/yaroslavafenkin/projects/jenkinsci/jenkins/war/work/workspace/zxc_PR-4@script/7f84c2c049bbfafce90d9d2d0fe3b04ae23118ea5c2526236601377762ad2088 to read Jenkinsfile
The recommended git tool is: NONE
using credential whatever
 > git rev-parse --resolve-git-dir /Users/yaroslavafenkin/projects/jenkinsci/jenkins/war/work/workspace/zxc_PR-4@script/7f84c2c049bbfafce90d9d2d0fe3b04ae23118ea5c2526236601377762ad2088/.git # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://yafenkin-org@bitbucket.org/yafenkin-org/test.git # timeout=10
Fetching without tags
Fetching upstream changes from https://yafenkin-org@bitbucket.org/yafenkin-org/test.git
 > git --version # timeout=10
 > git --version # 'git version 2.39.3 (Apple Git-146)'
using GIT_ASKPASS to set credentials whatever
 > git fetch --no-tags --force --progress -- https://yafenkin-org@bitbucket.org/yafenkin-org/test.git +refs/heads/Yaroslav-Afenkin/jenkinsfile-edited-online-with-bitbucket-1709733248367:refs/remotes/origin/Yaroslav-Afenkin/jenkinsfile-edited-online-with-bitbucket-1709733248367 +refs/heads/main:refs/remotes/origin/main # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from https://yafenkin-org@bitbucket.org/yafenkin-org/test.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:999)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1241)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1305)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:136)
	at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:167)
	at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:143)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:311)
	at hudson.model.ResourceController.execute(ResourceController.java:101)
	at hudson.model.Executor.run(Executor.java:442)
Caused by: hudson.plugins.git.GitException: Command "git fetch --no-tags --force --progress -- https://yafenkin-org@bitbucket.org/yafenkin-org/test.git +refs/heads/Yaroslav-Afenkin/jenkinsfile-edited-online-with-bitbucket-1709733248367:refs/remotes/origin/Yaroslav-Afenkin/jenkinsfile-edited-online-with-bitbucket-1709733248367 +refs/heads/main:refs/remotes/origin/main" returned status code 128:
stdout: 
stderr: remote: Invalid credentials
fatal: Authentication failed for 'https://bitbucket.org/yafenkin-org/test.git/'

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2872)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2211)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:637)
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:997)
	... 8 more
ERROR: Error fetching remote repo 'origin'
ERROR: Maximum checkout retry attempts reached, aborting
ERROR: Error fetching remote repo 'origin'
ERROR: Maximum checkout retry attempts reached, aborting
[Bitbucket] Notifying pull request build result
Can not determine Jenkins root URL or Jenkins URL is not a valid URL regarding Bitbucket API. Commit status notifications are disabled until a root URL is configured in Jenkins global configuration. 
IllegalStateException: Jenkins URL cannot start with http://localhost/
Finished: FAILURE

I switched to a terminal and did git clone https://yafenkin-org@bitbucket.org/yafenkin-org/test.git. Was asked for a password. Captured the token via the debugger, pasted it in and the repo was cloned successfully. Went back to Jenkins and restarted the build for the PR. It now succeeds (issuing a new token for every build, so it's a not a particular token issue). Doesn't seem to happen without the fix, so likely caused by it :/

After that comment he changed the line in question to stop passing credentialsId, which seemed to fix the issue for him when testing.

He doesn't think that your change would regress the security fix, the question is just whether the error that he saw when the code was like you have it in the PR now is representative of a real bug that would affect users or is just some kind of environmental or configuration issue.

@Dohbedoh
Copy link
Contributor Author

Dohbedoh commented Aug 2, 2024

Thanks!

  1. I can't reproduce the security problem.
  2. Reproducible. Actually that comment made me think about credentials.helper, and when getting rid of it, actually even a branch build checkout scm fails. Looking into this.

@Dohbedoh
Copy link
Contributor Author

Dohbedoh commented Aug 2, 2024

Allright, so actually this cannot work .. In GitSCM, the GitSCMExtension decorates the GitClient first and then it set the credentials if a credentials ID is provided https://github.com/jenkinsci/git-plugin/blob/git-5.2.2/src/main/java/hudson/plugins/git/GitSCM.java#L929. It's been the case since 2013...

I wonder why we don't decorate the client at the end of the GitClient#createClient method 🤔. We can maybe propose this.. I guess a way to solve this particular problem is to override the GitSCMExtension#beforeCheckout to override the credentials added to the GitClient... I tested this and it works.

If unit tests are required, I would need more time to provide them..

@Dohbedoh
Copy link
Contributor Author

Dohbedoh commented Aug 28, 2024

@Dohbedoh
Copy link
Contributor Author

git-plugin 5.5.0 has been released.
Added a temporary dependency in the pom until that version of git is included in the BOM.
Updated the BOM.

cc @jenkinsci/bitbucket-branch-source-plugin-developers

@Dohbedoh Dohbedoh force-pushed the JENKINS-73471 branch 3 times, most recently from 6a23a48 to fb52504 Compare September 23, 2024 12:14
@jglick
Copy link
Member

jglick commented Sep 25, 2024

Alternately, revert ad359b3 and just use a https://javadoc.jenkins.io/plugin/workflow-api/org/jenkinsci/plugins/workflow/log/TaskListenerDecorator.Factory.html which is triggered by the use of this SCM source and which masks any x-token-auth:{…}.

@Dohbedoh
Copy link
Contributor Author

Dohbedoh commented Oct 9, 2024

@jglick The decorator would solve the leakage in the console output but I recall another source.. I think the "Changes" page that of a job with GitSCM that displays the remote URL. Using secrets in the GitSCM remote URL is a problem.

@Dohbedoh
Copy link
Contributor Author

Dohbedoh commented Oct 9, 2024

I was thinking of splitting this into 2 PRs.

  • PR-1: One PR that remove the GitClientAuthenticatorExtension when using SSHCheckoutTrait.
  • PR-2: One PR that re-add the credentials ID

We need to make sure theGitClientAuthenticatorExtension is not present for SSHCheckoutTrait otherwise it would break when applying #867 in the git-plugin. So we can apply and release git-plugin without breaking user's environment.

We would coordinate the fix in the following order::

WDYT @jenkinsci/bitbucket-branch-source-plugin-developers ?

@jglick
Copy link
Member

jglick commented Oct 10, 2024

the "Changes" page that of a job

Make https://github.com/jenkinsci/git-plugin/blob/f7b32740f2abfc148692fe0d4628c094eed31798/src/main/java/hudson/plugins/git/util/BuildData.java#L92 strip any authentication fields out of URLs before storing them.

@nfalco79
Copy link
Member

nfalco79 commented Nov 2, 2024

@Dohbedoh the changes merged in git plugin has been reverted.

In our production environment we use a OAuth 2 credential to discover branches and SSH trait to perform git operation (clone, pull, ...).
A pipeline library defined by git SCM with an App password.
I'm configuring one of existing organization folder to use OAuth2 (with wipeout & force clone) to perform git operations covering all scenarios.

I had update plugin to latest version, now I will install build from this branch and I will have a look to the console to see if secret appear in some logs.

@nfalco79
Copy link
Member

nfalco79 commented Nov 2, 2024

Well, discover branches and clone using OAuth2 credential fail:

The recommended git tool is: NONE
using credential bitbucket.oauth
Wiping out workspace first.
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command "git fetch --no-tags --force --progress -- https://<bb_owner>@bitbucket.org/<bb_owner>/<reposlug>.git +refs/heads/support/1.2.x:refs/remotes/origin/support/1.2.x" returned status code 128:
stdout: 
stderr: remote: Invalid credentials
fatal: Authentication failed for 'https://bitbucket.org/<bb_owner>/<reposlug>.git/'

	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2848)
	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2184)
	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:635)
	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:877)
	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:170)
	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
	at hudson.remoting.UserRequest.perform(UserRequest.java:225)
	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
	at hudson.remoting.Request$2.run(Request.java:391)
	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:81)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:140)
	at java.base/java.lang.Thread.run(Unknown Source)
	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 10.244.1.102/10.244.1.102:43228
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1923)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:384)
		at hudson.remoting.Channel.call(Channel.java:1112)
		at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:153)
		at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
		at java.base/java.lang.reflect.Method.invoke(Unknown Source)
		at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:138)
		at PluginClassLoader for git-client/jdk.proxy75/jdk.proxy75.$Proxy170.execute(Unknown Source)
		at PluginClassLoader for git//hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1220)
		at PluginClassLoader for git//hudson.plugins.git.GitSCM._checkout(GitSCM.java:1310)
		at PluginClassLoader for git//hudson.plugins.git.GitSCM.checkout(GitSCM.java:1277)
		at PluginClassLoader for workflow-scm-step//org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:136)
		at PluginClassLoader for workflow-scm-step//org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:101)
		at PluginClassLoader for workflow-scm-step//org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:88)
		at PluginClassLoader for workflow-step-api//org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
		at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
		at java.base/java.lang.Thread.run(Unknown Source)
ERROR: Error cloning remote repo 'origin'
ERROR: Maximum checkout retry attempts reached, aborting

Without changes of this PR the checkout using OAuth2 credentials works:

The recommended git tool is: NONE
No credentials specified
Wiping out workspace first.
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository https://<bb_owner>@bitbucket.org/<bb_owner>/<reposlug>.git
 > git init /home/jenkins/agent/workspace/PND_<reposlug>_support_1.2.x # timeout=10
Fetching upstream changes from https://<bb_owner>@bitbucket.org/<bb_owner>/<reposlug>.git
 > git --version # timeout=10
 > git --version # 'git version 2.39.5'
using GIT_ASKPASS to set credentials 
 > git fetch --no-tags --force --progress -- https://<bb_owner>@bitbucket.org/<bb_owner>/<reposlug>.git +refs/heads/support/1.2.x:refs/remotes/origin/support/1.2.x # timeout=10
Avoid second fetch
Checking out Revision c862a77a3b97dd7159c52b4cf303d60ad139106a (support/1.2.x)
Commit message: "[maven-release-plugin] prepare for next development iteration"
 > git config remote.origin.url https://<bb_owner>@bitbucket.org/<bb_owner>/<reposlug>.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/support/1.2.x:refs/remotes/origin/support/1.2.x # timeout=10
 > git config core.sparsecheckout # timeout=10
 > git checkout -f c862a77a3b97dd7159c52b4cf303d60ad139106a # timeout=10
 > git branch -a -v --no-abbrev # timeout=10
 > git checkout -b support/1.2.x c862a77a3b97dd7159c52b4cf303d60ad139106a # timeout=10
 > git rev-list --no-walk c862a77a3b97dd7159c52b4cf303d60ad139106a # timeout=10
Cleaning workspace

Discover branches using OAuth2 credential and clone using SSH trait works:

The recommended git tool is: NONE
using credential builder.openssh
Wiping out workspace first.
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository git@bitbucket.org:<bb_owner>/<reposlug>.git
 > git init /home/jenkins/agent/workspace/PND_<reposlug>_support_1.2.x # timeout=10
Fetching upstream changes from git@bitbucket.org:<bb_owner>/<reposlug>.git
 > git --version # timeout=10
 > git --version # 'git version 2.39.5'
using GIT_SSH to set credentials Builder OpenSSH key
Verifying host key using known hosts file, will automatically accept unseen keys
 > git fetch --no-tags --force --progress -- git@bitbucket.org:<bb_owner>/<reposlug>.git +refs/heads/support/1.2.x:refs/remotes/origin/support/1.2.x # timeout=10
Avoid second fetch
Checking out Revision c862a77a3b97dd7159c52b4cf303d60ad139106a (support/1.2.x)
 > git config remote.origin.url git@bitbucket.org:<bb_owner>/<reposlug>.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/support/1.2.x:refs/remotes/origin/support/1.2.x # timeout=10
Commit message: "[maven-release-plugin] prepare for next development iteration"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f c862a77a3b97dd7159c52b4cf303d60ad139106a # timeout=10
 > git branch -a -v --no-abbrev # timeout=10
 > git checkout -b support/1.2.x c862a77a3b97dd7159c52b4cf303d60ad139106a # timeout=10
 > git rev-list --no-walk c862a77a3b97dd7159c52b4cf303d60ad139106a # timeout=10
Cleaning workspace

@nfalco79 nfalco79 self-requested a review November 2, 2024 02:47
@nfalco79
Copy link
Member

nfalco79 commented Nov 2, 2024

Now I understand the issue. If the BitbucketGitSCMBuilder is built with credentialsId, it will build a GitClient performing a lookup a StandardUsernameCredentials for the given credentialsId. This override what the extension GitClientAuthenticatorExtension contribute, because it decorate the GitClient in GitSCM before than it configure all UserRemoteConfigs. And that why you make a PR to the git-plugin to move extensions after UserRemoteConfigs configurations.

In case of OAuth2 the authorization header is enough, and that why with credentilsId = null no StandardUsernameCredentials is applied for the URL to clone and it works.

@nfalco79
Copy link
Member

nfalco79 commented Nov 3, 2024

@Dohbedoh as suggested the first part of this fix has been merged (set to null credentials in case of SSHTrait).
Now we can move to reapply jenkinsci/git-plugin#1649
Than rebase and merge this PR

@nfalco79 nfalco79 added the bug label Nov 3, 2024
@Dohbedoh
Copy link
Contributor Author

Dohbedoh commented Nov 4, 2024

@nfalco79 Thanks for the help. Can we get a release of the first part ? The automated release did not kick in because it misses required labels.
Then I can open a PR for git plugin again and update Mark about this.

@Dohbedoh
Copy link
Contributor Author

Dohbedoh commented Nov 4, 2024

@nfalco79 turns out git-plugin already moved to 2.479.1 baseline so I had to bump this requirement here too.

@nfalco79
Copy link
Member

nfalco79 commented Nov 4, 2024

@nfalco79 Thanks for the help. Can we get a release of the first part ? The automated release did not kick in because it misses required labels. Then I can open a PR for git plugin again and update Mark about this.

Could you point me to some help page about this required labels? Other plugins I'm the mantainer I perform manual release

@nfalco79
Copy link
Member

nfalco79 commented Nov 4, 2024

Unfortunately in jenkins-infra exclusive JEP-229 is enabled, so deny upload permission to the mantainers

@jglick
Copy link
Member

jglick commented Nov 4, 2024

Could you point me to some help page about this required labels?

Do you mean https://github.com/jenkinsci/.github/blob/f1a34987f2919f039282a13b8741e3c463d18bc6/.github/release-drafter.yml#L9-L55 ?

@Dohbedoh
Copy link
Contributor Author

Dohbedoh commented Nov 5, 2024

From https://www.jenkins.io/doc/developer/publishing/releasing-cd/#releasing:

You can also trigger a deployment explicitly, if the current commit has a passing check from Jenkins. Visit https://github.com/jenkinsci/your-plugin/actions?query=workflow%3Acd and click Run workflow.

@nfalco79
Copy link
Member

nfalco79 commented Nov 5, 2024

Could you point me to some help page about this required labels?

Do you mean https://github.com/jenkinsci/.github/blob/f1a34987f2919f039282a13b8741e3c463d18bc6/.github/release-drafter.yml#L9-L55 ?

I mean is there a specific label on a pull request that trigger a release?

@nfalco79
Copy link
Member

nfalco79 commented Nov 5, 2024

From https://www.jenkins.io/doc/developer/publishing/releasing-cd/#releasing:

You can also trigger a deployment explicitly, if the current commit has a passing check from Jenkins. Visit https://github.com/jenkinsci/your-plugin/actions?query=workflow%3Acd and click Run workflow.

I saw that documentation and I had already try but Release stage is disable and there is not way (I found) to trigger a deploy
immagine

@jglick
Copy link
Member

jglick commented Nov 5, 2024

is there a specific label on a pull request that trigger a release

Yes, any of the interesting ones, here I guess bug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Version 887 breaks usage of scm.userRemoteConfigs[].credentialsId
4 participants