Install sfdx plugins for Jenkins or github action

[gambe94]

Issue

I experienced couple of times that sfdx plugin installation not work properly under the CI user.

In Jenkins pipeline

In Jenkins always run the agent with a Jenkins user.
And when we create our sfdx image where we want to install some plugins, it's tricky. Plugins are only available just for the user who installed them.
My solution in the docker file:

#Install sfdx latest version with npm as root user
RUN npm install sfdx-cli --location=global
RUN sfdx --version


RUN groupadd --gid 1000 jenkins \
   && useradd --uid 1000 --gid jenkins --shell /bin/bash --create-home jenkins
RUN echo 'jenkins ALL=(ALL) NOPASSWD: ALL' | sudo EDITOR='tee -a' visudo

#Switch to jenkins user
USER jenkins

#sfdx plugin goigng to be installed .local directory under the current user
#sfdx plugin need to be installed under the running user!!

#ADD sfdx-git-delta
RUN echo "y" | sfdx plugins:install sfdx-git-delta
RUN sfdx plugins
WORKDIR home/jenkins

Github Action workflow

Option1

You need to include some environment variables in docker file before you install the plugins.
I used this image as a reference
My docker file:

#Install sfdx latest version with npm
RUN npm install sfdx-cli --location=global
RUN sfdx --version

#Set XDG environment variables explicitly so that GitHub Actions does not apply
#default paths that do not point to the plugins directory
#https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html
ENV XDG_DATA_HOME=/sfdx_plugins/.local/share
ENV XDG_CONFIG_HOME=/sfdx_plugins/.config
ENV XDG_CACHE_HOME=/sfdx_plugins/.cache

RUN echo "y" | sfdx plugins:install sfpowerkit
#Add sfdx essentials
RUN echo 'y' | sfdx plugins:install sfdx-essentials
#ADD sfdx-git-delta
RUN echo 'y' | sfdx plugins:install sfdx-git-delta

Option2

Here my solution is more difficult, because I cannot manage to install the plugins inside of the docker container.
The problem is: when github action runs the container its true that is run as the root user. But its mount the workspace to root(docker) directory -> I think that's cause the issue that preinstalled plugin not working.
Check the official docs

How I managed to make it work:
Plugin not need to be installed in the docker image in this case

1. Install the plugin as npm package locally: npm install --save-dev sfdx-git-delta
2. Added to package.json and package-lock.json also created
3. In the CI run npm install and cache node/modules link
4. Run the following command to link the plugin from the created node modules lib sfdx plugins:link node_modules/sfdx-git-delta

Example code for github action:

name: CI job
on:
  workflow_dispatch:
  push:
    branches:
      - main
#  pull_request:
jobs:
  dev-flow:
    runs-on: ubuntu-latest
    container:
      image: docker://<myimage>
      credentials:
        username: ${{ secrets.DOCKER_USER }}
        password: ${{ secrets.DOCKER_TOKEN }}

    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          fetch-depth: 0

      - name: Cache node modules
        id: cache-npm
        uses: actions/cache@v3
        env:
          cache-name: cache-node-modules
        with:
          # npm cache files are stored in `~/.npm` on Linux/macOS
          path: ~/.npm
          key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
          restore-keys: |
            ${{ runner.os }}-build-${{ env.cache-name }}-
            ${{ runner.os }}-build-
            ${{ runner.os }}-

      - if: ${{ steps.cache-npm.outputs.cache-hit == 'false' }}
        name: List the state of node modules
        continue-on-error: true
        run: npm list

      - name: Install dependencies
        run: npm install

      - name: Link plugins
        run: sfdx plugins:link node_modules/sfdx-git-delta

[scolladon]

Thank you very much, very helpful
Here is the #315 where this discussion started

[mikjaf]

Hello @scolladon & @gambe94
Hope you're doing good.

I'm having an issue using Git delta inside a Jenkins Pipeline.
Let me quick resume my actions:

• I built a docker image with SFDX cli and Git Delta installed
• Pushed the image to my docker hub
• Edit my already existing Jenkins Pipeline to connect to my new image (was using SFDX docker image before)
• Add the git delta command in 1 step
• Get the error : "error": "--to is not a valid sha pointer: 'HEAD', --from is not a valid sha pointer: 'HEAD~1'",

I think I might miss a step in my pipeline but I don't what ...
If you could help me, that would be awesome!!!

Here's my script:

pipeline {
agent {
        kubernetes {
            defaultContainer 'salesforcedx'
            yaml """
                name: jnlp-slave
                apiVersion: v1
                kind: Pod
                spec:
                    containers:
                    - name: jnlp
                      image: 'jenkins/inbound-agent'
                      command:
                      - "/bin/sh"
                      - "-c"
                      args:
                      - "while ! /usr/local/bin/jenkins-slave ; do sleep 5 ; done"
                      tty: true
                      resources:
                        limits:
                            memory: 256Mi
                    - name: salesforcedx
                      image: myRepoDocker/image:latest
                      tty: true
                      resources:
                        limits:
                            memory: 256Mi
                      command: ['sleep']
                      args: ['infinity']
             """
        }
    }
    
	environment {
        CONSUMER_KEY_INT = credentials('toto')
        USERNAME = credentials('tata')
        JWT = credentials('titi')
        
	}

	stages {
        /**
        Checkout metadata from develop branch.
        */
	    stage ('Checkout') {
		    steps {
				git branch: "develop", credentialsId: 'tutu', url: 'blablabla/repo.git'
			}
		}
	    
        /**
        Authorize to INT sandbox with JWT-Flow.
        */
        stage('Authorize') {
            steps {
                sh """
                   sfdx force:auth:jwt:grant --setalias aliasEnv --clientid ${CONSUMER_KEY_INT} --username ${USERNAME} --jwtkeyfile ${JWT} -r https://test.salesforce.com
                """
            }
        }

        /**
        Run the deployment.
        */
        stage('Deploy') {
            steps {
                catchError(buildResult: 'FAILURE', stageResult: 'FAILURE') {
                    sh """
			sfdx sgd:source:delta --to "HEAD" --from "HEAD~1" --output manifest
                        sfdx force:source:deploy -x manifest/package/package.xml -u aliasEnv -l RunLocalTests -c
                    """
                }
            }
        }
    
        /**
        Logout from all Salesforce orgs.
        */
        stage('Logout') {
            steps {
                sh """
                    sfdx force:auth:logout --all -p
                """
            }
        }
	}
	
}

[mikjaf]

Alright, thanks for the reply.

1/ I checked my "Check out" stage and it looks like I'm getting what I need, the branch, the commit, etc.

2/ I added : --repo=https://MyUrl.com/stash/scm/projectName/salesforce-repository.git
Not sure if it is the right syntax.

[scolladon]

Little more explanation about step 2: The --repo parameter takes a directory where the sfdx repository is checked out. It does not take url.

Let's see an example.
I want to see the diff of my sfdx repository using sgd.
Ok then first I install sfdx and sgd:

$ npm i -g sfdx-cli
$ echo y | sfdx plugins:install sfdx-git-delta

Now I checkout the sfdx repository (named cats and dogs) and navigate to it

# it checkout the content of the repository in the cats-and-dogs folder
$ git clone my/repository/url/cats-and-dogs.git
$ cd cats-and-dogs

Now the current folder is my sfdx sources I can run sgd

# first I make sure the manifest folder exist
$ mkdir manifest 
# then I execute the command
$ sfdx sgd:source:delta -f "HEAD~1" -o manifest

[mikjaf]

What I don't understand is where to put these commands:

• in the dockerfile or the pipeline?

I just launched my Jenkins pipeline without the Git delta command, just using the SFDX source deploy, and it worked. So I guess the image is ok.
The Delta plugin is well installed as well otherwise I would not have the error message.

Now, as you mentioned before :

• or I don't fetch the right things ... but when I inspected and here's what is displaying:

Cloning the remote Git repository
Cloning repository https://MyUrl.com/stash/scm/projectName/salesforce-repository.git
 > git init /home/jenkins/agent/workspace/DEV/DEV_DeployTest # timeout=10
Fetching upstream changes from https://MyUrl.com/stash/scm/projectName/salesforce-repository.git
 > git --version # timeout=10
 > git --version # 'git version 2.30.2'
using GIT_ASKPASS to set credentials User Technique / Bitbucket / jenkins
 > git fetch --tags --force --progress -- https://MyUrl.com/stash/scm/projectName/salesforce-repository.git +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://MyUrl.com/stash/scm/rockyvgf/salesforce-repository.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
Checking out Revision fb1ec44a496431ef911c44e06d610008cdbaeeef (refs/remotes/origin/testDelta)
 > git rev-parse refs/remotes/origin/testDelta^{commit} # timeout=10
 > git config core.sparsecheckout # timeout=10
 > git checkout -f fb1ec44a496431ef911c44e06d610008cdbaeeef # timeout=10
 > git branch -a -v --no-abbrev # timeout=10
 > git checkout -b testDelta fb1ec44a496431ef911c44e06d610008cdbaeeef # timeout=10
Commit message: "add spaces debug ApexClassName"
 > git rev-list --no-walk fb1ec44a496431ef911c44e06d610008cdbaeeef # timeout=10

• or Delta and sfdx are not in the same repository but to be honest I did not really understand how that really works and how to solution it. I'm lacking programming skills here.

Thanks a lot anyway.

[scolladon]

I would put the installation of sfdx and the plugin in the docker and the git clone and sgd execution commands in the pipeline, this way the docker file could be reused by multiple repository

To me it looks the sfdx and sgd installtion part is ok.
What I think could not be ok is

• the git fetching (but it seems right seeing the logs)
• the sgd command is executed at the wrong directory (not from the sfdx repository)

Could you help me reproduce it ?

• What is the docker file ?
• What is the jenkins conf ?
• What version of jenkins and how to install it ?

[mikjaf]

Agreed on the steps.

Docker file is very simple and looks like this:

FROM salesforce/salesforcedx:latest-rc-slim

RUN echo y | sfdx plugins:install sfdx-git-delta
RUN sfdx plugins

Here is the docker file used to build the salesforce/salesforcedx:latest-rc-slim image.
https://github.com/salesforcecli/sfdx-cli/blob/main/dockerfiles/Dockerfile_slim

FROM heroku/heroku:20

ENV DEBIAN_FRONTEND=noninteractive
ARG DOWNLOAD_URL=https://developer.salesforce.com/media/salesforce-cli/sfdx/channels/nightly/sfdx-linux-x64.tar.xz
RUN apt-get update
# install cli from linux tarball with bundled node
RUN curl -s $DOWNLOAD_URL --output sfdx-linux-x64.tar.xz \
  && mkdir -p /usr/local/sfdx \
  && tar xJf sfdx-linux-x64.tar.xz -C /usr/local/sfdx --strip-components 1 \
  && rm sfdx-linux-x64.tar.xz

RUN apt-get install --assume-yes \
  openjdk-11-jdk-headless

RUN apt-get autoremove --assume-yes \
  && apt-get clean --assume-yes \
  && rm -rf /var/lib/apt/lists/*

ENV PATH="/usr/local/sfdx/bin:$PATH"
ENV SFDX_CONTAINER_MODE true
ENV DEBIAN_FRONTEND=dialog
ENV SHELL /bin/bash

Jenkins conf
Not sure what you need here. Unfortunately, I'm not the Admin of the whole configuration. I'm juste responsible for the Jobs that concerns my scope. Don't know the version, I did not install it.
The way it works is:

• I connect to Jenkins though my browser.
• I build a job that contains a pipeline script (the one that I shared in my 1st comment)
  That's it.