Cannot deploy new jenkins with install-plugins.sh

[zi-adis]

Describe the bug
Jenkins released a new version that they do not support installing plugins with install-plugins.sh
jenkinsci/docker@9b54caa

since then, the operator failed to deploy new Jenkins, with the following error:

Installing plugins required by Operator - begin

[[ -z ‘’ ]]
install-plugins.sh
WARN: install-plugins.sh has been removed, please switch to jenkins-plugin-cli

To Reproduce
Steps to reproduce the behavior.
deploy any new version of jenkins (all version changed in the last 3 days)

Additional information

Kubernetes version: 1.22
Jenkins Operator version: any (0.7.1/0.7.0)

Add error logs about the problem here (operator logs and Kubernetes events).

[surmabck]

it has been fixed already in #755. No official release yet, but you can use this one -> virtuslab/jenkins-operator:bef796a

[chrisgrove-keysight]

I am testing the virtuslab/jenkins-operator:bef796a image and Jenkins jenkins/jenkins:2.346.2-lts-jdk11 the behavior is different from virtuslab/jenkins-operator:29c5b1e.

virtuslab/jenkins-operator:29c5b1e behavior

Installing plugins required by user - begin
+ [[ -z '' ]]
+ /var/jenkins/scripts/install-plugins.sh
WARN: install-plugins.sh is deprecated, please switch to jenkins-plugin-cli

Jenkins exits here and Jenkins Operator restarts the pod.

virtuslab/jenkins-operator:bef796a behavior

Installing plugins required by user - begin
+ [[ -z '' ]]
+ /var/jenkins/scripts/install-plugins.sh
WARN: install-plugins.sh is deprecated, please switch to jenkins-plugin-cli
Cleaning up locks
Creating initial locks...
Analyzing war /usr/share/jenkins/jenkins.war...
Registering preinstalled plugins...
Using version-specific update center: https://updates.jenkins.io/dynamic-2.346/...
Downloading plugins...

WAR bundled plugins:
--- snip ---
Installing plugins required by user - end
***************************************************************************
* WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING *
***************************************************************************
Please update your scripts to use /usr/bin/tini going forward.
The previous path has been preserved for backwards compatibility
but WILL BE REMOVED in the future (around Jenkins >= 2.345+).

Now sleeping 2 minutes...

Then over in Jenkins Operator log -

2022-08-02T20:32:09.093Z	INFO	controller-jenkins	Creating a new Jenkins Master Pod jenkins-instances/jenkins-chrisgdev	{"cr": "chrisgdev"}
2022-08-02T20:34:11.306Z	INFO	controller-jenkins	Jenkins master pod restarted by kubernetes: Invalid Jenkins pod phase '{Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-08-02 20:32:09 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-08-02 20:32:09 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [jenkins-master]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-08-02 20:32:09 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [jenkins-master]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-08-02 20:32:09 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:192.168.1.220 PodIP:10.42.0.34 PodIPs:[{IP:10.42.0.34}] StartTime:2022-08-02 20:32:09 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:jenkins-master State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:137,Signal:0,Reason:Error,Message:,StartedAt:2022-08-02 20:32:09 +0000 UTC,FinishedAt:2022-08-02 20:34:10 +0000 UTC,ContainerID:containerd://38d5d123bc734f5d380d82bedfb589463ae35813f690c025f424e3058e698009,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:docker.io/jenkins/jenkins:2.346.2-lts-jdk11 ImageID:docker.io/jenkins/jenkins@sha256:c878e1aac1f5152a6234b33a10542c7f694b7c5c37de27191d1c173800853b93 ContainerID:containerd://38d5d123bc734f5d380d82bedfb589463ae35813f690c025f424e3058e698009 Started:0xc0006c879d}] QOSClass:Burstable EphemeralContainerStatuses:[]}'	{"cr": "chrisgdev"}
2022-08-02T20:34:11.547Z	INFO	controller-jenkins	Creating a new Jenkins Master Pod jenkins-instances/jenkins-chrisgdev	{"cr": "chrisgdev"}

Is there another daily build we should be testing?

[UkonnRa]

Any update on the official release about this fix?

[olen2006]

use jenkins-plugin-cli, read the official documentation
https://github.com/jenkinsci/docker/blob/master/README.md#usage-1

[UkonnRa]

use jenkins-plugin-cli, read the official documentation https://github.com/jenkinsci/docker/blob/master/README.md#usage-1

Can we do anything on the user side and hack the operator to make it using jenkins-plugin-cli rather than install-plugins.sh?

[zi-adis]

@UkonnRa I'm bulding by own Dockerfile to support it

FROM jenkins/jenkins:2.346.2
COPY --chown=jenkins:jenkins plugins.txt /usr/share/jenkins/ref/plugins.txt
RUN jenkins-plugin-cli -f /usr/share/jenkins/ref/plugins.txt
RUN  git config --global http.sslVerify false

# from a derived Dockerfile, can use `RUN install-plugins.sh active.txt` to setup $REF/plugins from a support bundle
COPY install-plugins.sh /usr/local/bin/install-plugins.sh

install-plugins.sh

#!/bin/bash -eu

# Resolve dependencies and download plugins given on the command line
#
# FROM jenkins
# RUN install-plugins.sh docker-slaves github-branch-source
#
# Environment variables:
# REF: directory with preinstalled plugins. Default: /usr/share/jenkins/ref/plugins
# JENKINS_WAR: full path to the jenkins.war. Default: /usr/share/jenkins/jenkins.war
# JENKINS_UC: url of the Update Center. Default: ""
# JENKINS_UC_EXPERIMENTAL: url of the Experimental Update Center for experimental versions of plugins. Default: ""
# JENKINS_INCREMENTALS_REPO_MIRROR: url of the incrementals repo mirror. Default: ""
# JENKINS_UC_DOWNLOAD: download url of the Update Center. Default: JENKINS_UC/download
# CURL_OPTIONS When downloading the plugins with curl. Curl options. Default: -sSfL
# CURL_CONNECTION_TIMEOUT When downloading the plugins with curl. <seconds> Maximum time allowed for connection. Default: 20
# CURL_RETRY When downloading the plugins with curl. Retry request if transient problems occur. Default: 3
# CURL_RETRY_DELAY When downloading the plugins with curl. <seconds> Wait time between retries. Default: 0
# CURL_RETRY_MAX_TIME When downloading the plugins with curl. <seconds> Retry only within this period. Default: 60

set -o pipefail


JENKINS_WAR=${JENKINS_WAR:-/usr/share/jenkins/jenkins.war}

. /usr/local/bin/jenkins-support

REF_DIR="${REF}/plugins"
FAILED="$REF_DIR/failed-plugins.txt"

getLockFile() {
    printf '%s' "$REF_DIR/${1}.lock"
}

getArchiveFilename() {
    printf '%s' "$REF_DIR/${1}.jpi"
}

download() {
    local plugin originalPlugin version lock ignoreLockFile url
    plugin="$1"
    version="${2:-latest}"
    ignoreLockFile="${3:-}"
    url="${4:-}"
    lock="$(getLockFile "$plugin")"

    if [[ $ignoreLockFile ]] || mkdir "$lock" &>/dev/null; then
        if ! doDownload "$plugin" "$version" "$url"; then
            # some plugin don't follow the rules about artifact ID
            # typically: docker-plugin
            originalPlugin="$plugin"
            plugin="${plugin}-plugin"
            if ! doDownload "$plugin" "$version" "$url"; then
                echo "Failed to download plugin: $originalPlugin or $plugin" >&2
                echo "Not downloaded: ${originalPlugin}" >> "$FAILED"
                return 1
            fi
        fi

        if ! checkIntegrity "$plugin"; then
            echo "Downloaded file is not a valid ZIP: $(getArchiveFilename "$plugin")" >&2
            echo "Download integrity: ${plugin}" >> "$FAILED"
            return 1
        fi

        resolveDependencies "$plugin"
    fi
}

doDownload() {
    local plugin version url jpi
    plugin="$1"
    version="$2"
    url="$3"
    jpi="$(getArchiveFilename "$plugin")"

    # If plugin already exists and is the same version do not download
    if test -f "$jpi" && unzip -p "$jpi" META-INF/MANIFEST.MF | tr -d '\r' | grep "^Plugin-Version: ${version}$" > /dev/null; then
        echo "Using provided plugin: $plugin"
        return 0
    fi

    if [[ -n $url ]] ; then
        echo "Will use url=$url"
    elif [[ "$version" == "latest" && -n "$JENKINS_UC_LATEST" ]]; then
        # If version-specific Update Center is available, which is the case for LTS versions,
        # use it to resolve latest versions.
        url="$JENKINS_UC_LATEST/latest/${plugin}.hpi"
    elif [[ "$version" == "experimental" && -n "$JENKINS_UC_EXPERIMENTAL" ]]; then
        # Download from the experimental update center
        url="$JENKINS_UC_EXPERIMENTAL/latest/${plugin}.hpi"
    elif [[ "$version" == incrementals* ]] ; then
        # Download from Incrementals repo: https://jenkins.io/blog/2018/05/15/incremental-deployment/
        # Example URL: https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/workflow/workflow-support/2.19-rc289.d09828a05a74/workflow-support-2.19-rc289.d09828a05a74.hpi
        local groupId incrementalsVersion
        # add a trailing ; so the \n gets added to the end
        readarray -t "-d;" arrIN <<<"${version};";
        unset 'arrIN[-1]';
        groupId=${arrIN[1]}
        incrementalsVersion=${arrIN[2]}
        url="${JENKINS_INCREMENTALS_REPO_MIRROR}/$(echo "${groupId}" | tr '.' '/')/${plugin}/${incrementalsVersion}/${plugin}-${incrementalsVersion}.hpi"
    else
        JENKINS_UC_DOWNLOAD=${JENKINS_UC_DOWNLOAD:-"$JENKINS_UC/download"}
        url="$JENKINS_UC_DOWNLOAD/plugins/$plugin/$version/${plugin}.hpi"
    fi

    echo "Downloading plugin: $plugin from $url"
    # We actually want to allow variable value to be split into multiple options passed to curl.
    # This is needed to allow long options and any options that take value.
    # shellcheck disable=SC2086
    retry_command curl ${CURL_OPTIONS:--sSfL} --connect-timeout "${CURL_CONNECTION_TIMEOUT:-20}" --retry "${CURL_RETRY:-3}" --retry-delay "${CURL_RETRY_DELAY:-0}" --retry-max-time "${CURL_RETRY_MAX_TIME:-60}" "$url" -o "$jpi"
    return $?
}

checkIntegrity() {
    local plugin jpi
    plugin="$1"
    jpi="$(getArchiveFilename "$plugin")"

    unzip -t -qq "$jpi" >/dev/null
    return $?
}

resolveDependencies() {
    local plugin jpi dependencies
    plugin="$1"
    jpi="$(getArchiveFilename "$plugin")"

    dependencies="$(unzip -p "$jpi" META-INF/MANIFEST.MF | tr -d '\r' | tr '\n' '|' | sed -e 's#| ##g' | tr '|' '\n' | grep "^Plugin-Dependencies: " | sed -e 's#^Plugin-Dependencies: ##')"

    if [[ ! $dependencies ]]; then
        echo " > $plugin has no dependencies"
        return
    fi

    echo " > $plugin depends on $dependencies"

    IFS=',' read -r -a array <<< "$dependencies"

    for d in "${array[@]}"
    do
        plugin="$(cut -d':' -f1 - <<< "$d")"
        if [[ $d == *"resolution:=optional"* ]]; then
            echo "Skipping optional dependency $plugin"
        else
            local pluginInstalled
            if pluginInstalled="$(echo -e "${bundledPlugins}\n${installedPlugins}" | grep "^${plugin}:")"; then
                pluginInstalled="${pluginInstalled//[$'\r']}"
                local versionInstalled; versionInstalled=$(versionFromPlugin "${pluginInstalled}")
                local minVersion; minVersion=$(versionFromPlugin "${d}")
                if versionLT "${versionInstalled}" "${minVersion}"; then
                    echo "Upgrading bundled dependency $d ($minVersion > $versionInstalled)"
                    download "$plugin" &
                else
                    echo "Skipping already installed dependency $d ($minVersion <= $versionInstalled)"
                fi
            else
                download "$plugin" &
            fi
        fi
    done
    wait
}

bundledPlugins() {
    if [ -f "$JENKINS_WAR" ]
    then
        TEMP_PLUGIN_DIR=/tmp/plugintemp.$$
        for i in $(jar tf "$JENKINS_WAR" | grep -E '[^detached-]plugins.*\..pi' | sort)
        do
            rm -fr $TEMP_PLUGIN_DIR
            mkdir -p $TEMP_PLUGIN_DIR
            PLUGIN=$(basename "$i"|cut -f1 -d'.')
            (cd $TEMP_PLUGIN_DIR;jar xf "$JENKINS_WAR" "$i";jar xvf "$TEMP_PLUGIN_DIR/$i" META-INF/MANIFEST.MF >/dev/null 2>&1)
            VER=$(grep -E -i Plugin-Version "$TEMP_PLUGIN_DIR/META-INF/MANIFEST.MF"|cut -d: -f2|sed 's/ //')
            echo "$PLUGIN:$VER"
        done
        rm -fr $TEMP_PLUGIN_DIR
    else
        echo "war not found, installing all plugins: $JENKINS_WAR"
    fi
}

versionFromPlugin() {
    local plugin=$1
    if [[ $plugin =~ .*:.* ]]; then
        echo "${plugin##*:}"
    else
        echo "latest"
    fi

}

installedPlugins() {
    for f in "$REF_DIR"/*.jpi; do
        echo "$(basename "$f" | sed -e 's/\.jpi//'):$(get_plugin_version "$f")"
    done
}

jenkinsMajorMinorVersion() {
    if [[ -f "$JENKINS_WAR" ]]; then
        local version major minor
        version="$(java -jar "$JENKINS_WAR" --version)"
        major="$(echo "$version" | cut -d '.' -f 1)"
        minor="$(echo "$version" | cut -d '.' -f 2)"
        echo "$major.$minor"
    else
        echo ""
    fi
}

main() {
    local plugin jenkinsVersion
    local plugins=()

    mkdir -p "$REF_DIR" || exit 1
    rm -f "$FAILED"

    # Read plugins from stdin or from the command line arguments
    if [[ ($# -eq 0) ]]; then
        while read -r line || [ "$line" != "" ]; do
            # Remove leading/trailing spaces, comments, and empty lines
            plugin=$(echo "${line}" | tr -d '\r' | sed -e 's/^[ \t]*//g' -e 's/[ \t]*$//g' -e 's/[ \t]*#.*$//g' -e '/^[ \t]*$/d')

            # Avoid adding empty plugin into array
            if [ ${#plugin} -ne 0 ]; then
                plugins+=("${plugin}")
            fi
        done
    else
        plugins=("$@")
    fi

    # Create lockfile manually before first run to make sure any explicit version set is used.
    echo "Creating initial locks..."
    for plugin in "${plugins[@]}"; do
        mkdir "$(getLockFile "${plugin%%:*}")"
    done

    echo "Analyzing war $JENKINS_WAR..."
    bundledPlugins="$(bundledPlugins)"

    echo "Registering preinstalled plugins..."
    installedPlugins="$(installedPlugins)"

    # Check if there's a version-specific update center, which is the case for LTS versions
    jenkinsVersion="$(jenkinsMajorMinorVersion)"
    if curl -fsL -o /dev/null "$JENKINS_UC/$jenkinsVersion"; then
        JENKINS_UC_LATEST="$JENKINS_UC/$jenkinsVersion"
        echo "Using version-specific update center: $JENKINS_UC_LATEST..."
    else
        JENKINS_UC_LATEST=
    fi

    echo "Downloading plugins..."
    for plugin in "${plugins[@]}"; do
        local reg='^([^:]+):?([^:]+)?:?([^:]+)?:?(http.+)?'
        if [[ $plugin =~ $reg ]]; then
            local pluginId="${BASH_REMATCH[1]}"
            local version="${BASH_REMATCH[2]}"
            local lock="${BASH_REMATCH[3]}"
            local url="${BASH_REMATCH[4]}"
            download "$pluginId" "$version" "${lock:-true}" "${url}" &
        else
          echo "Skipping the line '${plugin}' as it does not look like a reference to a plugin"
        fi
    done
    wait

    echo
    echo "WAR bundled plugins:"
    echo "${bundledPlugins}"
    echo
    echo "Installed plugins:"
    installedPlugins

    if [[ -f $FAILED ]]; then
        echo "Some plugins failed to download!" "$(<"$FAILED")" >&2
        exit 1
    fi

    echo "Cleaning up locks"
    find "$REF_DIR" -regex ".*.lock" | while read -r filepath; do
        rm -r "$filepath"
    done

}

main "$@"

[UkonnRa]

@UkonnRa I'm bulding by own Dockerfile to support it

FROM jenkins/jenkins:2.346.2
COPY --chown=jenkins:jenkins plugins.txt /usr/share/jenkins/ref/plugins.txt
RUN jenkins-plugin-cli -f /usr/share/jenkins/ref/plugins.txt
RUN  git config --global http.sslVerify false

# from a derived Dockerfile, can use `RUN install-plugins.sh active.txt` to setup $REF/plugins from a support bundle
COPY install-plugins.sh /usr/local/bin/install-plugins.sh

install-plugins.sh

#!/bin/bash -eu

# Resolve dependencies and download plugins given on the command line
#
# FROM jenkins
# RUN install-plugins.sh docker-slaves github-branch-source
#
# Environment variables:
# REF: directory with preinstalled plugins. Default: /usr/share/jenkins/ref/plugins
# JENKINS_WAR: full path to the jenkins.war. Default: /usr/share/jenkins/jenkins.war
# JENKINS_UC: url of the Update Center. Default: ""
# JENKINS_UC_EXPERIMENTAL: url of the Experimental Update Center for experimental versions of plugins. Default: ""
# JENKINS_INCREMENTALS_REPO_MIRROR: url of the incrementals repo mirror. Default: ""
# JENKINS_UC_DOWNLOAD: download url of the Update Center. Default: JENKINS_UC/download
# CURL_OPTIONS When downloading the plugins with curl. Curl options. Default: -sSfL
# CURL_CONNECTION_TIMEOUT When downloading the plugins with curl. <seconds> Maximum time allowed for connection. Default: 20
# CURL_RETRY When downloading the plugins with curl. Retry request if transient problems occur. Default: 3
# CURL_RETRY_DELAY When downloading the plugins with curl. <seconds> Wait time between retries. Default: 0
# CURL_RETRY_MAX_TIME When downloading the plugins with curl. <seconds> Retry only within this period. Default: 60

set -o pipefail


JENKINS_WAR=${JENKINS_WAR:-/usr/share/jenkins/jenkins.war}

. /usr/local/bin/jenkins-support

REF_DIR="${REF}/plugins"
FAILED="$REF_DIR/failed-plugins.txt"

getLockFile() {
    printf '%s' "$REF_DIR/${1}.lock"
}

getArchiveFilename() {
    printf '%s' "$REF_DIR/${1}.jpi"
}

download() {
    local plugin originalPlugin version lock ignoreLockFile url
    plugin="$1"
    version="${2:-latest}"
    ignoreLockFile="${3:-}"
    url="${4:-}"
    lock="$(getLockFile "$plugin")"

    if [[ $ignoreLockFile ]] || mkdir "$lock" &>/dev/null; then
        if ! doDownload "$plugin" "$version" "$url"; then
            # some plugin don't follow the rules about artifact ID
            # typically: docker-plugin
            originalPlugin="$plugin"
            plugin="${plugin}-plugin"
            if ! doDownload "$plugin" "$version" "$url"; then
                echo "Failed to download plugin: $originalPlugin or $plugin" >&2
                echo "Not downloaded: ${originalPlugin}" >> "$FAILED"
                return 1
            fi
        fi

        if ! checkIntegrity "$plugin"; then
            echo "Downloaded file is not a valid ZIP: $(getArchiveFilename "$plugin")" >&2
            echo "Download integrity: ${plugin}" >> "$FAILED"
            return 1
        fi

        resolveDependencies "$plugin"
    fi
}

doDownload() {
    local plugin version url jpi
    plugin="$1"
    version="$2"
    url="$3"
    jpi="$(getArchiveFilename "$plugin")"

    # If plugin already exists and is the same version do not download
    if test -f "$jpi" && unzip -p "$jpi" META-INF/MANIFEST.MF | tr -d '\r' | grep "^Plugin-Version: ${version}$" > /dev/null; then
        echo "Using provided plugin: $plugin"
        return 0
    fi

    if [[ -n $url ]] ; then
        echo "Will use url=$url"
    elif [[ "$version" == "latest" && -n "$JENKINS_UC_LATEST" ]]; then
        # If version-specific Update Center is available, which is the case for LTS versions,
        # use it to resolve latest versions.
        url="$JENKINS_UC_LATEST/latest/${plugin}.hpi"
    elif [[ "$version" == "experimental" && -n "$JENKINS_UC_EXPERIMENTAL" ]]; then
        # Download from the experimental update center
        url="$JENKINS_UC_EXPERIMENTAL/latest/${plugin}.hpi"
    elif [[ "$version" == incrementals* ]] ; then
        # Download from Incrementals repo: https://jenkins.io/blog/2018/05/15/incremental-deployment/
        # Example URL: https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/workflow/workflow-support/2.19-rc289.d09828a05a74/workflow-support-2.19-rc289.d09828a05a74.hpi
        local groupId incrementalsVersion
        # add a trailing ; so the \n gets added to the end
        readarray -t "-d;" arrIN <<<"${version};";
        unset 'arrIN[-1]';
        groupId=${arrIN[1]}
        incrementalsVersion=${arrIN[2]}
        url="${JENKINS_INCREMENTALS_REPO_MIRROR}/$(echo "${groupId}" | tr '.' '/')/${plugin}/${incrementalsVersion}/${plugin}-${incrementalsVersion}.hpi"
    else
        JENKINS_UC_DOWNLOAD=${JENKINS_UC_DOWNLOAD:-"$JENKINS_UC/download"}
        url="$JENKINS_UC_DOWNLOAD/plugins/$plugin/$version/${plugin}.hpi"
    fi

    echo "Downloading plugin: $plugin from $url"
    # We actually want to allow variable value to be split into multiple options passed to curl.
    # This is needed to allow long options and any options that take value.
    # shellcheck disable=SC2086
    retry_command curl ${CURL_OPTIONS:--sSfL} --connect-timeout "${CURL_CONNECTION_TIMEOUT:-20}" --retry "${CURL_RETRY:-3}" --retry-delay "${CURL_RETRY_DELAY:-0}" --retry-max-time "${CURL_RETRY_MAX_TIME:-60}" "$url" -o "$jpi"
    return $?
}

checkIntegrity() {
    local plugin jpi
    plugin="$1"
    jpi="$(getArchiveFilename "$plugin")"

    unzip -t -qq "$jpi" >/dev/null
    return $?
}

resolveDependencies() {
    local plugin jpi dependencies
    plugin="$1"
    jpi="$(getArchiveFilename "$plugin")"

    dependencies="$(unzip -p "$jpi" META-INF/MANIFEST.MF | tr -d '\r' | tr '\n' '|' | sed -e 's#| ##g' | tr '|' '\n' | grep "^Plugin-Dependencies: " | sed -e 's#^Plugin-Dependencies: ##')"

    if [[ ! $dependencies ]]; then
        echo " > $plugin has no dependencies"
        return
    fi

    echo " > $plugin depends on $dependencies"

    IFS=',' read -r -a array <<< "$dependencies"

    for d in "${array[@]}"
    do
        plugin="$(cut -d':' -f1 - <<< "$d")"
        if [[ $d == *"resolution:=optional"* ]]; then
            echo "Skipping optional dependency $plugin"
        else
            local pluginInstalled
            if pluginInstalled="$(echo -e "${bundledPlugins}\n${installedPlugins}" | grep "^${plugin}:")"; then
                pluginInstalled="${pluginInstalled//[$'\r']}"
                local versionInstalled; versionInstalled=$(versionFromPlugin "${pluginInstalled}")
                local minVersion; minVersion=$(versionFromPlugin "${d}")
                if versionLT "${versionInstalled}" "${minVersion}"; then
                    echo "Upgrading bundled dependency $d ($minVersion > $versionInstalled)"
                    download "$plugin" &
                else
                    echo "Skipping already installed dependency $d ($minVersion <= $versionInstalled)"
                fi
            else
                download "$plugin" &
            fi
        fi
    done
    wait
}

bundledPlugins() {
    if [ -f "$JENKINS_WAR" ]
    then
        TEMP_PLUGIN_DIR=/tmp/plugintemp.$$
        for i in $(jar tf "$JENKINS_WAR" | grep -E '[^detached-]plugins.*\..pi' | sort)
        do
            rm -fr $TEMP_PLUGIN_DIR
            mkdir -p $TEMP_PLUGIN_DIR
            PLUGIN=$(basename "$i"|cut -f1 -d'.')
            (cd $TEMP_PLUGIN_DIR;jar xf "$JENKINS_WAR" "$i";jar xvf "$TEMP_PLUGIN_DIR/$i" META-INF/MANIFEST.MF >/dev/null 2>&1)
            VER=$(grep -E -i Plugin-Version "$TEMP_PLUGIN_DIR/META-INF/MANIFEST.MF"|cut -d: -f2|sed 's/ //')
            echo "$PLUGIN:$VER"
        done
        rm -fr $TEMP_PLUGIN_DIR
    else
        echo "war not found, installing all plugins: $JENKINS_WAR"
    fi
}

versionFromPlugin() {
    local plugin=$1
    if [[ $plugin =~ .*:.* ]]; then
        echo "${plugin##*:}"
    else
        echo "latest"
    fi

}

installedPlugins() {
    for f in "$REF_DIR"/*.jpi; do
        echo "$(basename "$f" | sed -e 's/\.jpi//'):$(get_plugin_version "$f")"
    done
}

jenkinsMajorMinorVersion() {
    if [[ -f "$JENKINS_WAR" ]]; then
        local version major minor
        version="$(java -jar "$JENKINS_WAR" --version)"
        major="$(echo "$version" | cut -d '.' -f 1)"
        minor="$(echo "$version" | cut -d '.' -f 2)"
        echo "$major.$minor"
    else
        echo ""
    fi
}

main() {
    local plugin jenkinsVersion
    local plugins=()

    mkdir -p "$REF_DIR" || exit 1
    rm -f "$FAILED"

    # Read plugins from stdin or from the command line arguments
    if [[ ($# -eq 0) ]]; then
        while read -r line || [ "$line" != "" ]; do
            # Remove leading/trailing spaces, comments, and empty lines
            plugin=$(echo "${line}" | tr -d '\r' | sed -e 's/^[ \t]*//g' -e 's/[ \t]*$//g' -e 's/[ \t]*#.*$//g' -e '/^[ \t]*$/d')

            # Avoid adding empty plugin into array
            if [ ${#plugin} -ne 0 ]; then
                plugins+=("${plugin}")
            fi
        done
    else
        plugins=("$@")
    fi

    # Create lockfile manually before first run to make sure any explicit version set is used.
    echo "Creating initial locks..."
    for plugin in "${plugins[@]}"; do
        mkdir "$(getLockFile "${plugin%%:*}")"
    done

    echo "Analyzing war $JENKINS_WAR..."
    bundledPlugins="$(bundledPlugins)"

    echo "Registering preinstalled plugins..."
    installedPlugins="$(installedPlugins)"

    # Check if there's a version-specific update center, which is the case for LTS versions
    jenkinsVersion="$(jenkinsMajorMinorVersion)"
    if curl -fsL -o /dev/null "$JENKINS_UC/$jenkinsVersion"; then
        JENKINS_UC_LATEST="$JENKINS_UC/$jenkinsVersion"
        echo "Using version-specific update center: $JENKINS_UC_LATEST..."
    else
        JENKINS_UC_LATEST=
    fi

    echo "Downloading plugins..."
    for plugin in "${plugins[@]}"; do
        local reg='^([^:]+):?([^:]+)?:?([^:]+)?:?(http.+)?'
        if [[ $plugin =~ $reg ]]; then
            local pluginId="${BASH_REMATCH[1]}"
            local version="${BASH_REMATCH[2]}"
            local lock="${BASH_REMATCH[3]}"
            local url="${BASH_REMATCH[4]}"
            download "$pluginId" "$version" "${lock:-true}" "${url}" &
        else
          echo "Skipping the line '${plugin}' as it does not look like a reference to a plugin"
        fi
    done
    wait

    echo
    echo "WAR bundled plugins:"
    echo "${bundledPlugins}"
    echo
    echo "Installed plugins:"
    installedPlugins

    if [[ -f $FAILED ]]; then
        echo "Some plugins failed to download!" "$(<"$FAILED")" >&2
        exit 1
    fi

    echo "Cleaning up locks"
    find "$REF_DIR" -regex ".*.lock" | while read -r filepath; do
        rm -r "$filepath"
    done

}

main "$@"

Thanks for your response! Finally I choose to use virtuslab/jenkins-operator:94b1473 rather than a release version. Hoping this feature can be released soon.

[jay-johnson]

I was able to get passed this error, but it's hacky:

On jenkins-master (docker.io/jenkins/jenkins:2.366), the container is starting up with:

/var/jenkins/scripts/init.sh

and that fails with a non-zero exit status code due to what I believe is the in-progress migration to the jenkins-plugin-cli.

Defaulted container "jenkins-master" out of: jenkins-master, backup
+ '[' '' == true ']'
+ echo 'To print debug messages set environment variable '\''DEBUG_JENKINS_OPERATOR'\'' to '\''true'\'''
+ mkdir -p /var/lib/jenkins/init.groovy.d
To print debug messages set environment variable 'DEBUG_JENKINS_OPERATOR' to 'true'
+ cp -n /var/jenkins/init-configuration/createOperatorUser.groovy /var/lib/jenkins/init.groovy.d
+ mkdir -p /var/lib/jenkins/scripts
+ cp /var/jenkins/scripts/init.sh /var/jenkins/scripts/install-plugins.sh /var/lib/jenkins/scripts
+ chmod +x /var/lib/jenkins/scripts/init.sh /var/lib/jenkins/scripts/install-plugins.sh
+ echo 'Installing plugins required by Operator - begin'
Installing plugins required by Operator - begin
+ cat
+ [[ -z '' ]]
+ install-plugins.sh
WARN: install-plugins.sh has been removed, please switch to jenkins-plugin-cli

^ at this point the pod would enter the Terminating state and the operator would reap/restart the jenkins-jenkins pod in a loop. I believe this is due to a few code sections that need to migrate to the new jenkins-plugin-cli. Here's a snippet for reference:

kubernetes-operator/pkg/configuration/base/resources/scripts_configmap.go

Lines 288 to 292 in 3a143a0

	if [[ -z "${OPENSHIFT_JENKINS_IMAGE_VERSION}" ]]; then
	{{ $installPluginsCommand }} < {{ .JenkinsHomePath }}/user-plugins
	else
	{{ $installPluginsCommand }} {{ .JenkinsHomePath }}/user-plugins
	fi

I am opening a pr, but I do not know if the OpenShift case will work or not because I only tested this on minikube (1.24.3).

For now, I already installed all my plugins using jenkins-plugin-cli within my jenkins-master docker image:

jenkins@jenkins-jenkins:/$ jenkins-plugin-cli -f /usr/share/jenkins/ref/plugins.txt
Done
jenkins@jenkins-jenkins:/$ echo $?
0

So I can bypass the error by overriding the jenkins-master's default entrypoint/command:

    containers:
    - name: jenkins-master
      image: <custom jenkins image built in an aws ecr>
      imagePullPolicy: Always
      command:
      - sh
      - -c
      - |
        #!/usr/bin/env bash -e
        echo "initializing: /var/jenkins/scripts/init.sh" 
        /var/jenkins/scripts/init.sh
        last_status="$$?" 
        echo "/var/jenkins/scripts/init.sh exited with status=$${last_status}"
        echo "starting jenkins: exec /usr/bin/tini -s -- /usr/local/bin/jenkins.sh"
        exec /usr/bin/tini -s -- /usr/local/bin/jenkins.sh

^ logs this on startup:

Defaulted container "jenkins-master" out of: jenkins-master, backup
initializing: /var/jenkins/scripts/init.sh
To print debug messages set environment variable 'DEBUG_JENKINS_OPERATOR' to 'true'
+ '[' '' == true ']'
+ echo 'To print debug messages set environment variable '\''DEBUG_JENKINS_OPERATOR'\'' to '\''true'\'''
+ mkdir -p /var/lib/jenkins/init.groovy.d
+ cp -n /var/jenkins/init-configuration/createOperatorUser.groovy /var/lib/jenkins/init.groovy.d
+ mkdir -p /var/lib/jenkins/scripts
+ cp /var/jenkins/scripts/init.sh /var/jenkins/scripts/install-plugins.sh /var/lib/jenkins/scripts
+ chmod +x /var/lib/jenkins/scripts/init.sh /var/lib/jenkins/scripts/install-plugins.sh
+ echo 'Installing plugins required by Operator - begin'
Installing plugins required by Operator - begin
+ cat
+ [[ -z '' ]]
+ install-plugins.sh
WARN: install-plugins.sh has been removed, please switch to jenkins-plugin-cli
/var/jenkins/scripts/init.sh exited with status=1
starting jenkins: exec /usr/bin/tini -s -- /usr/local/bin/jenkins.sh
Running from: /usr/share/jenkins/jenkins.war

^ with the plugins already installed in the docker image, the jenkins-jenkins pod is now able to startup.

Note there was an additional downstream error/sleep that I fixed by changing the path for tini. Previously this led to a 2 minute sleep (see block below). That's why I added both commands to the command yaml in the jenkins-master yaml section.

Before I found the tini error:

exec /sbin/tini -s -- /usr/local/bin/jenkins.sh
***************************************************************************
* WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING *
***************************************************************************
Please update your scripts to use /usr/bin/tini going forward.
The previous path has been preserved for backwards compatibility
but WILL BE REMOVED in the future (around Jenkins >= 2.345+).

Now sleeping 2 minutes...

^ after seeing this sleep message, I migrated to /usr/bin/tini and here's the pod's process table:

jenkins@jenkins-jenkins:/$ ps auwwx
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
jenkins        1  0.0  0.0   2316   572 ?        Ss   19:03   0:00 /usr/bin/tini -s -- /usr/local/bin/jenkins.sh
jenkins       15  8.6  0.5 7925132 2472648 ?     Sl   19:03   2:18 java -Duser.home=/var/lib/jenkins -Djenkins.install.runSetupWizard=false -Djava.awt.headless=true -Djenkins.model.Jenkins.slaveAgentPort=50000 -Dhudson.lifecycle=hudson.lifecycle.ExitLifecycle -jar /usr/share/jenkins/jenkins.war
jenkins     1611  0.0  0.0   5992  3768 pts/0    Ss   19:16   0:00 bash
jenkins     1722  0.0  0.0   8592  3252 pts/0    R+   19:29   0:00 ps auwwx

[2002batman]

tried several fixes and hit many errors like

• WARN: install-plugins.sh has been removed, please switch to jenkins-plugin-cli
• Failed Loading plugin JavaBeans Activation Framework (JAF) API v1.2.0-5 (javax-activation-api)
  Jenkins (2.332.1) or higher required
• bash: line 1: /sbin/tini: No such file or directory

finally composed a values.yaml that works

jenkins:
  name: op08

  image: jenkins/jenkins:lts-alpine-jdk11

  basePlugins:
  - name: kubernetes
    version: "3985.vd26d77b_2a_48a_"
  - name: workflow-job
    version: "1316.vd2290d3341a_f"
  - name: workflow-aggregator
    version: "596.v8c21c963d92d"
  - name: git
    version: "5.2.0"
  - name: job-dsl
    version: "1.84"
  - name: configuration-as-code
    version: "1647.ve39ca_b_829b_42"
  - name: kubernetes-credentials-provider
    version: "1.225.v14f9e6b_28f53"    
  - name: build-pipeline-plugin
    version: "1.5.8"    
  - name: github-branch-source
    version: "1732.v3f1889a_c475b_"


operator:
 replicaCount: 1

 image: virtuslab/jenkins-operator:bef796a

helm install jenkins-op08 jenkins/jenkins-operator -f values.yaml