feat: build both agent and inbound agent

[lemeurherve]

This PR allows to build both agent and inbound-agent container images in this repository.

Note: each type of image ("agent" or "inbound-agent") has its corresponding tests suite.

I've splitted this pull requests in atomic operations:

• rename docker-inbound-agent files before import: lemeurherve@ab173cf
• rename docker-agent files before import: 77850e6
• import prepared docker-inbound-agent files: cfc94cc (merge commit to keep imported files history)
• feat: build both agent and inbound-agent images: 0ab91ff (this is the main one to review)

New/renamed/imported/modified files list:

New file

• README.md

Modified

• .gitignore
• alpine/Dockerfile
• build.ps1
• build.sh
• debian/Dockerfile
• debian/preview/Dockerfile
• docker-bake.hcl
• Jenkinsfile
• Makefile
• tests/agent.Tests.ps1
• tests/test_helpers.bash
• tests/test_helpers.psm1
• windows/nanoserver/Dockerfile
• windows/windowsservercore/Dockerfile

Renamed from agent

• README_agent.md

Renamed and modified from agent

• tests/tests_agent.bats

Imported from inbound-agent

• jenkins-agent
• jenkins-agent.ps1
• tests/netcat-helper/Dockerfile
• tests/netcat-helper/Dockerfile-windows

Imported and renamed from inbound-agent

• README_inbound-agent.md

Imported and modified from inbound-agent

• tests/inbound-agent.Tests.ps1

Imported, renamed and modified from inbound-agent

• .github/workflows/update-dockerhub-description.yaml (was sync-dockerhub-readme.yml)
• tests/tests_inbound-agent.bats

How I imported files from inbound-agent with their git history:

---

I created a script (adapted from https://gist.github.com/tsayen/f1c1c4d62d4fda77abf1586bd39f9b74) with the following content to retrieve git history of a file as a set of email patches and to replace `#n` references by `jenkinsci/docker-inbound-agent#n`:

#! /bin/bash
# Usage:
# ./git-patch.sh path/to/file/or/dir path/to/destination/repo
echo "creating patch for path ${1}"
git log --name-only --pretty="format:" --follow  "${1}" \
 | sort -u | \
 xargs git log --pretty=email --patch-with-stat --reverse --full-index --binary -m --first-parent -- > "${2}/_patch_" \
&& cat "${2}/_patch_" | sed 's/Merge pull request #/Merge pull request jenkinsci\/docker-inbound-agent#/' >> "${2}/_all_patches_"

As applying this script on multiple files at once resulted in a unordered history, I applied this script on every imported inbound-agent file from the list above, one after the other, manually (and painfully) reordering each generated patch in the correct order.

When all files have been proceed, I've got as a result a big patch file that I applied to the agent repository folder with the following command:

git am --committer-date-is-author-date --keep < ../_all_patches_

Here is the corresponding patches file (renammed in .txt to be attached here): [_all_patches_.txt](https://github.com/jenkinsci/docker-agent/files/13601317/_all_patches_.txt)

State of the agent repository after applying these patches and before renaming/modifying files if you want to check that the content of these files haven't been changed by this process and that no unwanted modifications have been introduced:
https://github.com/lemeurherve/docker-agent/tree/aa4262466bab1782f9c946357205c76a596c862f

The downside of this process is that even if I'm not the author of these inbound-agent commits, I didn't manage not to be the (co-)commiter of them.

---

EDIT: I wrote a script using https://github.com/newren/git-filter-repo allowing me to rework this PR and this import to get a proper history.

#!/usr/bin/env bash

set -ex

# Cleanup
rm -rf docker-agent
rm -rf docker-inbound-agent
rm -rf pullrequest

# Retrieve content of the pull request from my fork, will be used later
git clone https://github.com/lemeurherve/docker-agent pullrequest
cd pullrequest
git fetch --all
git checkout feat-build-both-agent-and-inbound-agent

cd ..

# Commit message replacement
echo "Merge pull request #==>Merge pull request jenkinsci/docker-inbound-agent#" > commits-msg-replacement.txt

# Prepare docker-inbound-agent for import
git clone https://github.com/jenkinsci/docker-inbound-agent

cd docker-inbound-agent

# Trim repo to keep only files to import, and replace GitHub pull request references in commit messages
git filter-repo \
    --path tests/netcat-helper/Dockerfile \
    --path tests/netcat-helper/Dockerfile-windows \
    --path tests/inboundAgent.Tests.ps1 \
    --path tests/tests.bats \
    --path .github/workflows/sync-dockerhub-readme.yaml \
    --path jenkins-agent \
    --path jenkins-agent.ps1 \
    --path README.md \
    --replace-message ../commits-msg-replacement.txt

# Rename files before import to avoid collisions
git mv README.md README_inbound-agent.md
git mv .github/workflows/sync-dockerhub-readme.yaml .github/workflows/update-dockerhub-description.yaml
git mv tests/inboundAgent.Tests.ps1 tests/inbound-agent.Tests.ps1
git mv tests/tests.bats tests/tests_inbound-agent.bats

git status
git commit -m "rename \`docker-inbound-agent\` files before import"

cd ..


# Prepare docker-agent before import
git clone https://github.com/jenkinsci/docker-agent

cd docker-agent

# Rename files before import to avoid collisions
git mv README.md README_agent.md
git mv tests/tests.bats tests/tests_agent.bats

git status
git commit -m "rename \`docker-agent\` files before import"

# Import prepared docker-inbound-agent into docker-agent'
git remote add docker-inbound-agent ../docker-inbound-agent
git fetch docker-inbound-agent --no-tags
git merge --allow-unrelated-histories docker-inbound-agent/master -m "import prepared \`docker-inbound-agent\` files"
git remote remove docker-inbound-agent

### Import finished
### Below are the commands used to retrieve the modifications from this pull request automatically

# Update docker-agent with the modified content from the pull request
## New file
cp ../pullrequest/README.md README.md

## Modified in agent
cp ../pullrequest/.gitignore .gitignore
cp ../pullrequest/alpine/Dockerfile alpine/Dockerfile
cp ../pullrequest/build.ps1 build.ps1
cp ../pullrequest/build.sh build.sh
cp ../pullrequest/debian/Dockerfile debian/Dockerfile
cp ../pullrequest/debian/preview/Dockerfile debian/preview/Dockerfile
cp ../pullrequest/docker-bake.hcl docker-bake.hcl
cp ../pullrequest/Jenkinsfile Jenkinsfile
cp ../pullrequest/Makefile Makefile
cp ../pullrequest/tests/agent.Tests.ps1 tests/agent.Tests.ps1
cp ../pullrequest/tests/test_helpers.bash tests/test_helpers.bash
cp ../pullrequest/tests/test_helpers.psm1 tests/test_helpers.psm1
cp ../pullrequest/windows/nanoserver/Dockerfile windows/nanoserver/Dockerfile
cp ../pullrequest/windows/windowsservercore/Dockerfile windows/windowsservercore/Dockerfile

## Renamed and modified from agent
cp ../pullrequest/tests/tests_agent.bats tests/tests_agent.bats

## Imported and modified from inbound-agent
cp ../pullrequest/tests/inbound-agent.Tests.ps1 tests/inbound-agent.Tests.ps1

## Imported, renamed and modified from inbound-agent
cp ../pullrequest/.github/workflows/update-dockerhub-description.yaml .github/workflows/update-dockerhub-description.yaml
cp ../pullrequest/tests/tests_inbound-agent.bats tests/tests_inbound-agent.bats

# Commit changes from pull request
git status
git add .
git commit -m "feat: build both \`agent\` and \`inbound-agent\` images"

# Create verification branch
git checkout -b verification-build-both-agent-and-inbound-agent

# Create branch for the pull request update
git checkout -b feat-build-both-agent-and-inbound-agent

# Set appropriate remotes
git remote remove origin
git remote add origin https://github.com/lemeurherve/docker-agent.git
git remote add upstream https://github.com/jenkinsci/docker-agent.git

As a bonus, I can reapply this script whenever I want then I just have to git push origin feat-build-both-agent-and-inbound-agent --force to update this PR.

Ref and details about proposed procedure for the next steps:

• Build both agent and inbound-agent container images in this repository #569

Testing done

• Ran on a Macintosh M2 for Linux images and on a Windows VM with the same image used by agents for Windows images.
• CI build: https://ci.jenkins.io/job/Packaging/job/docker-agent/job/PR-570/10/
• Compared files lists from sudo find / between these images and the existing ones
• Used docker scout compare between these images and the existing ones: https://docs.docker.com/engine/reference/commandline/scout_compare/
• As suggested by @basil, ran a basic job using https://github.com/jenkinsci/docker-plugin with both "Launch attached" (agent) and "Launch via JNLP" (inbound-agent) with the existing images, then with these images

Submitter checklist

Preview Give feedback

1. Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
2. Ensure that the pull request title represents the desired changelog entry
3. Please describe what you did
4. Link to relevant issues in GitHub or Jira
5. Link to relevant pull requests, esp. upstream and downstream changes
6. Ensure you have provided tests - that demonstrates feature works or fixes the issue

[basil]

Thank you, this sounds fine to me.

[basil]

(My comments about the merge of the other repository's commits remain unaddressed.)

[lemeurherve]

Currently trying git-filter-repo to see if I can properly reimport inbound-agent files and their git history without too much time and effort.

[lemeurherve]

Currently trying git-filter-repo

Got it working, thanks for the hint @basil.

I'm going to push with force on this branch to keep existing comments here, only timja review should be discarded.

EDIT: nothing lost, just broke my own review comments not visible anymore in the "Files" tabs. I can repost them if needed.

[lemeurherve]

Check the updated PR body if you want to see the script I used to get a proper import of docker-inbound-agent files and their history.

Force pushed twice as I initially forgot to replace Merge pull request # by Merge pull request jenkinsci/docker-inbound-agent# in imported docker-inbound-agent commits messages to get proper GitHub references.

[basil]

Very nice!

[lemeurherve]

Release of https://github.com/jenkinsci/docker-agent/releases/tag/3206.vb_15dcf73f6a_9-2 for both agent and inbound-agent successful!

The first build on trusted.ci.jenkins.io failed publishing the "linux" images with the following error:

unexpected status from HEAD request to https://registry-1.docker.io/v2/jenkins/agent/blobs/...: 429 Too Many Requests

After a few minutes, I replayed a build only on "linux" images, it finished with success.

--- old/Jenkinsfile
+++ new/Jenkinsfile
@@ -24,7 +24,7 @@
                 axes {
                     axis {
                         name 'IMAGE_TYPE'
-                        values 'linux', 'nanoserver-1809', 'nanoserver-ltsc2019', 'nanoserver-ltsc2022', 'windowsservercore-1809', 'windowsservercore-ltsc2019', 'windowsservercore-ltsc2022'
+                        values 'linux'
                     }
                 }

https://hub.docker.com/r/jenkins/agent/tags
https://hub.docker.com/r/jenkins/inbound-agent/tags

[ksalerno99]

Great work, guys! This eliminates 11 lines from my test build environment!

docker exec -ti docker-buildx-$ARCH /bin/bash -c "cd; \
			git clone https://github.com/jenkinsci/docker-inbound-agent.git; \
			cd docker-inbound-agent && \
			TAG=\$(grep -A1 PARENT_IMAGE_VERSION docker-bake.hcl | \
				head -2 | tail -1 | awk '{ print \$NF }' | \
				tr -d '\"') && \
			docker image tag jenkins/agent:jdk17 \
				jenkins/agent:\$TAG-jdk11 && \
			docker image tag jenkins/agent:jdk21 \
				jenkins/agent:\$TAG-jdk17 && \
			time make build"