Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: use docker bake to generate docker compose file #415

Merged
merged 1 commit into from
Jun 19, 2024

Conversation

lemeurherve
Copy link
Member

@lemeurherve lemeurherve commented Jun 7, 2024

This PR uses docker bake to generate the content of the docker compose files for Windows images from the same docker-bake.hcl used for Linux images.

Fixes:

The JSON output of docker buildx bake windows --print is converted with yq into the YAML format expected by docker compose to build the corresponding images. (We can't use bake for building as it's not functional on Windows yet.)

The docker compose files are generated (here) from a Linux agent as it avoids tedious yq queries escaping for PowerShell and installing docker buildx on Windows agents.

EDIT:
As docker buildx bake windows --print can be used on Windows, make isn't required for contributors.
And as yq is already required, the only new requirement for contributors is https://github.com/docker/buildx included in Docker Desktop, one of the most common docker installation types on Windows.

Example of resulting build-windows.yaml (note that no arg depends on any variable)
services:
  nanoserver-1809_jdk11:
    image: docker.io/jenkins/ssh-agent:1.2.3-nanoserver-1809-jdk11
    build:
      context: .
      dockerfile: windows/nanoserver/Dockerfile
      args:
        JAVA_HOME: C:/openjdk-11
        JAVA_VERSION: 11.0.23+9
        TOOLS_WINDOWS_VERSION: "1809"
        WINDOWS_VERSION_TAG: "1809"
      tags:
        - docker.io/jenkins/ssh-agent:1.2.3-nanoserver-1809-jdk11
        - docker.io/jenkins/ssh-agent:nanoserver-1809-jdk11
  nanoserver-1809_jdk17:
    image: docker.io/jenkins/ssh-agent:1.2.3-nanoserver-1809-jdk17
    build:
      context: .
      dockerfile: windows/nanoserver/Dockerfile
      args:
        JAVA_HOME: C:/openjdk-17
        JAVA_VERSION: 17.0.11+9
        TOOLS_WINDOWS_VERSION: "1809"
        WINDOWS_VERSION_TAG: "1809"
      tags:
        - docker.io/jenkins/ssh-agent:1.2.3-nanoserver-1809-jdk17
        - docker.io/jenkins/ssh-agent:1.2.3-nanoserver-1809
        - docker.io/jenkins/ssh-agent:nanoserver-1809
        - docker.io/jenkins/ssh-agent:nanoserver-1809-jdk17
  nanoserver-1809_jdk21:
    image: docker.io/jenkins/ssh-agent:1.2.3-nanoserver-1809-jdk21
    build:
      context: .
      dockerfile: windows/nanoserver/Dockerfile
      args:
        JAVA_HOME: C:/openjdk-21
        JAVA_VERSION: 21.0.3+9
        TOOLS_WINDOWS_VERSION: "1809"
        WINDOWS_VERSION_TAG: "1809"
      tags:
        - docker.io/jenkins/ssh-agent:1.2.3-nanoserver-1809-jdk21
        - docker.io/jenkins/ssh-agent:nanoserver-1809-jdk21
  nanoserver-ltsc2019_jdk11:
    image: docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2019-jdk11
    build:
      context: .
      dockerfile: windows/nanoserver/Dockerfile
      args:
        JAVA_HOME: C:/openjdk-11
        JAVA_VERSION: 11.0.23+9
        TOOLS_WINDOWS_VERSION: "1809"
        WINDOWS_VERSION_TAG: ltsc2019
      tags:
        - docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2019-jdk11
        - docker.io/jenkins/ssh-agent:nanoserver-ltsc2019-jdk11
  nanoserver-ltsc2019_jdk17:
    image: docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2019-jdk17
    build:
      context: .
      dockerfile: windows/nanoserver/Dockerfile
      args:
        JAVA_HOME: C:/openjdk-17
        JAVA_VERSION: 17.0.11+9
        TOOLS_WINDOWS_VERSION: "1809"
        WINDOWS_VERSION_TAG: ltsc2019
      tags:
        - docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2019-jdk17
        - docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2019
        - docker.io/jenkins/ssh-agent:nanoserver-ltsc2019
        - docker.io/jenkins/ssh-agent:nanoserver-ltsc2019-jdk17
  nanoserver-ltsc2019_jdk21:
    image: docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2019-jdk21
    build:
      context: .
      dockerfile: windows/nanoserver/Dockerfile
      args:
        JAVA_HOME: C:/openjdk-21
        JAVA_VERSION: 21.0.3+9
        TOOLS_WINDOWS_VERSION: "1809"
        WINDOWS_VERSION_TAG: ltsc2019
      tags:
        - docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2019-jdk21
        - docker.io/jenkins/ssh-agent:nanoserver-ltsc2019-jdk21
  nanoserver-ltsc2022_jdk11:
    image: docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2022-jdk11
    build:
      context: .
      dockerfile: windows/nanoserver/Dockerfile
      args:
        JAVA_HOME: C:/openjdk-11
        JAVA_VERSION: 11.0.23+9
        TOOLS_WINDOWS_VERSION: ltsc2022
        WINDOWS_VERSION_TAG: ltsc2022
      tags:
        - docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2022-jdk11
        - docker.io/jenkins/ssh-agent:nanoserver-ltsc2022-jdk11
  nanoserver-ltsc2022_jdk17:
    image: docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2022-jdk17
    build:
      context: .
      dockerfile: windows/nanoserver/Dockerfile
      args:
        JAVA_HOME: C:/openjdk-17
        JAVA_VERSION: 17.0.11+9
        TOOLS_WINDOWS_VERSION: ltsc2022
        WINDOWS_VERSION_TAG: ltsc2022
      tags:
        - docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2022-jdk17
        - docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2022
        - docker.io/jenkins/ssh-agent:nanoserver-ltsc2022
        - docker.io/jenkins/ssh-agent:nanoserver-ltsc2022-jdk17
  nanoserver-ltsc2022_jdk21:
    image: docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2022-jdk21
    build:
      context: .
      dockerfile: windows/nanoserver/Dockerfile
      args:
        JAVA_HOME: C:/openjdk-21
        JAVA_VERSION: 21.0.3+9
        TOOLS_WINDOWS_VERSION: ltsc2022
        WINDOWS_VERSION_TAG: ltsc2022
      tags:
        - docker.io/jenkins/ssh-agent:1.2.3-nanoserver-ltsc2022-jdk21
        - docker.io/jenkins/ssh-agent:nanoserver-ltsc2022-jdk21
  windowsservercore-ltsc2019_jdk11:
    image: docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2019-jdk11
    build:
      context: .
      dockerfile: windows/windowsservercore/Dockerfile
      args:
        JAVA_HOME: C:/openjdk-11
        JAVA_VERSION: 11.0.23+9
        TOOLS_WINDOWS_VERSION: "1809"
        WINDOWS_VERSION_TAG: ltsc2019
      tags:
        - docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2019-jdk11
        - docker.io/jenkins/ssh-agent:windowsservercore-ltsc2019-jdk11
  windowsservercore-ltsc2019_jdk17:
    image: docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2019-jdk17
    build:
      context: .
      dockerfile: windows/windowsservercore/Dockerfile
      args:
        JAVA_HOME: C:/openjdk-17
        JAVA_VERSION: 17.0.11+9
        TOOLS_WINDOWS_VERSION: "1809"
        WINDOWS_VERSION_TAG: ltsc2019
      tags:
        - docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2019-jdk17
        - docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2019
        - docker.io/jenkins/ssh-agent:windowsservercore-ltsc2019
        - docker.io/jenkins/ssh-agent:windowsservercore-ltsc2019-jdk17
  windowsservercore-ltsc2019_jdk21:
    image: docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2019-jdk21
    build:
      context: .
      dockerfile: windows/windowsservercore/Dockerfile
      args:
        JAVA_HOME: C:/openjdk-21
        JAVA_VERSION: 21.0.3+9
        TOOLS_WINDOWS_VERSION: "1809"
        WINDOWS_VERSION_TAG: ltsc2019
      tags:
        - docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2019-jdk21
        - docker.io/jenkins/ssh-agent:windowsservercore-ltsc2019-jdk21
  windowsservercore-ltsc2022_jdk11:
    image: docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2022-jdk11
    build:
      context: .
      dockerfile: windows/windowsservercore/Dockerfile
      args:
        JAVA_HOME: C:/openjdk-11
        JAVA_VERSION: 11.0.23+9
        TOOLS_WINDOWS_VERSION: ltsc2022
        WINDOWS_VERSION_TAG: ltsc2022
      tags:
        - docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2022-jdk11
        - docker.io/jenkins/ssh-agent:windowsservercore-ltsc2022-jdk11
  windowsservercore-ltsc2022_jdk17:
    image: docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2022-jdk17
    build:
      context: .
      dockerfile: windows/windowsservercore/Dockerfile
      args:
        JAVA_HOME: C:/openjdk-17
        JAVA_VERSION: 17.0.11+9
        TOOLS_WINDOWS_VERSION: ltsc2022
        WINDOWS_VERSION_TAG: ltsc2022
      tags:
        - docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2022-jdk17
        - docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2022
        - docker.io/jenkins/ssh-agent:windowsservercore-ltsc2022
        - docker.io/jenkins/ssh-agent:windowsservercore-ltsc2022-jdk17
  windowsservercore-ltsc2022_jdk21:
    image: docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2022-jdk21
    build:
      context: .
      dockerfile: windows/windowsservercore/Dockerfile
      args:
        JAVA_HOME: C:/openjdk-21
        JAVA_VERSION: 21.0.3+9
        TOOLS_WINDOWS_VERSION: ltsc2022
        WINDOWS_VERSION_TAG: ltsc2022
      tags:
        - docker.io/jenkins/ssh-agent:1.2.3-windowsservercore-ltsc2022-jdk21
        - docker.io/jenkins/ssh-agent:windowsservercore-ltsc2022-jdk21

Obtained with:

ON_TAG=true VERSION=1.2.3 docker buildx bake -f docker-bake.hcl windows --print \
| yq -P '.target[] | del(.platforms) | {(. | key): {"image": .tags[0], "build": .}}'
| yq '{"services": .}'

Note: this command works for Linux/MacOS, Windows (or more specifically PowerShell) requires backslashes in front of every double quote and setting environment variables on their own.

Enhancements:

  • Regroup all Linux and Windows image definitions in a common docker-bake.hcl file containing every parameters and helper functions
  • Remove build-windows.yaml
  • Simplify Jenkinsfile
  • Simplify build.ps1 by not having to massage parameters to deal with the fact that there are no mcr.microsoft.com/powershell ltsc2019 images, only 1809 ones
  • Simplify updatecli manifests
  • Prepare the terrain for when docker buildx will be functional to build Windows images

Images content remains untouched hence the "chore" label.

Testing done

Local tests and CI.

Submitter checklist

  • Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
  • Ensure that the pull request title represents the desired changelog entry
  • Please describe what you did
  • Link to relevant issues in GitHub or Jira
  • Link to relevant pull requests, esp. upstream and downstream changes
  • Ensure you have provided tests - that demonstrates feature works or fixes the issue

@lemeurherve lemeurherve changed the title chore(Windows): use docker bake to generate the docker compose file chore(Windows): use docker bake Jun 7, 2024
@lemeurherve lemeurherve force-pushed the docker-bake-windows branch from 1ecdf6d to 478b69e Compare June 8, 2024 04:00
@lemeurherve lemeurherve changed the title chore(Windows): use docker bake chore(Windows): generate docker compose from docker bake Jun 8, 2024
@lemeurherve lemeurherve force-pushed the docker-bake-windows branch 15 times, most recently from bb9760a to 53e511a Compare June 8, 2024 20:53
@lemeurherve lemeurherve marked this pull request as ready for review June 8, 2024 20:55
@lemeurherve lemeurherve requested a review from a team as a code owner June 8, 2024 20:55
@lemeurherve lemeurherve changed the title chore(Windows): generate docker compose from docker bake chore: generate docker compose files from docker bake Jun 8, 2024
Copy link
Contributor

@dduportal dduportal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you update https://github.com/jenkinsci/docker-ssh-agent?tab=readme-ov-file#building-and-testing-on-windows as part of this PR please?

Given

The docker compose files are generated from a Linux agent as it avoids tedious yq queries escaping for PowerShell and installing docker buildx on Windows agents.

It looks like you need a lot of additional tools if you are on Windows to be able to build the image (including GNU make and yq): am I correct? If yes, wether we agree or not, this PR need the doc. to be updated to guide contributors

docker-bake.hcl Outdated Show resolved Hide resolved
docker-bake.hcl Outdated Show resolved Hide resolved
Copy link
Contributor

@dduportal dduportal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a really interesting PR technically speaking, showing a lot of different usages and improvements over the current HCL usage.

  • There are a lot of tiny improvement which looks really useful without any doubts, such as factorizations to variables in the HCL => do you mind splitting into PR with different goals to make it easier for othger maintainers to review please (I'm not willing to give and approval with so much changes at the same time).
  • Only blocker for now is the missing updated documentation for windows contributor: this PR introduces way more requirements for them and it must be documented.
  • I'm not sure the end result is easier: it goes on the opposite style as what we used to do: trying to remove the abstraction of Make/make.ps1 . I won't block for this as it is philosophical
  • Could you explain what is the "problem" you want to solve with this PR? As much as it looks interesting and fun technically, I fail to see the gain

@lemeurherve
Copy link
Member Author

lemeurherve commented Jun 9, 2024

Could you explain what is the "problem" you want to solve with this PR? As much as it looks interesting and fun technically, I fail to see the gain

The main problem I have with the current status is that we don't have a "pure" definition of Windows images with the docker compose file as it is, we need to massage data with build.ps1 in order to obtain the correct value for TOOLS_WINDOWS_VERSION for example.

This PR proposes having "pure" images definitions, inside a single common source of truth (the bake file), containing all the helper functions (without having to replicate them in build.ps1) and all the parameters needed to generate the images.

It also paves the way for when docker build bake will be usable to build Windows images.

Jenkinsfile Outdated Show resolved Hide resolved
@lemeurherve lemeurherve force-pushed the docker-bake-windows branch from dd78325 to 7bcc98f Compare June 15, 2024 12:46
@lemeurherve
Copy link
Member Author

lemeurherve commented Jun 15, 2024

It looks like you need a lot of additional tools if you are on Windows to be able to build the image (including GNU make and yq): am I correct?

I reworked the PR to generate the docker compose file from a Windows agent instead of a Linux one, make isn't required on Windows anymore.

As yq is already required, the only new requirement is https://github.com/docker/buildx, added to the agents with jenkins-infra/packer-images#1215.

There are a lot of tiny improvement which looks really useful without any doubts, such as factorizations to variables in the HCL => do you mind splitting into PR with different goals to make it easier for othger maintainers to review please (I'm not willing to give and approval with so much changes at the same time).

Extracted in:

Could you explain what is the "problem" you want to solve with this PR? As much as it looks interesting and fun technically, I fail to see the gain

I've detailed everything in #426.

Only blocker for now is the missing updated documentation for windows contributor: this PR introduces way more requirements for them and it must be documented.

Currently working on it, the rest is in 7bcc98f

I'll pass the PR in ready for review when I'll add the documentation.

@lemeurherve lemeurherve force-pushed the docker-bake-windows branch from 7bcc98f to 54d6230 Compare June 15, 2024 13:24
@lemeurherve lemeurherve changed the title chore: generate docker compose files from docker bake chore: use docker bake to generate docker compose file Jun 15, 2024
@lemeurherve lemeurherve force-pushed the docker-bake-windows branch from 54d6230 to ed751ae Compare June 15, 2024 22:24
@lemeurherve lemeurherve marked this pull request as ready for review June 15, 2024 22:34
@lemeurherve
Copy link
Member Author

lemeurherve commented Jun 15, 2024

Pull request ready for review, this commit especially: f93da6a?w=1

The rest are cherry-picks from the other pull requests, amongst those #419 and #424 are required to be merged before merging this one.

Or eventually, this PR can also be merged as it is, superseding all extracted PRs if they're OK for maintainers.

@lemeurherve lemeurherve force-pushed the docker-bake-windows branch 2 times, most recently from 4c2d2ba to f93da6a Compare June 16, 2024 21:55
@lemeurherve
Copy link
Member Author

lemeurherve commented Jun 16, 2024

As I put the docker compose file generation into a function, I realized that moving Pester initialization to its own function would result in a more readable build.ps1 code
(can be extracted to its own PR if deemed preferable)

Resulting build.ps1 "main" script

docker-ssh-agent/build.ps1

Lines 186 to 258 in e93d536

Test-CommandExists 'docker'
Test-CommandExists 'docker-compose'
Test-CommandExists 'docker buildx'
Test-CommandExists 'yq'
# Generate the docker compose file if it doesn't exists or if the parameter OverwriteDockerComposeFile is set
if ((Test-Path $dockerComposeFile) -and -not $OverwriteDockerComposeFile) {
Write-Host "= PREPARE: The docker compose file '$dockerComposeFile' containing the image definitions already exists."
} else {
Write-Host "= PREPARE: Initialize the docker compose file '$dockerComposeFile' containing the image definitions."
Initialize-DockerComposeFile
}
Write-Host '= PREPARE: List of images and tags to be processed:'
Invoke-Expression "$baseDockerCmd config"
Write-Host '= BUILD: Building all images...'
switch ($DryRun) {
$true { Write-Host "(dry-run) $baseDockerBuildCmd" }
$false { Invoke-Expression $baseDockerBuildCmd }
}
Write-Host '= BUILD: Finished building all images.'
if($lastExitCode -ne 0) {
exit $lastExitCode
}
if($target -eq 'test') {
if ($DryRun) {
Write-Host '= TEST: (dry-run) test harness'
} else {
Write-Host '= TEST: Starting test harness'
$configuration = Initialize-Pester
Write-Host '= TEST: Testing all images...'
# Only fail the run afterwards in case of any test failures
$testFailed = $false
$definitions = Invoke-Expression "$baseDockerCmd config" | yq --unwrapScalar --output-format json '.services' | ConvertFrom-Json
foreach ($definition in $definitions.PSObject.Properties) {
$sut = '{0}|{1}' -f $definition.Value.image, $definition.Value.build.args.JAVA_VERSION
$testFailed = $testFailed -or (Test-Image $sut -Configuration $configuration)
}
# Fail if any test failures
if($testFailed -ne $false) {
Write-Error '= TEST: stage failed!'
exit 1
} else {
Write-Host '= TEST: stage passed!'
}
}
}
if($target -eq 'publish') {
Write-Host '= PUBLISH: push all images and tags'
switch($DryRun) {
$true { Write-Host "(dry-run) $baseDockerCmd push" }
$false { Invoke-Expression "$baseDockerCmd push" }
}
# Fail if any issues when publising the docker images
if($lastExitCode -ne 0) {
Write-Error '= PUBLISH: failed!'
exit 1
}
}
if($lastExitCode -ne 0) {
Write-Error 'Build failed!'
} else {
Write-Host 'Build finished successfully'
}
exit $lastExitCode

EDIT: droped, can't make it work on ci.jenkins.io while it works locally, will try later.

@lemeurherve lemeurherve force-pushed the docker-bake-windows branch 6 times, most recently from c74738d to f93da6a Compare June 17, 2024 00:29
dduportal
dduportal previously approved these changes Jun 18, 2024
Copy link
Contributor

@dduportal dduportal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good \o/

Devil in the details; I've added a few nitpicks or non blocking comments, I let you decide if you want to keep them or not.

Great job and documentation!

Jenkinsfile Outdated Show resolved Hide resolved
Makefile Outdated Show resolved Hide resolved
Jenkinsfile Outdated Show resolved Hide resolved
Makefile Outdated Show resolved Hide resolved
Jenkinsfile Outdated
} else {
powershell "& ./build.ps1 -VersionTag ${env.TAG_NAME} publish"
powershell '& ./build.ps1 publish'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️

README.md Outdated Show resolved Hide resolved
README.md Show resolved Hide resolved
Copy link
Contributor

@dduportal dduportal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: I'm transplanting the comments in #421

@lemeurherve
Copy link
Member Author

lemeurherve commented Jun 19, 2024

As the cherry-picked commits and the corresponding (merged) pull requests have diverged, I'm rebasing this pull request into a clean commit.

@lemeurherve lemeurherve force-pushed the docker-bake-windows branch from 13d6e93 to a305ab1 Compare June 19, 2024 11:28
@lemeurherve lemeurherve requested review from a team and dduportal June 19, 2024 11:47
@lemeurherve lemeurherve merged commit f58ab60 into jenkinsci:master Jun 19, 2024
9 checks passed
@lemeurherve lemeurherve deleted the docker-bake-windows branch July 17, 2024 22:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants