Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[master] feat: build Fedora 41 RPM packages (carry #1055) #1058

Merged
merged 2 commits into from
Sep 4, 2024

Conversation

thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Sep 3, 2024

- What I did

It's nearing that time again. A new Fedora release.
This is part 2 of 2 for packaging Docker CE for Fedora 41. Part 1 is awaiting review in the docker/containerd-packaging repository.

Fedora have not (yet) set a concrete date for the EOL of Fedora 41, however it will be in or around November 2025.

- How I did it

The same as the previous two releases - added a line for Fedora 41 in the Jenkinsfile, updated the list of OSes in the Makefile and created a new Dockerfile using the new Fedora 41 image.

- How to verify it

I have ran make fedora from the ./rpms directory, and a RPM file was successfully created.

- Description for the changelog
Enable Docker CE builds for Fedora 41

- A picture of a cute animal (not mandatory but encouraged)

kali

@thaJeztah thaJeztah changed the title feat: build Fedora 41 RPM packages [master] feat: build Fedora 41 RPM packages (carry #1055) Sep 3, 2024
@thaJeztah
Copy link
Member Author

Let's see if CI is happy, but it' possible we may need something similar as;

@thaJeztah
Copy link
Member Author

Interesting; compiling works, but looks like the verify script needs an update;

othing to do.
+ install_rpm_containerd
+ '[' prod = stage ']'
+ REPO_URL=https://download.docker.com/linux/fedora/docker-ce.repo
+ echo '[DEBUG] Installing engine dependencies from https://download.docker.com/linux/fedora/docker-ce.repo'
+ dnf --version
[DEBUG] Installing engine dependencies from https://download.docker.com/linux/fedora/docker-ce.repo
dnf5 version 5.2.5.0
dnf5 plugin API version 2.0
libdnf5 version 5.2.5.0
libdnf5 plugin API version 2.0

Loaded dnf5 plugins:
  name: builddep
  version: 1.0.0
  API version: 2.0

  name: changelog
  version: 1.0.0
  API version: 2.0

  name: config-manager
  version: 0.1.0
  API version: 2.0

  name: copr
  version: 0.1.0
  API version: 2.0

  name: needs_restarting
  version: 1.0.0
  API version: 2.0

  name: repoclosure
  version: 1.0.0
  API version: 2.0
+ dnf config-manager --add-repo https://download.docker.com/linux/fedora/docker-ce.repo
Unknown argument "--add-repo" for command "config-manager". Add "--help" for more information about the arguments.
make: *** [Makefile:95: verify] Error 2
script returned exit code 2

@thaJeztah
Copy link
Member Author

thaJeztah commented Sep 3, 2024

Ugh... looks like Fedora 41 ships with dnf5 (https://github.com/rpm-software-management/dnf5), which is a complete rewrite of the command?
https://www.reddit.com/r/Fedora/comments/1f3o6s4/problems_when_installing_docker_engine_with_dnf/

On Fedora 40, dnf config-manager is an optional install;

docker run -it --rm fedora:41

dnf config-manager --help
No such command: config-manager. Please use /usr/bin/dnf --help
It could be a DNF plugin command, try: "dnf install 'dnf-command(config-manager)'"

dnf install 'dnf-command(config-manager)'

dnf config-manager --help
usage: dnf config-manager [-c [config file]] [-q] [-v] [--version] [--installroot [path]] [--nodocs] [--noplugins] [--enableplugin [plugin]] [--disableplugin [plugin]] [--releasever RELEASEVER] [--setopt SETOPTS] [--skip-broken] [-h] [--allowerasing] [-b | --nobest] [-C]
[-R [minutes]] [-d [debug level]] [--debugsolver] [--showduplicates] [-e ERRORLEVEL] [--obsoletes] [--rpmverbosity [debug level name]] [-y] [--assumeno] [--enablerepo [repo]] [--disablerepo [repo] | --repo [repo]] [--enable | --disable] [-x [package]]
[--disableexcludes [repo]] [--repofrompath [repo,path]] [--noautoremove] [--nogpgcheck] [--color COLOR] [--refresh] [-4] [-6] [--destdir DESTDIR] [--downloadonly] [--comment COMMENT] [--bugfix] [--enhancement] [--newpackage] [--security]
[--advisory ADVISORY] [--bz BUGZILLA] [--cve CVES] [--sec-severity {Critical,Important,Moderate,Low}] [--forcearch ARCH] [--save] [--add-repo URL] [--dump] [--dump-variables] [--set-enabled | --set-disabled]
[section ...]

manage dnf configuration options and repositories

General DNF options:
  -c [config file], --config [config file]
                        config file location
  -q, --quiet           quiet operation
  -v, --verbose         verbose operation
  --version             show DNF version and exit
  --installroot [path]  set install root
  --nodocs              do not install documentations
  --noplugins           disable all plugins
  --enableplugin [plugin]
                        enable plugins by name
  --disableplugin [plugin]
                        disable plugins by name
  --releasever RELEASEVER
                        override the value of $releasever in config and repo files
  --setopt SETOPTS      set arbitrary config and repo options
  --skip-broken         resolve depsolve problems by skipping packages
  -h, --help, --help-cmd
                        show command help
  --allowerasing        allow erasing of installed packages to resolve dependencies
  -b, --best            try the best available package versions in transactions.
  --nobest              do not limit the transaction to the best candidate
  -C, --cacheonly       run entirely from system cache, don't update cache
  -R [minutes], --randomwait [minutes]
                        maximum command wait time
  -d [debug level], --debuglevel [debug level]
                        debugging output level
  --debugsolver         dumps detailed solving results into files
  --showduplicates      show duplicates, in repos, in list/search commands
  -e ERRORLEVEL, --errorlevel ERRORLEVEL
                        error output level
  --obsoletes           enables dnf's obsoletes processing logic for upgrade or display capabilities that the package obsoletes for info, list and repoquery
  --rpmverbosity [debug level name]
                        debugging output level for rpm
  -y, --assumeyes       automatically answer yes for all questions
  --assumeno            automatically answer no for all questions
  --enablerepo [repo]   Temporarily enable repositories for the purpose of the current dnf command. Accepts an id, a comma-separated list of ids, or a glob of ids. This option can be specified multiple times.
  --disablerepo [repo]  Temporarily disable active repositories for the purpose of the current dnf command. Accepts an id, a comma-separated list of ids, or a glob of ids. This option can be specified multiple times, but is mutually exclusive with `--repo`.
  --repo [repo], --repoid [repo]
                        enable just specific repositories by an id or a glob, can be specified multiple times
  --enable              enable repos with config-manager command (automatically saves)
  --disable             disable repos with config-manager command (automatically saves)
  -x [package], --exclude [package], --excludepkgs [package]
                        exclude packages by name or glob
  --disableexcludes [repo], --disableexcludepkgs [repo]
                        disable excludepkgs
  --repofrompath [repo,path]
                        label and path to an additional repository to use (same path as in a baseurl), can be specified multiple times.
  --noautoremove        disable removal of dependencies that are no longer used
  --nogpgcheck          disable gpg signature checking (if RPM policy allows)
  --color COLOR         control whether color is used
  --refresh             set metadata as expired before running the command
  -4                    resolve to IPv4 addresses only
  -6                    resolve to IPv6 addresses only
  --destdir DESTDIR, --downloaddir DESTDIR
                        set directory to copy packages to
  --downloadonly        only download packages
  --comment COMMENT     add a comment to transaction
  --bugfix              Include bugfix relevant packages, in updates
  --enhancement         Include enhancement relevant packages, in updates
  --newpackage          Include newpackage relevant packages, in updates
  --security            Include security relevant packages, in updates
  --advisory ADVISORY, --advisories ADVISORY
                        Include packages needed to fix the given advisory, in updates
  --bz BUGZILLA, --bzs BUGZILLA
                        Include packages needed to fix the given BZ, in updates
  --cve CVES, --cves CVES
                        Include packages needed to fix the given CVE, in updates
  --sec-severity {Critical,Important,Moderate,Low}, --secseverity {Critical,Important,Moderate,Low}
                        Include security relevant packages matching the severity, in updates
  --forcearch ARCH      Force the use of an architecture

Config-manager command-specific options:
  --save                save the current options (useful with --setopt)
  --add-repo URL        add (and enable) the repo from the specified file or url
  --dump                print current configuration values to stdout
  --dump-variables      print variable values to stdout
  --set-enabled         enable repos (automatically saves)
  --set-disabled        disable repos (automatically saves)

  section               configuration sections to manage, "main" or repoid

Fedora 41:

docker run -it --rm fedora:41 dnf config-manager --help
Usage:
  dnf5 [GLOBAL OPTIONS] config-manager <COMMAND> ...

Description:
  Manage main and repositories configuration, variables and add new repositories.

Commands:
  addrepo                       Add repositories from the specified configuration file or define a new repository using user options
  setopt                        Set configuration and repositories options
  unsetopt                      Unset/remove configuration and repositories options
  setvar                        Set variables
  unsetvar                      Unset/remove variables

@thaJeztah
Copy link
Member Author

so I started updating the verify script to work with dnf5; ran into some issues;

dnf config-manager --set-disabled docker-ce-*
dnf config-manager --set-enabled docker-ce-test

It looks like dnf itself still has --enablerepo and --disablerepo flags (aliases for --enable-repo and --disable-repo;

--enable-repo=REPO_ID,...              Enable additional repositories. List option. Supports globs, can be specified multiple times.
  --disable-repo=REPO_ID,...             Disable repositories. List option. Supports globs, can be specified multiple times.

But if those are the equivalent to the dnf / dnf4 command, they won't persist changes, so it's only for a single invocation;

--enablerepo [repo]   Temporarily enable repositories for the purpose of the current dnf command. Accepts an id, a comma-separated list of ids, or a glob of ids. This option can be specified multiple
                        times.
  --disablerepo [repo]  Temporarily disable active repositories for the purpose of the current dnf command. Accepts an id, a comma-separated list of ids, or a glob of ids. This option can be specified
                        multiple times, but is mutually exclusive with `--repo`.

So the alternative is to use dnf5 config-manger setopt repo_id.enabled=0; https://github.com/rpm-software-management/dnf5/blob/58cfa3f3e5768665f2d6a1b6f06eb1ec3451a5fc/doc/commands/repo.8.rst#examples

However, that doesn't support glob / wildcards, so we may need to do a dnf repo list to get the IDs of the repositories to disable;

dnf repo list --all docker-ce-*
repo id                                                                                                        repo name                                                                                                                                                         status
docker-ce-stable                                                                                               Docker CE Stable - aarch64                                                                                                                                       enabled
docker-ce-stable-debuginfo                                                                                     Docker CE Stable - Debuginfo aarch64                                                                                                                            disabled

Perhaps using --json;

dnf repo list --json --enabled docker-ce-*
[
  {
    "id":"docker-ce-stable",
    "name":"Docker CE Stable - aarch64",
    "is_enabled":true
  }
]

However, I ran into an issue; it looks like dnf5 is not compatible with the existing .repo files we have;

dnf config-manager addrepo --from-repofile="https://download.docker.com/linux/fedora/docker-ce.repo"
Error in added repository configuration file. Cannot set repository option "#1=
": Option "#1" not found

After some debugging, it looks like the problem is in empty lines in the file; individual entries work (so it's not an issue in any of those), but the empty lines between them; i.e., this fails;

[docker-ce-stable]
name=Docker CE Stable - $basearch
baseurl=https://download.docker.com/linux/fedora/$releasever/$basearch/stable
enabled=1
gpgcheck=1
gpgkey=https://download.docker.com/linux/fedora/gpg

[docker-ce-stable-debuginfo]
name=Docker CE Stable - Debuginfo $basearch
baseurl=https://download.docker.com/linux/fedora/$releasever/debug-$basearch/stable
enabled=0
gpgcheck=1
gpgkey=https://download.docker.com/linux/fedora/gpg
dnf config-manager addrepo --from-repofile=./test.repo --overwrite
Error in added repository configuration file. Cannot set repository option "#1=
": Option "#1" not found

And this works;

[docker-ce-stable]
name=Docker CE Stable - $basearch
baseurl=https://download.docker.com/linux/fedora/$releasever/$basearch/stable
enabled=1
gpgcheck=1
gpgkey=https://download.docker.com/linux/fedora/gpg
[docker-ce-stable-debuginfo]
name=Docker CE Stable - Debuginfo $basearch
baseurl=https://download.docker.com/linux/fedora/$releasever/debug-$basearch/stable
enabled=0
gpgcheck=1
gpgkey=https://download.docker.com/linux/fedora/gpg

@thaJeztah
Copy link
Member Author

And it looks like it's a confirmed bug, and .... 😂 😂 the reproducer is.. our .repo file;

So; confirmed at least that the issue is because of the empty lines, but not sure yet when the fix will ship, so possibly we have to fix it up;

  • replace the repo-files with empty lines removed?
  • download to a temporary file and strip empty lines?
  • manually add the repositories (but this requires all options to be set separately)

@thaJeztah
Copy link
Member Author

On a fun sidetone; I discovered that dnf5 has support for releasever_major - wondering if we could use that for the RHEL 8/9 issue that we ran into with minor versions (which we tried to solve with redirects); not sure if it's also supported by dnf4 though;

@jrohel
Copy link

jrohel commented Sep 4, 2024

@thaJeztah

I discovered that dnf5 has support for releasever_major
not sire if it's also supported by dnf4 though

Yes, it is supported in dnf4 since version 4.18.0 PR rpm-software-management/dnf#1989

wondering if we could use that for the RHEL 8/9

No. If I know correctly, in RHEL 8 there is an older version of dnf4 4.7.0 and in RHEL 9 there is an older version of dnf4 4.14.0.

Note: It can be used in RHEL 10, there is dnf4 4.20.0.

@jrohel
Copy link

jrohel commented Sep 4, 2024

@thaJeztah

However, that doesn't support glob / wildcards, so we may need to do a dnf repo list to get the IDs of the repositories to disable;

Globs are supported in the repository id. And it documentation for config-manager https://dnf5.readthedocs.io/en/latest/dnf5_plugins/config-manager.8.html is written Reposiory ID may contain globs.

See the examples at the end of the documentation. I purposely wrote the examples with globs in the id repository, hoping it will help someone.

@thaJeztah
Copy link
Member Author

Yes, it is supported in dnf4 since version 4.18.0 PR rpm-software-management/dnf#1989

Ah, thanks! I'm not a user of a rpm-based distro myself, so sometimes need to do a bit more digging for the right place to search

No. If I know correctly, in RHEL 8 there is an older version of dnf4 4.7.0 and in RHEL 9 there is an older version of dnf4 4.14.0.

Note: It can be used in RHEL 10, there is dnf4 4.20.0.

Yes, that's usually the fun challenge; we're packaging for various distros and distro-versions, so where possible, we try to use options that are common between versions. Scripts we use in packaging don't always have to be "polished", we can do some detection there, but for our documentation (and installation script), it m more important, because, well, users don't read documentation (so may be copying the wrong commands), and we also know many users try to install packages on various derivative distros (most of which we don't actively "support", and at least don't test)

Globs are supported in the repository id. And it documentation for config-manager https://dnf5.readthedocs.io/en/latest/dnf5_plugins/config-manager.8.html is written Reposiory ID may contain globs.

See the examples at the end of the documentation. I purposely wrote the examples with globs in the id repository, hoping it will help someone.

Oh, perfect! Thank you for that; I didn't spot that part, that's useful.

Interesting; looks like setopt is taking a slightly different approach, and doesn't update the original .repo file (/etc/yum.repos.d/docker-ce.repo) but instead uses override files (/usr/share/dnf5/repos.override.d or /etc/dnf/repos.override.d).

@@ -27,7 +27,12 @@ function install_rpm_containerd() {
if command -v dnf5; then
dnf --version

dnf config-manager addrepo --save-filename=docker-ce.repo --from-repofile="${REPO_URL}"
# FIXME(thaJeztah); strip empty lines as workaround for https://github.com/rpm-software-management/dnf5/issues/1603
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dnf5 config-manager bug affects empty lines and comment lines (starting with the # character).

dnf config-manager addrepo --save-filename=docker-ce.repo --from-repofile="${REPO_URL}"
# FIXME(thaJeztah); strip empty lines as workaround for https://github.com/rpm-software-management/dnf5/issues/1603
TMP_REPO_FILE="$(mktemp --dry-run)"
curl -fsSL "${REPO_URL}" | tr -s '\n' > "${TMP_REPO_FILE}"
Copy link

@jrohel jrohel Sep 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tr -s '\n' only removes empty lines. Comment lines must also be removed. May be replace tr -s '\n' with grep '^[^#]'.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Yes, I was considering comment-lines as well, but our .repo files don't have them, so I ultimately only accounted for the empty lines.

It should (hopefully 🤞) just be temporary, and we can revert the commit once fixed.

FWIW: I lifted those commits to a separate PR (mostly for visibility); and we just merged them; #1059

p5 and others added 2 commits September 4, 2024 14:55
Signed-off-by: Robert Sturla <robertsturla@outlook.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This is the equivalent of docker/containerd-packaging#390
for containerd packages, but unlike for containerd packages, we currently do
not run into this issue when building docker-ce packages. We're installing
this as a precaution, but perhaps it's not needed.

go1.21 and up have a patch that enforces the use of ld.gold to work around
a bug in GNU binutils. See;

- golang/go#22040.
- golang/go@cd77738

Fedora 41 and up has a fixed version of binutils, and no longer requires that
patch, but may fail without ld.gold installed;

    /usr/bin/gcc -Wl,-z,now -Wl,-z,nocopyreloc -fuse-ld=gold -o $WORK/b001/exe/a.out -rdynamic /tmp/go-link-1738353519/go.o /tmp/go-link-1738353519/000000.o /tmp/go-link-1738353519/000001.o /tmp/go-link-1738353519/000002.o /tmp/go-link-1738353519/000003.o /tmp/go-link-1738353519/000004.o /tmp/go-link-1738353519/000005.o /tmp/go-link-1738353519/000006.o /tmp/go-link-1738353519/000007.o /tmp/go-link-1738353519/000008.o /tmp/go-link-1738353519/000009.o /tmp/go-link-1738353519/000010.o /tmp/go-link-1738353519/000011.o /tmp/go-link-1738353519/000012.o /tmp/go-link-1738353519/000013.o /tmp/go-link-1738353519/000014.o /tmp/go-link-1738353519/000015.o /tmp/go-link-1738353519/000016.o /tmp/go-link-1738353519/000017.o /tmp/go-link-1738353519/000018.o /tmp/go-link-1738353519/000019.o /tmp/go-link-1738353519/000020.o /tmp/go-link-1738353519/000021.o /tmp/go-link-1738353519/000022.o /tmp/go-link-1738353519/000023.o /tmp/go-link-1738353519/000024.o -O2 -g -lresolv -O2 -g -lpthread -O2 -g -ldl -O2 -g
    collect2: fatal error: cannot find 'ld'

Fedora's build of Go carries a patch for that, but it's not (yet) in upstream;
- https://src.fedoraproject.org/rpms/golang/blob/a867bd88a656c1d6e91e7b18bab696dc3fcf1e77/f/0006-Default-to-ld.bfd-on-ARM64.patch
- https://src.fedoraproject.org/rpms/golang/c/a867bd88a656c1d6e91e7b18bab696dc3fcf1e77?branch=rawhide

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah thaJeztah marked this pull request as ready for review September 4, 2024 14:19
Comment on lines +22 to +45
# FIXME(thaJeztah): workaround for building on Fedora 41 on arm64
#
# This is the equivalent of https://github.com/docker/containerd-packaging/pull/390
# for containerd packages, but unlike for containerd packages, we currently do
# not run into this issue when building docker-ce packages. We're installing
# this as a precaution, but perhaps it's not needed.
#
# go1.21 and up have a patch that enforces the use of ld.gold to work around
# a bug in GNU binutils. See;
# - https://github.com/golang/go/issues/22040.
# - https://github.com/golang/go/commit/cd77738198ffe0c4a1db58352c89f9b2d2a4e85e
#
# Fedora 41 and up has a fixed version of binutils, and no longer requires that
# patch, but may fail without ld.gold installed;
#
# /usr/bin/gcc -Wl,-z,now -Wl,-z,nocopyreloc -fuse-ld=gold -o $WORK/b001/exe/a.out -rdynamic /tmp/go-link-1738353519/go.o /tmp/go-link-1738353519/000000.o /tmp/go-link-1738353519/000001.o /tmp/go-link-1738353519/000002.o /tmp/go-link-1738353519/000003.o /tmp/go-link-1738353519/000004.o /tmp/go-link-1738353519/000005.o /tmp/go-link-1738353519/000006.o /tmp/go-link-1738353519/000007.o /tmp/go-link-1738353519/000008.o /tmp/go-link-1738353519/000009.o /tmp/go-link-1738353519/000010.o /tmp/go-link-1738353519/000011.o /tmp/go-link-1738353519/000012.o /tmp/go-link-1738353519/000013.o /tmp/go-link-1738353519/000014.o /tmp/go-link-1738353519/000015.o /tmp/go-link-1738353519/000016.o /tmp/go-link-1738353519/000017.o /tmp/go-link-1738353519/000018.o /tmp/go-link-1738353519/000019.o /tmp/go-link-1738353519/000020.o /tmp/go-link-1738353519/000021.o /tmp/go-link-1738353519/000022.o /tmp/go-link-1738353519/000023.o /tmp/go-link-1738353519/000024.o -O2 -g -lresolv -O2 -g -lpthread -O2 -g -ldl -O2 -g
# collect2: fatal error: cannot find 'ld'
#
# Fedora's build of Go carries a patch for that, but it's not (yet) in upstream;
# - https://src.fedoraproject.org/rpms/golang/blob/a867bd88a656c1d6e91e7b18bab696dc3fcf1e77/f/0006-Default-to-ld.bfd-on-ARM64.patch
# - https://src.fedoraproject.org/rpms/golang/c/a867bd88a656c1d6e91e7b18bab696dc3fcf1e77?branch=rawhide
#
# As a workaround; install binutils-gold
RUN if [ "$(arch)" = 'aarch64' ]; then dnf -y install binutils-gold; fi
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vvoland this one should be ready for review; I pushed a commit to install the binutils-gold, but mostly as a precaution; happy to drop it as well (at least made it a separate commit).

@thaJeztah thaJeztah merged commit 5e90bd3 into docker:master Sep 4, 2024
7 checks passed
@thaJeztah thaJeztah deleted the carry_1055_enable_f41_builds branch September 4, 2024 14:50
@p5
Copy link
Contributor

p5 commented Sep 4, 2024

Apologies, if I saw this thread I'd have helped out.
Thank you for pushing this through!

@thaJeztah
Copy link
Member Author

You're welcome! I had to carry the PR because Jenkins is configured to not allow external contributors to make changes in the Jenkinsfile itself, so it wouldn't test it in CI.

The dnf5 change made me go a bit down the rabbit-hole though 😂

@p5
Copy link
Contributor

p5 commented Sep 5, 2024

Is there something I should have ran from my local machine before submitting the PR in the first place? Some way that I could have identified the dnf5 issue before CI?

The build was fine, but I wasn't aware of any validation scripts

Edit:
Ah, looking at the repo, there's a ./verify script with instructions on how to run this inside a Docker container. Will be sure to run this next time

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants