feat: dnf module

[xynydev]

A dnf module will eventually replace the rpm-ostree module. We should start investigating usage of dnf5 on atomic Fedora systems to determine a reasonable timeline for development of such a module.

https://fedoramagazine.org/whats-new-for-fedora-atomic-desktops-in-fedora-41/#first-step-towards-bootable-containers-dnf5-and-bootc

[fiftydinar]

It functions similarly to rpm-ostree when I tested it.

The only advantage I see is that we can enable copr repos with simpler command dnf copr enable maintainer/repo instead of manual curls.

Basically, when we would replace rpm-ostree occurences with dnf in the current module, users wouldn't notice any change in functionality except a bit different logs.

[xynydev]

Basically, when we would replace rpm-ostree occurences with dnf in the current module

I would rather create a new module called dnf. We may keep most of the configuration and even code the same, but there's no reason to keep the legacy name of rpm-ostree around.

[fiftydinar]

I'm in for making a new dnf module & announcing it to the users in blog, Discord, Mastodon & BlueSky (if it's used?).

Current structure from rpm-ostree module looks good & can be ported to dnf, except there needs to be some minor corrections in recipe format & some code logic replacements are needed.

Minor recipe format corrections

Repos

Currently, in rpm-ostree we just download the repos with curl directly to /etc/yum.repos.d/.

dnf has copr plugin to download & enable COPR repos.

• We can continue to use the same method of downloading repos using curl to /etc/yum.repos.d/
• Or switch to more native method of using dnf5 -y copr enable user/package
  • In case of dnf5 copr command usage, we can either
    • keep the same structure of repos array in recipe, so when it detects that it's copr URL, it will use dnf5 copr command
    • same as above, but users can just name the array copr: user/package instead of the full URL & we will detect that (this one is the best approach imo).
    • or add the new YAML array called repos-copr or similar, where users would just input user & package

Replacing packages

Currently, we use rpm-ostree override replace, which supports replacing packages from repo, direct URL & maybe local RPMs too.

It seems that there is no alternative for replacing packages in dnf.

dnf5 swap "${rem_packages}" "${inst_packages}" only does
rpm-ostree override remove "${rem_packages}" --install "${inst_packages}".

Everything else

Everything else looks relatively native.

[tulilirockz]

dnf5 has behavioural differences from rpm-ostree, as it probably does not install Recommends: packages by default! I ended up figuring that out when I was testing out the dnf5 migration on https://github.com/ublue-os/main and https://github.com/ublue-os/bluefin.

[tulilirockz]

rpm-ostree override replace can be replaced by dnf -y remove "${rem_packages}" && dnf -y install "${inst_packages}"

[tulilirockz]

Moving to dnf copr should be completely fine, too! There are some repos that sadly do not work rn like the ublue-os/akmods one that gets funky w/ the dnf copr behaviour (see: ublue-os/bluefin#1954)

[fiftydinar]

dnf5 has behavioural differences from rpm-ostree, as it probably does not install Recommends: packages by default! I ended up figuring that out when I was testing out the dnf5 migration on https://github.com/ublue-os/main and https://github.com/ublue-os/bluefin.

According to dnf defaults, the default is to install weak dependencies by default, just like rpm-ostree.
install_weak_deps=True

If  enabled, when a new package is about to be installed, all packages linked by weak dependency relation
(Recommends or Supplements flags) with this package will be pulled into the transaction.

I also personally didn't notice any difference in this behavior when I tested it a bit.

rpm-ostree override replace can be replaced by dnf -y remove "${rem_packages}" && dnf -y install "${inst_packages}"

That's basically what dnf swap does (except swap only supports swapping 1 package as I see now...).

But if there is no alternative, then sure, that can be also used & adjusted to match the replace behavior from recipe, without user noticing any difference.

Moving to dnf copr should be completely fine, too! There are some repos that sadly do not work rn like the ublue-os/akmods one that gets funky w/ the dnf copr behaviour (see: ublue-os/bluefin#1954)

Interesting, thanks for noting.

[fiftydinar]

I noticed in this Bazzite's PR:
https://github.com/ublue-os/bazzite/pull/1905/files

That they are using dnf5 -y upgrade --repo for replacing packages from the repo, so I think we found the command replacement for this actually.

[xynydev]

dnf has copr plugin to download & enable COPR repos.

I think we could do this simple change:

• keep the repos: array
• for each element, check if it starts with http(s)://, and use the curl method if so
• if not, give the repo to dnf5 -y copr enable and see what happens
  • if it was configured correctly, the repo should be enabled, but if there's a problem it should error out and we can print a nice error message in the logs

[fiftydinar]

• if not, give the repo to dnf5 -y copr enable and see what happens

I think a flaw in this method is support for repo files which are directly provided in /files/rpm-ostree/

So I think that my suggestion of using copr: user/package format is better.

[fiftydinar]

@gmpinder We will probably need to add --mount=type=cache,dst=/var/cache/libdnf5 in Containerfile when this gets merged

[gmpinder]

@gmpinder We will probably need to add --mount=type=cache,dst=/var/cache/libdnf5 in Containerfile when this gets merged

Sounds good, that'll be an easy addition.

[xynydev]

I think a flaw in this method is support for repo files which are directly provided in /files/rpm-ostree/

Oh right, I forgot that. If we assume copr repo names don't include the .repo suffix, but repo files all do, we use that as another way to detect it.

Adding another key would work too I guess, but then we'd have to debate the whole config structure related to that again lol.

[fiftydinar]

We should hold a bit releasing dnf module until all issues are resolved with it. rpm-ostree should be still well supported for F42 at minimum.

ublue-os/bluefin#1954 (comment)

Draft PR can be made though, to get us more ready.

[gmpinder]

So I think that my suggestion of using copr: user/package format is better.

I'm still of the opinion that we should have a separate top-level copr property that takes a list. Right now it's coded to be in the repos section and requires the user to define it like COPR repo/namespace. This, imho, could get confusing for users and be error prone if extra whitespace is added. I think we should encode as much semantics into the spec as we can to allow for easier readability and validation.

[tulilirockz]

I am also with @gmpinder in this one. It's both kind of confusing and having some typing-strict way to check if the repo is a COPR would be much better. Like, I think the best way would be to do something like this

# (...)
repos:
    - name: "myuser/chongus"
      copr: true # or something dunno
    - "other-regular-repo.repo"
    - anotherone.repo
# (...)

[fiftydinar]

I'm still of the opinion that we should have a separate top-level copr property that takes a list. Right now it's coded to be in the repos section and requires the user to define it like COPR repo/namespace. This, imho, could get confusing for users and be error prone if extra whitespace is added. I think we should encode as much semantics into the spec as we can to allow for easier readability and validation.

I thought about adding manipulation flag options to repos, like we have for install, remove & replace.

I think that having distinct copr: would be better in that case. We would use dnf copr to provide enable & disable repo options.

For repos:, those would be enable, disable, changing priority & maybe something else too.

I am also with @gmpinder in this one. It's both kind of confusing and having some typing-strict way to check if the repo > is a COPR would be much better. Like, I think the best way would be to do something like this

# (...)
repos:
    - name: "myuser/chongus"
      copr: true # or something dunno
    - "other-regular-repo.repo"
    - anotherone.repo
# (...)

I think that distinct copr: is better, as I proposed having repo manipulation as outlined above.

[gmpinder]

I thought about adding manipulation flag options to repos, like we have for install, remove & replace.

I think that having distinct copr: would be better in that case. We would use dnf copr to provide enable & disable repo options.

For repos:, those would be enable, disable, changing priority & maybe something else too.

I'd be down with this. Especially if there are flows where you'd want to enable/disable repos.

[fiftydinar]

I thought about adding manipulation flag options to repos, like we have for install, remove & replace.
I think that having distinct copr: would be better in that case. We would use dnf copr to provide enable & disable repo options.
For repos:, those would be enable, disable, changing priority & maybe something else too.

I'd be down with this. Especially if there are flows where you'd want to enable/disable repos.

I added separate copr: array now.

I'll add some manipulation options to copr: & repos: later. Recipe format will need to change a bit for this for those arrays, so any suggestion for that would be great & which options should be added.

[gmpinder]

So for the copr: property, we could have enable, disable, and remove.

type: dnf
copr:
  enable: # could also be 'add' since `dnf copr enable` is used for both adding and enabling
    - some/repo
  disable: # use `dnf copr disable $REPO`
    - disabled/repo
  remove: # use `dnf copr remove $REPO`
    - removed/repo

That way it matches what is available under the command.

$ dnf5 copr --help
Usage:
  dnf5 [GLOBAL OPTIONS] copr [OPTIONS] <COMMAND> ...

Description:
  Manage Copr repositories (add-ons provided by users/community/third-party)

Commands:
  list                          list Copr repositories
  enable                        download the repository info from a Copr server and install it as a /etc/yum.repos.d/*.repo file
  disable                       disable specified Copr repository (if exists), keep /etc/yum.repos.d/*.repo file - just mark enabled=0
  remove                        remove specified Copr repository from the system (removes the /etc/yum.repos.d/*.repo file)
  debug                         print useful info about the system, useful for debugging

Options:
  --hub=HOSTNAME                Copr hub (the web-UI/API server) hostname

[gmpinder]

As for the repo: section, we could have an add, enable, disable. We could add a remove, but that would require figuring out which repo file is the correct one. You might be able to figure that out with dnf repo info --json <repo_name> | jq -cr '.[0].repo_file_path'.

type: dnf
repos:
  add: # uses `dnf config-manager addrepo`
    - https://pkg.earthly.dev/earthly.repo
    - https://cli.github.com/packages/rpm/gh-cli.repo
  disable: # uses `dnf config-manager setopt "$REPO.enabled=false"`
    - earthly-stable
  remove: # uses `rm -f "$(dnf repo info --json $REPO | jq -cr '.[0].repo_file_path')"`
    - google-cloud-cli
  enable: # uses `dnf config-manager setopt "$REPO.enabled=true"`
    - fedora-steam
install:
  packages:
    - earthly
    - gh
    - google-cloud-cli
    - steam

For the disable, remove, and enable properties, you will have to require the name of the repo as read by dnf.

[gmpinder]

Another question would be, do we want to have removing/disabling of repos be done at the beginning of the script like with adding, or should we do it at the end so that you could:

1. Add the repo
2. Install the packages you want
3. Remove/disable the repo to not interfere with future calls

I personally think it would be cleaner for the kind of flows I've seen to remove/disable after installation of packages so that you could keep all of that logic contained in a single module call.

[xynydev]

It seems repo management is one major pain point to discuss before the implementation of the module can be finished.

Here are the features I would definitely like to support:

• Adding repos from http(s) URLs
• Adding repos from COPR
• Cleaning up repos after install

Here are the features that I am still unsure about:

• Adding repos from files local to the repo (can easily be done with the files module, by just placing the file in whichever directory corresponds to /etc/yum.repos.d/ in the git repo)
• Removing repos, disabling repos, enabling disabled repos (more complicated to script in a way that works for everyone than to just use a script snippet to do the specific thing you want)

Here are two solutions I can come up with right now:

A: Magic

type: dnf
clean-up-repos: true # whether to remove added repos after module call is done
repos: # these repos are added & enabled
  - https://example.org/fedora-example.repo # starts with 'http(s)://', treated as URL
  - custom/myrepo.repo # ends with '.repo', treated as local file
  - username/myrepo # does not match previous, contains one '/', treated as COPR repo

B: Straightforward (with nesting)

type: dnf
repos:
  clean-up: true
  copr:
    - username/myrepo
  file:
    - https://example.org/fedora-example.repo # starts with 'http(s)://', treated as URL
    - custom/myrepo.repo # does not match previous, treated as local file

B.1: All operations

type: dnf
repos:
  copr:
    - username/myrepo
    - name: username/myrepo2
      operation: remove

B.2: All operations

type: dnf
repos:
  copr:
    add:
      - user/myrepo
    disable:
      - user/myrepo2

[xynydev]

Another idea

type: dnf
packages:
  - repo: fedora
    install:
      - micro
  - repo: terra
     install:
       - prismlauncher 
  - repo: myuser/myrepo # COPR (same magic as described above or something ig)
     install:
       - custompackage1
       - custompackage2
     replace:
        - kernel

All repos are cleaned out by default after packages are installed. This feels like an intuitive separation to me, but also an opinionated restriction.

[gmpinder]

Another idea

type: dnf
packages:
  - repo: fedora
    install:
      - micro
  - repo: terra
     install:
       - prismlauncher 
  - repo: myuser/myrepo # COPR (same magic as described above or something ig)
     install:
       - custompackage1
       - custompackage2
     replace:
        - kernel

All repos are cleaned out by default after packages are installed. This feels like an intuitive separation to me, but also an opinionated restriction.

This feels too complex imho. I'm all for making things easy to the point it feels like "magic", but to make these operations make more sense, we should have the structure map more closely with the commands that are being ran.

B.2: All operations

type: dnf
repos:
  copr:
    add:
      - user/myrepo
    disable:
      - user/myrepo2

This is a lot closer to how I was thinking about it. I wouldn't mind there being a section under repos for copr.

B.1: All operations

type: dnf
repos:
  copr:
    - username/myrepo
    - name: username/myrepo2
      operation: remove

Having something like this, while perfectly valid and a pattern I've seen with other yaml based systems, feels like it could introduce some cognitive complexity for the user. I think we should have a goal of trying to make our schemas relatively simple so that it can be understood by a wider range of users. We shouldn't be targeting just developer minded users.

B: Straightforward (with nesting)

type: dnf
repos:
  clean-up: true
  copr:
    - username/myrepo
  file:
    - https://example.org/fedora-example.repo # starts with 'http(s)://', treated as URL
    - custom/myrepo.repo # does not match previous, treated as local file

I think a clean-up flag could be useful, the only issue being needing to separate module calls if you don't want all the repos you're adding to be removed. Maybe we have a section under repos: called post-remove: or post-disable: that would be ran after everything is installed. This way it's clearer in the name of the property what the intent is.

[xynydev]

B.1

feels like it could introduce some cognitive complexity for the user

Sure, I didn't like the look of that either that much.

Another idea

This feels too complex imho. I'm all for making things easy to the point it feels like "magic", but to make these operations make more sense, we should have the structure map more closely with the commands that are being ran.

Sure. I still really like the notion of being able to install packages from a specific repo seamlessly. My recipe often has a long list of repos, and a long list of packages, but no way to tell which repo which package comes from. I think it would be nice to be able to tell from the structure of the recipe that these packages are all from rpmfusion, and this kernel is being replaced from this specific copr, and the rest are just from standard fedora repos.

I feel like we'd be adding more value by not just wrapping a YAML structure around a CLI interface, and by structuring it in such a way that it's easier to intuitively reason about. But I guess it's up to each user which structure feels more intuitive.

Maybe we have a section under repos: called post-remove: or post-disable: that would be ran after everything is installed. This way it's clearer in the name of the property what the intent is.

I don't find that clear at all. If I want to add some repos and not clean them up, while cleaning up others, with option A or B I would just use two module calls, or with the Another idea I would integrate that into the config structure.

---

Some of this could be combined together, too, though.

type: dnf
repos:
  clean-up-after-use: false # these repos i want to keep around (parameter name tbd)
  copr: # Array<String> | { enable: Array<String>, disable: Array<String> }
    - username/myrepo
  file: # Array<String> | { add: Array<String>, remove: Array<String> }
    - https://example.org/fedora-example.repo # starts with 'http(s)://', treated as URL
    - custom/myrepo.repo # does not match previous, treated as local file
install: 
  - fedora-package
  - copr-package
  - myrepo-pacage
remove:
  - firefox
# ---
package-sets: # optional, alternative installation method, where repos are specific and cleaned up by default
  - repos: fedora # default, may be omitted
    install:
       - fedora-package
  - repos: # "fedora" | "rpmfusion" | "stufflikethat" |  { copr: Array<String> }
      copr:
        - username/mykernelrepo
    install:
       - kernel
       - firmware-blob
  - repos: rpmfusion
    install:
       - nonfree-package

A user would choose to use whatever bits make most sense to them. Everything would work together, though, but one could choose not to learn how to use package sets if they so please.

---

This is all brainstorming, all ideas are appreciated.

[gmpinder]

package-sets: # optional, alternative installation method, where repos are specific and cleaned up by default
  - repos: fedora # default, may be omitted
    install:
       - fedora-package
  - repos: # "fedora" | "rpmfusion" | "stufflikethat" |  { copr: Array<String> }
      copr:
        - username/mykernelrepo
    install:
       - kernel
       - firmware-blob
  - repos: rpmfusion
    install:
       - nonfree-package

How would this even work? If it's disabling the repo that's defined, wouldn't that be bad for the standard Fedora repos? If they don't, the extra information isn't used for anything and is just extra fluff. If the repo is wrong, does the build fail?

I feel like we'd be adding more value by not just wrapping a YAML structure around a CLI interface, and by structuring it in such a way that it's easier to intuitively reason about. But I guess it's up to each user which structure feels more intuitive.

Sure, but the information the user provides in the yaml SHOULD be used directly and have a purpose. If a user wants to track for themselves which repo a package is from, they can add a comment in their recipe. That's why I'm advocating for keeping the schema simple and much more closely related to what is used to call dnf.

[xynydev]

How would this even work? If it's disabling the repo that's defined, wouldn't that be bad for the standard Fedora repos? If they don't, the extra information isn't used for anything and is just extra fluff. If the repo is wrong, does the build fail?

Regarding how it works? I don't know the specifics, but the general idea I guess is that we'd install packages with --repo. And if a user tries to use a repo that is not added yet, say rpmfusion, we first add that and then remove that. When using the repos that come with the image then, nothing would be added or removed regarding the repo.

If the specified repo is unavailable, or the packages specified don't exist in said repo, of course the build would then fail.

Sure, but the information the user provides in the yaml SHOULD be used directly and have a purpose. If a user wants to track for themselves which repo a package is from, they can add a comment in their recipe.

I think I covered covered this already above.

That's why I'm advocating for keeping the schema simple and much more closely related to what is used to call dnf.

And I'm not saying that's the wrong approach, we should probably support and keep supporting that. My additional proposal isn't really high-stakes, but it could be a useful way to structure package installations for some.

[gmpinder]

Ok I've come around on some things after seeing this message from one of our users.

Is there any way to specify where to install a package from? For example, I want to download Steam but it seems like it is installing through another third-party repo instead of rpmfusion.

So I think rather than having a separate section like package-sets, we could instead have install: be a list of either string or object like:

type: dnf
install:
  install-weak-dependencies: false
  packages:
    - name: earthly # package name
      repo: earthly-stable # repo to install from using `--repo`
    - neovim # Install based on priorities in .repo files

While I did say earlier

This feels too complex imho. I'm all for making things easy to the point it feels like "magic", but to make these operations make more sense, we should have the structure map more closely with the commands that are being ran.

I think that this schema style would be relatively easy to understand. We would then continue with the usual top-level properties of group-install, remove, and replace as defined so far in modules/dnf/module.yml.

type: dnf
repos:
  clean-up-after-use: false # these repos i want to keep around (parameter name tbd)
  copr: # Array<String> | { enable: Array<String>, disable: Array<String> }
    - username/myrepo
  file: # Array<String> | { add: Array<String>, remove: Array<String> }
    - https://example.org/fedora-example.repo # starts with 'http(s)://', treated as URL
    - custom/myrepo.repo # does not match previous, treated as local file

I think this is a good schema for handling repos.

[fiftydinar]

Ok I've come around on some things after seeing this message from one of our users.

Is there any way to specify where to install a package from? For example, I want to download Steam but it seems like it is installing through another third-party repo instead of rpmfusion.

So I think rather than having a separate section like package-sets, we could instead have install: be a list of either string or object like:

type: dnf
install:
  install-weak-dependencies: false
  packages:
    - name: earthly # package name
      repo: earthly-stable # repo to install from using `--repo`
    - neovim # Install based on priorities in .repo files

I like this, I approve.

While I did say earlier

This feels too complex imho. I'm all for making things easy to the point it feels like "magic", but to make these operations make more sense, we should have the structure map more closely with the commands that are being ran.

I think that this schema style would be relatively easy to understand. We would then continue with the usual top-level properties of group-install, remove, and replace as defined so far in modules/dnf/module.yml.

type: dnf
repos:
  clean-up-after-use: false # these repos i want to keep around (parameter name tbd)
  copr: # Array<String> | { enable: Array<String>, disable: Array<String> }
    - username/myrepo
  file: # Array<String> | { add: Array<String>, remove: Array<String> }
    - https://example.org/fedora-example.repo # starts with 'http(s)://', treated as URL
    - custom/myrepo.repo # does not match previous, treated as local file

I think this is a good schema for handling repos.

I like this schema also, just wondering how to handle clean-up-after-use when URL is provided?

[xynydev]

I like this schema also, just wondering how to handle clean-up-after-use when URL is provided?

Delete the file that was downloaded? Isn't there some way to store that filename?

So I think rather than having a separate section like package-sets, we could instead have install: be a list of either string or object like:

Sure, but then if you have to install multiple related packages from a specific repo there's duplication again. If you're open to it being Array<String | Object> couldn't we do it like so?

type: dnf
install:
  install-weak-dependencies: false
  packages:
    - repo: mycustom-repo # repo to install from using `--repo`
      packages:
        - kernel
        - driver-package
    - neovim # Install based on priorities in .repo files

And one thing I'm confused about here is if the specified repo is installed with the install step like I proposed or if it also has to be added under repos:...

Also, if we're already making install: an array of objects, group installation could also work like below, where it would also be possible to specify a repo and whatever other options we choose to support there. Could even allow changing the installation of recommended packages or weak deps on a case-by-case basis.

install:
  - group: XFCE Desktop

I guess this goes back in a more complicated direction in exchange for all the power...

[fiftydinar]

I like this schema also, just wondering how to handle clean-up-after-use when URL is provided?

Delete the file that was downloaded? Isn't there some way to store that filename?

I'll check

So I think rather than having a separate section like package-sets, we could instead have install: be a list of either string or object like:

Sure, but then if you have to install multiple related packages from a specific repo there's duplication again. If you're open to it being Array<String | Object> couldn't we do it like so?

type: dnf
install:
  install-weak-dependencies: false
  packages:
    - repo: mycustom-repo # repo to install from using `--repo`
      packages:
        - kernel
        - driver-package
    - neovim # Install based on priorities in .repo files

Makes sense, I approve.

And one thing I'm confused about here is if the specified repo is installed with the install step like I proposed or if it also has to be added under repos:...

It needs to be added under repos:.

Also, if we're already making install: an array of objects, group installation could also work like below, where it would also be possible to specify a repo and whatever other options we choose to support there. Could even allow changing the installation of recommended packages or weak deps on a case-by-case basis.

install:
  - group: XFCE Desktop

I guess this goes back in a more complicated direction in exchange for all the power...

I'm all in to have all the power!

[xynydev]

It needs to be added under repos:.

I think it would be nicer if this wasn't the case, but I guess it would make more sense implementation-wise.

[gmpinder]

OK, I just got done converting the existing changes to nushell. Now I'll work on setting everything up with the agreed upon schema.