Add support for repos configuration overrides #820

jrohel · 2023-08-15T11:32:23Z

After the repositories configuration is loaded, new code is run that loads the override files from the /usr/share/dnf5/repos.override.d and /etc/dnf/repos.overide.d directories.

The format of override files is the same as regular ".repo" files. However, override files can modify the settings of already existing repositories. They cannot create new repositories.
Overide files support glob in the repository ID (in the configuration file section name) to support bulk modification of repositories parameters.

The order of processing the files with overrides:

Creates an alphabetically sorted list of all filenames with overrides in the distribution override directory ("/usr/share/dnf5/repos.override.d") and the user override directory ("/etc/dnf/repos.overide.d"). If a file with the same name is present in both override directories, only the file from the user override directory is added to the sorted list. So, the distribution override file can be simply masked by creating a file with the same name in the user override directory.
Retrieves the overrides from the files in the sorted list. Follows the order. The override from the next file overrides the previous one - the last override value wins.

The PR also includes a commit that adds support for the ']' character in the section header name in IniParser. Allows to use more complex glob patterns for repository IDs.

Example of the repos configuration override file:

# Enable `skip_if_unavailable` for all repositories
[*]
skip_if_unavailable = true

# And then disable `skip_if_unavailable` for repositories with id prefix "fedora"
[fedora*]
skip_if_unavailable = false

An example showing the order in which override files are processed:

Files with user repos overrides:
/etc/dnf/repos.overide.d/20-user-overrides.repo
/etc/dnf/repos.overide.d/60-something2.repo
/etc/dnf/repos.overide.d/80-user-overrides.repo
/etc/dnf/repos.overide.d/99-config-manager.repo

Files with distribution repos overrides:
/usr/share/dnf5/repos.overide.d/50-something2.repo
/usr/share/dnf5/repos.overide.d/60-something2.repo
/usr/share/dnf5/repos.overide.d/90-something2.repo

Resulting file processing order:

/etc/dnf/repos.overide.d/20-user-overrides.repo
/usr/share/dnf5/repos.overide.d/50-something2.repo
/etc/dnf/repos.overide.d/60-something2.repo
/etc/dnf/repos.overide.d/80-user-overrides.repo
/usr/share/dnf5/repos.overide.d/90-something2.repo
/etc/dnf/repos.overide.d/99-config-manager.repo

libdnf5/utils/iniparser.cpp

jrohel · 2023-08-28T08:46:59Z

I added CI tests rpm-software-management/ci-dnf-stack#1368

Conan-Kudo · 2023-08-28T16:12:30Z

We need to be able to move packaged repos out of /etc too, but this is a good start.

jrohel · 2023-09-01T14:52:20Z

I rebased the code to resolve conflicts and deduplicate create_sorted_file_list function.

libdnf5/utils/iniparser.cpp

ppisar · 2023-09-01T16:08:02Z

Otherwise it looks good for my eye untrained for C++.

ppisar

I don't see any mistake. Yet I did not test the code.

j-mracek · 2023-09-11T10:22:38Z

include/libdnf5/repo/repo_sack.hpp

    void create_repos_from_system_configuration();

+    /// Loads repositories configuration overrides from drop-in directories. No new repositories are created.
+    /// Only the configuration of the coresponding existing repositories is modified.
+    void load_repos_configuration_overrides();


What about to make it private?

OK. Good idea.

j-mracek · 2023-09-11T10:27:37Z

libdnf5/base/base.cpp

@@ -38,45 +39,6 @@ along with libdnf.  If not, see <https://www.gnu.org/licenses/>.

 namespace fs = std::filesystem;

-namespace {


I am not 100% sure but the commit might be merged with previous commit or is there any reason why you would prefer the split?

j-mracek · 2023-09-11T10:36:51Z

libdnf5/utils/iniparser.cpp

+
+// Returns the input string with the undoubled characters ']'.
+// A sequence of two chars ']' is returned as a single char ']'.
+std::string section_name_decode_bracket(const std::string & section_name) {


I guess it is required for glob support in section header like [fedora[2-8]]. @jrohel Is it correct?

What about to use the sema logic like we have in NEVRA parser - when the second [ then the parser skips everything until the next ]. What do you think?

@j-mracek OK. I changed the logic to be compatible with the NEVRA parser.

After the repositories configuration is loaded, new code is run that loads the override files from the "/usr/share/dnf5/repos.override.d" and "/etc/dnf/repos.overide.d" directories. The format of override files is the same as regular ".repo" files. However, override files can modify the settings of already existing repositories. They cannot create new repositories. Overide files support glob in the repository ID (in the configuration file section name) to support bulk modification of repositories parameters. The order of processing the files with overrides: 1. Creates an alphabetically sorted list of all filenames with overrides in the distribution override directory ("/usr/share/dnf5/repos.override.d") and the user override directory ("/etc/dnf/repos.overide.d"). If a file with the same name is present in both override directories, only the file from the user override directory is added to the sorted list. So, the distribution override file can be simply masked by creating a file with the same name in the user override directory. 2. Retrieves the overrides from the files in the sorted list. Follows the order. The override from the next file overrides the previous one - the last override value wins. ------------------------------------------------- Example of the repos configuration override file: [*] skip_if_unavailable = true [fedora*] skip_if_unavailable = false ------------------------------------------------------------------- An example showing the order in which override files are processed: Files with user repos overrides: /etc/dnf/repos.overide.d/20-user-overrides.repo /etc/dnf/repos.overide.d/60-something2.repo /etc/dnf/repos.overide.d/80-user-overrides.repo /etc/dnf/repos.overide.d/99-config-manager.repo Files with distribution repos overrides: /usr/share/dnf5/repos.overide.d/50-something2.repo /usr/share/dnf5/repos.overide.d/60-something2.repo /usr/share/dnf5/repos.overide.d/90-something2.repo Resulting file processing order: 1. /etc/dnf/repos.overide.d/20-user-overrides.repo 2. /usr/share/dnf5/repos.overide.d/50-something2.repo 3. /etc/dnf/repos.overide.d/60-something2.repo 4. /etc/dnf/repos.overide.d/80-user-overrides.repo 5. /usr/share/dnf5/repos.overide.d/90-something2.repo 6. /etc/dnf/repos.overide.d/99-config-manager.repo

Older changes (Use utils::fs::File instead of streams) caused part of the code to become unreachable (dead).

The section header name is between '[', ']' characters. So the ']' character is interpreted as the end of the section name. This patch adds support for the ']' character in the section name. Doubling the character ']' - the sequence "]]" - means inserting a single character ']' into the section name.

Some lines in the raw strings ended with spaces. And some editors remove trailing spaces during code reformatting. Using a normal string is more robust.

j-mracek · 2023-09-11T14:02:48Z

There is one problem - PR does not contain documentation, but it can be delivered later, but it requires to create an issue.

Conan-Kudo · 2023-09-11T14:22:04Z

There is one problem - PR does not contain documentation, but it can be delivered later, but it requires to create an issue.

I've created an issue to track this as #883

j-mracek · 2023-09-11T14:49:23Z

LGTM

jrohel force-pushed the feature/conf_repos_overrides branch 2 times, most recently from fede15a to b18dd05 Compare August 16, 2023 06:24

jrohel force-pushed the feature/conf_repos_overrides branch from b18dd05 to b691c56 Compare August 25, 2023 10:36

ppisar reviewed Aug 25, 2023

View reviewed changes

libdnf5/utils/iniparser.cpp Show resolved Hide resolved

jrohel force-pushed the feature/conf_repos_overrides branch 2 times, most recently from fcc3657 to 72cb511 Compare August 27, 2023 14:33

jrohel mentioned this pull request Aug 28, 2023

Test configuration files with repository overrides rpm-software-management/ci-dnf-stack#1368

Merged

jrohel added the ready for review label Aug 28, 2023

jrohel force-pushed the feature/conf_repos_overrides branch from 72cb511 to d219818 Compare September 1, 2023 14:49

ppisar reviewed Sep 1, 2023

View reviewed changes

libdnf5/utils/iniparser.cpp Outdated Show resolved Hide resolved

jrohel force-pushed the feature/conf_repos_overrides branch from d219818 to 1ff5a3a Compare September 4, 2023 05:14

ppisar approved these changes Sep 4, 2023

View reviewed changes

j-mracek self-assigned this Sep 11, 2023

j-mracek reviewed Sep 11, 2023

View reviewed changes

jrohel added 9 commits September 11, 2023 14:01

Deduplicate code - move func create_sorted_file_list to utils

2bda439

Add directories for repos configuration overrides to the package

3fef7ee

IniParser: Remove unreachable code

8e760cd

Older changes (Use utils::fs::File instead of streams) caused part of the code to become unreachable (dead).

unit test: IniParser: code deduplication

00cfd3e

unit test: IniParser: Test support for ']' char in section header name

0c1cc29

unit test: IniParser: Replace raw strings by normal strings

26d5b78

Some lines in the raw strings ended with spaces. And some editors remove trailing spaces during code reformatting. Using a normal string is more robust.

unit test: IniParser: Add tests for exceptions on incorrect input

cfc03e9

jrohel force-pushed the feature/conf_repos_overrides branch from 1ff5a3a to cfc03e9 Compare September 11, 2023 13:32

Conan-Kudo mentioned this pull request Sep 11, 2023

Document new repository and configuration paths #883

Closed

Conan-Kudo approved these changes Sep 11, 2023

View reviewed changes

j-mracek added this pull request to the merge queue Sep 11, 2023

Merged via the queue into rpm-software-management:main with commit 1ce386d Sep 11, 2023
7 checks passed

jrohel mentioned this pull request Nov 7, 2023

DNF config-manager Plugin #405

Closed

3 tasks

Conan-Kudo mentioned this pull request Dec 17, 2023

Tracking upstream projects that do not support hermetic-usr for configuration uapi-group/specifications#76

Open

37 tasks

keszybz mentioned this pull request Dec 17, 2023

RFE: allow listing protected packages under /usr #1107

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for repos configuration overrides #820

Add support for repos configuration overrides #820

jrohel commented Aug 15, 2023 •

edited

Loading

jrohel commented Aug 28, 2023

Conan-Kudo commented Aug 28, 2023

jrohel commented Sep 1, 2023

ppisar commented Sep 1, 2023

ppisar left a comment

j-mracek Sep 11, 2023

jrohel Sep 11, 2023

j-mracek Sep 11, 2023

j-mracek Sep 11, 2023

j-mracek Sep 11, 2023

jrohel Sep 11, 2023

j-mracek commented Sep 11, 2023

Conan-Kudo commented Sep 11, 2023

j-mracek commented Sep 11, 2023

		@@ -38,45 +39,6 @@ along with libdnf. If not, see <https://www.gnu.org/licenses/>.

		namespace fs = std::filesystem;

		namespace {

Add support for repos configuration overrides #820

Add support for repos configuration overrides #820

Conversation

jrohel commented Aug 15, 2023 • edited Loading

Example of the repos configuration override file:

An example showing the order in which override files are processed:

jrohel commented Aug 28, 2023

Conan-Kudo commented Aug 28, 2023

jrohel commented Sep 1, 2023

ppisar commented Sep 1, 2023

ppisar left a comment

Choose a reason for hiding this comment

j-mracek Sep 11, 2023

Choose a reason for hiding this comment

jrohel Sep 11, 2023

Choose a reason for hiding this comment

j-mracek Sep 11, 2023

Choose a reason for hiding this comment

j-mracek Sep 11, 2023

Choose a reason for hiding this comment

j-mracek Sep 11, 2023

Choose a reason for hiding this comment

jrohel Sep 11, 2023

Choose a reason for hiding this comment

j-mracek commented Sep 11, 2023

Conan-Kudo commented Sep 11, 2023

j-mracek commented Sep 11, 2023

jrohel commented Aug 15, 2023 •

edited

Loading