Configuration parser issues and possible improvements #2846
Labels
Comments
DaanDeMeyer
changed the title
DaanDeMeyer
added a commit
to DaanDeMeyer/mkosi
that referenced
this issue
Jul 7, 2024
As explained in systemd#2846, there are multiple issues with the current implementation of mkosi.images. Let's take what we learned from the default initrd and the default tools tree and apply it mkosi.images. Specifically, all issues arise from the fact that we apply every option from the global configuration (including CLI arguments) to the images from mkosi.images/. To avoid the issues that arise from this (e.g --package abc installing abc in all images), we made configuration values override CLI arguments again so that we could override faulty CLI arguments again in subimages so that they would only apply to the main image (e.g. set Format= explicitly for each subimage so that --format on the command line only applies to the main image). Because we still wanted to allow configurable settings that can be modified via the command line, we introduced the default specifier '@' which can be prefixed to a setting to set a default value instead of overriding the value. The '@' specifier is generally used in the global image independent configuration to specify default values that can be overridden from the command line. This specifier has led to a lot of confusion, along with the behavior that the CLI does not override the configuration. From the default tools tree and default initrd, we learned that what works very well is to only have specific settings from the main image configuration apply to the default tools tree and default initrd. For example, the distribution, release, mirror and architecture should be the same for the main image and the initrd, but the packages from the main image should not all be installed in the initrd. We can apply this idea to the images from mkosi.images/ as well, if we introduce the assumption that all images defined in mkosi.images are subimages intended to be included in some way or form in the main image. This assumption allows us to divide all settings into either image specific settings or "universal" settings that should apply to the main image and all its subimages. The universal settings are passed on to each subimage. The image specific settings are not. This idea also allows us to define the "main" image outside of mkosi.images again. Since only "universal" settings are passed on, we can safely define an output format and such again in the global configuration, as we know this won't be passed on to subimages. It also allows us to make CLI arguments override configuration again. Since there is no need anymore for subimages to override the CLI configuration as inappropriate CLI configuration such as extra packages will only apply to the main image and not any subimages from mkosi.images/. Because CLI configuration overrides file configuration again, we also don't need the '@' specifier anymore, as default values can simply be set without '@', since the CLI will override the configuration file values by default. We also lose the need for --append, because the sole use for --append was again to override file based configuration. This commit implements all of what's mentioned above, specifically: - CLI configuration now always trumps file based configuration. - The '@' specifier is removed - The main image is now always added from global configuration, even if there are images in mkosi.images/. The main image is always built last, and cannot be used as a dependency in the Dependencies= setting for images defined in mkosi.images/. - The Dependencies= setting for the main image now is used to specify which subimages from mkosi.images/ to build. By default all subimages are built. - A universal tag is introduced for settings and appropriate settings are marked as universal. Universal settings are passed on from the main image configuration to subimage configuration. - The Images= setting is removed, as it's role is replaced by Dependencies=. - The old name mkosi.presets and the Preset section name are removed as they have been deprecated for a long time now. - The config parsing tests are extended to cover more cases. - All builtin configuration is adapted to stop using the '@' specifier. - The documentation is updated in accordance with the changes.
DaanDeMeyer
added a commit
to DaanDeMeyer/mkosi
that referenced
this issue
Jul 8, 2024
As explained in systemd#2846, there are multiple issues with the current implementation of mkosi.images. Let's take what we learned from the default initrd and the default tools tree and apply it mkosi.images. Specifically, all issues arise from the fact that we apply every option from the global configuration (including CLI arguments) to the images from mkosi.images/. To avoid the issues that arise from this (e.g --package abc installing abc in all images), we made configuration values override CLI arguments again so that we could override faulty CLI arguments again in subimages so that they would only apply to the main image (e.g. set Format= explicitly for each subimage so that --format on the command line only applies to the main image). Because we still wanted to allow configurable settings that can be modified via the command line, we introduced the default specifier '@' which can be prefixed to a setting to set a default value instead of overriding the value. The '@' specifier is generally used in the global image independent configuration to specify default values that can be overridden from the command line. This specifier has led to a lot of confusion, along with the behavior that the CLI does not override the configuration. From the default tools tree and default initrd, we learned that what works very well is to only have specific settings from the main image configuration apply to the default tools tree and default initrd. For example, the distribution, release, mirror and architecture should be the same for the main image and the initrd, but the packages from the main image should not all be installed in the initrd. We can apply this idea to the images from mkosi.images/ as well, if we introduce the assumption that all images defined in mkosi.images are subimages intended to be included in some way or form in the main image. This assumption allows us to divide all settings into either image specific settings or "universal" settings that should apply to the main image and all its subimages. The universal settings are passed on to each subimage. The image specific settings are not. This idea also allows us to define the "main" image outside of mkosi.images again. Since only "universal" settings are passed on, we can safely define an output format and such again in the global configuration, as we know this won't be passed on to subimages. It also allows us to make CLI arguments override configuration again. Since there is no need anymore for subimages to override the CLI configuration as inappropriate CLI configuration such as extra packages will only apply to the main image and not any subimages from mkosi.images/. Because CLI configuration overrides file configuration again, we also don't need the '@' specifier anymore, as default values can simply be set without '@', since the CLI will override the configuration file values by default. We also lose the need for --append, because the sole use for --append was again to override file based configuration. This commit implements all of what's mentioned above, specifically: - CLI configuration now always trumps file based configuration. - The '@' specifier is removed - The main image is now always added from global configuration, even if there are images in mkosi.images/. The main image is always built last, and cannot be used as a dependency in the Dependencies= setting for images defined in mkosi.images/. - The Dependencies= setting for the main image now is used to specify which subimages from mkosi.images/ to build. By default all subimages are built. - A universal tag is introduced for settings and appropriate settings are marked as universal. Universal settings are passed on from the main image configuration to subimage configuration. - The Images= setting is removed, as it's role is replaced by Dependencies=. - The old name mkosi.presets and the Preset section name are removed as they have been deprecated for a long time now. - The config parsing tests are extended to cover more cases. - All builtin configuration is adapted to stop using the '@' specifier. - The documentation is updated in accordance with the changes.
DaanDeMeyer
added a commit
to DaanDeMeyer/mkosi
that referenced
this issue
Jul 8, 2024
As explained in systemd#2846, there are multiple issues with the current implementation of mkosi.images. Let's take what we learned from the default initrd and the default tools tree and apply it mkosi.images. Specifically, all issues arise from the fact that we apply every option from the global configuration (including CLI arguments) to the images from mkosi.images/. To avoid the issues that arise from this (e.g --package abc installing abc in all images), we made configuration values override CLI arguments again so that we could override faulty CLI arguments again in subimages so that they would only apply to the main image (e.g. set Format= explicitly for each subimage so that --format on the command line only applies to the main image). Because we still wanted to allow configurable settings that can be modified via the command line, we introduced the default specifier '@' which can be prefixed to a setting to set a default value instead of overriding the value. The '@' specifier is generally used in the global image independent configuration to specify default values that can be overridden from the command line. This specifier has led to a lot of confusion, along with the behavior that the CLI does not override the configuration. From the default tools tree and default initrd, we learned that what works very well is to only have specific settings from the main image configuration apply to the default tools tree and default initrd. For example, the distribution, release, mirror and architecture should be the same for the main image and the initrd, but the packages from the main image should not all be installed in the initrd. We can apply this idea to the images from mkosi.images/ as well, if we introduce the assumption that all images defined in mkosi.images are subimages intended to be included in some way or form in the main image. This assumption allows us to divide all settings into either image specific settings or "universal" settings that should apply to the main image and all its subimages. The universal settings are passed on to each subimage. The image specific settings are not. This idea also allows us to define the "main" image outside of mkosi.images again. Since only "universal" settings are passed on, we can safely define an output format and such again in the global configuration, as we know this won't be passed on to subimages. It also allows us to make CLI arguments override configuration again. Since there is no need anymore for subimages to override the CLI configuration as inappropriate CLI configuration such as extra packages will only apply to the main image and not any subimages from mkosi.images/. Because CLI configuration overrides file configuration again, we also don't need the '@' specifier anymore, as default values can simply be set without '@', since the CLI will override the configuration file values by default. We also lose the need for --append, because the sole use for --append was again to override file based configuration. This commit implements all of what's mentioned above, specifically: - CLI configuration now always trumps file based configuration. - The '@' specifier is removed - The main image is now always added from global configuration, even if there are images in mkosi.images/. The main image is always built last, and cannot be used as a dependency in the Dependencies= setting for images defined in mkosi.images/. - The Dependencies= setting for the main image now is used to specify which subimages from mkosi.images/ to build. By default all subimages are built. - A universal tag is introduced for settings and appropriate settings are marked as universal. Universal settings are passed on from the main image configuration to subimage configuration. - The Images= setting is removed, as it's role is replaced by Dependencies=. - The old name mkosi.presets and the Preset section name are removed as they have been deprecated for a long time now. - The config parsing tests are extended to cover more cases. - All builtin configuration is adapted to stop using the '@' specifier. - The documentation is updated in accordance with the changes.
DaanDeMeyer
added a commit
to DaanDeMeyer/mkosi
that referenced
this issue
Jul 8, 2024
As explained in systemd#2846, there are multiple issues with the current implementation of mkosi.images. Let's take what we learned from the default initrd and the default tools tree and apply it mkosi.images. Specifically, all issues arise from the fact that we apply every option from the global configuration (including CLI arguments) to the images from mkosi.images/. To avoid the issues that arise from this (e.g --package abc installing abc in all images), we made configuration values override CLI arguments again so that we could override faulty CLI arguments again in subimages so that they would only apply to the main image (e.g. set Format= explicitly for each subimage so that --format on the command line only applies to the main image). Because we still wanted to allow configurable settings that can be modified via the command line, we introduced the default specifier '@' which can be prefixed to a setting to set a default value instead of overriding the value. The '@' specifier is generally used in the global image independent configuration to specify default values that can be overridden from the command line. This specifier has led to a lot of confusion, along with the behavior that the CLI does not override the configuration. From the default tools tree and default initrd, we learned that what works very well is to only have specific settings from the main image configuration apply to the default tools tree and default initrd. For example, the distribution, release, mirror and architecture should be the same for the main image and the initrd, but the packages from the main image should not all be installed in the initrd. We can apply this idea to the images from mkosi.images/ as well, if we introduce the assumption that all images defined in mkosi.images are subimages intended to be included in some way or form in the main image. This assumption allows us to divide all settings into either image specific settings or "universal" settings that should apply to the main image and all its subimages. The universal settings are passed on to each subimage. The image specific settings are not. This idea also allows us to define the "main" image outside of mkosi.images again. Since only "universal" settings are passed on, we can safely define an output format and such again in the global configuration, as we know this won't be passed on to subimages. It also allows us to make CLI arguments override configuration again. Since there is no need anymore for subimages to override the CLI configuration as inappropriate CLI configuration such as extra packages will only apply to the main image and not any subimages from mkosi.images/. Because CLI configuration overrides file configuration again, we also don't need the '@' specifier anymore, as default values can simply be set without '@', since the CLI will override the configuration file values by default. We also lose the need for --append, because the sole use for --append was again to override file based configuration. This commit implements all of what's mentioned above, specifically: - CLI configuration now always trumps file based configuration. - The '@' specifier is dropped automatically during parsing - The main image is now always added from global configuration, even if there are images in mkosi.images/. The main image is always built last, and cannot be used as a dependency in the Dependencies= setting for images defined in mkosi.images/. - The Dependencies= setting for the main image now is used to specify which subimages from mkosi.images/ to build. By default all subimages are built. - A universal tag is introduced for settings and appropriate settings are marked as universal. Universal settings are passed on from the main image configuration to subimage configuration. - The Images= setting is removed, as it's role is replaced by Dependencies=. - The old name mkosi.presets and the Preset section name are removed as they have been deprecated for a long time now. - The config parsing tests are extended to cover more cases. - All builtin configuration is adapted to stop using the '@' specifier. - The documentation is updated in accordance with the changes.
DaanDeMeyer
added a commit
to DaanDeMeyer/mkosi
that referenced
this issue
Jul 8, 2024
As explained in systemd#2846, there are multiple issues with the current implementation of mkosi.images. Let's take what we learned from the default initrd and the default tools tree and apply it mkosi.images. Specifically, all issues arise from the fact that we apply every option from the global configuration (including CLI arguments) to the images from mkosi.images/. To avoid the issues that arise from this (e.g --package abc installing abc in all images), we made configuration values override CLI arguments again so that we could override faulty CLI arguments again in subimages so that they would only apply to the main image (e.g. set Format= explicitly for each subimage so that --format on the command line only applies to the main image). Because we still wanted to allow configurable settings that can be modified via the command line, we introduced the default specifier '@' which can be prefixed to a setting to set a default value instead of overriding the value. The '@' specifier is generally used in the global image independent configuration to specify default values that can be overridden from the command line. This specifier has led to a lot of confusion, along with the behavior that the CLI does not override the configuration. From the default tools tree and default initrd, we learned that what works very well is to only have specific settings from the main image configuration apply to the default tools tree and default initrd. For example, the distribution, release, mirror and architecture should be the same for the main image and the initrd, but the packages from the main image should not all be installed in the initrd. We can apply this idea to the images from mkosi.images/ as well, if we introduce the assumption that all images defined in mkosi.images are subimages intended to be included in some way or form in the main image. This assumption allows us to divide all settings into either image specific settings or "universal" settings that should apply to the main image and all its subimages. The universal settings are passed on to each subimage. The image specific settings are not. This idea also allows us to define the "main" image outside of mkosi.images again. Since only "universal" settings are passed on, we can safely define an output format and such again in the global configuration, as we know this won't be passed on to subimages. It also allows us to make CLI arguments override configuration again. Since there is no need anymore for subimages to override the CLI configuration as inappropriate CLI configuration such as extra packages will only apply to the main image and not any subimages from mkosi.images/. Because CLI configuration overrides file configuration again, we also don't need the '@' specifier anymore, as default values can simply be set without '@', since the CLI will override the configuration file values by default. We also lose the need for --append, because the sole use for --append was again to override file based configuration. This commit implements all of what's mentioned above, specifically: - CLI configuration now always trumps file based configuration. - The '@' specifier is dropped automatically during parsing - The main image is now always added from global configuration, even if there are images in mkosi.images/. The main image is always built last, and cannot be used as a dependency in the Dependencies= setting for images defined in mkosi.images/. - The Dependencies= setting for the main image now is used to specify which subimages from mkosi.images/ to build. By default all subimages are built. - A universal tag is introduced for settings and appropriate settings are marked as universal. Universal settings are passed on from the main image configuration to subimage configuration. - The Images= setting is removed, as it's role is replaced by Dependencies=. - The old name mkosi.presets and the Preset section name are removed as they have been deprecated for a long time now. - The config parsing tests are extended to cover more cases. - All builtin configuration is adapted to stop using the '@' specifier. - The documentation is updated in accordance with the changes.
DaanDeMeyer
added a commit
to DaanDeMeyer/mkosi
that referenced
this issue
Jul 8, 2024
As explained in systemd#2846, there are multiple issues with the current implementation of mkosi.images. Let's take what we learned from the default initrd and the default tools tree and apply it mkosi.images. Specifically, all issues arise from the fact that we apply every option from the global configuration (including CLI arguments) to the images from mkosi.images/. To avoid the issues that arise from this (e.g --package abc installing abc in all images), we made configuration values override CLI arguments again so that we could override faulty CLI arguments again in subimages so that they would only apply to the main image (e.g. set Format= explicitly for each subimage so that --format on the command line only applies to the main image). Because we still wanted to allow configurable settings that can be modified via the command line, we introduced the default specifier '@' which can be prefixed to a setting to set a default value instead of overriding the value. The '@' specifier is generally used in the global image independent configuration to specify default values that can be overridden from the command line. This specifier has led to a lot of confusion, along with the behavior that the CLI does not override the configuration. From the default tools tree and default initrd, we learned that what works very well is to only have specific settings from the main image configuration apply to the default tools tree and default initrd. For example, the distribution, release, mirror and architecture should be the same for the main image and the initrd, but the packages from the main image should not all be installed in the initrd. We can apply this idea to the images from mkosi.images/ as well, if we introduce the assumption that all images defined in mkosi.images are subimages intended to be included in some way or form in the main image. This assumption allows us to divide all settings into either image specific settings or "universal" settings that should apply to the main image and all its subimages. The universal settings are passed on to each subimage. The image specific settings are not. This idea also allows us to define the "main" image outside of mkosi.images again. Since only "universal" settings are passed on, we can safely define an output format and such again in the global configuration, as we know this won't be passed on to subimages. It also allows us to make CLI arguments override configuration again. Since there is no need anymore for subimages to override the CLI configuration as inappropriate CLI configuration such as extra packages will only apply to the main image and not any subimages from mkosi.images/. Because CLI configuration overrides file configuration again, we also don't need the '@' specifier anymore, as default values can simply be set without '@', since the CLI will override the configuration file values by default. We also lose the need for --append, because the sole use for --append was again to override file based configuration. This commit implements all of what's mentioned above, specifically: - CLI configuration now always trumps file based configuration. - The '@' specifier is dropped automatically during parsing - The main image is now always added from global configuration, even if there are images in mkosi.images/. The main image is always built last, and cannot be used as a dependency in the Dependencies= setting for images defined in mkosi.images/. - The Dependencies= setting for the main image now is used to specify which subimages from mkosi.images/ to build. By default all subimages are built. - A universal tag is introduced for settings and appropriate settings are marked as universal. Universal settings are passed on from the main image configuration to subimage configuration. - The Images= setting is removed, as it's role is replaced by Dependencies=. - The old name mkosi.presets and the Preset section name are removed as they have been deprecated for a long time now. - The config parsing tests are extended to cover more cases. - All builtin configuration is adapted to stop using the '@' specifier. - The documentation is updated in accordance with the changes.
DaanDeMeyer
added a commit
to DaanDeMeyer/mkosi
that referenced
this issue
Jul 8, 2024
As explained in systemd#2846, there are multiple issues with the current implementation of mkosi.images. Let's take what we learned from the default initrd and the default tools tree and apply it mkosi.images. Specifically, all issues arise from the fact that we apply every option from the global configuration (including CLI arguments) to the images from mkosi.images/. To avoid the issues that arise from this (e.g --package abc installing abc in all images), we made configuration values override CLI arguments again so that we could override faulty CLI arguments again in subimages so that they would only apply to the main image (e.g. set Format= explicitly for each subimage so that --format on the command line only applies to the main image). Because we still wanted to allow configurable settings that can be modified via the command line, we introduced the default specifier '@' which can be prefixed to a setting to set a default value instead of overriding the value. The '@' specifier is generally used in the global image independent configuration to specify default values that can be overridden from the command line. This specifier has led to a lot of confusion, along with the behavior that the CLI does not override the configuration. From the default tools tree and default initrd, we learned that what works very well is to only have specific settings from the main image configuration apply to the default tools tree and default initrd. For example, the distribution, release, mirror and architecture should be the same for the main image and the initrd, but the packages from the main image should not all be installed in the initrd. We can apply this idea to the images from mkosi.images/ as well, if we introduce the assumption that all images defined in mkosi.images are subimages intended to be included in some way or form in the main image. This assumption allows us to divide all settings into either image specific settings or "universal" settings that should apply to the main image and all its subimages. The universal settings are passed on to each subimage. The image specific settings are not. This idea also allows us to define the "main" image outside of mkosi.images again. Since only "universal" settings are passed on, we can safely define an output format and such again in the global configuration, as we know this won't be passed on to subimages. It also allows us to make CLI arguments override configuration again. Since there is no need anymore for subimages to override the CLI configuration as inappropriate CLI configuration such as extra packages will only apply to the main image and not any subimages from mkosi.images/. Because CLI configuration overrides file configuration again, we also don't need the '@' specifier anymore, as default values can simply be set without '@', since the CLI will override the configuration file values by default. We also lose the need for --append, because the sole use for --append was again to override file based configuration. This commit implements all of what's mentioned above, specifically: - CLI configuration now always trumps file based configuration. - The '@' specifier is dropped automatically during parsing - The main image is now always added from global configuration, even if there are images in mkosi.images/. The main image is always built last, and cannot be used as a dependency in the Dependencies= setting for images defined in mkosi.images/. - The Dependencies= setting for the main image now is used to specify which subimages from mkosi.images/ to build. By default all subimages are built. - A universal tag is introduced for settings and appropriate settings are marked as universal. Universal settings are passed on from the main image configuration to subimage configuration. - The Images= setting is removed, as it's role is replaced by Dependencies=. - The old name mkosi.presets and the Preset section name are removed as they have been deprecated for a long time now. - The config parsing tests are extended to cover more cases. - All builtin configuration is adapted to stop using the '@' specifier. - The documentation is updated in accordance with the changes.
DaanDeMeyer
added a commit
to DaanDeMeyer/mkosi
that referenced
this issue
Jul 8, 2024
As explained in systemd#2846, there are multiple issues with the current implementation of mkosi.images. Let's take what we learned from the default initrd and the default tools tree and apply it mkosi.images. Specifically, all issues arise from the fact that we apply every option from the global configuration (including CLI arguments) to the images from mkosi.images/. To avoid the issues that arise from this (e.g --package abc installing abc in all images), we made configuration values override CLI arguments again so that we could override faulty CLI arguments again in subimages so that they would only apply to the main image (e.g. set Format= explicitly for each subimage so that --format on the command line only applies to the main image). Because we still wanted to allow configurable settings that can be modified via the command line, we introduced the default specifier '@' which can be prefixed to a setting to set a default value instead of overriding the value. The '@' specifier is generally used in the global image independent configuration to specify default values that can be overridden from the command line. This specifier has led to a lot of confusion, along with the behavior that the CLI does not override the configuration. From the default tools tree and default initrd, we learned that what works very well is to only have specific settings from the main image configuration apply to the default tools tree and default initrd. For example, the distribution, release, mirror and architecture should be the same for the main image and the initrd, but the packages from the main image should not all be installed in the initrd. We can apply this idea to the images from mkosi.images/ as well, if we introduce the assumption that all images defined in mkosi.images are subimages intended to be included in some way or form in the main image. This assumption allows us to divide all settings into either image specific settings or "universal" settings that should apply to the main image and all its subimages. The universal settings are passed on to each subimage. The image specific settings are not. This idea also allows us to define the "main" image outside of mkosi.images again. Since only "universal" settings are passed on, we can safely define an output format and such again in the global configuration, as we know this won't be passed on to subimages. It also allows us to make CLI arguments override configuration again. Since there is no need anymore for subimages to override the CLI configuration as inappropriate CLI configuration such as extra packages will only apply to the main image and not any subimages from mkosi.images/. Because CLI configuration overrides file configuration again, we also don't need the '@' specifier anymore, as default values can simply be set without '@', since the CLI will override the configuration file values by default. We also lose the need for --append, because the sole use for --append was again to override file based configuration. Note that configuration from mkosi.local.conf is special in that it should override settings from other configuration files, but not settings that are specified on the CLI. This commit implements all of what's mentioned above, specifically: - CLI configuration now always trumps file based configuration. - The '@' specifier is dropped automatically during parsing - The main image is now always added from global configuration, even if there are images in mkosi.images/. The main image is always built last, and cannot be used as a dependency in the Dependencies= setting for images defined in mkosi.images/. - The Dependencies= setting for the main image now is used to specify which subimages from mkosi.images/ to build. By default all subimages are built. - A universal tag is introduced for settings and appropriate settings are marked as universal. Universal settings are passed on from the main image configuration to subimage configuration. - The Images= setting is removed, as it's role is replaced by Dependencies=. - The old name mkosi.presets and the Preset section name are removed as they have been deprecated for a long time now. - The config parsing tests are extended to cover more cases. - All builtin configuration is adapted to stop using the '@' specifier. - The documentation is updated in accordance with the changes.
DaanDeMeyer
added a commit
to DaanDeMeyer/mkosi
that referenced
this issue
Jul 8, 2024
As explained in systemd#2846, there are multiple issues with the current implementation of mkosi.images. Let's take what we learned from the default initrd and the default tools tree and apply it mkosi.images. Specifically, all issues arise from the fact that we apply every option from the global configuration (including CLI arguments) to the images from mkosi.images/. To avoid the issues that arise from this (e.g --package abc installing abc in all images), we made configuration values override CLI arguments again so that we could override faulty CLI arguments again in subimages so that they would only apply to the main image (e.g. set Format= explicitly for each subimage so that --format on the command line only applies to the main image). Because we still wanted to allow configurable settings that can be modified via the command line, we introduced the default specifier '@' which can be prefixed to a setting to set a default value instead of overriding the value. The '@' specifier is generally used in the global image independent configuration to specify default values that can be overridden from the command line. This specifier has led to a lot of confusion, along with the behavior that the CLI does not override the configuration. From the default tools tree and default initrd, we learned that what works very well is to only have specific settings from the main image configuration apply to the default tools tree and default initrd. For example, the distribution, release, mirror and architecture should be the same for the main image and the initrd, but the packages from the main image should not all be installed in the initrd. We can apply this idea to the images from mkosi.images/ as well, if we introduce the assumption that all images defined in mkosi.images are subimages intended to be included in some way or form in the main image. This assumption allows us to divide all settings into either image specific settings or "universal" settings that should apply to the main image and all its subimages. The universal settings are passed on to each subimage. The image specific settings are not. This idea also allows us to define the "main" image outside of mkosi.images again. Since only "universal" settings are passed on, we can safely define an output format and such again in the global configuration, as we know this won't be passed on to subimages. It also allows us to make CLI arguments override configuration again. Since there is no need anymore for subimages to override the CLI configuration as inappropriate CLI configuration such as extra packages will only apply to the main image and not any subimages from mkosi.images/. Because CLI configuration overrides file configuration again, we also don't need the '@' specifier anymore, as default values can simply be set without '@', since the CLI will override the configuration file values by default. We also lose the need for --append, because the sole use for --append was again to override file based configuration. Note that configuration from mkosi.local.conf is special in that it should override settings from other configuration files, but not settings that are specified on the CLI. This commit implements all of what's mentioned above, specifically: - CLI configuration now always trumps file based configuration. - The '@' specifier is dropped automatically during parsing - The main image is now always added from global configuration, even if there are images in mkosi.images/. The main image is always built last, and cannot be used as a dependency in the Dependencies= setting for images defined in mkosi.images/. - The Dependencies= setting for the main image now is used to specify which subimages from mkosi.images/ to build. By default all subimages are built. - A universal tag is introduced for settings and appropriate settings are marked as universal. Universal settings are passed on from the main image configuration to subimage configuration. - The Images= setting is removed, as it's role is replaced by Dependencies=. - The old name mkosi.presets and the Preset section name are removed as they have been deprecated for a long time now. - The config parsing tests are extended to cover more cases. - All builtin configuration is adapted to stop using the '@' specifier. - The documentation is updated in accordance with the changes.
DaanDeMeyer
added a commit
to DaanDeMeyer/mkosi
that referenced
this issue
Jul 8, 2024
As explained in systemd#2846, there are multiple issues with the current implementation of mkosi.images. Let's take what we learned from the default initrd and the default tools tree and apply it mkosi.images. Specifically, all issues arise from the fact that we apply every option from the global configuration (including CLI arguments) to the images from mkosi.images/. To avoid the issues that arise from this (e.g --package abc installing abc in all images), we made configuration values override CLI arguments again so that we could override faulty CLI arguments again in subimages so that they would only apply to the main image (e.g. set Format= explicitly for each subimage so that --format on the command line only applies to the main image). Because we still wanted to allow configurable settings that can be modified via the command line, we introduced the default specifier '@' which can be prefixed to a setting to set a default value instead of overriding the value. The '@' specifier is generally used in the global image independent configuration to specify default values that can be overridden from the command line. This specifier has led to a lot of confusion, along with the behavior that the CLI does not override the configuration. From the default tools tree and default initrd, we learned that what works very well is to only have specific settings from the main image configuration apply to the default tools tree and default initrd. For example, the distribution, release, mirror and architecture should be the same for the main image and the initrd, but the packages from the main image should not all be installed in the initrd. We can apply this idea to the images from mkosi.images/ as well, if we introduce the assumption that all images defined in mkosi.images are subimages intended to be included in some way or form in the main image. This assumption allows us to divide all settings into either image specific settings or "universal" settings that should apply to the main image and all its subimages. The universal settings are passed on to each subimage. The image specific settings are not. This idea also allows us to define the "main" image outside of mkosi.images again. Since only "universal" settings are passed on, we can safely define an output format and such again in the global configuration, as we know this won't be passed on to subimages. It also allows us to make CLI arguments override configuration again. Since there is no need anymore for subimages to override the CLI configuration as inappropriate CLI configuration such as extra packages will only apply to the main image and not any subimages from mkosi.images/. Because CLI configuration overrides file configuration again, we also don't need the '@' specifier anymore, as default values can simply be set without '@', since the CLI will override the configuration file values by default. We also lose the need for --append, because the sole use for --append was again to override file based configuration. Note that configuration from mkosi.local.conf is special in that it should override settings from other configuration files, but not settings that are specified on the CLI. This commit implements all of what's mentioned above, specifically: - CLI configuration now always trumps file based configuration. - The '@' specifier is dropped automatically during parsing - The main image is now always added from global configuration, even if there are images in mkosi.images/. The main image is always built last, and cannot be used as a dependency in the Dependencies= setting for images defined in mkosi.images/. - The Dependencies= setting for the main image now is used to specify which subimages from mkosi.images/ to build. By default all subimages are built. - A universal tag is introduced for settings and appropriate settings are marked as universal. Universal settings are passed on from the main image configuration to subimage configuration. - The Images= setting is removed, as it's role is replaced by Dependencies=. - The old name mkosi.presets and the Preset section name are removed as they have been deprecated for a long time now. - The config parsing tests are extended to cover more cases. - All builtin configuration is adapted to stop using the '@' specifier. - The documentation is updated in accordance with the changes.
DaanDeMeyer
added a commit
to DaanDeMeyer/mkosi
that referenced
this issue
Jul 8, 2024
As explained in systemd#2846, there are multiple issues with the current implementation of mkosi.images. Let's take what we learned from the default initrd and the default tools tree and apply it mkosi.images. Specifically, all issues arise from the fact that we apply every option from the global configuration (including CLI arguments) to the images from mkosi.images/. To avoid the issues that arise from this (e.g --package abc installing abc in all images), we made configuration values override CLI arguments again so that we could override faulty CLI arguments again in subimages so that they would only apply to the main image (e.g. set Format= explicitly for each subimage so that --format on the command line only applies to the main image). Because we still wanted to allow configurable settings that can be modified via the command line, we introduced the default specifier '@' which can be prefixed to a setting to set a default value instead of overriding the value. The '@' specifier is generally used in the global image independent configuration to specify default values that can be overridden from the command line. This specifier has led to a lot of confusion, along with the behavior that the CLI does not override the configuration. From the default tools tree and default initrd, we learned that what works very well is to only have specific settings from the main image configuration apply to the default tools tree and default initrd. For example, the distribution, release, mirror and architecture should be the same for the main image and the initrd, but the packages from the main image should not all be installed in the initrd. We can apply this idea to the images from mkosi.images/ as well, if we introduce the assumption that all images defined in mkosi.images are subimages intended to be included in some way or form in the main image. This assumption allows us to divide all settings into either image specific settings or "universal" settings that should apply to the main image and all its subimages. The universal settings are passed on to each subimage. The image specific settings are not. This idea also allows us to define the "main" image outside of mkosi.images again. Since only "universal" settings are passed on, we can safely define an output format and such again in the global configuration, as we know this won't be passed on to subimages. It also allows us to make CLI arguments override configuration again. Since there is no need anymore for subimages to override the CLI configuration as inappropriate CLI configuration such as extra packages will only apply to the main image and not any subimages from mkosi.images/. Because CLI configuration overrides file configuration again, we also don't need the '@' specifier anymore, as default values can simply be set without '@', since the CLI will override the configuration file values by default. We also lose the need for --append, because the sole use for --append was again to override file based configuration. Note that configuration from mkosi.local.conf is special in that it should override settings from other configuration files, but not settings that are specified on the CLI. This commit implements all of what's mentioned above, specifically: - CLI configuration now always trumps file based configuration. - The '@' specifier is dropped automatically during parsing - The main image is now always added from global configuration, even if there are images in mkosi.images/. The main image is always built last, and cannot be used as a dependency in the Dependencies= setting for images defined in mkosi.images/. - The Dependencies= setting for the main image now is used to specify which subimages from mkosi.images/ to build. By default all subimages are built. - A universal tag is introduced for settings and appropriate settings are marked as universal. Universal settings are passed on from the main image configuration to subimage configuration. - The Images= setting is removed, as it's role is replaced by Dependencies=. - The old name mkosi.presets and the Preset section name are removed as they have been deprecated for a long time now. - The config parsing tests are extended to cover more cases. - All builtin configuration is adapted to stop using the '@' specifier. - The documentation is updated in accordance with the changes.
DaanDeMeyer
added a commit
to DaanDeMeyer/mkosi
that referenced
this issue
Jul 8, 2024
As explained in systemd#2846, there are multiple issues with the current implementation of mkosi.images. Let's take what we learned from the default initrd and the default tools tree and apply it mkosi.images. Specifically, all issues arise from the fact that we apply every option from the global configuration (including CLI arguments) to the images from mkosi.images/. To avoid the issues that arise from this (e.g --package abc installing abc in all images), we made configuration values override CLI arguments again so that we could override faulty CLI arguments again in subimages so that they would only apply to the main image (e.g. set Format= explicitly for each subimage so that --format on the command line only applies to the main image). Because we still wanted to allow configurable settings that can be modified via the command line, we introduced the default specifier '@' which can be prefixed to a setting to set a default value instead of overriding the value. The '@' specifier is generally used in the global image independent configuration to specify default values that can be overridden from the command line. This specifier has led to a lot of confusion, along with the behavior that the CLI does not override the configuration. From the default tools tree and default initrd, we learned that what works very well is to only have specific settings from the main image configuration apply to the default tools tree and default initrd. For example, the distribution, release, mirror and architecture should be the same for the main image and the initrd, but the packages from the main image should not all be installed in the initrd. We can apply this idea to the images from mkosi.images/ as well, if we introduce the assumption that all images defined in mkosi.images are subimages intended to be included in some way or form in the main image. This assumption allows us to divide all settings into either image specific settings or "universal" settings that should apply to the main image and all its subimages. The universal settings are passed on to each subimage. The image specific settings are not. This idea also allows us to define the "main" image outside of mkosi.images again. Since only "universal" settings are passed on, we can safely define an output format and such again in the global configuration, as we know this won't be passed on to subimages. It also allows us to make CLI arguments override configuration again. Since there is no need anymore for subimages to override the CLI configuration as inappropriate CLI configuration such as extra packages will only apply to the main image and not any subimages from mkosi.images/. Because CLI configuration overrides file configuration again, we also don't need the '@' specifier anymore, as default values can simply be set without '@', since the CLI will override the configuration file values by default. We also lose the need for --append, because the sole use for --append was again to override file based configuration. Note that configuration from mkosi.local.conf is special in that it should override settings from other configuration files, but not settings that are specified on the CLI. This commit implements all of what's mentioned above, specifically: - CLI configuration now always trumps file based configuration. - The '@' specifier is dropped automatically during parsing - The main image is now always added from global configuration, even if there are images in mkosi.images/. The main image is always built last, and cannot be used as a dependency in the Dependencies= setting for images defined in mkosi.images/. - The Dependencies= setting for the main image now is used to specify which subimages from mkosi.images/ to build. By default all subimages are built. - A universal tag is introduced for settings and appropriate settings are marked as universal. Universal settings are passed on from the main image configuration to subimage configuration. - The Images= setting is removed, as it's role is replaced by Dependencies=. - The old name mkosi.presets and the Preset section name are removed as they have been deprecated for a long time now. - The config parsing tests are extended to cover more cases. - All builtin configuration is adapted to stop using the '@' specifier. - The documentation is updated in accordance with the changes.
DaanDeMeyer
added a commit
to DaanDeMeyer/mkosi
that referenced
this issue
Jul 8, 2024
As explained in systemd#2846, there are multiple issues with the current implementation of mkosi.images. Let's take what we learned from the default initrd and the default tools tree and apply it mkosi.images. Specifically, all issues arise from the fact that we apply every option from the global configuration (including CLI arguments) to the images from mkosi.images/. To avoid the issues that arise from this (e.g --package abc installing abc in all images), we made configuration values override CLI arguments again so that we could override faulty CLI arguments again in subimages so that they would only apply to the main image (e.g. set Format= explicitly for each subimage so that --format on the command line only applies to the main image). Because we still wanted to allow configurable settings that can be modified via the command line, we introduced the default specifier '@' which can be prefixed to a setting to set a default value instead of overriding the value. The '@' specifier is generally used in the global image independent configuration to specify default values that can be overridden from the command line. This specifier has led to a lot of confusion, along with the behavior that the CLI does not override the configuration. From the default tools tree and default initrd, we learned that what works very well is to only have specific settings from the main image configuration apply to the default tools tree and default initrd. For example, the distribution, release, mirror and architecture should be the same for the main image and the initrd, but the packages from the main image should not all be installed in the initrd. We can apply this idea to the images from mkosi.images/ as well, if we introduce the assumption that all images defined in mkosi.images are subimages intended to be included in some way or form in the main image. This assumption allows us to divide all settings into either image specific settings or "universal" settings that should apply to the main image and all its subimages. The universal settings are passed on to each subimage. The image specific settings are not. This idea also allows us to define the "main" image outside of mkosi.images again. Since only "universal" settings are passed on, we can safely define an output format and such again in the global configuration, as we know this won't be passed on to subimages. It also allows us to make CLI arguments override configuration again. Since there is no need anymore for subimages to override the CLI configuration as inappropriate CLI configuration such as extra packages will only apply to the main image and not any subimages from mkosi.images/. Because CLI configuration overrides file configuration again, we also don't need the '@' specifier anymore, as default values can simply be set without '@', since the CLI will override the configuration file values by default. We also lose the need for --append, because the sole use for --append was again to override file based configuration. Note that configuration from mkosi.local.conf is special in that it should override settings from other configuration files, but not settings that are specified on the CLI. This commit implements all of what's mentioned above, specifically: - CLI configuration now always trumps file based configuration. - The '@' specifier is dropped automatically during parsing - The main image is now always added from global configuration, even if there are images in mkosi.images/. The main image is always built last, and cannot be used as a dependency in the Dependencies= setting for images defined in mkosi.images/. - The Dependencies= setting for the main image now is used to specify which subimages from mkosi.images/ to build. By default all subimages are built. - A universal tag is introduced for settings and appropriate settings are marked as universal. Universal settings are passed on from the main image configuration to subimage configuration. - The Images= setting is removed, as it's role is replaced by Dependencies=. - The old name mkosi.presets and the Preset section name are removed as they have been deprecated for a long time now. - The config parsing tests are extended to cover more cases. - All builtin configuration is adapted to stop using the '@' specifier. - The documentation is updated in accordance with the changes.
DaanDeMeyer
added a commit
to DaanDeMeyer/mkosi
that referenced
this issue
Jul 8, 2024
As explained in systemd#2846, there are multiple issues with the current implementation of mkosi.images. Let's take what we learned from the default initrd and the default tools tree and apply it mkosi.images. Specifically, all issues arise from the fact that we apply every option from the global configuration (including CLI arguments) to the images from mkosi.images/. To avoid the issues that arise from this (e.g --package abc installing abc in all images), we made configuration values override CLI arguments again so that we could override faulty CLI arguments again in subimages so that they would only apply to the main image (e.g. set Format= explicitly for each subimage so that --format on the command line only applies to the main image). Because we still wanted to allow configurable settings that can be modified via the command line, we introduced the default specifier '@' which can be prefixed to a setting to set a default value instead of overriding the value. The '@' specifier is generally used in the global image independent configuration to specify default values that can be overridden from the command line. This specifier has led to a lot of confusion, along with the behavior that the CLI does not override the configuration. From the default tools tree and default initrd, we learned that what works very well is to only have specific settings from the main image configuration apply to the default tools tree and default initrd. For example, the distribution, release, mirror and architecture should be the same for the main image and the initrd, but the packages from the main image should not all be installed in the initrd. We can apply this idea to the images from mkosi.images/ as well, if we introduce the assumption that all images defined in mkosi.images are subimages intended to be included in some way or form in the main image. This assumption allows us to divide all settings into either image specific settings or "universal" settings that should apply to the main image and all its subimages. The universal settings are passed on to each subimage. The image specific settings are not. This idea also allows us to define the "main" image outside of mkosi.images again. Since only "universal" settings are passed on, we can safely define an output format and such again in the global configuration, as we know this won't be passed on to subimages. It also allows us to make CLI arguments override configuration again. Since there is no need anymore for subimages to override the CLI configuration as inappropriate CLI configuration such as extra packages will only apply to the main image and not any subimages from mkosi.images/. Because CLI configuration overrides file configuration again, we also don't need the '@' specifier anymore, as default values can simply be set without '@', since the CLI will override the configuration file values by default. We also lose the need for --append, because the sole use for --append was again to override file based configuration. Note that configuration from mkosi.local.conf is special in that it should override settings from other configuration files, but not settings that are specified on the CLI. This commit implements all of what's mentioned above, specifically: - CLI configuration now always trumps file based configuration. - The '@' specifier is dropped automatically during parsing - The main image is now always added from global configuration, even if there are images in mkosi.images/. The main image is always built last, and cannot be used as a dependency in the Dependencies= setting for images defined in mkosi.images/. - The Dependencies= setting for the main image now is used to specify which subimages from mkosi.images/ to build. By default all subimages are built. - A universal tag is introduced for settings and appropriate settings are marked as universal. Universal settings are passed on from the main image configuration to subimage configuration. - The Images= setting is removed, as it's role is replaced by Dependencies=. - The old name mkosi.presets and the Preset section name are removed as they have been deprecated for a long time now. - The config parsing tests are extended to cover more cases. - All builtin configuration is adapted to stop using the '@' specifier. - The documentation is updated in accordance with the changes.
DaanDeMeyer
added a commit
to DaanDeMeyer/mkosi
that referenced
this issue
Jul 8, 2024
As explained in systemd#2846, there are multiple issues with the current implementation of mkosi.images. Let's take what we learned from the default initrd and the default tools tree and apply it mkosi.images. Specifically, all issues arise from the fact that we apply every option from the global configuration (including CLI arguments) to the images from mkosi.images/. To avoid the issues that arise from this (e.g --package abc installing abc in all images), we made configuration values override CLI arguments again so that we could override faulty CLI arguments again in subimages so that they would only apply to the main image (e.g. set Format= explicitly for each subimage so that --format on the command line only applies to the main image). Because we still wanted to allow configurable settings that can be modified via the command line, we introduced the default specifier '@' which can be prefixed to a setting to set a default value instead of overriding the value. The '@' specifier is generally used in the global image independent configuration to specify default values that can be overridden from the command line. This specifier has led to a lot of confusion, along with the behavior that the CLI does not override the configuration. From the default tools tree and default initrd, we learned that what works very well is to only have specific settings from the main image configuration apply to the default tools tree and default initrd. For example, the distribution, release, mirror and architecture should be the same for the main image and the initrd, but the packages from the main image should not all be installed in the initrd. We can apply this idea to the images from mkosi.images/ as well, if we introduce the assumption that all images defined in mkosi.images are subimages intended to be included in some way or form in the main image. This assumption allows us to divide all settings into either image specific settings or "universal" settings that should apply to the main image and all its subimages. The universal settings are passed on to each subimage. The image specific settings are not. This idea also allows us to define the "main" image outside of mkosi.images again. Since only "universal" settings are passed on, we can safely define an output format and such again in the global configuration, as we know this won't be passed on to subimages. It also allows us to make CLI arguments override configuration again. Since there is no need anymore for subimages to override the CLI configuration as inappropriate CLI configuration such as extra packages will only apply to the main image and not any subimages from mkosi.images/. Because CLI configuration overrides file configuration again, we also don't need the '@' specifier anymore, as default values can simply be set without '@', since the CLI will override the configuration file values by default. We also lose the need for --append, because the sole use for --append was again to override file based configuration. Note that configuration from mkosi.local.conf is special in that it should override settings from other configuration files, but not settings that are specified on the CLI. This commit implements all of what's mentioned above, specifically: - CLI configuration now always trumps file based configuration. - The '@' specifier is dropped automatically during parsing - The main image is now always added from global configuration, even if there are images in mkosi.images/. The main image is always built last, and cannot be used as a dependency in the Dependencies= setting for images defined in mkosi.images/. - The Dependencies= setting for the main image now is used to specify which subimages from mkosi.images/ to build. By default all subimages are built. - A universal tag is introduced for settings and appropriate settings are marked as universal. Universal settings are passed on from the main image configuration to subimage configuration. - The Images= setting is removed, as it's role is replaced by Dependencies=. - The old name mkosi.presets and the Preset section name are removed as they have been deprecated for a long time now. - The config parsing tests are extended to cover more cases. - All builtin configuration is adapted to stop using the '@' specifier. - The documentation is updated in accordance with the changes.
DaanDeMeyer
added a commit
to DaanDeMeyer/mkosi
that referenced
this issue
Jul 8, 2024
As explained in systemd#2846, there are multiple issues with the current implementation of mkosi.images. Let's take what we learned from the default initrd and the default tools tree and apply it mkosi.images. Specifically, all issues arise from the fact that we apply every option from the global configuration (including CLI arguments) to the images from mkosi.images/. To avoid the issues that arise from this (e.g --package abc installing abc in all images), we made configuration values override CLI arguments again so that we could override faulty CLI arguments again in subimages so that they would only apply to the main image (e.g. set Format= explicitly for each subimage so that --format on the command line only applies to the main image). Because we still wanted to allow configurable settings that can be modified via the command line, we introduced the default specifier '@' which can be prefixed to a setting to set a default value instead of overriding the value. The '@' specifier is generally used in the global image independent configuration to specify default values that can be overridden from the command line. This specifier has led to a lot of confusion, along with the behavior that the CLI does not override the configuration. From the default tools tree and default initrd, we learned that what works very well is to only have specific settings from the main image configuration apply to the default tools tree and default initrd. For example, the distribution, release, mirror and architecture should be the same for the main image and the initrd, but the packages from the main image should not all be installed in the initrd. We can apply this idea to the images from mkosi.images/ as well, if we introduce the assumption that all images defined in mkosi.images are subimages intended to be included in some way or form in the main image. This assumption allows us to divide all settings into either image specific settings or "universal" settings that should apply to the main image and all its subimages. The universal settings are passed on to each subimage. The image specific settings are not. This idea also allows us to define the "main" image outside of mkosi.images again. Since only "universal" settings are passed on, we can safely define an output format and such again in the global configuration, as we know this won't be passed on to subimages. It also allows us to make CLI arguments override configuration again. Since there is no need anymore for subimages to override the CLI configuration as inappropriate CLI configuration such as extra packages will only apply to the main image and not any subimages from mkosi.images/. Because CLI configuration overrides file configuration again, we also don't need the '@' specifier anymore, as default values can simply be set without '@', since the CLI will override the configuration file values by default. We also lose the need for --append, because the sole use for --append was again to override file based configuration. Note that configuration from mkosi.local.conf is special in that it should override settings from other configuration files, but not settings that are specified on the CLI. This commit implements all of what's mentioned above, specifically: - CLI configuration now always trumps file based configuration. - The '@' specifier is dropped automatically during parsing - The main image is now always added from global configuration, even if there are images in mkosi.images/. The main image is always built last, and cannot be used as a dependency in the Dependencies= setting for images defined in mkosi.images/. - The Dependencies= setting for the main image now is used to specify which subimages from mkosi.images/ to build. By default all subimages are built. - A universal tag is introduced for settings and appropriate settings are marked as universal. Universal settings are passed on from the main image configuration to subimage configuration. - The Images= setting is removed, as it's role is replaced by Dependencies=. - The old name mkosi.presets and the Preset section name are removed as they have been deprecated for a long time now. - The config parsing tests are extended to cover more cases. - All builtin configuration is adapted to stop using the '@' specifier. - The documentation is updated in accordance with the changes.
DaanDeMeyer
added a commit
to DaanDeMeyer/mkosi
that referenced
this issue
Jul 8, 2024
As explained in systemd#2846, there are multiple issues with the current implementation of mkosi.images. Let's take what we learned from the default initrd and the default tools tree and apply it mkosi.images. Specifically, all issues arise from the fact that we apply every option from the global configuration (including CLI arguments) to the images from mkosi.images/. To avoid the issues that arise from this (e.g --package abc installing abc in all images), we made configuration values override CLI arguments again so that we could override faulty CLI arguments again in subimages so that they would only apply to the main image (e.g. set Format= explicitly for each subimage so that --format on the command line only applies to the main image). Because we still wanted to allow configurable settings that can be modified via the command line, we introduced the default specifier '@' which can be prefixed to a setting to set a default value instead of overriding the value. The '@' specifier is generally used in the global image independent configuration to specify default values that can be overridden from the command line. This specifier has led to a lot of confusion, along with the behavior that the CLI does not override the configuration. From the default tools tree and default initrd, we learned that what works very well is to only have specific settings from the main image configuration apply to the default tools tree and default initrd. For example, the distribution, release, mirror and architecture should be the same for the main image and the initrd, but the packages from the main image should not all be installed in the initrd. We can apply this idea to the images from mkosi.images/ as well, if we introduce the assumption that all images defined in mkosi.images are subimages intended to be included in some way or form in the main image. This assumption allows us to divide all settings into either image specific settings or "universal" settings that should apply to the main image and all its subimages. The universal settings are passed on to each subimage. The image specific settings are not. This idea also allows us to define the "main" image outside of mkosi.images again. Since only "universal" settings are passed on, we can safely define an output format and such again in the global configuration, as we know this won't be passed on to subimages. It also allows us to make CLI arguments override configuration again. Since there is no need anymore for subimages to override the CLI configuration as inappropriate CLI configuration such as extra packages will only apply to the main image and not any subimages from mkosi.images/. Because CLI configuration overrides file configuration again, we also don't need the '@' specifier anymore, as default values can simply be set without '@', since the CLI will override the configuration file values by default. We also lose the need for --append, because the sole use for --append was again to override file based configuration. Note that configuration from mkosi.local.conf is special in that it should override settings from other configuration files, but not settings that are specified on the CLI. This commit implements all of what's mentioned above, specifically: - CLI configuration now always trumps file based configuration. - The '@' specifier is dropped automatically during parsing - The main image is now always added from global configuration, even if there are images in mkosi.images/. The main image is always built last, and cannot be used as a dependency in the Dependencies= setting for images defined in mkosi.images/. - The Dependencies= setting for the main image now is used to specify which subimages from mkosi.images/ to build. By default all subimages are built. - A universal tag is introduced for settings and appropriate settings are marked as universal. Universal settings are passed on from the main image configuration to subimage configuration. - The Images= setting is removed, as it's role is replaced by Dependencies=. - The old name mkosi.presets and the Preset section name are removed as they have been deprecated for a long time now. - The config parsing tests are extended to cover more cases. - All builtin configuration is adapted to stop using the '@' specifier. - The documentation is updated in accordance with the changes.
DaanDeMeyer
added a commit
to DaanDeMeyer/mkosi
that referenced
this issue
Jul 8, 2024
As explained in systemd#2846, there are multiple issues with the current implementation of mkosi.images. Let's take what we learned from the default initrd and the default tools tree and apply it mkosi.images. Specifically, all issues arise from the fact that we apply every option from the global configuration (including CLI arguments) to the images from mkosi.images/. To avoid the issues that arise from this (e.g --package abc installing abc in all images), we made configuration values override CLI arguments again so that we could override faulty CLI arguments again in subimages so that they would only apply to the main image (e.g. set Format= explicitly for each subimage so that --format on the command line only applies to the main image). Because we still wanted to allow configurable settings that can be modified via the command line, we introduced the default specifier '@' which can be prefixed to a setting to set a default value instead of overriding the value. The '@' specifier is generally used in the global image independent configuration to specify default values that can be overridden from the command line. This specifier has led to a lot of confusion, along with the behavior that the CLI does not override the configuration. From the default tools tree and default initrd, we learned that what works very well is to only have specific settings from the main image configuration apply to the default tools tree and default initrd. For example, the distribution, release, mirror and architecture should be the same for the main image and the initrd, but the packages from the main image should not all be installed in the initrd. We can apply this idea to the images from mkosi.images/ as well, if we introduce the assumption that all images defined in mkosi.images are subimages intended to be included in some way or form in the main image. This assumption allows us to divide all settings into either image specific settings or "universal" settings that should apply to the main image and all its subimages. The universal settings are passed on to each subimage. The image specific settings are not. This idea also allows us to define the "main" image outside of mkosi.images again. Since only "universal" settings are passed on, we can safely define an output format and such again in the global configuration, as we know this won't be passed on to subimages. It also allows us to make CLI arguments override configuration again. Since there is no need anymore for subimages to override the CLI configuration as inappropriate CLI configuration such as extra packages will only apply to the main image and not any subimages from mkosi.images/. Because CLI configuration overrides file configuration again, we also don't need the '@' specifier anymore, as default values can simply be set without '@', since the CLI will override the configuration file values by default. We also lose the need for --append, because the sole use for --append was again to override file based configuration. Note that configuration from mkosi.local.conf is special in that it should override settings from other configuration files, but not settings that are specified on the CLI. This commit implements all of what's mentioned above, specifically: - CLI configuration now always trumps file based configuration. - The '@' specifier is dropped automatically during parsing - The main image is now always added from global configuration, even if there are images in mkosi.images/. The main image is always built last, and cannot be used as a dependency in the Dependencies= setting for images defined in mkosi.images/. - The Dependencies= setting for the main image now is used to specify which subimages from mkosi.images/ to build. By default all subimages are built. - A universal tag is introduced for settings and appropriate settings are marked as universal. Universal settings are passed on from the main image configuration to subimage configuration. - The Images= setting is removed, as it's role is replaced by Dependencies=. - The old name mkosi.presets and the Preset section name are removed as they have been deprecated for a long time now. - The config parsing tests are extended to cover more cases. - All builtin configuration is adapted to stop using the '@' specifier. - The documentation is updated in accordance with the changes.
DaanDeMeyer
added a commit
to DaanDeMeyer/mkosi
that referenced
this issue
Jul 8, 2024
As explained in systemd#2846, there are multiple issues with the current implementation of mkosi.images. Let's take what we learned from the default initrd and the default tools tree and apply it mkosi.images. Specifically, all issues arise from the fact that we apply every option from the global configuration (including CLI arguments) to the images from mkosi.images/. To avoid the issues that arise from this (e.g --package abc installing abc in all images), we made configuration values override CLI arguments again so that we could override faulty CLI arguments again in subimages so that they would only apply to the main image (e.g. set Format= explicitly for each subimage so that --format on the command line only applies to the main image). Because we still wanted to allow configurable settings that can be modified via the command line, we introduced the default specifier '@' which can be prefixed to a setting to set a default value instead of overriding the value. The '@' specifier is generally used in the global image independent configuration to specify default values that can be overridden from the command line. This specifier has led to a lot of confusion, along with the behavior that the CLI does not override the configuration. From the default tools tree and default initrd, we learned that what works very well is to only have specific settings from the main image configuration apply to the default tools tree and default initrd. For example, the distribution, release, mirror and architecture should be the same for the main image and the initrd, but the packages from the main image should not all be installed in the initrd. We can apply this idea to the images from mkosi.images/ as well, if we introduce the assumption that all images defined in mkosi.images are subimages intended to be included in some way or form in the main image. This assumption allows us to divide all settings into either image specific settings or "universal" settings that should apply to the main image and all its subimages. The universal settings are passed on to each subimage. The image specific settings are not. This idea also allows us to define the "main" image outside of mkosi.images again. Since only "universal" settings are passed on, we can safely define an output format and such again in the global configuration, as we know this won't be passed on to subimages. It also allows us to make CLI arguments override configuration again. Since there is no need anymore for subimages to override the CLI configuration as inappropriate CLI configuration such as extra packages will only apply to the main image and not any subimages from mkosi.images/. Because CLI configuration overrides file configuration again, we also don't need the '@' specifier anymore, as default values can simply be set without '@', since the CLI will override the configuration file values by default. We also lose the need for --append, because the sole use for --append was again to override file based configuration. Note that configuration from mkosi.local.conf is special in that it should override settings from other configuration files, but not settings that are specified on the CLI. This commit implements all of what's mentioned above, specifically: - CLI configuration now always trumps file based configuration. - The '@' specifier is dropped automatically during parsing - The main image is now always added from global configuration, even if there are images in mkosi.images/. The main image is always built last, and cannot be used as a dependency in the Dependencies= setting for images defined in mkosi.images/. - The Dependencies= setting for the main image now is used to specify which subimages from mkosi.images/ to build. By default all subimages are built. - A universal tag is introduced for settings and appropriate settings are marked as universal. Universal settings are passed on from the main image configuration to subimage configuration. - The Images= setting is removed, as it's role is replaced by Dependencies=. - The old name mkosi.presets and the Preset section name are removed as they have been deprecated for a long time now. - The config parsing tests are extended to cover more cases. - All builtin configuration is adapted to stop using the '@' specifier. - The documentation is updated in accordance with the changes.
DaanDeMeyer
added a commit
to DaanDeMeyer/mkosi
that referenced
this issue
Jul 8, 2024
As explained in systemd#2846, there are multiple issues with the current implementation of mkosi.images. Let's take what we learned from the default initrd and the default tools tree and apply it mkosi.images. Specifically, all issues arise from the fact that we apply every option from the global configuration (including CLI arguments) to the images from mkosi.images/. To avoid the issues that arise from this (e.g --package abc installing abc in all images), we made configuration values override CLI arguments again so that we could override faulty CLI arguments again in subimages so that they would only apply to the main image (e.g. set Format= explicitly for each subimage so that --format on the command line only applies to the main image). Because we still wanted to allow configurable settings that can be modified via the command line, we introduced the default specifier '@' which can be prefixed to a setting to set a default value instead of overriding the value. The '@' specifier is generally used in the global image independent configuration to specify default values that can be overridden from the command line. This specifier has led to a lot of confusion, along with the behavior that the CLI does not override the configuration. From the default tools tree and default initrd, we learned that what works very well is to only have specific settings from the main image configuration apply to the default tools tree and default initrd. For example, the distribution, release, mirror and architecture should be the same for the main image and the initrd, but the packages from the main image should not all be installed in the initrd. We can apply this idea to the images from mkosi.images/ as well, if we introduce the assumption that all images defined in mkosi.images are subimages intended to be included in some way or form in the main image. This assumption allows us to divide all settings into either image specific settings or "universal" settings that should apply to the main image and all its subimages. The universal settings are passed on to each subimage. The image specific settings are not. This idea also allows us to define the "main" image outside of mkosi.images again. Since only "universal" settings are passed on, we can safely define an output format and such again in the global configuration, as we know this won't be passed on to subimages. It also allows us to make CLI arguments override configuration again. Since there is no need anymore for subimages to override the CLI configuration as inappropriate CLI configuration such as extra packages will only apply to the main image and not any subimages from mkosi.images/. Because CLI configuration overrides file configuration again, we also don't need the '@' specifier anymore, as default values can simply be set without '@', since the CLI will override the configuration file values by default. We also lose the need for --append, because the sole use for --append was again to override file based configuration. Note that configuration from mkosi.local.conf is special in that it should override settings from other configuration files, but not settings that are specified on the CLI. This commit implements all of what's mentioned above, specifically: - CLI configuration now always trumps file based configuration. - The '@' specifier is dropped automatically during parsing - The main image is now always added from global configuration, even if there are images in mkosi.images/. The main image is always built last, and cannot be used as a dependency in the Dependencies= setting for images defined in mkosi.images/. - The Dependencies= setting for the main image now is used to specify which subimages from mkosi.images/ to build. By default all subimages are built. - A universal tag is introduced for settings and appropriate settings are marked as universal. Universal settings are passed on from the main image configuration to subimage configuration. - The Images= setting is removed, as it's role is replaced by Dependencies=. - The old name mkosi.presets and the Preset section name are removed as they have been deprecated for a long time now. - The config parsing tests are extended to cover more cases. - All builtin configuration is adapted to stop using the '@' specifier. - The documentation is updated in accordance with the changes.
DaanDeMeyer
added a commit
to DaanDeMeyer/mkosi
that referenced
this issue
Jul 8, 2024
As explained in systemd#2846, there are multiple issues with the current implementation of mkosi.images. Let's take what we learned from the default initrd and the default tools tree and apply it mkosi.images. Specifically, all issues arise from the fact that we apply every option from the global configuration (including CLI arguments) to the images from mkosi.images/. To avoid the issues that arise from this (e.g --package abc installing abc in all images), we made configuration values override CLI arguments again so that we could override faulty CLI arguments again in subimages so that they would only apply to the main image (e.g. set Format= explicitly for each subimage so that --format on the command line only applies to the main image). Because we still wanted to allow configurable settings that can be modified via the command line, we introduced the default specifier '@' which can be prefixed to a setting to set a default value instead of overriding the value. The '@' specifier is generally used in the global image independent configuration to specify default values that can be overridden from the command line. This specifier has led to a lot of confusion, along with the behavior that the CLI does not override the configuration. From the default tools tree and default initrd, we learned that what works very well is to only have specific settings from the main image configuration apply to the default tools tree and default initrd. For example, the distribution, release, mirror and architecture should be the same for the main image and the initrd, but the packages from the main image should not all be installed in the initrd. We can apply this idea to the images from mkosi.images/ as well, if we introduce the assumption that all images defined in mkosi.images are subimages intended to be included in some way or form in the main image. This assumption allows us to divide all settings into either image specific settings or "universal" settings that should apply to the main image and all its subimages. The universal settings are passed on to each subimage. The image specific settings are not. This idea also allows us to define the "main" image outside of mkosi.images again. Since only "universal" settings are passed on, we can safely define an output format and such again in the global configuration, as we know this won't be passed on to subimages. It also allows us to make CLI arguments override configuration again. Since there is no need anymore for subimages to override the CLI configuration as inappropriate CLI configuration such as extra packages will only apply to the main image and not any subimages from mkosi.images/. Because CLI configuration overrides file configuration again, we also don't need the '@' specifier anymore, as default values can simply be set without '@', since the CLI will override the configuration file values by default. We also lose the need for --append, because the sole use for --append was again to override file based configuration. Note that configuration from mkosi.local.conf is special in that it should override settings from other configuration files, but not settings that are specified on the CLI. This commit implements all of what's mentioned above, specifically: - CLI configuration now always trumps file based configuration. - The '@' specifier is dropped automatically during parsing - The main image is now always added from global configuration, even if there are images in mkosi.images/. The main image is always built last, and cannot be used as a dependency in the Dependencies= setting for images defined in mkosi.images/. - The Dependencies= setting for the main image now is used to specify which subimages from mkosi.images/ to build. By default all subimages are built. - A universal tag is introduced for settings and appropriate settings are marked as universal. Universal settings are passed on from the main image configuration to subimage configuration. - The Images= setting is removed, as it's role is replaced by Dependencies=. - The old name mkosi.presets and the Preset section name are removed as they have been deprecated for a long time now. - The config parsing tests are extended to cover more cases. - All builtin configuration is adapted to stop using the '@' specifier. - The documentation is updated in accordance with the changes.
DaanDeMeyer
added a commit
to DaanDeMeyer/mkosi
that referenced
this issue
Jul 8, 2024
As explained in systemd#2846, there are multiple issues with the current implementation of mkosi.images. Let's take what we learned from the default initrd and the default tools tree and apply it mkosi.images. Specifically, all issues arise from the fact that we apply every option from the global configuration (including CLI arguments) to the images from mkosi.images/. To avoid the issues that arise from this (e.g --package abc installing abc in all images), we made configuration values override CLI arguments again so that we could override faulty CLI arguments again in subimages so that they would only apply to the main image (e.g. set Format= explicitly for each subimage so that --format on the command line only applies to the main image). Because we still wanted to allow configurable settings that can be modified via the command line, we introduced the default specifier '@' which can be prefixed to a setting to set a default value instead of overriding the value. The '@' specifier is generally used in the global image independent configuration to specify default values that can be overridden from the command line. This specifier has led to a lot of confusion, along with the behavior that the CLI does not override the configuration. From the default tools tree and default initrd, we learned that what works very well is to only have specific settings from the main image configuration apply to the default tools tree and default initrd. For example, the distribution, release, mirror and architecture should be the same for the main image and the initrd, but the packages from the main image should not all be installed in the initrd. We can apply this idea to the images from mkosi.images/ as well, if we introduce the assumption that all images defined in mkosi.images are subimages intended to be included in some way or form in the main image. This assumption allows us to divide all settings into either image specific settings or "universal" settings that should apply to the main image and all its subimages. The universal settings are passed on to each subimage. The image specific settings are not. This idea also allows us to define the "main" image outside of mkosi.images again. Since only "universal" settings are passed on, we can safely define an output format and such again in the global configuration, as we know this won't be passed on to subimages. It also allows us to make CLI arguments override configuration again. Since there is no need anymore for subimages to override the CLI configuration as inappropriate CLI configuration such as extra packages will only apply to the main image and not any subimages from mkosi.images/. Because CLI configuration overrides file configuration again, we also don't need the '@' specifier anymore, as default values can simply be set without '@', since the CLI will override the configuration file values by default. We also lose the need for --append, because the sole use for --append was again to override file based configuration. Note that configuration from mkosi.local.conf is special in that it should override settings from other configuration files, but not settings that are specified on the CLI. This commit implements all of what's mentioned above, specifically: - CLI configuration now always trumps file based configuration. - The '@' specifier is dropped automatically during parsing - The main image is now always added from global configuration, even if there are images in mkosi.images/. The main image is always built last, and cannot be used as a dependency in the Dependencies= setting for images defined in mkosi.images/. - The Dependencies= setting for the main image now is used to specify which subimages from mkosi.images/ to build. By default all subimages are built. - A universal tag is introduced for settings and appropriate settings are marked as universal. Universal settings are passed on from the main image configuration to subimage configuration. - The Images= setting is removed, as it's role is replaced by Dependencies=. - The old name mkosi.presets and the Preset section name are removed as they have been deprecated for a long time now. - The config parsing tests are extended to cover more cases. - All builtin configuration is adapted to stop using the '@' specifier. - The documentation is updated in accordance with the changes.
DaanDeMeyer
added a commit
to DaanDeMeyer/mkosi
that referenced
this issue
Jul 8, 2024
As explained in systemd#2846, there are multiple issues with the current implementation of mkosi.images. Let's take what we learned from the default initrd and the default tools tree and apply it mkosi.images. Specifically, all issues arise from the fact that we apply every option from the global configuration (including CLI arguments) to the images from mkosi.images/. To avoid the issues that arise from this (e.g --package abc installing abc in all images), we made configuration values override CLI arguments again so that we could override faulty CLI arguments again in subimages so that they would only apply to the main image (e.g. set Format= explicitly for each subimage so that --format on the command line only applies to the main image). Because we still wanted to allow configurable settings that can be modified via the command line, we introduced the default specifier '@' which can be prefixed to a setting to set a default value instead of overriding the value. The '@' specifier is generally used in the global image independent configuration to specify default values that can be overridden from the command line. This specifier has led to a lot of confusion, along with the behavior that the CLI does not override the configuration. From the default tools tree and default initrd, we learned that what works very well is to only have specific settings from the main image configuration apply to the default tools tree and default initrd. For example, the distribution, release, mirror and architecture should be the same for the main image and the initrd, but the packages from the main image should not all be installed in the initrd. We can apply this idea to the images from mkosi.images/ as well, if we introduce the assumption that all images defined in mkosi.images are subimages intended to be included in some way or form in the main image. This assumption allows us to divide all settings into either image specific settings or "universal" settings that should apply to the main image and all its subimages. The universal settings are passed on to each subimage. The image specific settings are not. This idea also allows us to define the "main" image outside of mkosi.images again. Since only "universal" settings are passed on, we can safely define an output format and such again in the global configuration, as we know this won't be passed on to subimages. It also allows us to make CLI arguments override configuration again. Since there is no need anymore for subimages to override the CLI configuration as inappropriate CLI configuration such as extra packages will only apply to the main image and not any subimages from mkosi.images/. Because CLI configuration overrides file configuration again, we also don't need the '@' specifier anymore, as default values can simply be set without '@', since the CLI will override the configuration file values by default. We also lose the need for --append, because the sole use for --append was again to override file based configuration. Note that configuration from mkosi.local.conf is special in that it should override settings from other configuration files, but not settings that are specified on the CLI. This commit implements all of what's mentioned above, specifically: - CLI configuration now always trumps file based configuration. - The '@' specifier is dropped automatically during parsing - The main image is now always added from global configuration, even if there are images in mkosi.images/. The main image is always built last, and cannot be used as a dependency in the Dependencies= setting for images defined in mkosi.images/. - The Dependencies= setting for the main image now is used to specify which subimages from mkosi.images/ to build. By default all subimages are built. - A universal tag is introduced for settings and appropriate settings are marked as universal. Universal settings are passed on from the main image configuration to subimage configuration. - The Images= setting is removed, as it's role is replaced by Dependencies=. - The old name mkosi.presets and the Preset section name are removed as they have been deprecated for a long time now. - The config parsing tests are extended to cover more cases. - All builtin configuration is adapted to stop using the '@' specifier. - The documentation is updated in accordance with the changes.
|
Fixed by #2847 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We've stuck with the new parser for quite a while now, and I've discovered a few fundamental issues with it:
mkosi.imageswhen the directory exists was a mistake.mkosi.imagesshould purely be about sub images that are included in the main image (initrd, sysext, ...). This allows us to make much more assumptions about the images inmkosi.imageswhich makes the next point possible. The main image would be defined using the global configuration instead. If multiple "main" images are needed, profiles + invoking mkosi multiple times can be used instead.mkosi.imagesand the issues that brought with it made us make various changes to config parsing that wouldn't have been necessary otherwise. For example we would have never had to introduce the@specifier for default values. What we should do is what we do when building the default tools tree and the default initrd, we pass through specific "global" settings (distribution, release, mirror, architecture, ...) instead of all of them. This also solves the issue where currently images inmkosi.imageshave to explicitly override various settings that they don't want to apply to them if specified on the CLI (for example settingPackages=before defining their own packages to make sure packages specified on the CLI aren't included in the subimage).The text was updated successfully, but these errors were encountered: