Add high-level subcommands for customizing live ISO images and PXE initrds #706

bgilbert · 2021-12-07T08:13:16Z

Fully customizing the installation process can involve a lot of moving parts, including installer config files, NetworkManager key files, kernel arguments, and nested Ignition configs. Add user-friendly iso customize and pxe customize wrappers allowing common customizations to be performed with individual command-line options.

There are a lot of options here, and I'd like feedback on whether they provide the right set of functionality and whether they're clearly named.

This tool is helpful for visualizing Ignition configs generated by these subcommands.

cc @dustymabe for the networking parts in particular.

jlebon

A couple of comments, but this looks great overall!

jlebon · 2021-12-08T15:19:00Z

src/cmdline.rs

+#[derive(Debug, StructOpt)]
+pub struct CommonCustomizeConfig {
+    /// Script to run before installation
+    #[structopt(long, value_name = "path")]


Hmm, I thought we landed on capitalized strings for value names, but now looking at the current cmdline.rs, it looks like a mix. We should probably decide on something and be consistent.

xref #305 (comment) 😁

Consistency seems good, yes. I still find SCREAMING CASE less readable, and it seems unnecessary since clap wraps value names in <> anyway. But if you feel strongly about it, I can convert these over.

src/io/ignition.rs

src/live.rs

src/cmdline.rs

src/live.rs

src/cmdline.rs

dustymabe · 2021-12-08T21:20:51Z

Naming is hard...

I'm not particularly in love with the use of dest but I can't really think of a much better term. final maybe?
Is dest required in --dest-device. There's no --live-device.
--ignition-ca. Applies to both live and final system by default? Should we say that somewhere?

bgilbert · 2021-12-08T23:50:09Z

I'm not particularly in love with the use of dest but I can't really think of a much better term. final maybe?

I'm not thrilled with it either; better ideas welcome. final doesn't obviously relate to the installation device IMO.

Is dest required in --dest-device. There's no --live-device.

It applies to the destination system just as the other --dest-* options do, so I added the dest prefix for clarity/consistency. Also, this keeps the name consistent with the installer config file option, which is also called dest-device.

bgilbert · 2021-12-15T09:05:06Z

Rebased, refactored for recent developments, added long help. Not yet tested.

We're going to have some large subcommand structs, but we allocate one of them per program, so it's fine.

Add a convenience command that combines `iso ignition remove`, `iso network remove`, and `iso kargs reset`.

Serialize and embed an empty Ignition config in the ISO.

bgilbert · 2021-12-15T18:28:24Z

In particular, pxe customize now takes a CoreOS initramfs and prepends it to the customizations. This is consistent with the iso customize semantics, and allows both subcommands to use the new features.json to validate that the OS image supports the customization features we're going to use.

To validate that the input is a CoreOS live initrd, we check for /etc/coreos-live-initramfs, which was previously an internal API between coreos-assembler and fedora-coreos-config. This makes the existence of that file into a stable API.

I originally implemented in-place modification of the input, as iso customize does by default, but implementing in-place reversion to a pristine image would be tricky. To avoid the footgun, I omitted in-place customization for now. I left the subcommand syntax parallel with iso customize (-o is required, -o - for stdout) so we could add this later.

bgilbert · 2021-12-15T19:20:18Z

Also of note, some options want to inform the install process but not necessarily initiate it. For example, --network-keyfile shouldn't, by itself, cause coreos-installer to run inside the live system, which would happen if we unconditionally set copy-network in an installer config. So, in some cases, we need to defer processing the dest portion of an option until all the options have been scanned.

bgilbert · 2021-12-16T09:46:50Z

Switched to reading features.json directly from the ISO to improve performance
- buildextend-live: add features.json directly to ISO coreos-assembler#2617
Renamed dest karg options for consistency with live karg options
Added tests

jlebon

Really nice, LGTM! The split commits are helpful.

Mostly just some tiny nits. But I'd be fine getting this in as is too and fixing things up as follow-ups.

src/io/tee.rs

docs/cmd/iso.md

src/live.rs

jlebon · 2021-12-16T17:36:26Z

src/cmdline.rs

@@ -484,6 +484,13 @@ pub struct CommonCustomizeConfig {
    /// shell.
    #[structopt(long, value_name = "path")]
    pub post_install: Vec<String>,
+    /// Installer config file


"Installer YAML config file" ? 😁

The user will need to look up (or know) the schema anyway, so "YAML" doesn't add anything here IMO.

tests/customize.sh

jlebon · 2021-12-16T19:22:53Z

tests/customize.sh

+    "${prog}" pxe customize src-initrd -o initrd "$@"
+}
+
+qemu_common() {


Would've been nice to leverage kola qemuexec for this, but yeah... it'd require a bunch of changes. Might be worth thinking about long-term doing the enhancements necessary and switching over. E.g. I'd love to be able to do kola qemuexec -p pxe.

Serialize and embed an empty Ignition config in a PXE initrd image. Unlike the pxe wrap commands, this subcommand takes a CoreOS initramfs and prepends it to the customizations. This is consistent with the "iso customize" semantics, and will allow us to validate that the OS image supports the customization features we're going to use. We verify that the input is a CoreOS live initrd, and that it hasn't already been customized. The former is done by checking for /etc/coreos-live-initramfs, which was previously an internal API between coreos-assembler and fedora-coreos-config; this makes that file's existence into a stable API. It's possible to implement in-place modification of the input, as "iso customize" does by default, but implementing in-place reversion to a pristine image would be tricky. (We'd need to truncate the image to the start of the concatenated CPIO archive that contains our customizations.) To avoid the footgun, omit in-place customization. However, the subcommand syntax is parallel with "iso customize" ("-o -" is required for stdout) so we could add this later.

Since they're user code connected with the sole reason for the live boot, and they may do something complex that the user might need to debug, direct their stdout/stderr to kmsg+console.

Read supported OS features from /etc/coreos/features.json in the live initramfs image or /coreos/features.json the live ISO.

Call the options --dest-karg-{append,delete} instead of --dest-{append,delete}-karg. This is inconsistent with the install subcommand and its config file, but provides cleaner UX for the customize subcommands.

Some options engage the installer and some don't. The docs try to make this clear, but it doesn't hurt to show a warning.

It helps debug failing tests.

jlebon reviewed Dec 8, 2021

View reviewed changes

jlebon mentioned this pull request Dec 11, 2021

POC/RFC: Support reinstalls via kexec #712

Closed

bgilbert changed the title ~~RFC: Add high-level subcommands for customizing live ISO images and PXE initrds~~ Add high-level subcommands for customizing live ISO images and PXE initrds Dec 15, 2021

bgilbert added 3 commits December 15, 2021 12:55

cmdline: disable clippy large_enum_variant lint

b9b3fbc

We're going to have some large subcommand structs, but we allocate one of them per program, so it's fine.

live: add iso reset command

9f4dca6

Add a convenience command that combines `iso ignition remove`, `iso network remove`, and `iso kargs reset`.

live: add initial "iso customize" subcommand

060af63

Serialize and embed an empty Ignition config in the ISO.

bgilbert marked this pull request as ready for review December 16, 2021 09:46

jlebon approved these changes Dec 16, 2021

View reviewed changes

jlebon reviewed Dec 16, 2021

View reviewed changes

bgilbert added 13 commits December 16, 2021 14:59

customize: support adding pre-install and post-install hooks

1b7b583

Since they're user code connected with the sole reason for the live boot, and they may do something complex that the user might need to debug, direct their stdout/stderr to kmsg+console.

customize: support merging Ignition child configs for live environment

bbfc563

customize: add OS feature detection

9b77740

Read supported OS features from /etc/coreos/features.json in the live initramfs image or /coreos/features.json the live ISO.

customize: support custom installer config files

8b23cd6

customize: support destination device

b324f98

customize: support destination Ignition configs

21e34ce

customize: support NetworkManager keyfiles

f6b636a

customize: support modifying destination kargs

40d3f35

Call the options --dest-karg-{append,delete} instead of --dest-{append,delete}-karg. This is inconsistent with the install subcommand and its config file, but provides cleaner UX for the customize subcommands.

customize: support modifying ISO kargs

05ae611

customize: support CA bundles for Ignition

6401ad4

customize: warn when installer will run automatically

505344d

Some options engage the installer and some don't. The docs try to make this clear, but it doesn't hurt to show a warning.

tests: print line numbers in execution trace

31b1aa2

It helps debug failing tests.

bgilbert enabled auto-merge December 16, 2021 20:06

tests: add iso/pxe customize tests

4cde093

bgilbert merged commit 9ce1899 into coreos:main Dec 17, 2021

bgilbert deleted the customize branch December 17, 2021 01:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add high-level subcommands for customizing live ISO images and PXE initrds #706

Add high-level subcommands for customizing live ISO images and PXE initrds #706

bgilbert commented Dec 7, 2021 •

edited

Loading

jlebon left a comment

jlebon Dec 8, 2021

bgilbert Dec 9, 2021

dustymabe commented Dec 8, 2021

bgilbert commented Dec 8, 2021

bgilbert commented Dec 15, 2021

bgilbert commented Dec 15, 2021

bgilbert commented Dec 15, 2021

bgilbert commented Dec 16, 2021

jlebon left a comment

jlebon Dec 16, 2021

bgilbert Dec 16, 2021

jlebon Dec 16, 2021 •

edited

Loading

Add high-level subcommands for customizing live ISO images and PXE initrds #706

Add high-level subcommands for customizing live ISO images and PXE initrds #706

Conversation

bgilbert commented Dec 7, 2021 • edited Loading

jlebon left a comment

Choose a reason for hiding this comment

jlebon Dec 8, 2021

Choose a reason for hiding this comment

bgilbert Dec 9, 2021

Choose a reason for hiding this comment

dustymabe commented Dec 8, 2021

bgilbert commented Dec 8, 2021

bgilbert commented Dec 15, 2021

bgilbert commented Dec 15, 2021

bgilbert commented Dec 15, 2021

bgilbert commented Dec 16, 2021

jlebon left a comment

Choose a reason for hiding this comment

jlebon Dec 16, 2021

Choose a reason for hiding this comment

bgilbert Dec 16, 2021

Choose a reason for hiding this comment

jlebon Dec 16, 2021 • edited Loading

Choose a reason for hiding this comment

bgilbert commented Dec 7, 2021 •

edited

Loading

jlebon Dec 16, 2021 •

edited

Loading