-
-
Notifications
You must be signed in to change notification settings - Fork 42
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RFC: Redesign of grml-live #267
Comments
As stated before, I'd like to explore using a "config .deb" approach for most of the work that FAI previously did. |
Thanks for the write-up on what grml-live does today, but I'd like to ask: what should it do in the future? I think it's obvious that it needs to continue to create an installed ISO from scratch. But do we need to keep the update feature |
ACK, that's the reason for RFC! 😄
ACK
So what I see as use cases and what I'm using them for myself:
It feels to me that The I'm not 100% sure whether the
Yes! This new release mode is definitely related to my python based release script that you saw in a paste of mine and something I'm looking forward. :) |
I think supporting Dunno about -b/-B, when I did the template updates recreating the chroot was fast enough 😅 Maybe we can see how a refactor would look like and then decide how easy it is to keep them. |
FWIW, I had also been thinking about a simplified FAI replacement (that would only be able to deploy files, not install Debian packages, but support a class system like FAI's). Specification of a FAI replacementGoals:
Specific features wanted
Why not just use ansible?
Why support symlinkingThis enables a use-case where you keep a (possibly sparse) checkout of the class hierarchy in e.g. Template structureNaïve example with nonhierarchical classes, to demonstrate an issue:
Problem: how do we know whether the Possible approaches:
Alternative layout, to better support sparse checkouts at the cost of duplicating subtrees:
MetadataIt seems to me that attempting to rely solely on the filesystem for instructions like "append" or "delete" or "symlink" or "hardlink" would require unintuitive, complex contortions (such as "magical" filename extensions that the logic then removes, such as "resolv.conf.append", except when they're escaped). Instead, it seems better to handle non-default cases (where files don't just need to be copied to the destination, overwriting whatever is there) using metadata. I'm deliberately not specifying the metadata backend at this point, only the structure of the data itself; I only assume that whatever way is chosen to store the metadata will support storing arbitrary key-value pairs. Ideally, it should be possible to leverage metadata inheritance if it's supported by the backend (set something on a directory, perhaps at a class level, and all child objects under that hierarchy inherit the value). Inheritable metadata should also be settable on the command line, both as a default that individual entries can override, and as a forced value that overrides entry metadata. Special instructions and their metadata encoding:
Possible metadata backends
Actual implementations of this specification SHOULD support at least one of these mechanisms, and if they support more than one, SHOULD support converting between them as well as using all at the same time, with some order of precedence. EDITS:
|
I want to clarify that #281 is an interim step, achieving a number of things:
Notably it doesn't change any of the basic concepts. This redesign RFC however probably wants to change the basic concepts :) |
I updated my comment to add a different directory layout that I think would be better, even though it differs from what FAI uses. |
I like the "alternate layout" a lot. |
I also think that this is a good and worthwhile idea.
This might have been influenced by cfengine (see its class docs at https://docs.cfengine.com/docs/3.25/reference-language-concepts-classes.html) which FAI was targeting since its very beginnings. From a user perspective it could be easier to have:
and list it then, looking like:
Whereas with the suggested layout we'd get:
With the present and expected class preference behavior (later class names have precedence over former ones) I'm not sure which approach is easier to follow through (as in: find out which files/classes are relevant and what's the end result)? |
I think the tool should have a dry run feature that tells the user what would be deployed from where, and how. |
Now that #281 has been merged, we could think about further redesign we'd like to take care of? :) |
Executing it converts our existing config/* from FAI layout (e.g. config/files/$FILENAME/$CLASS) into the new minifai layout (e.g. config/files/$CLASS/$FILENAME). See #267
Redesign of grml-live
Main tasks of grml-live
grml-live is a build system for creating a Grml and Debian based Linux Live system.
grml-live includes the following main stages:
grml_chroot
directory)grml_cd
directory)grml_iso
directory)We don't plan to replace any of
mksquashfs
norxorriso
, though consider getting rid ofFAI
(with itsdirinstall
feature and itsfcopy
tool that we currently use within grml-live).Rational for getting rid of FAI
FAI is a tool for unattended mass deployment of Linux, which is available since 1999. Especially the class system of FAI was a compelling and promising reason to build grml-live on top of FAI. grml-live therefore was based on FAI since its early beginnings in 2007.
grml-live was designed for building Debian based live systems and relied on FAI's
dirinstall
feature underneath. The way live systems are built might look similar to disk based installations. But with grml-live we have specific needs for our release process that FAI was never designed for. We also provide ISO specific customization options and need to comply with GPL needs. Neither of those requirements are fullfilled by FAI.We also had to deal with behavior changes within FAI (e.g. shell.log vs. scripts.log: 74c4bce, make-fai-nfsroot.conf: 75e65f3, ignoring packages: 121b348).
As with any software, we hit bugs also with FAI, like e.g. #760133 (
'mount --make-runbindable' breaks underlying device for unrelated operations
), #989547 (ROOTCMD relies on specific unshare features
), and #1056151 (which makes /usr/sbin/init vanish in /usr-move conditions
).For certain issues we implemented our own workarounds (e.g. 5c3e795, 4e93b8e, e5e4578, ef74e87, 8352df0 AKA https://bugs.debian.org/928981, e9da330, 4e027c7, 8cfed16, 4bc598a).
Usage of FAI within containers also isn't straight forward, we need to e.g. patch hardcoded behavior in
/usr/lib/fai/subroutines
(seegrml-live/build-driver/build.py
Line 150 in 694d50f
We also hit problems with FAI's base tar files approach (see #143 (comment) + https://bugs.debian.org/881829), which we relied on for performance reasons (as bootstrapping a Debian system via https://wiki.debian.org/Debootstrap is quite slow).
Throughout the last years mmdebstrap became the go-to implementation for bootstrapping Debian systems, being a very fast and also customizable tool which we want to utilise for our needs and also avoid need for base files.
FAI hardcodes usage of
debootstrap
withinfai-make-nfsroot
(also see e.g. f365415), preventing us to switch to mmdebstrap therefore.Last but not least, we also have several workarounds in place for dealing with the way FAI handles logfiles and exit codes, that we would like to get rid of.
Re-Design Goal
What's going underneath FAI within grml-live usage
We invoke
FAI
from withingrml-live
via:With a
grml-live
command like this:The resulting
fai dirinstall
command line then looks like this:grml-live with FAI then underneath invokes:
grml-live with
-b
(build only) invokes:grml-live with
-u
(update) invokes:Migration plan
Currently existing FAI related files + scripts in grml-live and their purpose:
config/debconf/GRMLBASE
: debconf preseeding (idea: provide script which invokes debconf-set-selections within mmdebstrap?)config/class/GRMLBASE.var
: setting environment variables (idea: should be set by default by grml-live and be customizable via grml-live.conf?)config/files
: system specific configuration for e.g. apt, initramfs-tools + systemdconfig/grml/squashfs-excludes
: grml-live specific configuration to exclude files during mksquashfs execution (idea: move to etc/grml/?)config/hooks/instsoft.ZFS
: build zfs modules (idea: convert into a script for usage within mmdebstrap?)config/hooks/updatebase.GRMLBASE
: mostly workarounds to handle FAI in different build modes (idea: most shouldn't be relevant without FAI, relevant steps could be handled directly within grml-live or mmdebstrap scripts)config/hooks/instsoft.GRMLBASE
: in build-only mode (-b) doesn't do anything, otherwise only legacy actions or FAI workarounds take place (idea: most shouldn't be relevant without FAI, relevant steps could be handled directly within grml-live or mmdebstrap scripts)config/hooks/savelog.LAST.source
: FAI specific for log files related to error.log (idea: shouldn't be relevant with deprecation of FAI?)config/package_config/*
: which Debian packages should be installed in which grml flavour, also depending on Debian release (idea: investigate whether to keep as is or reconsider file format, also research what options could we have with apt/aptitude to easily ignore packages on-demand? how to implement class approach on our own?)config/scripts
: executing tools to apply configuration changes, updates etc (idea: drop scripts that only invoke FAI'sfcopy
to install configuration files, provide$ROOTCMD
+$target
viagrml-live
to keep scripts functional as-isFeedback
Constructive comments, feedback and suggestions highly welcome!
The text was updated successfully, but these errors were encountered: