Enable the use of the OverlayFS for the LiveOS root filesystem. #107

FGrose · 2015-12-20T03:43:32Z

Integrate the option to use an OverlayFS as the root filesystem
into the 90dmsquash-live module for testing purposes.

The rd.live.overlay.overlayfs option allows one to request an
OverlayFS overlay. If a persistent overlay is detected at the
standard LiveOS path, the overlay & type detected will be used.

Tested primarily with transient in-RAM overlay boots on vfat-
formatted Live USB devices, with persistent overlay directories
on ext4-formatted Live USB devices, and with embedded, persistent
overlay directories on vfat-formatted devices. (Persistent overlay
directories on a vfat-formatted device must be in an embedded
filesystem that supports the creation of trusted.* extended attributes,
and must provide valid d_type in readdir responses.)

The rd.live.overlay.readonly option, which allows a persistent
overlayfs to be mounted read only through a higher level transient
overlay directory, has been implemented through the multiple lower
layers feature of OverlayFS.

This implementation currently requires the enforcing=0 kernel command
line option. Further SELinux tuning may change this limitation.
(SELinux is supported in OverlayFS since kernel 4.8.)

The default transient DM overlay size has been adjusted up to 32 GiB.
This change supports comparison of transient Device-mapper vs.
transient OverlayFS overlay performance. A transient DM overlay
is a sparse file in memory, so this setting does not consume more
RAM for legacy applications. It does permit a user to use all of
the available root filesystem storage, and fails gently when it is
consumed, as the available free root filesystem storage on a typical
LiveOS build is only a few GiB. Thus, when booted on other-
than-small RAM systems, the transient DM overlay should not overflow.

OverlayFS offers the potential to use all of the available free RAM
or all of the available free disc storage (on non-vfat-devices)
in its overlay, even beyond the root filesystem available space,
because the OverlayFS root filesystem is a union of directories on
two different partitions.

FGrose · 2015-12-20T04:10:06Z

Mailing list post: http://article.gmane.org/gmane.linux.kernel.initramfs/4308

bcl · 2015-12-22T01:10:09Z

This is worth exploring, but if it can't support SELinux it can't be used for live installations. We also need to make sure that /dev/mapper/live-base or something similar points to the original base filesystem so that local changes don't get copied to the target during installation.

FGrose · 2015-12-22T02:52:24Z

This version continues to provide /dev/mapper/live-base even with the rd.live.overlay.overlayfs option in order to be compatible with the current installation structures. Some users may eventually want to avoid any Device-mapper overhead, and the OverlayFS does depend on the read-only base image, ext3fs.img or rootfs.img, mounted at /run/rootfsbase.

The purpose of this version is to provide a means to optionally boot an OverlayFS version of LiveOS in Fedora Rawhide so that others can test its features and limitations, with the hope that those with expertise in SELinux and other affected components could explore adaptations to complete the integration.

centos-ci · 2016-10-20T12:53:10Z

Can one of the admins verify this patch?

centos-ci · 2016-10-20T15:18:06Z

Can one of the admins verify this patch?

haraldh · 2016-10-20T15:42:35Z

ok to test

Conan-Kudo · 2017-01-04T04:46:54Z

@bcl @FGrose My understanding is that this should work with SELinux now? I think Dan Walsh made it work in recent kernels so that Docker+SELinux+OverlayFS worked.

centos-ci · 2017-01-04T04:46:54Z

Can one of the admins verify this patch?

haraldh · 2017-01-16T10:01:21Z

@centos-ci
ok to test

haraldh · 2017-01-16T13:16:13Z

@centos-ci
retest this please

FGrose · 2017-01-16T22:26:52Z

I force pushed amended commits to simplify checking the overlayfs commandline argument for those cases where only one check is needed.

FGrose · 2017-01-20T04:53:40Z

I force pushed a couple of fixes: 1. to better deal with the readonly_overlay code path for OverlayFS, and 2. to take advantage of the $sz variable while setting up live-osimg-min.
Here was the diff:

diff --git a/modules.d/90dmsquash-live/dmsquash-live-root.sh b/modules.d/90dmsquash-live/dmsquash-live-root.sh
index 987065b..60ef6c1 100755
--- a/modules.d/90dmsquash-live/dmsquash-live-root.sh
+++ b/modules.d/90dmsquash-live/dmsquash-live-root.sh
@@ -164,13 +164,15 @@ do_live_overlay() {
 
     # set up the snapshot
     sz=$(blockdev --getsz $BASE_LOOPDEV)
-    if [ -n "$readonly_overlay" ]; then
-        echo 0 $sz snapshot $BASE_LOOPDEV $OVERLAY_LOOPDEV P 8 | dmsetup create --readonly live-ro
-        base="/dev/mapper/live-ro"
-        over=$RO_OVERLAY_LOOPDEV
-    else
-        base=$BASE_LOOPDEV
-        over=$OVERLAY_LOOPDEV
+    if [ -z "$overlayfs" ]; then
+        if [ -n "$readonly_overlay" ]; then
+            echo 0 $sz snapshot $BASE_LOOPDEV $OVERLAY_LOOPDEV P 8 | dmsetup create --readonly live-ro
+            base="/dev/mapper/live-ro"
+            over=$RO_OVERLAY_LOOPDEV
+        else
+            base=$BASE_LOOPDEV
+            over=$OVERLAY_LOOPDEV
+        fi
     fi
 
     if [ -n "$thin_snapshot" ]; then
@@ -295,7 +297,7 @@ fi
 if [ -b "$OSMIN_LOOPDEV" ]; then
     # set up the devicemapper snapshot device, which will merge
     # the normal live fs image, and the delta, into a minimzied fs image
-    echo "0 $( blockdev --getsz $BASE_LOOPDEV ) snapshot $BASE_LOOPDEV $OSMIN_LOOPDEV P 8" | dmsetup create --readonly live-osimg-min
+    echo "0 $sz snapshot $BASE_LOOPDEV $OSMIN_LOOPDEV P 8" | dmsetup create --readonly live-osimg-min
 fi
 
 ROOTFLAGS="$(getarg rootflags)"
@@ -306,8 +308,8 @@ fi
 if [ -n "$overlayfs" ]; then
     mkdir -m 0755 /run/rootfsbase
     if [ -n "$reset_overlay" ] && [ -L /run/overlayfs ]; then
-        rm -r -- $(readlink /run/overlayfs)
-        mkdir -m 0755 $(readlink /run/overlayfs)
+        ovlfs=$(readlink /run/overlayfs)
+        rm -r -- ${ovlfs}/* ${ovlfs}/.*
     fi
     if [ -n "$readonly_overlay" ]; then
         mkdir -m 0755 /run/rootfsbase-r

haraldh · 2017-01-20T07:49:13Z

rm -r -- ${ovlfs}/* ${ovlfs}/.*

Wouldn't that lead to error messages like:

rm: refusing to remove '.' or '..' directory: skipping […]
rm: refusing to remove '.' or '..' directory: skipping […]

haraldh · 2017-01-20T13:45:42Z

@centos-ci
retest please

haraldh · 2017-01-20T14:23:00Z

@centos-ci
retest this please

FGrose · 2017-01-21T08:02:34Z

@haraldh re: rm -r -- ${ovlfs}/* ${ovlfs}/.* -- Yes, 2 message lines in several pages. This version drops them into /dev/null. (The directory must be kept to maintain the root_t context.)

I've force pushed a couple more fixes: 1. to deal properly with a missing OverlayFS overlay or an empty/zeroed DM overlay, and 2. to die if rd.live.overlay.overlayfs is specified with a Device-mapper overlay file. 3. Also shortened the execution path slightly for the newer images using the 'rootfs.img' name.
The diff from before is the following:

diff --git a/usr/lib/dracut/modules.d/90dmsquash-live/dmsquash-live-root.sh b/modules.d/90dmsquash-live/dmsquash-live-root.sh
index 6a5d0c2..0933130 100755
--- a/usr/lib/dracut/modules.d/90dmsquash-live/dmsquash-live-root.sh
+++ b/modules.d/90dmsquash-live/dmsquash-live-root.sh
@@ -118,8 +118,12 @@ do_live_overlay() {
             losetup $opt $OVERLAY_LOOPDEV /run/initramfs/overlayfs$pathspec
             umount -l /run/initramfs/overlayfs || :
             oltype=$(det_img_fs $OVERLAY_LOOPDEV)
-            if [ $oltype = DM_snapshot_cow ]; then
-                if [ -n "$reset_overlay" ]; then
+            if [ -z "$oltype" ] || [ "$oltype" = DM_snapshot_cow ]; then
+                if [ -n "$overlayfs" ]; then
+                    die "Found a Device-mapper overlay--not OverlayFS... " \
+                        "rd.live.overlay.overlayfs is not valid."
+                elif [ -n "$reset_overlay" ]; then
+                    info "Resetting the Device-mapper overlay."
                     dd if=/dev/zero of=$OVERLAY_LOOPDEV bs=64k count=1 conv=fsync 2>/dev/null
                 fi
             else
@@ -127,11 +131,12 @@ do_live_overlay() {
                 ln -s /run/initramfs/overlayfs/overlayfs /run/overlayfs$opt
                 ln -s /run/initramfs/overlayfs/ovlwork /run/ovlwork$opt
             fi
+            setup="yes"
         elif [ -d /run/initramfs/overlayfs$pathspec ]; then
             ln -s /run/initramfs/overlayfs$pathspec /run/overlayfs$opt
             ln -s /run/initramfs/overlayfs$pathspec/../ovlwork /run/ovlwork$opt
+            setup="yes"
         fi
-        setup="yes"
     fi
     if [ -n "$overlayfs" ]; then
         modprobe overlay
@@ -230,8 +235,7 @@ fi
 if [ -e /run/initramfs/live/${live_dir}/${squash_image} ]; then
     SQUASHED="/run/initramfs/live/${live_dir}/${squash_image}"
 fi
-
-if [ -e "$SQUASHED" ] ; then
+if [ -e "$SQUASHED" ]; then
     if [ -n "$live_ram" ]; then
         echo 'Copying live image to RAM...' > /dev/kmsg
         echo ' (this may take a minute)' > /dev/kmsg
@@ -245,17 +249,17 @@ if [ -e "$SQUASHED" ] ; then
     mkdir -m 0755 -p /run/initramfs/squashfs
     mount -n -t squashfs -o ro $SQUASHED_LOOPDEV /run/initramfs/squashfs
 
-    if [ -f /run/initramfs/squashfs/LiveOS/ext3fs.img ]; then
-        FSIMG="/run/initramfs/squashfs/LiveOS/ext3fs.img"
-    elif [ -f /run/initramfs/squashfs/LiveOS/rootfs.img ]; then
+    if [ -f /run/initramfs/squashfs/LiveOS/rootfs.img ]; then
         FSIMG="/run/initramfs/squashfs/LiveOS/rootfs.img"
+    elif [ -f /run/initramfs/squashfs/LiveOS/ext3fs.img ]; then
+        FSIMG="/run/initramfs/squashfs/LiveOS/ext3fs.img"
     fi
 else
     # we might have an embedded fs image to use as rootfs (uncompressed live)
-    if [ -e /run/initramfs/live/${live_dir}/ext3fs.img ]; then
-        FSIMG="/run/initramfs/live/${live_dir}/ext3fs.img"
-    elif [ -e /run/initramfs/live/${live_dir}/rootfs.img ]; then
+    if [ -e /run/initramfs/live/${live_dir}/rootfs.img ]; then
         FSIMG="/run/initramfs/live/${live_dir}/rootfs.img"
+    elif [ -e /run/initramfs/live/${live_dir}/ext3fs.img ]; then
+        FSIMG="/run/initramfs/live/${live_dir}/ext3fs.img"
     fi
     if [ -n "$live_ram" ]; then
         echo 'Copying live image to RAM...' > /dev/kmsg
@@ -309,7 +313,8 @@ if [ -n "$overlayfs" ]; then
     mkdir -m 0755 /run/rootfsbase
     if [ -n "$reset_overlay" ] && [ -L /run/overlayfs ]; then
         ovlfs=$(readlink /run/overlayfs)
-        rm -r -- ${ovlfs}/* ${ovlfs}/.*
+        info "Resetting the OverlayFS overlay directory."
+        rm -r -- ${ovlfs}/* ${ovlfs}/.* >/dev/null 2>&1
     fi
     if [ -n "$readonly_overlay" ]; then
         mkdir -m 0755 /run/rootfsbase-r
(END)

FGrose · 2017-01-23T21:29:09Z

This rebased commit accomplishes the OverlayFS feature and includes the fixes from pull request #190, superseding it, if this commit is accepted.

See also some testing of sudo dnf upgrade on a LiveOS with OverlayFS at issue #104.

haraldh · 2017-01-25T11:45:14Z

@centos-ci
retest this please

Conan-Kudo · 2017-03-08T13:40:22Z

@FGrose @haraldh How does this look now?

haraldh · 2017-03-08T13:55:49Z

@centos-ci
retest this please

FGrose · 2017-03-09T18:52:34Z

Added documentation for the OverlayFS overlays.

Added feedback for missing overlays. But I couldn't find a more generic test for an active plymouth splash, such as rhgb, in the DRACUT_SYSTEMD case.

To avoid two separate plymouth calls, I tried:
systemd-ask-password --timeout=0 message, but the rhgb case fails to show the message prompt (a bug, I think, in plymouth). If I could sendkey <Esc>, this would show the startup console.

I also tried

cat > /run/systemd/ask-password/ask.0000<<EOF
[Ask]
Socket=/dev/null
Message=${m[*]}
timeout=0
EOF

but the message is limited to a single line and the notification doesn't pause and is lost in a lot of other messages at the startup console.

The revised commit below, covers the case where plymouth is not available.

Integrate the option to use an OverlayFS as the root filesystem into the 90dmsquash-live module for testing purposes. The rd.live.overlay.overlayfs option allows one to request an OverlayFS overlay. If a persistent overlay is detected at the standard LiveOS path, the overlay & type detected will be used. Tested primarily with transient, in-RAM overlay boots on vfat- formatted Live USB devices, with persistent overlay directories on ext4-formatted Live USB devices, and with embedded, persistent overlay directories on vfat-formatted devices. (Persistent overlay directories on a vfat-formatted device must be in an embedded filesystem that supports the creation of trusted.* extended attributes, and must provide valid d_type in readdir responses.) The rd.live.overlay.readonly option, which allows a persistent overlayfs to be mounted read only through a higher level transient overlay directory, has been implemented through the multiple lower layers feature of OverlayFS. The default transient DM overlay size has been adjusted up to 32 GiB. This change supports comparison of transient Device-mapper vs. transient OverlayFS overlay performance. A transient DM overlay is a sparse file in memory, so this setting does not consume more RAM for legacy applications. It does permit a user to use all of the available root filesystem storage, and fails gently when it is consumed, as the available free root filesystem storage on a typical LiveOS build is only a few GiB. Thus, when booted on other- than-small RAM systems, the transient DM overlay should not overflow. OverlayFS offers the potential to use all of the available free RAM or all of the available free disc storage (on non-vfat-devices) in its overlay, even beyond the root filesystem available space, because the OverlayFS root filesystem is a union of directories on two different partitions. This patch also cleans up some message spew at shutdown, shortens the execution path in a couple of places, and uses persistent DM targets where required. Documentation is updated for these changes.

Provide a more prominent alert to the user if an overlay is missing or the overlay module is not available and a temporary overlay will be provided. This, to avoid losing data intended to persist.

Support the rd.live.overlay.readonly option for writable images, such as rd.live.overlay=none and rd.writable.fsimg.

Conan-Kudo · 2017-03-22T13:40:50Z

@haraldh It looks like the current test failures are not related to this PR, and the latest test run in master appears to be broken on the same tests.

At this point, could you consider it acceptable to merge so that it could be available in Fedora 26? @FGrose has prepared changes for my project (livecd-tools) to be able to leverage overlayfs for live media, and I'd like to pull that in and make it available for livecd-tools v26.

haraldh · 2017-03-30T08:55:53Z

tested locally and pushed manually

FGrose · 2017-04-09T01:27:34Z

See pull request #217 for a fix needed for newer mount utilities on Fedora 26.

FGrose mentioned this pull request Dec 20, 2015

Overlayfs instead of device-mapper #104

Closed

FGrose force-pushed the overlayfs branch 3 times, most recently from ce9fe44 to 52fb5cf Compare January 8, 2017 05:15

FGrose force-pushed the overlayfs branch from 88845c8 to 245881c Compare January 16, 2017 22:10

FGrose force-pushed the overlayfs branch from 245881c to 11435c4 Compare January 20, 2017 04:30

FGrose force-pushed the overlayfs branch 2 times, most recently from e818087 to ea30a6d Compare January 21, 2017 07:43

FGrose force-pushed the overlayfs branch 5 times, most recently from d04c836 to 37d150c Compare January 23, 2017 21:00

FGrose force-pushed the overlayfs branch 5 times, most recently from 4a4a876 to 572a6e2 Compare February 1, 2017 05:08

FGrose force-pushed the overlayfs branch from 572a6e2 to 06b6695 Compare February 2, 2017 20:19

FGrose force-pushed the overlayfs branch from 06b6695 to 4e2359c Compare March 9, 2017 18:01

FGrose force-pushed the overlayfs branch 2 times, most recently from 6a60a4c to d906d66 Compare March 10, 2017 01:14

FGrose added 2 commits March 11, 2017 00:42

dmsquash-live-root: Provide enhanced feedback on missing overlay.

f1b65e9

Provide a more prominent alert to the user if an overlay is missing or the overlay module is not available and a temporary overlay will be provided. This, to avoid losing data intended to persist.

FGrose force-pushed the overlayfs branch from d906d66 to f1b65e9 Compare March 11, 2017 05:44

dmsquash-live-root: Extend rd.live.overlay.readonly support

3c6337f

Support the rd.live.overlay.readonly option for writable images, such as rd.live.overlay=none and rd.writable.fsimg.

FGrose force-pushed the overlayfs branch from b346fa2 to 3c6337f Compare March 12, 2017 00:15

haraldh merged commit 3c6337f into dracutdevs:master Mar 30, 2017

Conan-Kudo mentioned this pull request Mar 30, 2017

dmsquash-live-root: Use persistent DM targets where needed. #190

Closed

FGrose deleted the overlayfs branch March 30, 2017 17:47

FGrose mentioned this pull request Aug 24, 2017

Refactor live iso layout to be able to use dracut OSInside/kiwi#470

Closed

FGrose mentioned this pull request Feb 27, 2023

New 90overlayfs module does not set up the overlay in three cases where it happened before (breaks Fedora/RHEL installer image boot) #2232

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable the use of the OverlayFS for the LiveOS root filesystem. #107

Enable the use of the OverlayFS for the LiveOS root filesystem. #107

FGrose commented Dec 20, 2015 •

edited

Loading

FGrose commented Dec 20, 2015

bcl commented Dec 22, 2015

FGrose commented Dec 22, 2015

centos-ci commented Oct 20, 2016

centos-ci commented Oct 20, 2016

haraldh commented Oct 20, 2016

Conan-Kudo commented Jan 4, 2017

centos-ci commented Jan 4, 2017

haraldh commented Jan 16, 2017

haraldh commented Jan 16, 2017

FGrose commented Jan 16, 2017 •

edited

Loading

FGrose commented Jan 20, 2017

haraldh commented Jan 20, 2017

haraldh commented Jan 20, 2017

haraldh commented Jan 20, 2017

FGrose commented Jan 21, 2017 •

edited

Loading

FGrose commented Jan 23, 2017 •

edited

Loading

haraldh commented Jan 25, 2017

Conan-Kudo commented Mar 8, 2017

haraldh commented Mar 8, 2017

FGrose commented Mar 9, 2017 •

edited

Loading

Conan-Kudo commented Mar 22, 2017

haraldh commented Mar 30, 2017

FGrose commented Apr 9, 2017

Enable the use of the OverlayFS for the LiveOS root filesystem. #107

Enable the use of the OverlayFS for the LiveOS root filesystem. #107

Conversation

FGrose commented Dec 20, 2015 • edited Loading

FGrose commented Dec 20, 2015

bcl commented Dec 22, 2015

FGrose commented Dec 22, 2015

centos-ci commented Oct 20, 2016

centos-ci commented Oct 20, 2016

haraldh commented Oct 20, 2016

Conan-Kudo commented Jan 4, 2017

centos-ci commented Jan 4, 2017

haraldh commented Jan 16, 2017

haraldh commented Jan 16, 2017

FGrose commented Jan 16, 2017 • edited Loading

FGrose commented Jan 20, 2017

haraldh commented Jan 20, 2017

haraldh commented Jan 20, 2017

haraldh commented Jan 20, 2017

FGrose commented Jan 21, 2017 • edited Loading

FGrose commented Jan 23, 2017 • edited Loading

haraldh commented Jan 25, 2017

Conan-Kudo commented Mar 8, 2017

haraldh commented Mar 8, 2017

FGrose commented Mar 9, 2017 • edited Loading

Conan-Kudo commented Mar 22, 2017

haraldh commented Mar 30, 2017

FGrose commented Apr 9, 2017

FGrose commented Dec 20, 2015 •

edited

Loading

FGrose commented Jan 16, 2017 •

edited

Loading

FGrose commented Jan 21, 2017 •

edited

Loading

FGrose commented Jan 23, 2017 •

edited

Loading

FGrose commented Mar 9, 2017 •

edited

Loading