Overlayfs instead of device-mapper #104
Comments
|
Here is something that is beginning to work for me: Now of course it would be nice if something like this would officially be integrated or even made the default (not in the hackish way I used) after some thorough testing. In my experience, the resulting live system is way more stable and usable than without my change. |
|
thanks @probonopd it looks promising; please continue push till it can become a default in next fedora live and I would like a test |
|
@FGrose : what do you think? |
|
I have been running this for quite some time now and I can say that the Fedora 23 live system now is much more stable than before. It is my main OS right now, which simply wasn't possible before. |
|
Well, there is much misunderstanding of Device-mapper snapshot behavior in LiveOS images, but the OverlayFS does have the potential to avoid some disadvantages of currently configured LiveOS images. If Fedora LiveOS images were built with large root filesystems and defaulted to use large transient overlays, most of the complaints about overflowing overlays invalidating the system might be eliminated. (The new Persistent with Overflow dm snapshot store type, in kernel 4.3+, will also make persistent LiveUSB systems more robust.) Since transient dm overlay files are sparse files on an ext4 tmpfs, and since the LiveOS root filesystem is sparsed before compression, carrying the extra reserved partition table space on the root filesystem would be a good compromise in exchange for the magnitude of frustration that users have when their large RAM stores cannot be used. (By the way, the storage space of files added to a dm snapshot can be reused by deleting the file. Only files deleted from the base, read-only, root filesystem cannot have their space reused without merging the overlay.) In any case, I have prepared a test commit that enables the use of the OverlayFS for Fedora LiveOS root filesystems, see pull request #107. It would be good to see some comparative performance testing for the two types of overlays. Before submitting a pull request, more testing with other boot configurations, such as persistent overlays on vfat, ext4, btrfs, and other filesystems is needed. I'm hoping others can jump in here. |
|
Thank you very much, this is awesome! So far I can say that my Fedora 23 live ISO runs very well from a fat32 USB drive booted with GRUB2 loop-mount using overlayfs and |
|
I've force pushed some updates to the OverlayFS commit, see pull request #107. The patch now supports and has been tested with persistent overlays on vfat and ext4-formatted devices. I'm still looking for a way to allow the initramfs to sense and use a persistent overlay of either OverlayFS or DM type on the boot device with and without dracut-systemd. |
|
Does it possible to use live squashfs image on buttom and tmpfs in upper layer via overlayfs ? |
|
Yes, by requesting |
|
@FGrose i don't see mounting tmpfs in you patch, how this can works? |
|
Notice that the transient overlay directories are placed in /run/overlayfs and /run/ovlwork (at lines 150 & 151). /run is a tmpfs. See also line 67 in dmsquash-generator. |
|
@FGrose thanks! i miss that =)! |
|
See this trial version of livecd-iso-to-disk https://github.com/FGrose/livecd-tools/blob/litd/tools/livecd-iso-to-disk, which allows one to optionally configure a Live USB with an OverlayFS. To use the new script on an existing LiveCD/.iso, use the |
|
2015-12-03 13:17 GMT+03:00 Frederick Grose notifications@github.com:
THanks! Vasiliy Tolstov, |
|
I've force pushed an updated to the OverlayFS commit, see pull request #107. This version implements rd.live.overlay.readonly for OverlayFS using the multiple lower layer mount option for OverlayFS. This allows a device with a persistent overlay to be mounted with a transient overlay stacked over the persistent overlay. This version now has all the features for OverlayFS as for Device-mapper overlays. |
|
Nice! |
|
Pull request submitted: #107 Mailing list post: http://article.gmane.org/gmane.linux.kernel.initramfs/4308 |
|
Excellent, thank you! |
|
Hello, |
|
@yopito : I suspect that what you want could be achieved with appropriate kernel command line options. See https://github.com/haraldh/dracut/blob/master/dracut.cmdline.7.asc . |
|
Here is a new version of livecd-tools/liveimage-mount that supports LiveOS devices with OverlayFS. And here is a new version of livecd-tools/editliveos.py that supports editing LiveOS images that use OverlayFS. |
|
@FGrose : thanks for your reply. |
|
Can someone please confirm that |
|
Doesn't work for me using Fedora-MATE_Compiz-Live-x86_64-Rawhide-20160628.n.0.iso; can't find the persistent overlay |
|
in FAI (Fully Automatic Installation) we use dracut an overlayfs with a ramdisk. I've created a new dracut module which puts a ramdisk on top of the nfsroot using a ramdisk. You can find it here: |
|
Further references to this issue:
Isn't it about time to dump this unreliable device-mapper stuff entirely and just use overlayfs exclusively? (Yes, I know it had been discussed 10 years ago when unionfs was still bleeding edge). Can it be done now, using overlayfs? |
|
@probonopd My understanding is that OverlayFS should now work with SELinux, as a result of Dan Walsh's work for Docker stuff. Are there any other blockers to switching to OverlayFS for live images? |
|
With pull request #107, Fedora 24 (updated) and Fedora 25 LiveOS images will boot and run with a root OverlayFS and SELinux enforcing. This constitutes a proof of concept. There remain differences and some immaturity with OverlayFS as a root filesystem. For example, below are some errors that appear on running LiveUSB with OverlayFS with a transient overlay:
booted with kernel 4.9.3-200.fc25.x86_64 on a 32 GiB USB 3.0 flash drive:
Some errors in the transcript of sudo dnf upgrade:
Upgrading : pcre-8.40-1.fc25.x86_64 17/974
libsemanage.semanage_commit_sandbox: Error while renaming /var/lib/selinux/targeted/active to /var/lib/selinux/targeted/previous. (Invalid cross-device link).
semodule: Failed!
Upgrading : selinux-policy-3.13.1-225.6.fc25.noarch 308/974
libsemanage.semanage_commit_sandbox: Error while renaming /var/lib/selinux/targeted/active to /var/lib/selinux/targeted/previous. (Invalid cross-device link).
semodule: Failed!
Upgrading : selinux-policy-targeted-3.13.1-225.6.fc25.noarch 376/974
libsemanage.semanage_commit_sandbox: Error while renaming /var/lib/selinux/targeted/active to /var/lib/selinux/targeted/previous. (Invalid cross-device link).
/usr/sbin/semodule: Failed!These are caused by a known difficiency (see https://patchwork.kernel.org/patch/9373113/). There are others (a partial report): Upgrading : NetworkManager-bluetooth-1:1.4.2-2.fc25.x86_64 377/974
Upgrading : NetworkManager-adsl-1:1.4.2-2.fc25.x86_64 378/974
/var/tmp/rpm-tmp.kYo3cn: line 1: 4020 Segmentation fault /usr/sbin/groupadd -r nm-openconnect &> /dev/null
/var/tmp/rpm-tmp.kYo3cn: line 4: 4021 Segmentation fault /usr/sbin/useradd -r -s /sbin/nologin -d / -M -c 'NetworkManager user for OpenConnect' -g nm-openconnect nm-openconnect &> /dev/null
Upgrading : NetworkManager-openconnect-1.2.4-3.fc25.x86_64 379/974
Upgrading : libnl3-cli-3.2.29-1.fc25.x86_64 463/974
warning: %post(libnl3-cli-3.2.29-1.fc25.x86_64) scriptlet failed, exit status 127
Non-fatal POSTIN scriptlet failure in rpm package libnl3-cli
Non-fatal POSTIN scriptlet failure in rpm package libnl3-cli
Upgrading : perl-Time-Local-1:1.250-1.fc25.noarch 464/974
Upgrading : lua-posix-33.3.1-3.fc25.x86_64 465/974
Upgrading : adwaita-qt4-0.97-1.fc25.x86_64 466/974
Upgrading : libphodav-2.1-1.fc25.x86_64 467/974
warning: %post(libphodav-2.1-1.fc25.x86_64) scriptlet failed, exit status 127
Non-fatal POSTIN scriptlet failure in rpm package libphodav
Non-fatal POSTIN scriptlet failure in rpm package libphodav
Upgrading : augeas-libs-1.7.0-1.fc25.x86_64 468/974
warning: %post(augeas-libs-1.7.0-1.fc25.x86_64) scriptlet failed, exit status 127
Non-fatal POSTIN scriptlet failure in rpm package augeas-libs
Non-fatal POSTIN scriptlet failure in rpm package augeas-libs
Upgrading : libev-4.24-1.fc25.x86_64 469/974
warning: %post(libev-4.24-1.fc25.x86_64) scriptlet failed, exit status 127
Non-fatal POSTIN scriptlet failure in rpm package libev
Non-fatal POSTIN scriptlet failure in rpm package libev
Upgrading : libsss_autofs-1.14.2-2.fc25.x86_64 470/974
Upgrading : libsss_sudo-1.14.2-2.fc25.x86_64 471/974
warning: %post(libsss_sudo-1.14.2-2.fc25.x86_64) scriptlet failed, exit status 127
Non-fatal POSTIN scriptlet failure in rpm package libsss_sudo
Non-fatal POSTIN scriptlet failure in rpm package libsss_sudo
Upgrading : microcode_ctl-2:2.1-13.1.fc25.x86_64 472/974
Upgrading : npth-1.3-1.fc25.x86_64 473/974
warning: %post(npth-1.3-1.fc25.x86_64) scriptlet failed, exit status 127
Non-fatal POSTIN scriptlet failure in rpm package npth
Non-fatal POSTIN scriptlet failure in rpm package npth
Upgrading : pigz-2.3.4-1.fc25.x86_64 474/974
Upgrading : slang-2.3.0-7.fc25.x86_64 475/974
warning: %post(slang-2.3.0-7.fc25.x86_64) scriptlet failed, exit status 127
Non-fatal POSTIN scriptlet failure in rpm package slang
Non-fatal POSTIN scriptlet failure in rpm package slang
Upgrading : xfsprogs-4.9.0-1.fc25.x86_64 476/974
warning: %post(xfsprogs-4.9.0-1.fc25.x86_64) scriptlet failed, exit status 127
Non-fatal POSTIN scriptlet failure in rpm package xfsprogs
Non-fatal POSTIN scriptlet failure in rpm package xfsprogs
Cleanup : pcre-8.39-6.fc25.x86_64 968/974
warning: %postun(pcre-8.39-6.fc25.x86_64) scriptlet failed, exit status 127
Non-fatal POSTUN scriptlet failure in rpm package pcre
Non-fatal POSTUN scriptlet failure in rpm package pcre
Cleanup : libselinux-2.5-12.fc25.x86_64 969/974
warning: %postun(libselinux-2.5-12.fc25.x86_64) scriptlet failed, exit status 127
Non-fatal POSTUN scriptlet failure in rpm package libselinux
Non-fatal POSTUN scriptlet failure in rpm package libselinux
Cleanup : glibc-common-2.24-3.fc25.x86_64 970/974
Cleanup : glibc-all-langpacks-2.24-3.fc25.x86_64 971/974
Cleanup : glibc-2.24-3.fc25.x86_64 972/974
warning: %postun(glibc-2.24-3.fc25.x86_64) scriptlet failed, exit status 127
Non-fatal POSTUN scriptlet failure in rpm package glibc
Non-fatal POSTUN scriptlet failure in rpm package glibcAnd when a new kernel needed a new initramfs, this error occurred: dracut-install: No SOURCE argument given
Usage: dracut-install -D DESTROOTDIR [OPTION]... -a SOURCE...
or: dracut-install -D DESTROOTDIR [OPTION]... SOURCE DEST
or: dracut-install -D DESTROOTDIR [OPTION]... -m KERNELMODULE [KERNELMODULE …]
Install SOURCE to DEST in DESTROOTDIR with all needed dependencies.
KERNELMODULE can have the format:
<absolute path> with a leading /
=<kernel subdir>[/<kernel subdir>…] like '=drivers/hid'
<module name>
-D --destrootdir Install all files to DESTROOTDIR as the root
-a --all Install all SOURCE arguments to DESTROOTDIR
-o --optional If SOURCE does not exist, do not fail
-d --dir SOURCE is a directory
-l --ldd Also install shebang executables and libraries
-L --logdir <DIR> Log files, which were installed from the host to <DIR>
-R --resolvelazy Only install shebang executables and libraries
for all SOURCE files
-H --hostonly Mark all SOURCE files as hostonly
-f --fips Also install all '.SOURCE.hmac' files
--module,-m Install kernel modules, instead of files
--kerneldir Specify the kernel module directory
--firmwaredirs Specify the firmware directory search path with : separation
--silent Don't display error messages for kernel module install
-o --optional If kernel module does not exist, do not fail
-p --mod-filter-path Filter kernel modules by path regexp
-P --mod-filter-nopath Exclude kernel modules by path regexp
-s --mod-filter-symbol Filter kernel modules by symbol regexp
-S --mod-filter-nosymbol Exclude kernel modules by symbol regexp
-N --mod-filter-noname Exclude kernel modules by name regexp
-v --verbose Show more output
--debug Show debug output
--version Show package version
-h --help Show this help
dracut: FAILED: /usr/lib/dracut/dracut-install -D /var/tmp/dracut.M40ZkZ/initramfs --kerneldir /lib/modules/4.9.3-200.fc25.x86_64/ -o -m |
|
Would disabling selinux solve these? (It's the first thing I do on Fedora-type systems anyways.) |
|
With Journals for the boots and |
|
i'm always using the overlay might still have its own issues, but it resolved more issues than remaining; so could fedora liveiso change to overlay as default ? instead of device-mapper |
|
This issue is being marked as stale because it has not had any recent activity. It will be closed if no further activity occurs. If this is still an issue in the latest release of Dracut and you would like to keep it open please comment on this issue within the next 7 days. Thank you for your contributions. |
|
I loaded an updated Fedora 32, F32-WORK-x86_64-LIVE-20200804.iso, from https://dl.fedoraproject.org/pub/alt/live-respins/, onto a 64-GB LiveUSB flash disk using this command line: sudo ./livecd-iso-to-disk.sh --format --overlayfs /path/to/F32-WORK-x86_64-LIVE-20200804.iso /dev/sdcThis produces an ext4 formatted partition with a persistent OverlayFS overlay directory under /LiveOS/.
Running scriptlet: flatpak-selinux-1.6.5-1.fc32.noarch 37/122
libsemanage.semanage_commit_sandbox: Error while renaming /var/lib/selinux/targeted/active to /var/lib/selinux/targeted/previous. (Invalid cross-device link).
/usr/sbin/semodule: Failed!
For comparison, the same image was loaded a second time on the same LiveUSB disk with this command line: sudo ./livecd-iso-to-disk.sh --multi --livedir DM --overlay-size-mb 4096 /path/to/F32-WORK-x86_64-LIVE-20200804.iso /dev/sdc1When Here is the sudo df -TB 1
Filesystem Type 1B-blocks Used Available Use% Mounted on
/dev/loop0p1 iso9660 2103607296 2103607296 0 100% /run/media/fgrose/F32-WORK-x86_64-Live-20200804
/dev/sdc1 ext4 62705913856 15985045504 43505147904 27% /run/media/fgrose/San64
/dev/loop1 squashfs 1970143232 1970143232 0 100% /tmp/utxr11tc-mp-squash-w1va2hqg
/dev/mapper/live-rw ext4 6802661376 5652496384 1063596032 85% /mnt/a
/dev/loop4 squashfs 1970143232 1970143232 0 100% /tmp/gku_3jpd-mp-squash-drr35d7p
/dev/mapper/live-rw.6spkiuse ext4 6802661376 6043152384 672940032 90% /mnt/b
/dev/loop7 squashfs 1970143232 1970143232 0 100% /tmp/t6ceqxyd-mp-squash-w43z2q9s
/dev/loop8 ext4 6802661376 5652488192 1063604224 85% /tmp/t6ceqxyd-fsbase-7r1cssv0
LiveOS_rootfs overlay 62705913856 15985045504 43505147904 27% /mnt/c
|
|
It would still be good to see some comparative performance testing for the two types of overlays. Issue livecd-tools/livecd-tools#135 addresses switching the default overlay, and seems satisfied with the current options. There has also been some discussion of changing the SquashFS image layout, dracut already supports a flattened squashfs.img, |
|
Flattened squashfs.img sounds great! 👍 |
|
@johannbg what is needed in order to get this un-staled? :-) |
johannbg
added
enhancement
|
@probonopd simply responding to this issue was enough. ;) |
|
@FGrose, @probonopd what's the current status on this issue and what is left to do/missing in dracut from the original reported issue in comment 1? |
feat: ease decryption setup with TPM2 or FIDO2
|
We now have a test case for overlayfs - https://github.com/dracutdevs/dracut/blob/master/test/TEST-16-DMSQUASH/test.sh. This is the oldest issue still open for the project :-) @FGrose, @probonopd - can we close this now and as needed open more specific new issues ? |
|
This issue can be closed. An OverlayFS boot is available. Distribution builders can choose what they like. |
|
The original issue was that "dracut-based Fedora live images" crash after a while due to running out of space. Thanks for having added OverlayFS boot. Once OverlayFS is used by default on Fedora Live ISOs then this can be closed. Just curious though, why keep any other options than OverlayFS around at all? Do we have to persuade each distribution individually to switch to OverlayFS? Especially Fedora? |
I think dracut should advocate and support for choice (e.g. musl vs glibc or systemd vs openrc) and dracut should not be a vehicle to force changes downstream. Closing based on earlier comments. from @FGrose and @probonopd . |
|
For what it's worth, @iammattcoleman and I have proposed a change in Fedora to switch to OverlayFS: https://fedoraproject.org/wiki/Changes/ModernizeLiveMedia |
I have the exact same issue as described on https://lists.fedoraproject.org/pipermail/users/2015-July/463627.html and https://lists.fedoraproject.org/pipermail/devel/2015-July/212956.html
The live system crashes when installing large packages despite having 8GB or RAM in the system. Other live systems such as casper do not have this issue. As far as I can see, there is no solution for this yet.
The difference for me as a user is that I can use casper-based live systems for my actual daily work (do not crash) while I cannot do the same with dracut-based Fedora live images (always crash).
Please make it possible to overlay-mount a large(!) amount of RAM so that the live system can handle large file system operations without crashing.
Thanks for your consideration.
The text was updated successfully, but these errors were encountered: