(C) Martin Väth (martin at mvath.de)
This project is under the BSD license 2.0 (“3-clause BSD license”). SPDX-License-Identifier: BSD-3-Clause
Init and management script for mounting rewritable squashfs-compressed data
This is the successor of the squash_dir project
It is actually a ground-up rewrite of that project in perl with a highly improved control interface.
One of its aims is to be init system independent. For openrc and systemd ready-to-use init/service-files are provided; I gladly add init-files for other init systems if I receive (tested) patches.
squashmount works with systemd in “standard” setups, but due to conceptional bugs of systemd in some setups (see section “Installation”), systemd is no longer officially supported.
This is squashmount, a generic initscript, user interface and management tool for keeping directories compressed by squashfs but simultaneously keep them writable using some of (depending on the configuration and what is available):
- overlay (AKA overlayfs for linux kernels 3.18 and newer)
- overlayfs, a variant for older linux kernels, see http://git.kernel.org/?p=linux/kernel/git/mszeredi/vfs.git
- aufs, see http://aufs.sourceforge.net
- fuse-overlayfs, see https://github.com/containers/fuse-overlayfs
- unionfs-fuse, see http://podgorny.cz/moin/UnionFsFuse (unionfs-fuse-0.25 or newer is required)
- unionfs, see http://www.fsl.cs.sunysb.edu/project-unionfs.html
- funionfs, see http://bugs.gentoo.org/show_bug.cgi?id=151673
Since squashmount v15.0.0 there is also a choice between
- squashfs (the linux kernel module)
- squashfuse, the fuse module, see https://github.com/vasi/squashfuse
The idea is that, as a rule, on shutdown the data is recompressed (and the temporary modified data removed). This approach is originally due to synss' script from http://forums.gentoo.org/viewtopic-t-465367-highlight-.html In that forum thread you can also ask for help about this project.
For some mount points different rules than “resquash and delete on umount” might be desired (see the examples section below), and moreover, it might be necessary to override these rules temporarily. For such things a powerful user interface is provided.
This project can be useful for any linux distribution. Historically, the main motivations was to keep the Gentoo main repository compressed. This is still one of the most striking examples:
If the Gentoo main repository (/usr/portage or /var/db/repos/gentoo)
is compressed with squashmount (without DISTDIR which you should store
somewhere else when using this script), the required disk space is only
about 50-100 MB (depending on your compression method), instead of
200-400 MB (or much higher, the actual space requirement
depending essentially on the filesystem, e.g. how inodes are used).
Usually, also the access is much faster.
It is possible to combine squashmount with portage's
sync-type = squashdelta to mount the Gentoo repository writable.
The script requires of course that squashfs support is activated in the kernel (and supports the COMPRESSION method), that the mksquashfs tool is available, and also that some of the above mentioned unionfs-type tools is available and supported by the kernel.
Since v17.0.0/v10.0.0 , squashmount defaults to the COMPRESSION method
zstd/lz4. This method is available only in linux-4.14/3.19 or higher
or in squashfuse-0.1.101_alpha20170917/0.1.100_alpha20140523 or higher.
So take care to either use a sufficiently new kernel/squashfuse version
or to change the default!
Moreover, you need a decently new version of perl5 together with some of
its standard modules (which might need to be installed separately if your
perl5 version should be too old). Decently new perl versions should have the
TERM::ANSIColor module; you need this if you want to see nicely
colored output.
It is also strongly recommended to install the File::Which module
(although there are some fallbacks if it is not available).
If you want that the hard status line is set, also the title script from https://github.com/vaeth/runtitle (version >=2.3) is required in your path.
If you are a Gentoo user, you can just emerge squashmount from the mv overlay.
Otherwise you just have to copy bin/squashmount into /usr/bin/squashmount
or any other directory of your $PATH.
For zsh completion support also copy zsh/_squashmount into a directory of
your zsh's $fpath.
It is strongly recommended to put
alias squashmount='noglob squashmount'
into your ~/.zshrc or /etc/zsh/zshrc or /etc/zshrc,
so that things like
squashmount start *
will work in your zsh as intended without the need to quote *. (I assume that you do not use any poor shell instead of zsh.) ;)
If you use revdep-rebuild from Gentoo or similar distributions, and
if you use the default naming scheme, it is recommended to copy the content of
etc/revdep-rebuild into /etc/revdep-rebuild to cancel duplicate or obsolete
paths search of revdep-rebuild.
For openrc support copy openrc/init.d/squashmount to
/etc/init.d/squashmount and activate it in the usual way.
For systemd-support copy systemd/system/squashmount.service to your
systemd unit folder (pkg-config --variable=systemdsystemunitdir systemd,
usually /lib/systemd/system or /usr/lib/systemd/system)
and activate it in the usual way (or e.g. copy into /etc/systemd/system)
If you use systemd be sure to compile the mount binary
with systemd support
(this should be the case in most distributions providing systemd; in Gentoo
this means to enable USE=systemd for the util-linux package. If you compile
util-linux manually, make sure to pass --with-systemd to ./configure).
In this case, systemd will probably work for you in “standard” setups.
With systemd-219 (or newer?) and some unusual setups like --make-shared
on some partitions, it can happen nevertheless that mount appears to work,
but actually nothing is mounted if systemd is in use. This is related with
the fact that systemd tries to control all mounts instead of letting the
kernel do it alone. Of course, this breaks tools like squashmount
completely.
Bug systemd upstream about such problems, but not me: I am not planning
to add hacks to fix the breakage introduced by some ill-conceived systemd
concepts.
For systemd, you should set an appropriate timeout: There is no general
rule how long compression can take “maximally”, so the timeout is set to
infinity, by default. It is strongly recommended to set this to a realistic
value for you system and setting by giving a (generous) upper estimate for
your needs by copying the file etc/system/squashmount.service.d/timeout.conf
to /etc/systemd/squashmount/service.d/timeout.conf and editing appropriately.
If you copied the main script not to /usr/bin/squashmount, you should
put into the same directory a file with appropriate modified paths.
For instance, if you copied the main script to /sbin/squashmount then
create /etc/systemd/system/squashmount.service.d/exec.conf with the content
[Service]
ExecStart=
ExecStart=/sbin/squashmount start
ExecStop=
ExecStop=/sbin/squashmount -f --lsof=0 stop
Also copy tmpfiles.d/squashmount.conf to /usr/lib/tmpfiles.d, although this
is not absolutely necessary (squashmount will create the corresponding
directories anyway).
If you use an init-system which does not mount /run as a ramdisk,
you should cleanup /run/squashmount on every fresh start before
calling squashmount start.
Depending on your init-system, a way to achieve this might be to change the
first letter in the crucial line in /usr/lib/tmpfiles.d/squashmount.conf
from d to D! and to make sure that the processing of /usr/lib/tmpfiles.d
takes place before calling squashmount start.
(Since accidental cleaning can have very inconvenient consequences, and
currently only systemd supports the D! syntax, d is the default.)
See section Emergency Case what to do if /run/squashmount
is removed accidentally anyway.
If you use find_cruft, you might want to copy the content of
lib/find_cruft to /usr/lib/find_cruft or /etc and adapt it to your needs.
If you plan to use portage's sync-type = squashdelta, you might want to copy
the content of etc/portage/repo.postsync.d to /etc/portage/repo.postsync.d
Note that the hook-file in this directory treats the mount point “gentoo”
specially! See the example configuration in etc/squashmount.pl how to
setup an appropriate mount point “gentoo” for this setting.
In all cases you have to copy lib/squashmount.pl to /etc/squashmount.pl
and adapt it to your need! This is an essential point of squashmount,
and it is impossible to use squashmount without setting up the configuration.
You can optionally also copy lib/squashmount.pl to /lib/squashmount.pl or
/usr/lib/squashmount.pl to provide a system-wide example config.
Alternatively, you can also modify that file to use it as a fallback if
/etc/squashmount.pl is not readable.
Essentially, the init-system (or you) has to call
squashmount start
on start and
squashmount -f --lsof=0 stop
on shutdown (at a time when the local filesystems are already/still mounted). The provided installation files for systemd and openrc do just this.
This will cause all configured mount points to be mounted/umounted correspondingly. When umounting, by default the modified data is recompressed into the squash-files (but this can be customized).
The configuration of the mount points happens in the file /etc/squashmount.pl This is a perl file, so you can use perl code in this file to source other files at your discretion.
The provided example configuration file etc/squashmount.pl is rather realistic if you are a Gentoo user: It provides the following mount points
- (a) guest: A guest user's home directory /home/guest
- (b) tex: The installed files from texlive /usr/share/texmf-dist
- (c1) portage: The Gentoo repository /usr/portage
- (c2) gentoo: The Gentoo repository when using sync-type = squashdelta
- (d) db: The Gentoo database of installed packages /var/db
Further mount points are in the example config-file but not listed here.
For all the mount points it is reasonable to use squashmount with them for different reasons:
-
(a) The guest-user should be able to modify data in /home/guest, but its changes should usually be forgotten. (Sometimes you will not want to forget these changes, e.g. when you want to update the “default” home directory which the user sees; see below how to do this).
-
(b) The tex directory is huge, and it saves considerable space to keep it compressed on disk.
-
(c1) Keeping the Gentoo repository compressed does not only save an enormous amount of disk space but actually also speeds up portage considerable, because the disk access is faster on a single (squashed) file. Moreover, after changes with eix --sync you might want to compare the new files with previous versions in the squashed file which you can still access when you use squashmount.
-
(c2) When you prefer to use portage's
sync-type = squashdelta, you already have the advantage of a compressed portage tree. However, you might want to use squashmount to make this portage tree writable (e.g. in order to temporarily fix a broken Manifest file locally). -
(d) The db directory is short but its data is very sensible and the number files is huge: Keeping it in a compressed file gives a lot of disk space and speedup, and it makes sense to keep a compressed backup of the last mounted version.
In these examples, additional features of squashmount are used in the example configuration:
-
For (a), it is obviously necessary to use a different treatment: Normally, the squash-file should not be generated, and the temporary data should be removed. A similar remark holds for (c2).
-
For (d), squashmount will keep a backup even of the squash-file for the db directory.
-
squashing of (c1) and (d) should happen only when a certain threshold of changes to the data is reached. The modified data will survive the reboot even if it is not resquashed, but it takes more diskspace of course, and there is no readonly version of the corresponding files.
-
No resquash of the tex directory when only certain files (like the automatically generated
ls-Rfile) were updated: In fact, if the only changes made to the directoy are in these files (it is optionally also checked that their content is not changed), the directory will be cleared when umounting/remounting/rebooting.
You can also call squashmount at runtime to resquash or clean certain directories manually or to change states for the above “default” actions on future umounts. For instance, if you changed the skeleton of the guest user in (a) you can call
squashmount --no-kill remount guest
to force immediate regeneration of the squashed file from the directory. Moreover, you can call squashmount to temporarily change the state of mounted directories. For example, if you want to change temporarily that the guest user's data is saved on remount, use
squashmount --no-kill set guest
A changed state will remain active until reset, restart, or stop is executed. Another example: If a lot of data would be resquashed at the next umount, but you want to reboot urgently, just call
squashmount --no-squash set
before rebooting. Conversely, if you want to squash the portage mount point despite its threshold is not reached (e.g. because you plan to make an experimental change which you plan to undo later), you can call
squashmount --squash restart portage
If afterwards you made your experimental change and want to undo it, call
squashmount --kill restart portage
If your changes take a longer time and you want to make sure that you do not forget to call the above command, you can temporarily change the state:
squashmount --kill set portage
Normally, this setting will survive a remount. In order to reset to the
original setting after remounting, use option -r with remount:
squashmount -r remount portage
This is the same as calling
-
squashmount remount portage squashmount reset portage
The above examples should perhaps be enough to give you an impression how to use squashmount. To get an exact description of the user interface and of the config file format just execute:
squashmount man
If you accidentally removed or corrupted /run/squashmount, e.g. due to a bug
in squashmount itself or in its configuration or if the init-system was
misconfigured and removed /run/squashmount after calling squashmount start,
you should try to umount unconditionally.
You can instruct squashmount to do this with
squashmount -fI umount
or (if you do not want resquashing in such a situation):
squashmount -nfI umount
This will work reliably unless you used temporary directories in your setup. It will even work with temporary directories if their name is still stored in /run/squashmount, i.e. if the information there was not lost completely.
It is in general rather safe to squash a directory, even a rather vital one:
Even if e.g. you boot from a kernel which has no support for some of
overlay|overlayfs|aufs|unionfs-fuse|unionfs |funionfs
to make the directory writable, squashmount will mount it at least as
read-only (using mount --bind if necessary).
Moreover, if everything goes wrong you can still use the unsquashfs tool
to unpack the directory manually.
Probably the only danger in packing “strange” directories are special files,
hard links (this information will usually get lost), or special devices
which are perhaps not supported by the used tools.
Since version 3.0, unless configured otherwise, squashmount will attempt to modprobe the modules
- squashfs
- aufs
- fuse
- overlay
- overlayfs
- unionfs
when they are required; optionally/alternatively, also the corresponding
kernel option can be checked in /sys/module or /proc/config.gz;
also the existence of required binaries can be checked before actually
the mounting is attempted.
It depends on your setting whether this is done and/or whether in case
of failure the corresponding tool is skipped tacitly without attempting
to mount.
If no tool mounts successfully, it is attempted to use mount --bind to get
the directory at least readonly on the expected place, so even in
this bad situation (which probably only happens if you boot from an
experimental kernel or a brand new kernel without corresponding support)
you can still access the directory read-only. Hence, also rather vital
directories can be compressed as long as it is not vital to write to them
(and as long as the relevant programs for mounting etc. are not contained
within these directories, of course).
If you rely only on the tool squashfuse and either unionfs-fuse or funionfs
(which are all based on the fuse userspace file system), then
squashmount can also be used with user permissions.
To support selecting only these tools and appropriate files and dirs from
the calling user, the option --user is supported since squashmount v15.0.0.
For details, call squashmount man and look for the option --user.
It is recommended to use a new version than 4.3 of squashfs-tools which
has the -quiet option. Cf. the description of --mksquash-verbose
and the $mksquash_verbose variable in ./squashmount man for details.
As a Gentoo user you can install such a recent version from the mv overlay.
However, this recommendation is only an eye-candy: squashmount will work
without any problems also with an unpatched version of squashfs-tools;
just the display when squashing will not be so nice.