Relatime is always turned on temporarily on ZFS root.

[crocket]

System information

Type	Version/Name
Distribution Name	Gentoo
Distribution Version	None
Linux Kernel	4.14.65-gentoo
Architecture	x86_64
ZFS Version	0.7.11-r0-gentoo
SPL Version	0.7.11-r0-gentoo

Describe the problem you're observing

I want to apply noatime to every single mount, but / is resisting my will.

$ findmnt | grep zfs
/                             os/ROOT/gentoo    zfs         rw,relatime,xattr,noacl
├─/home                       os/HOME           zfs         rw,noatime,xattr,noacl
├─/mnt/data                   user-data/data    zfs         rw,noatime,xattr,posixacl
├─/root                       os/HOME/root      zfs         rw,noatime,xattr,noacl
├─/usr/portage                os/GENTOO/portage zfs         rw,nosuid,noatime,xattr,noacl

$ cat /proc/mounts | grep zfs
os/ROOT/gentoo / zfs rw,relatime,xattr,noacl 0 0
os/HOME /home zfs rw,noatime,xattr,noacl 0 0
user-data/data /mnt/data zfs rw,noatime,xattr,posixacl 0 0
os/HOME/root /root zfs rw,noatime,xattr,noacl 0 0
os/GENTOO/portage /usr/portage zfs rw,nosuid,noatime,xattr,noacl 0 0

$ /sbin/zfs get all os/ROOT/gentoo | grep time
os/ROOT/gentoo  atime                 off                    inherited from os
os/ROOT/gentoo  relatime              on                     temporary

$ cat /etc/fstab
PARTUUID="e271ef53-7377-0545-9f0e-8467fafd1202" none swap sw 0 0
PARTUUID="d5179cfd-ea96-4183-8c0b-9ffc20a9b990" none swap sw 0 0
tmpfs /var/tmp/portage tmpfs size=14G,uid=portage,gid=portage,mode=775,noatime 0 0
tmpfs /tmp tmpfs size=8G,mode=1777,noatime 0 0
PARTUUID="7e95e77c-f08c-f34d-a2d3-9d540ff191c4" /boot btrfs rw,noatime,discard 0 0
PARTUUID="17485930-4257-41fc-bc1a-5921ea09c2a4" /boot/efi vfat rw,noatime,discard 0 0

$ /sbin/zfs list
NAME                           USED  AVAIL  REFER  MOUNTPOINT
os                            63.7G  46.7G    96K  none
os/GENTOO                     33.4G  46.7G    96K  none
os/GENTOO/portage             33.4G  46.7G  33.4G  /usr/portage
os/HOME                       20.9G  46.7G  19.8G  /home
os/HOME/root                  23.5M  46.7G  23.5M  /root
os/ROOT                       9.41G  46.7G    96K  none
os/ROOT/gentoo                9.41G  46.7G  9.41G  /
user-data                     1.23T  1.40T    96K  none
user-data/data                1.23T  1.40T  1.22T  /mnt/data

Describe how to reproduce the problem

Install root on a dataset, and set bootfs to that dataset. grub-mkconfig will take care of the rest.

[rlaager]

I can reproduce this on Ubuntu 18.04 too. Unless we have some universal fix, I think the initramfs script should work around this.

[GregorKopka]

man zpool states:

When a file system is mounted, either through mount(8) for legacy mounts or the zfs mount command for normal file systems, its mount options are set according to its properties. The correlation between properties and mount options is as follows:

PROPERTY MOUNT OPTION
atime atime/noatime
canmount auto/noauto
devices dev/nodev
exec exec/noexec
readonly ro/rw
relatime relatime/norelatime
setuid suid/nosuid
xattr xattr/noxattr

what's happening is

# zfs create tank/test -o atime=off -o canmount=noauto -o readonly=on
# mkdir /tmp/test
# mount -o zfsutil -t zfs tank/test /tmp/test/
# mount | grep test
tank/test on /tmp/test type zfs (rw,relatime,xattr,noacl)

The mount options are set according to its properties isn't happening as advertised when mounting using mount(8). Also

# umount /tmp/test/
# zfs set canmount=off tank/test
# mount -t zfs tank/test /tmp/test/
# mount | grep test
tank/test on /tmp/test type zfs (rw,relatime,xattr,noacl)

Which means also

canmount=on|off|noauto
If this property is set to off, the file system cannot be mounted, and is ignored by zfs mount -a.

as stated in man zfs isn't enforced (which it should, at least that's what I expect when reading that sentence).

[bunder2015]

FWIW, I've seen this on gentoo as well. I had hacked around it with a local.d script and I don't remember what turns relatime on.

#!/bin/bash

# Disable relatime=temporary on pool rootfs
ROOTFS=$(zpool get -o value -H bootfs)

zfs set relatime=off $ROOTFS

[beren12]

I believe the summary for your issue is: it doesn't matter. If atime is off, relatime has no meaning, because it is also off.

[ghfields]

@kpande You are right. It acts as relatime=on. Confirmed on Ubuntu 18.04 rpool exhibiting the relatime temporary issue. Of course it returned to noatime after issuing "mount -o remount -o noatime /"

[rlaager]

Basically, we need to revive #7948, support the other mount options, and extend that to the initramfs.

[quicktrick]

I confirm the issue on Void Linux. I used the workaround suggested by @bunder2015 : I added to /etc/rc.local the following command:

zfs set relatime=off $(zpool get -o value -H bootfs)

[bghira]

i submitted a fix but no one wanted it:

index 23c07af9e86..9bf80e14599 100755
--- a/contrib/dracut/90zfs/zfs-lib.sh.in
+++ b/contrib/dracut/90zfs/zfs-lib.sh.in
@@ -74,14 +74,22 @@ import_pool() {
 # mount_dataset DATASET
 #   mounts the given zfs dataset.
 mount_dataset() {
-        dataset="${1}"
+    dataset="${1}"
     mountpoint="$(zfs get -H -o value mountpoint "${dataset}")"
-
+    mountopts="strictatime"
+    ATIME_OPTION=$(zfs get -H -o value -s default,local,inherited atime ${dataset})
+    if [ "${ATIME_OPTION}" = "off" ]; then
+        mountopts="noatime"
+    fi
+    RELATIME_OPTION=$(zfs get -H -o value -s default,local,inherited relatime ${dataset})
+    if [ "${RELATIME_OPTION}" = "on" ]; then
+        mountopts="relatime"
+    fi
     # We need zfsutil for non-legacy mounts and not for legacy mounts.
     if [ "${mountpoint}" = "legacy" ] ; then
-        mount -t zfs "${dataset}" "${NEWROOT}"
+        mount -t zfs -o ${mountopts} "${dataset}" "${NEWROOT}"
     else
-        mount -o zfsutil -t zfs "${dataset}" "${NEWROOT}"
+        mount -o zfsutil -t zfs -o ${mountopts} "${dataset}" "${NEWROOT}"
     fi
 
     return $?

correction: it was viewed as a good idea, but people asked for waaaayyyy too many out-of-scope changes, so I withdrew the PR and said anyone could take the work up who wanted / needed it fixed. I've been maintaining my own local patch for this purpose.

[bghira]

@rlaager assigned it to himself and then did nothing with it, must have forgotten.

[rlaager]

That patch looks reasonable.

1. I'm not sure why the -s default,local,inherited.
2. I'm not sure how the mounting in _mount_dataset_cb() fits into this. Does that need to be modified too?
3. Theoretically, we should support other mount options. The list is: zfs list -H -o atime,relatime,devices,exec,readonly,setuid,nbmand That said, I realize that at least exec,readonly,setuid are extremely unlikely to be anything but defaults on the root fs. Though if we are mounting children (see 2 above), that could be different.
4. This is for dracut. We need support in initramfs too. I'm in a position to test that piece, so I can handle that piece if dracut is done.

This is probably what you meant by the change requests?

[bghira]

yep. i just wanted to fix one issue and everyone suggested changes that were well outside of both what i wanted to fix and what i was able to test. i've rewritten my patch for master:

diff --git a/contrib/dracut/90zfs/zfs-lib.sh.in b/contrib/dracut/90zfs/zfs-lib.sh.in
index defc0bfc8..2e09559d7 100755
--- a/contrib/dracut/90zfs/zfs-lib.sh.in
+++ b/contrib/dracut/90zfs/zfs-lib.sh.in
@@ -76,19 +76,26 @@ _mount_dataset_cb() {
 # mount_dataset DATASET
 #   mounts the given zfs dataset.
 mount_dataset() {
+    ret=0
     dataset="${1}"
     mountpoint="$(zfs get -H -o value mountpoint "${dataset}")"
-    ret=0
-
+    mountopts="strictatime"
+    ATIME_OPTION=$(zfs get -H -o value -s default,local,inherited atime ${dataset})
+    if [ "${ATIME_OPTION}" = "off" ]; then
+        mountopts="noatime"
+    fi
+    RELATIME_OPTION=$(zfs get -H -o value -s default,local,inherited relatime ${dataset})
+    if [ "${RELATIME_OPTION}" = "on" ]; then
+        mountopts="relatime"
+    fi
     # We need zfsutil for non-legacy mounts and not for legacy mounts.
     if [ "${mountpoint}" = "legacy" ] ; then
-        mount -t zfs "${dataset}" "${NEWROOT}" || ret=$?
+       mount -t zfs -o ${mountopts} "${dataset}" "${NEWROOT}" || ret=$?
     else
-        mount -o zfsutil -t zfs "${dataset}" "${NEWROOT}" || ret=$?
-
-        if [ "$ret" = "0" ]; then
-            for_relevant_root_children "${dataset}" _mount_dataset_cb || ret=$?
-        fi
+       mount -o zfsutil -t zfs -o ${mountopts} "${dataset}" "${NEWROOT}" || ret=$?
+    fi
+    if [ "$ret" = "0" ]; then
+        for_relevant_root_children "${dataset}" _mount_dataset_cb || ret=$?
     fi
 
     return ${ret}

[bghira]

it's been 3 years so I don't know why I used the -s arguments other than... something didn't work, and then it did.

[szubersk]

diff --git a/contrib/dracut/90zfs/zfs-lib.sh.in b/contrib/dracut/90zfs/zfs-lib.sh.in
index defc0bfc8..2e09559d7 100755
--- a/contrib/dracut/90zfs/zfs-lib.sh.in
+++ b/contrib/dracut/90zfs/zfs-lib.sh.in
@@ -76,19 +76,26 @@ _mount_dataset_cb() {
 # mount_dataset DATASET
 #   mounts the given zfs dataset.
 mount_dataset() {
+    ret=0
     dataset="${1}"
     mountpoint="$(zfs get -H -o value mountpoint "${dataset}")"
-    ret=0
-
+    mountopts="strictatime"

Why injecting here strictatime? It could overwrite the default mount options coming from the ZFS driver.

+    ATIME_OPTION=$(zfs get -H -o value -s default,local,inherited atime ${dataset})

Why -s in here?

+    if [ "${ATIME_OPTION}" = "off" ]; then
+        mountopts="noatime"
+    fi
+    RELATIME_OPTION=$(zfs get -H -o value -s default,local,inherited relatime ${dataset})
+    if [ "${RELATIME_OPTION}" = "on" ]; then
+        mountopts="relatime"
+    fi

noatime trumps relatime - those two ifstatements should be in a reverse order.

     # We need zfsutil for non-legacy mounts and not for legacy mounts.
     if [ "${mountpoint}" = "legacy" ] ; then
-        mount -t zfs "${dataset}" "${NEWROOT}" || ret=$?
+       mount -t zfs -o ${mountopts} "${dataset}" "${NEWROOT}" || ret=$?
     else
-        mount -o zfsutil -t zfs "${dataset}" "${NEWROOT}" || ret=$?
-
-        if [ "$ret" = "0" ]; then
-            for_relevant_root_children "${dataset}" _mount_dataset_cb || ret=$?
-        fi
+       mount -o zfsutil -t zfs -o ${mountopts} "${dataset}" "${NEWROOT}" || ret=$?
+    fi
+    if [ "$ret" = "0" ]; then
+        for_relevant_root_children "${dataset}" _mount_dataset_cb || ret=$?
     fi
 
     return ${ret}

[szubersk]

@rlaager I slept on this issue and decided to run few experiments. Looks like this issue is not strictly connected to initrd, boot process or root-on-ZFS. This is a problem in atime and relatime property handling within ZFS. The testscript was run on my laptop booted to graphical.target. Would you point the right direction to investigate this further?

System

% uname -a     
Linux laptop.local 5.14.0-2-amd64 #1 SMP Debian 5.14.9-2 (2021-10-03) x86_64 GNU/Linux
% zfs --version
zfs-2.0.6-1
zfs-kmod-2.1.1-1

Results

before mount
test/noatime-relatime   atime           off     local
test/noatime-relatime   relatime        on      local

after mount
test/noatime-relatime   atime           off     local
test/noatime-relatime   relatime        on      local

what Linux reports
test/noatime-relatime /tmp/noatime-relatime zfs rw,relatime,xattr,noacl 0 0
-----------------
before mount
test/noatime-norelatime atime           off     local
test/noatime-norelatime relatime        off     local

after mount
test/noatime-norelatime atime           off     local
test/noatime-norelatime relatime        on      temporary

what Linux reports
test/noatime-norelatime /tmp/noatime-norelatime zfs rw,relatime,xattr,noacl 0 0
-----------------
before mount
test/atime-relatime     atime           on      local
test/atime-relatime     relatime        on      local

after mount
test/atime-relatime     atime           on      local
test/atime-relatime     relatime        on      local

what Linux reports
test/atime-relatime /tmp/atime-relatime zfs rw,relatime,xattr,noacl 0 0
-----------------
before mount
test/atime-norelatime   atime           on      local
test/atime-norelatime   relatime        off     local

after mount
test/atime-norelatime   atime           on      local
test/atime-norelatime   relatime        on      temporary

what Linux reports
test/atime-norelatime /tmp/atime-norelatime zfs rw,relatime,xattr,noacl 0 0
-----------------

Test script

#!/bin/bash

set -euo pipefail

main() {
  umount noatime-relatime noatime-norelatime atime-relatime atime-norelatime &>/dev/null ||:
  zpool export -f test &>/dev/null ||:

  truncate -s 100M ./disk
  mkdir -p noatime-relatime noatime-norelatime atime-relatime atime-norelatime
  zpool create -f -O canmount=off -O mountpoint=none test $(pwd)/./disk
  zfs create -o canmount=noauto -o mountpoint=none -o atime=off -o relatime=on test/noatime-relatime
  zfs create -o canmount=noauto -o mountpoint=none -o atime=off -o relatime=off test/noatime-norelatime
  zfs create -o canmount=noauto -o mountpoint=none -o atime=on -o relatime=on test/atime-relatime
  zfs create -o canmount=noauto -o mountpoint=none -o atime=on -o relatime=off test/atime-norelatime

  for m in noatime-relatime noatime-norelatime atime-relatime atime-norelatime; do
    echo before mount
    zfs get atime,relatime -H -r test/$m
    mount -t zfs -o zfsutil test/$m $m

    echo
    echo after mount
    zfs get atime,relatime -H -r test/$m
    echo
    echo what Linux reports
    awk "/test\/$m/" /proc/mounts
    echo -----------------
  done

  echo
  echo DONE
}

main "$@"

[rlaager]

If you use the low level interface to mount, then you have to handle converting the properties into mount options yourself. That’s not a bug.

Ordinary mounting by users should be done using zfs mount.

[szubersk]

Richard, I agree that zfs mount should be used for all mounts but those with mountpoint=legacy. Our demise in here start with

zfs/contrib/initramfs/scripts/zfs

Lines 925 to 934 in 4b87c19

	# M O U N T F I L E S Y S T E M S
	# * Ideally, the root filesystem would be mounted like this:
	#
	# zpool import -R "$rootmnt" -N "$ZFS_RPOOL"
	# zfs mount -o mountpoint=/ "${ZFS_BOOTFS}"
	#
	# but the MOUNTPOINT prefix is preserved on descendent filesystem
	# after the pivot into the regular root, which later breaks things
	# like `zfs mount -a` and the /proc/self/mounts refresh.

(initramfs and dracut share their fate). That behavior makes us use mount -t zfs -o zfsutil to circumvent the mountpoint propagation.

Is it really true that we need to use mount -t zfs -o zfsutil instead of zfs mount there? If there's a way to use native ZFS solution, we should go for it. #2087 was written in 2014 and I'm not sure if all assumptions in there are still relevant in 2021. Could you, guys, come up with test cases to validate zfs mount usage in initrd scripts?

EDIT
I just run few tests and indeed, zpool import -R into zfs mount leads to wrong mountpoint property values in the imported pools after pivot_root call (pre-pivot value stays) causing further mount/remount operations to fail.

EDIT2
One the the most consistent ways of going about this issue is szubersk@200ffbf
It would enforce the zfs mount behavior for all but legacy mountpoints.

[rlaager]

@szubersk I don't really know much about all the low-level mount stuff.

[szubersk]

@rlaager Would you mind suggesting a project Member who could help evaluating this? Should I create a pull request to start the discussion?

[rlaager]

@szubersk Yeah, a PR would be a good way to go.

[quicktrick]

Is there any progress on this? Fresh installation of Void Linux, and the same problem again.

[bghira]

yeah, every time someone makes a suggested change, a few others come along to say it needs a huge number of unrelated changes. and those people are never the ones who are going to do that work. I've opened two PRs for this one and both times someone else asks for changes and we can currently blame @rlaager (or yourself, you didn't do much with the patch I proposed here)

[szubersk]

@bghira Would you mind sharing the PR you mentioned were rejected? I'm very interested in fixing this once and for all, I'm tired to rebasing local patches each time initramfs is updated.

[szubersk]

@rlaager, @bghira, @quicktrick and others in this thread please check if #13021 (the most recent master) fixed the issue for you. I hope we did not leave any loose ends.

[behlendorf]

Closing, as mentioned above this should be resolved in the master branch.