Docker build performance extremely slow due to lack of renameat2/overlayfs on ZFS

[dbavatar]

System information

Type	Version/Name
Distribution Name	Any
Distribution Version	Any
Linux Kernel	Any
Architecture	Any
ZFS Version	Any
SPL Version	Any

Describe the problem you're observing

This is more of a feature request than a bug. I can't find anyone actively tracking this problem, however I do know users of docker that have stopped using ZFS because of this after having stated there's something wrong with ZFS, which is unfortunate.

When using docker on a zfs dataset, the only option is docker's "zfs driver" which uses zfs dataset operations to create layered filesystems. On other filesystems such as ext4 and btrfs, the "overlay2 driver" (utilizing the kernel's overlayfs module) can be used which is extremely fast. Docker build times can go from a fraction of a second on ext4/overlay2 to several minutes on zfs/zfs.

overlay2 driver selection is currently blacklisted on zfs by docker:
https://github.com/moby/moby/blob/master/daemon/graphdriver/overlay2/overlay.go#L165

Note that btrfs added the necessary support in kernel 4.7:
https://github.com/moby/moby/blob/master/daemon/graphdriver/overlay2/overlay.go#L169

Describe how to reproduce the problem

From a performance perspective:
time the build of a Dockerfile with several layers as so:

time docker build . -t date_image

# date_image Dockerfile
FROM scratch
CMD [date]
CMD [date]
CMD [date]
CMD [date]
CMD [date]
# repeat date layer to magnify performance slowdown

From a functional perspective:

#include <errno.h>
#include <stdio.h>
#include <unistd.h>
#include <linux/fs.h>
#include <sys/syscall.h>

int main()
{
  int ret;

  ret = syscall(SYS_renameat2, 0, "/zfs/file1", 0, "/zfs/file2", RENAME_EXCHANGE);
  fprintf(stderr, "renameat2 ret=%d errno=%d\n", ret, errno);
  return ret;
}

Repeat test for RENAME_WHITEOUT flag as well.

I believe we should be able to use docker on ZFS w/overlay2 after both RENAME_EXCHANGE and RENAME_WHITEOUT are implemented in renameat2()

[cyphar]

I've been working on this and renameat2(2) flag support is not sufficient to get overlayfs-on-ZFS working. The main blocker is that currently ZFS uses d_revalidate which means overlayfs refuses to use it as an upperdir. In order to fix it (in addition to #8667) we will need to rework a fair amount of the ported dcache-interacting code so that d_revalidate is no longer needed.

[Frederick888]

I noticed that if I use ZFS as the lower layer of OverlayFS (upper, work, mount point in tmpfs), I cannot append contents to any existing files either.

Sorry for my ignorance, but may I know if this problem is also tracked by this ticket or #9414? Thanks.

[cyphar]

@Frederick888 As in O_APPEND doesn't work, or gives you an error? That sounds like a separate bug to me -- ZFS should work as a lowerdir for overlayfs and if it doesn't it's either a bug in overlayfs or ZFS.

[Frederick888]

@cyphar When using tee -a it reports Invalid argument, e.g.

$ zfs list | rg '/home'
tank/data/home     50.0G   367G     38.8G  /home
$ mkdir ~/lower

$ pwd
/tmp/test
$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            24G  308M   24G   2% /tmp
$ mkdir upper work merged

$ printf 'hello world\n' | tee ~/lower/test.txt
hello world
$ sudo mount -t overlay overlay -o lowerdir=/home/frederick/lower,upperdir=./upper,workdir=./work,redirect_dir=follow ./merged
$ cat merged/test.txt
hello world
$ printf 'hello world\n' | tee -a merged/test.txt
tee: merged/test.txt: Invalid argument
hello world

Similar when git pull, error message being e.g. error: cannot update the ref 'refs/remotes/origin/master': unable to append to '.git/logs/refs/remotes/origin/master': Invalid argument.

I also tested the same thing using ext4 as lower dir and it all worked normally.

[cyphar]

-EINVAL is definitely a weird response to get. I'll take a look -- it is possible it's related to the lack of renameat2 but that should only matter for upperdir not lowerdir...

[thoernle]

This should be closed because OverlayFS support has already been merged in #9414, right?

[galaxy001]

With zfs-dkms 2.1.9-1 and kernel 6.1.14-1-lts, I still got the error with podman:

[22424.607969] overlayfs: upper fs does not support RENAME_WHITEOUT.
[22424.608104] overlayfs: upper fs does not support file handles, falling back to index=off.
[22424.608109] overlayfs: fs on '/home/galaxy/.local/share/containers/storage/overlay/l/ZH4GPIMNPLZLZ5T2J5WAUPB77S' does not support file handles, falling back to xino=off.

[satmandu]

Overlayfs support is only in the master branch of openzfs at the moment. It should be in 2.2 when it gets released.