Unable to enumerate directories on RHEL 7.5 #7460

seidelma · 2018-04-18T20:43:33Z

System information

Type	Version/Name
Distribution Name	Red Hat Enterprise Linux
Distribution Version	7.5
Linux Kernel	3.10.0-862.el7
Architecture	x86_64
ZFS Version	0.7.8-1
SPL Version	0.7.8-1

Describe the problem you're observing

ZFS is unable to enumerate directories on RHEL 7.5, due to changes Red Hat has made to the kernel with 3.10.0-862. Kernel 3.10.0-693.21.1 from RHEL 7.4 does not experience this issue.

Describe how to reproduce the problem

Install RHEL 7.5
Install ZFS

# zpool create test /dev/sdb
# ls -alhtr /test
ls: reading directory /test: Not a directory
total 0

Include any warning/errors/backtraces from the system logs

This issue is caused by backported changes Red Hat made to their 3.10.0-862.el7 kernel. The getdents() system call now calls iterate_dir() (in fs/readdir.c) which also requires that the file be opened with f_mode & FMODE_KABI_ITERATE. In relevant part:

int iterate_dir(struct file *file, struct dir_context *ctx)
{
        struct inode *inode = file_inode(file);
        int res = -ENOTDIR;
        if (!file->f_op ||
            (!file->f_op->readdir && !(file->f_mode & FMODE_KABI_ITERATE)))
                goto out;
...

ZFS, for its part, #defines out the readdir() file op in module/zfs/zpl_file.c since it detects that the kernel supports iterate():

const struct file_operations zpl_dir_file_operations = {
        .llseek         = generic_file_llseek,
        .read           = generic_read_dir,
#ifdef HAVE_VFS_ITERATE_SHARED
        .iterate_shared = zpl_iterate,
#elif defined(HAVE_VFS_ITERATE)
        .iterate        = zpl_iterate,
#else
        .readdir        = zpl_readdir,
#endif
        .fsync          = zpl_fsync,
        .unlocked_ioctl = zpl_ioctl,
#ifdef CONFIG_COMPAT
        .compat_ioctl   = zpl_compat_ioctl,
#endif
};

Downgrading to RHEL's kernel-3.10.0-693.21.1.el7 package fixes the issue.

The text was updated successfully, but these errors were encountered:

behlendorf · 2018-04-18T21:06:14Z

@seidelma can you verify that falling back to .readdir works by undefining HAVE_VFS_ITERATE and HAVE_VFS_ITERATE_SHARED in zfs_config.h.

seidelma · 2018-04-18T22:06:11Z

@behlendorf Setting both #undef HAVE_VFS_ITERATE and #undef HAVE_VFS_ITERATE SHARED causes a bunch of redefinition errors in include/sys/zpl.h:

In file included from /usr/src/zfs-0.7.8/module/zfs/arc.c:281:0:
/usr/src/zfs-0.7.8/include/sys/zpl.h:135:16: error: redefinition of ‘struct dir_context’
 typedef struct dir_context {
                ^
In file included from include/linux/huge_mm.h:5:0,
                 from include/linux/mm.h:394,
                 from /usr/src/spl-0.7.8//include/sys/thread.h:29,
                 from /usr/src/spl-0.7.8//include/sys/taskq.h:35,
                 from /usr/src/spl-0.7.8//include/sys/kmem_cache.h:28,
                 from /usr/src/zfs-0.7.8/include/sys/zfs_context.h:41,
                 from /usr/src/zfs-0.7.8/include/sys/spa.h:36,
                 from /usr/src/zfs-0.7.8/module/zfs/arc.c:263:
include/linux/fs.h:1775:8: note: originally defined here

and so on.

Unsetting only one causes the preprocessor to nuke the definition of zpl_readdir() in module/zfs/zpl_file.c.

h1z1 · 2018-04-19T02:22:23Z

Is this RHEL only or CentOS as well? The rpm build is still a bit broken but as you indicated I can confirm at least CentOS 7.4 doesn't have this. zfs 0.7.8-1

seidelma · 2018-04-19T19:59:59Z

@h1z1 This is currently RHEL-only because kernel 3.10.0-862 hasn't been released on CentOS yet (though according to git.centos.org it is in the works, and the offending code in the CentOS version of fs/readdir.c is identical so it will be an issue with the CentOS-branded kernel).

I've submitted this patch, which fixes the issue, to Red Hat.
check_iterate_fileop.txt

As of RHEL 7.5 the mainline fops.iterate() method was added to the file_operations structure. This wouldn't have been a problem however in order to maintain KABI compatibility it was added to the very end of the structure. Callers intending to use this extended interface were additionally required to set the FMODE_KABI_ITERATE flag on the file struct when opening the directory. Update the ZPL to set this RHEL specific flag when required. Using this interface will mean that kmods built for RHEL 7.5 will not work on 7.4 without being rebuilt. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#7460

behlendorf · 2018-04-20T01:04:51Z

@seidelma thanks for looking in to this. I've opened #7463 with an proposed fix which will allow ZFS to use the provided RH (and CentOS) kernel. Please review and test if you have the time.

seidelma · 2018-04-20T15:51:01Z

@behlendorf Looks good with some rudimentary tests I've run. Thanks!

As of RHEL 7.5 the mainline fops.iterate() method was added to the file_operations structure. This wouldn't have been a problem however in order to maintain KABI compatibility it was added to the very end of the structure. Callers intending to use this extended interface were additionally required to set the FMODE_KABI_ITERATE flag on the file struct when opening the directory. Update the ZPL to set this RHEL specific flag when required. Using this interface will mean that kmods built for RHEL 7.5 will not work on 7.4 without being rebuilt. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#7460

truatpasteurdotfr · 2018-05-01T12:34:05Z

3.10.0-862 is released under the CR repository, and I can confirm that both 0.6.5.11 and 0.7.8 have the same issue when rebuilt under that kernel version.

ElCoyote27 · 2018-05-01T19:46:21Z

Hi,
Thanks for fixing this. Any idea when you'll be able to release updated el7.5 packages?
Thanks,
Vincent

ElCoyote27 · 2018-05-01T20:02:21Z

I've opened https://bugzilla.redhat.com/show_bug.cgi?id=1573633 to raise awareness.

As of RHEL 7.5 the mainline fops.iterate() method was added to the file_operations structure and is correctly detected by the configure script. Normally this is what we want, but in order to maintain KABI compatibility the RHEL change additionally does the following: * Requires that callers intending to use this extended interface set the FMODE_KABI_ITERATE flag on the file structure when opening the directory. * Adds the fops.iterate() method to the end of the structure, without removing fops.readdir(). This change updates the configure check to ignore the RHEL 7.5+ variant of fops.iterate() when detected. Instead fallback to the fops.readdir() interface which will be available. Finally, add the 'zpl_' prefix to the directory context wrappers to avoid colliding with the kernel provided symbols when both the fops.iterate() and fops.readdir() are provided by the kernel. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#7460

behlendorf · 2018-05-01T21:38:55Z

@ElCoyote27 we expect to have an updated point release this week which resolves the issue.

ElCoyote27 · 2018-05-01T21:44:56Z

Thanks Brian.
I'm currently rebuilding rpms of 0.7.8 with your latest patch to solve the issue for my environment.
Refs: https://github.com/zfsonlinux/zfs/commit/d177418748dc81279b080edd50cc6b4d3ed8f450.patch
Thank you very much for the quick fix.
Regards,
Vincent

As of RHEL 7.5 the mainline fops.iterate() method was added to the file_operations structure and is correctly detected by the configure script. Normally this is what we want, but in order to maintain KABI compatibility the RHEL change additionally does the following: * Requires that callers intending to use this extended interface set the FMODE_KABI_ITERATE flag on the file structure when opening the directory. * Adds the fops.iterate() method to the end of the structure, without removing fops.readdir(). This change updates the configure check to ignore the RHEL 7.5+ variant of fops.iterate() when detected. Instead fallback to the fops.readdir() interface which will be available. Finally, add the 'zpl_' prefix to the directory context wrappers to avoid colliding with the kernel provided symbols when both the fops.iterate() and fops.readdir() are provided by the kernel. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#7460

As of RHEL 7.5 the mainline fops.iterate() method was added to the file_operations structure and is correctly detected by the configure script. Normally this is what we want, but in order to maintain KABI compatibility the RHEL change additionally does the following: * Requires that callers intending to use this extended interface set the FMODE_KABI_ITERATE flag on the file structure when opening the directory. * Adds the fops.iterate() method to the end of the structure, without removing fops.readdir(). This change updates the configure check to ignore the RHEL 7.5+ variant of fops.iterate() when detected. Instead fallback to the fops.readdir() interface which will be available. Finally, add the 'zpl_' prefix to the directory context wrappers to avoid colliding with the kernel provided symbols when both the fops.iterate() and fops.readdir() are provided by the kernel. Reviewed-by: Olaf Faaland <faaland1@llnl.gov> Reviewed-by: Tony Hutter <hutter2@llnl.gov> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#7460 Closes openzfs#7463

As of RHEL 7.5 the mainline fops.iterate() method was added to the file_operations structure and is correctly detected by the configure script. Normally this is what we want, but in order to maintain KABI compatibility the RHEL change additionally does the following: * Requires that callers intending to use this extended interface set the FMODE_KABI_ITERATE flag on the file structure when opening the directory. * Adds the fops.iterate() method to the end of the structure, without removing fops.readdir(). This change updates the configure check to ignore the RHEL 7.5+ variant of fops.iterate() when detected. Instead fallback to the fops.readdir() interface which will be available. Finally, add the 'zpl_' prefix to the directory context wrappers to avoid colliding with the kernel provided symbols when both the fops.iterate() and fops.readdir() are provided by the kernel. Reviewed-by: Olaf Faaland <faaland1@llnl.gov> Reviewed-by: Tony Hutter <hutter2@llnl.gov> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes #7460 Closes #7463

behlendorf added the Type: Building Indicates an issue related to building binaries label Apr 18, 2018

behlendorf mentioned this issue Apr 20, 2018

RHEL 7.5 compat: FMODE_KABI_ITERATE #7463

Merged

13 tasks

behlendorf closed this as completed in 9464b95 May 2, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to enumerate directories on RHEL 7.5 #7460

Unable to enumerate directories on RHEL 7.5 #7460

seidelma commented Apr 18, 2018

behlendorf commented Apr 18, 2018

seidelma commented Apr 18, 2018

h1z1 commented Apr 19, 2018

seidelma commented Apr 19, 2018

behlendorf commented Apr 20, 2018

seidelma commented Apr 20, 2018

truatpasteurdotfr commented May 1, 2018

ElCoyote27 commented May 1, 2018

ElCoyote27 commented May 1, 2018

behlendorf commented May 1, 2018

ElCoyote27 commented May 1, 2018

Unable to enumerate directories on RHEL 7.5 #7460

Unable to enumerate directories on RHEL 7.5 #7460

Comments

seidelma commented Apr 18, 2018

System information

Describe the problem you're observing

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

behlendorf commented Apr 18, 2018

seidelma commented Apr 18, 2018

h1z1 commented Apr 19, 2018

seidelma commented Apr 19, 2018

behlendorf commented Apr 20, 2018

seidelma commented Apr 20, 2018

truatpasteurdotfr commented May 1, 2018

ElCoyote27 commented May 1, 2018

ElCoyote27 commented May 1, 2018

behlendorf commented May 1, 2018

ElCoyote27 commented May 1, 2018