Support for io_uring API?

[Harvie]

I've noticed this new io_uring buzz going on, as it was released in Linux 5.1. They claim "up to two times performance gain on CEPH". It should improve multiqueue scalability for NVMe drives. Also the new API is supposed to be easier to use than currently used AIO API. So naturaly as enthusiastic ZFS user, i can't do better but ask what are the consequences for ZFS on Linux.

Is it likely that we will get some performance boost from this?
Is this hard to implement at ZoL side?

• https://kernelnewbies.org/Linux_5.1#High-performance_asynchronous_I.2FO_with_io_uring
• https://lwn.net/Articles/776703/

[behlendorf]

This is definitely an interface we'll want to investigate supporting.

[Harvie]

Update:
https://www.phoronix.com/scan.php?page=news_item&px=IO_uring-755x-Performance-Fix
https://www.phoronix.com/scan.php?page=news_item&px=Linux-io_uring-Fast-Efficient

[JonKohler]

Stumbled upon this today, has anyone given this a poke?

A few interesting thoughts - now that 5.4 long term is out and cooking, more people may feel comfortable getting on to a long term version of linux that has io_uring. Secondarily, Ubuntu 20.04 said that they're standardizing around 5.4.

The "buzz" @Harvie mentioned seems to still be buzzing

[Harvie]

I think i still do not fully understand how io_uring works... Initialy i was thinking that io_uring would lay between ZFS and harddrive, but now somebody told me it is gonna lay between ZFS and userspace application, so i am bit confused. Do you know more about how this actualy works?

[maxximino]

Is this actually relevant for ZFS? AFAIU io_uring is an interface between userspace applications and the kernel.
ZFS data paths between userspace and the kernel that I'm aware of are:

• Filesystem IO - that's an interface managed by the kernel, and I can't spot in kernel 5.7 fs/ dir any filesystem-specific io_uring code.
  Looking deeper, https://github.com/torvalds/linux/blob/1ee08de1e234d95b5b4f866878b72fceb5372904/fs/io_uring.c#L2607 should call
  

  zfs/module/os/linux/zfs/zpl_file.c

  Lines 289 to 302 in 657ce25

  	zpl_iter_read(struct kiocb *kiocb, struct iov_iter *to)
  	{
  	ssize_t ret;
  	uio_seg_t seg = UIO_USERSPACE;
  	if (to->type & ITER_KVEC)
  	seg = UIO_SYSSPACE;
  	if (to->type & ITER_BVEC)
  	seg = UIO_BVEC;
  	ret = zpl_iter_read_common(kiocb, to->iov, to->nr_segs,
  	iov_iter_count(to), seg, to->iov_offset);
  	if (ret > 0)
  	iov_iter_advance(to, ret);
  	return (ret);
  	}

  (and it has a fallback in case we didn't implement it - so looks like doesn't expect integration work for every FS)

• Block devices/zvol IO - same as above, checked drivers/block/, no sign of anything special . Didn't follow the exact code path for block devices.

• ZFS send/receive: AFAIU that's one long-running IOCTL which dumps data directly into a file descriptor, so might not benefit from the reduced context switching of io_uring. (but this is just my speculation)

Ceph instead got huge performance gains because Ceph's ""backend"" is an userspace application which issues a lot of I/Os (see ceph/ceph#27392). Hypotethically, if ZoL was not a native kernel module but still zfs-fuse, that would gain performance.

[sistemas-sn]

Hi, how are you? any news about this?. This will be a great feature for improves zfs performance.

[recklessnl]

It's coming up to 3 years since this post - @behlendorf has there been any progress or updates with regards to io_uring?

[gigatexal]

BRTFS -- a COW fs -- has seen improvements from io_uring 🤷 https://www.phoronix.com/news/Linux-6.1-Btrfs

[KungFuJesus]

BRTFS -- a COW fs -- has seen improvements from io_uring 🤷 https://www.phoronix.com/news/Linux-6.1-Btrfs

This was an improvement to how async io was handled when coming from io_uring rather than utilizing io_uring itself.

[ghost]

OpenZFS supports io_uring, verified by the test suite: https://github.com/openzfs/zfs/blob/master/tests/zfs-tests/tests/functional/io/io_uring.ksh

[rincebrain]

I don't...think this should be closed, unless I'm grossly misunderstanding OP's goal.

That check just confirms that fio using its io_uring engine doesn't error out. That doesn't say anything about ZFS actually leveraging this interface in any way - which it does not, as of this writing.

Feel free to close this again if I'm mistaken, but "there's a ZTS test that fio using io_uring can use OpenZFS" and "OpenZFS leverages io_uring in any way, or decides it's not useful" don't seem like the same goal.

[wibed]

i could imagine zvol iops and io_uring are a good match
has anyone tested this, or any experience around it?

#11407

[Harvie]

https://www.phoronix.com/news/Linux-6.6-IOmap-Improvements

Does ZFS on Linux use iomap? There is some iomap improvement that leverages io_uring and memory folios to supposedly improve performance of filesystems which are using it.

https://lwn.net/Articles/935934/

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6016fc9162245c5b109305841f76cca59c20a273
https://lore.kernel.org/io-uring/20230720181310.71589-1-axboe@kernel.dk/