FreeBSD: Use unmapped I/O for scattered/gang ABD buffers.

[amotin]

Many FreeBSD disk drivers support "unmapped" I/O mode, when data
buffer represented not with a virtually contiguous KVA-mapped address
range, but with a list of physical memory pages. Originally it was
designed to do I/O from buffers without KVA mapping (unmapped). But
moving virtual addresses out of equation allows us to operate even
non-contiguous data buffers with one condition: all buffer discon-
tinuities must be aligned to memory page borders.

Doing I/O to capable GEOM device this patch traverses through non-
linear ABD buffers, validating the chunks borders. If the condition
is met, it supplues GEOM with the list of original physical memory
pages instead of copying the data into temporary contiguous buffer.
On capable hardware on pools with ashift=12 and default ABD chunk of
4KB it should handle all the I/O without additional memory copying.

How Has This Been Tested?

On 80-thread FreeBSD system doing ~15GB/s of 64KB ZVOLs reads profiler shows reduction of active CPU time by 34%. Doing ~8.5GB/s of 64KB ZVOLs writes shows the reduction by about 5% with 3.5% bandwidth increase.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[bwatkinson]

This is interesting. I was not not aware that FreeBSD was doing a copy of the ABD for the write. I am not an expert in FreeBSD or its internals, so I have never looked closely at the VDEV geom code before.

[amotin]

I was not not aware that FreeBSD was doing a copy of the ABD for the write.

All OS'es were copying before gang ABDs were added and specially optimized. File vdevs are still copying, despite they could use preadv()/pwritev() or their kernel equivalents.

[bwatkinson]

I was not not aware that FreeBSD was doing a copy of the ABD for the write.

All OS'es were copying before gang ABDs were added and specially optimized. File vdevs are still copying, despite they could use preadv()/pwritev() or their kernel equivalents.

Using a VDEV file, yes Linux does also copy over the ABD contents. With VDEV disks, the pages are mapped into the BIO. It seems that this patch also does that for FreeBSD as well correct?

[amotin]

With VDEV disks, the pages are mapped into the BIO. It seems that this patch also does that for FreeBSD as well correct?

Exactly. I've looked on the Linux code thinking about reducing the difference, but since on FreeBSD ABDs are always mapped to into KVA I found much easier to just reuse existing ABD iterator functions rather than reimplement traversal one more time as done for Linux.