Improve speculative prefetcher for block cloning

[amotin]

• Issue prescient prefetches for demand indirect blocks after the first one. It should be quite rare for reads/writes, but much more useful for cloning due to much bigger accesses (up to 1022 blocks, while default 32KB indirect block stores only 256 pointers). It covers the gap during the first couple accesses when we can not speculate yet, but we know what is needed right now. It reduces dbuf_hold() sync read delays in dmu_buf_hold_array_by_dnode().
• Increase maximum prefetch distance for indirect blocks from 64 to 128MB. It should cover the maximum 1022 blocks of block cloning access size in case of default 128KB recordsize used. In case of bigger recordsize the above prescient prefetch should also help. Indirects for 128MB take only 128KB (and even less with compression), so it should not cost us much.

How Has This Been Tested?

To simulate the worst possible scenario I created HDD pool with 200 files of 2048 block each, wiped their indirects from ARC and disk caches and cloned them. This patch reduced time needed to complete the cloning in half, from 9 to 4.5 seconds. Stats show that 63% of the issued prescient reads complete before demand, and other 37% while already in progress.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[behlendorf]

This makes good sense to me as a performance optimization.

[behlendorf]

@pjd can you take a look.