Parallelize several operations during zpool import #11470

asomers · 2021-01-16T04:28:56Z

Motivation and Context

zpool import is too slow for large pools on spinning disks. It can sometimes take over an hour. Most of that time is occupied by simply waiting for serialized disk I/O. The worst offender is metaslab_init, which must be repeated hundreds of times for each top-level vdev.

Description

This PR removes the worst bottlenecks during the import process:

Parallelizes vdev_load by using a taskqueue to load each top-level vdev.
Reads all four disk labels concurrently in vdev_label_read_config, similarly to what's already done in vdev_geom_read_config.
Parallelizes vdev_validate by using a taskqueue for all vdevs.

How Has This Been Tested?

Performance tested using various large pools on FreeBSD 13. Regression tested using the FreeBSD ZFS test suite. Shortens the zpool import time by about 6x for large pools. When combined by the changes in PRs #11469 and #11467 , shortens the import time by about 8x.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Performance enhancement (non-breaking change which improves efficiency)
Code cleanup (non-breaking change which makes code smaller or more readable)
Breaking change (fix or feature that would cause existing functionality to change)
Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
Documentation (a change to man pages or other documentation)

Checklist:

My code follows the OpenZFS code style requirements.
I have updated the documentation accordingly.
I have read the contributing document.
I have added tests to cover my changes.
I have run the ZFS Test Suite with this change applied.
All commit messages are properly formatted and contain Signed-off-by.

metaslab_init is the slowest part of importing a mature pool, and it must be repeated hundreds of times for each top-level vdev. But its speed is dominated by a few serialized disk accesses. That can lead to import times of > 1 hour for pools with many top-level vdevs on spinny disks. Speed up the import by using a taskqueue to parallelize vdev_load across all top-level vdevs. openzfs/zfs#11470 Sponsored by: Axcient

This is similar to what we already do in vdev_geom_read_config. openzfs/zfs#11470 Sponsored by: Axcient

The runtime of vdev_validate is dominated by the disk accesses in vdev_label_read_config. Speed it up by validating all vdevs in parallel. openzfs/zfs#11470 Sponsored by: Axcient

behlendorf · 2021-01-20T18:21:07Z

Great idea! But just in case you hadn't noticed this does appear to introduce a locking problem with the spa config lock.

http://build.zfsonlinux.org/builders/CentOS%207%20x86_64%20%28TEST%29/builds/14599/steps/shell_4/logs/console

asomers · 2021-01-20T18:33:38Z

Great idea! But just in case you hadn't noticed this does appear to introduce a locking problem with the spa config lock.

http://build.zfsonlinux.org/builders/CentOS%207%20x86_64%20%28TEST%29/builds/14599/steps/shell_4/logs/console

Yep. I didn't notice that at first because assertions were accidentally disabled in my build. I'm testing a fix now. AFAICT it's safe for multiple threads to call vdev_validate simultaneously, because each only access's one struct vdev. It doesn't try to modify the parents or children.

asomers · 2021-01-25T19:15:17Z

Of the test cases that failed, none of them failed on more than one run. So I think they're all intermittent. They are:
zfs_destroy_dev_removal_condense
l2arc/persist_l2arc_006_pos
cli_root/zfs_send/zfs_send-b
cli_root/zfs_send/zfs_send_003_pos
cli_root/zfs_send/zfs_send_005_pos
cli_root/zfs_send/zfs_send_encrypted
cli_root/zfs_send/zfs_send_raw
pyzfs/pyzfs_unittest
redacted_send/redacted_negative
removal/removal_with_send

behlendorf

Nice improvement! I agree the failures look unrelated but let's be sure. If you rebase this on the the latest master and force update the PR you should be a clean run including on FreeBSD HEAD.

asomers · 2021-01-25T22:32:21Z

Here we go!

behlendorf · 2021-01-25T23:10:29Z

@asomers thanks. Since this is all related any objections to us squashing this in to a single commit when merged. Keeping it split up is fine with me too as long as we squash the top two bug fixes f1221c0 in f11a497 the relevant previous commit.

asomers · 2021-01-25T23:12:47Z

Yes, I think it should be split into three commits. But the final commit actually needs to be squashed to the first, not the third. Would you like me to do it now, or wait until CI finishes?

behlendorf · 2021-01-25T23:18:36Z

@asomers it'd be great if you could sort it out now so each of the commits really can stand by itself.

metaslab_init is the slowest part of importing a mature pool, and it must be repeated hundreds of times for each top-level vdev. But its speed is dominated by a few serialized disk accesses. That can lead to import times of > 1 hour for pools with many top-level vdevs on spinny disks. Speed up the import by using a taskqueue to parallelize vdev_load across all top-level vdevs. This also requires adding mutex protection to metaslab_class_t.mc_historgram. The mc_histogram fields were unprotected when that code was first written in "Illumos 4976-4984 - metaslab improvements" (OpenZFS f3a7f66). The lock wasn't added until 3dfb57a, though it's unclear exactly which fields it's supposed to protect. In any case, it wasn't until vdev_load was parallelized that any code attempted concurrent access to those fields. Sponsored by: Axcient Signed-off-by: Alan Somers <asomers@gmail.com>

This is similar to what we already do in vdev_geom_read_config. Sponsored by: Axcient Signed-off-by: Alan Somers <asomers@gmail.com>

The runtime of vdev_validate is dominated by the disk accesses in vdev_label_read_config. Speed it up by validating all vdevs in parallel. Sponsored by: Axcient Signed-off-by: Alan Somers <asomers@gmail.com>

This is similar to what we already do in vdev_geom_read_config. Sponsored by: Axcient Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Alan Somers <asomers@gmail.com> Closes #11470

The runtime of vdev_validate is dominated by the disk accesses in vdev_label_read_config. Speed it up by validating all vdevs in parallel using a taskq. Sponsored by: Axcient Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Alan Somers <asomers@gmail.com> Closes #11470

metaslab_init is the slowest part of importing a mature pool, and it must be repeated hundreds of times for each top-level vdev. But its speed is dominated by a few serialized disk accesses. That can lead to import times of > 1 hour for pools with many top-level vdevs on spinny disks. Speed up the import by using a taskqueue to parallelize vdev_load across all top-level vdevs. This also requires adding mutex protection to metaslab_class_t.mc_historgram. The mc_histogram fields were unprotected when that code was first written in "Illumos 4976-4984 - metaslab improvements" (OpenZFS f3a7f66). The lock wasn't added until 3dfb57a, though it's unclear exactly which fields it's supposed to protect. In any case, it wasn't until vdev_load was parallelized that any code attempted concurrent access to those fields. Sponsored by: Axcient Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Alan Somers <asomers@gmail.com> Closes openzfs#11470

This is similar to what we already do in vdev_geom_read_config. Sponsored by: Axcient Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Alan Somers <asomers@gmail.com> Closes openzfs#11470

The runtime of vdev_validate is dominated by the disk accesses in vdev_label_read_config. Speed it up by validating all vdevs in parallel using a taskq. Sponsored by: Axcient Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Alan Somers <asomers@gmail.com> Closes openzfs#11470

metaslab_init is the slowest part of importing a mature pool, and it must be repeated hundreds of times for each top-level vdev. But its speed is dominated by a few serialized disk accesses. That can lead to import times of > 1 hour for pools with many top-level vdevs on spinny disks. Speed up the import by using a taskqueue to parallelize vdev_load across all top-level vdevs. This also requires adding mutex protection to metaslab_class_t.mc_historgram. The mc_histogram fields were unprotected when that code was first written in "Illumos 4976-4984 - metaslab improvements" (OpenZFS f3a7f66). The lock wasn't added until 3dfb57a, though it's unclear exactly which fields it's supposed to protect. In any case, it wasn't until vdev_load was parallelized that any code attempted concurrent access to those fields. Sponsored by: Axcient Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Alan Somers <asomers@gmail.com> Closes openzfs#11470

This is similar to what we already do in vdev_geom_read_config. Sponsored by: Axcient Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Alan Somers <asomers@gmail.com> Closes openzfs#11470

The runtime of vdev_validate is dominated by the disk accesses in vdev_label_read_config. Speed it up by validating all vdevs in parallel using a taskq. Sponsored by: Axcient Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Alan Somers <asomers@gmail.com> Closes openzfs#11470

asomers added a commit to asomers/freebsd-src that referenced this pull request Jan 19, 2021

zfs: read all disk labels concurrently in vdev_label_read_config

1053384

This is similar to what we already do in vdev_geom_read_config. openzfs/zfs#11470 Sponsored by: Axcient

behlendorf added the Status: Code Review Needed Ready for review and testing label Jan 20, 2021

asomers force-pushed the vdev_load branch from b00c309 to c8ba864 Compare January 22, 2021 22:44

behlendorf approved these changes Jan 25, 2021

View reviewed changes

asomers force-pushed the vdev_load branch from c8ba864 to f11a497 Compare January 25, 2021 22:32

behlendorf requested a review from sdimitro January 25, 2021 22:32

asomers added 3 commits January 25, 2021 16:22

zfs: read all disk labels concurrently in vdev_label_read_config

58ece9a

This is similar to what we already do in vdev_geom_read_config. Sponsored by: Axcient Signed-off-by: Alan Somers <asomers@gmail.com>

ZFS: Use a taskqueue to parallelize vdev_validate

2bc9bd5

The runtime of vdev_validate is dominated by the disk accesses in vdev_label_read_config. Speed it up by validating all vdevs in parallel. Sponsored by: Axcient Signed-off-by: Alan Somers <asomers@gmail.com>

asomers force-pushed the vdev_load branch from f11a497 to 2bc9bd5 Compare January 25, 2021 23:24

behlendorf added Status: Accepted Ready to integrate (reviewed, tested) and removed Status: Code Review Needed Ready for review and testing labels Jan 26, 2021

behlendorf closed this in a0e0199 Jan 27, 2021

asomers deleted the vdev_load branch January 27, 2021 16:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parallelize several operations during zpool import #11470

Parallelize several operations during zpool import #11470

asomers commented Jan 16, 2021

behlendorf commented Jan 20, 2021 •

edited

Loading

asomers commented Jan 20, 2021

asomers commented Jan 25, 2021

behlendorf left a comment

asomers commented Jan 25, 2021

behlendorf commented Jan 25, 2021

asomers commented Jan 25, 2021

behlendorf commented Jan 25, 2021

Parallelize several operations during zpool import #11470

Parallelize several operations during zpool import #11470

Conversation

asomers commented Jan 16, 2021

Motivation and Context

Description

How Has This Been Tested?

Types of changes

Checklist:

behlendorf commented Jan 20, 2021 • edited Loading

asomers commented Jan 20, 2021

asomers commented Jan 25, 2021

behlendorf left a comment

Choose a reason for hiding this comment

asomers commented Jan 25, 2021

behlendorf commented Jan 25, 2021

asomers commented Jan 25, 2021

behlendorf commented Jan 25, 2021

behlendorf commented Jan 20, 2021 •

edited

Loading