special device removal space accounting fixes

[ahrens]

Motivation and Context

The space in special devices is not included in spa_dspace (or
dsl_pool_adjustedsize(), or the zfs available property). Therefore
there is always at least as much free space in the normal class, as
there is allocated in the special class(es). And therefore, there is
always enough free space to remove a special device.

Description

However, the checks for free space when removing special devices did not
take this into account. This commit corrects that.

Additionally, when removing the 2nd-to-last special device, its space
would not be reallocated to the last remaining special device, because
mc_groups has already been decremented. That is also fixed.

Closes #9142

How Has This Been Tested?

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[don-brady]

LGTM

[behlendorf]

There was an unexpected ZTS failure of the alloc_class_012_pos test case on FreeBSD 12 and 13. This may be unrelated, but it is suspicious.

Tests with results other than PASS that are unexpected:
    FAIL alloc_class/alloc_class_012_pos (expected PASS)

Test: /usr/local/share/zfs/zfs-tests/tests/functional/alloc_class/alloc_class_012_pos (run as root) [00:06] [FAIL]
23:48:57.80 ASSERTION: Removing a special device from a pool succeeds.
23:48:57.82 SUCCESS: disk_setup
23:48:57.98 SUCCESS: zpool create testpool /mnt/device-0 /mnt/device-1 /mnt/device-2 special /mnt/device-3 special /mnt/device-4
23:49:00.07 SUCCESS: display_status testpool
23:49:00.17 SUCCESS: zfs create -o special_small_blocks=32K -o recordsize=32K testpool/testfs
23:49:00.40 SUCCESS: dd if=/dev/urandom of=/testpool/testfs/testfile.1 bs=1M count=25
23:49:00.84 SUCCESS: dd if=/dev/urandom of=/testpool/testfs/testfile.2 bs=1M count=50
23:49:02.18 SUCCESS: dd if=/dev/urandom of=/testpool/testfs/testfile.3 bs=1M count=75
23:49:03.77 SUCCESS: dd if=/dev/urandom of=/testpool/testfs/testfile.4 bs=1M count=100
23:49:04.01 SUCCESS: zpool sync testpool
23:49:04.02 SUCCESS: sync_pool testpool
23:49:04.02 NAME              SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
23:49:04.02 testpool         3.75G   269M  3.49G        -         -     0%     7%  1.00x    ONLINE  -
23:49:04.02   /mnt/device-0   960M     6M   954M        -         -     0%  0.62%      -  ONLINE  
23:49:04.02   /mnt/device-1   960M     6M   954M        -         -     0%  0.62%      -  ONLINE  
23:49:04.02   /mnt/device-2   960M  6.12M   954M        -         -     0%  0.63%      -  ONLINE  
23:49:04.02 special              -      -      -        -         -      -      -      -  -
23:49:04.02   /mnt/device-3   480M   121M   359M        -         -     2%  25.3%      -  ONLINE  
23:49:04.02   /mnt/device-4   480M   129M   351M        -         -     2%  26.9%      -  ONLINE  
23:49:04.03 SUCCESS: zpool list -v testpool
23:49:04.04 awk: can't open file { # find DVAs from string "offset level dva" only for L0 (data) blocks if (match($0,"L0 [0-9]+")) { dvas[0]=$3 dvas[1]=$4 dvas[2]=$5 for (i = 0; i < 3; ++i) { if (match(dvas[i],"([^:]+):.*")) { dva = substr(dvas[i], RSTART, RLENGTH); # parse DVA from string "vdev:offset:asize" if (split(dva,arr,":") != 3) { print "Error parsing DVA: <" dva ">"; exit 1; } # verify vdev is "special" if (arr[1] < d) { exit 1; } } } }} source line number 1
23:49:04.04 ERROR: file_in_special_vdev testpool/testfs 2 exited 2

[ahrens]

The FreeBSD failure is:

20:47:33.90 /usr/local/share/zfs/zfs-tests/tests/functional/cli_root/zpool_initialize/zpool_initialize_verify_initialized.ksh: line 79:  / 512: arithmetic syntax error

It looks like either $offset or $size is the empty string? Perhaps because zdb isn't outputting what it expects? I don't think this is related to my changes.

[behlendorf]

Yes, it's unrelated and has recently started failing consistently. There is draft PR #11328 open which is investigating the issue.