zfs_free_max_blocks=0 causes space to never be freed and txg_sync to do constant IO

[zviratko]

System information

Type	Version/Name
Distribution Name	Ubuntu
Distribution Version	16.04.5 LTS
Linux Kernel	4.15.0-30-generic
Architecture	amd64
ZFS Version	0.7.12-0york3~16.04
SPL Version	0.7.12-0york0~16.04

On a backup server receiving and storing lots of snapshots, I found it useful to increase zfs_free_max_blocks to speed up the receive.
Yesterday I upgraded this server to newest kernel/zfs versions and rebooted.
After that I simply set zfs_free_max_blocks=0, thinking this will just remove any limit.

Today after arriving at work, all the drives were doing lots of writes, like so:

                                                      capacity     operations     bandwidth
pool                                                alloc   free   read  write   read  write
--------------------------------------------------  -----  -----  -----  -----  -----  -----
backup                                              38.9T  4.63T      0  3.16K      0  73.5M
  mirror                                            3.27T   361G      0    246      0  3.85M
    scsi-350000c0f01352278                              -      -      0    125      0  1.92M
    scsi-350000c0f01e8496c                              -      -      0    121      0  1.92M
  mirror                                            3.26T   372G      0    301      0  3.97M
    wwn-0x50000c0f01f53d68                              -      -      0    151      0  1.98M
    scsi-350000c0f01ff280c                              -      -      0    150      0  1.98M
  mirror                                            3.26T   378G      0    359      0  5.40M
    wwn-0x50000c0f023d0ef4                              -      -      0    181      0  2.67M
    wwn-0x50000c0f023d17cc                              -      -      0    179      0  2.73M
  mirror                                            3.26T   372G      0    385      0  5.25M
    scsi-350000c0f023d1884                              -      -      0    194      0  2.61M
    wwn-0x50000c0f02effcbc                              -      -      0    191      0  2.64M
  mirror                                            3.26T   374G      0    138      0  2.74M
    scsi-350000c0f02f0086c                              -      -      0     69      0  1.37M
    scsi-350000c0f02f00b14                              -      -      0     68      0  1.37M
  mirror                                            1.72T  93.6G      0     99      0  1.23M
    wwn-0x50014ee20824808a                              -      -      0     49      0   629K
    wwn-0x50014ee2b2d8b53e                              -      -      0     49      0   629K
  mirror                                            4.84T   624G      0    376      0  10.6M
    wwn-0x5000cca255d63975                              -      -      0    188      0  5.29M
    wwn-0x5000cca255d63f4c                              -      -      0    188      0  5.29M
  mirror                                            4.85T   622G      0    330      0  13.9M
    wwn-0x5000cca255d63f18                              -      -      0    164      0  6.94M
    wwn-0x5000cca255d6397b                              -      -      0    166      0  6.94M
  mirror                                            4.84T   626G      0    686      0  18.2M
    wwn-0x5000cca255d60fc2                              -      -      0    340      0  8.86M
    wwn-0x5000cca271c2e0bb                              -      -      0    346      0  9.37M
  mirror                                            6.37T   918G      0    312      0  8.33M
    wwn-0x5000cca3b7cf0dac                              -      -      0    155      0  4.16M
    wwn-0x5000cca254e30cfa                              -      -      0    157      0  4.16M
cache                                                   -      -      -      -      -      -
  ata-INTEL_SSDSC2BB480G4_CVWL409600GN480QGN-part3   252G   185G      0      0      0      0
  ata-INTEL_SSDSC2BB480G4_CVWL410001BQ480QGN-part3   253G   184G      0      0      0      0

the process doing this IO was "txg_sync"
There was also 25GiB "freeing" in the pool that did not decrease.

After setting zfs_free_max_blocks=1000000 the problem disappeared immediately and the space freed.

There are no errors anywhere (apart from some hung_tasks from yesterday, this pool is overloaded until caches get filled)

I'm not sure what the behaviour of setting zfs_free_max_blocks=0 should be, this https://github.com/zfsonlinux/zfs/issues/2581#issuecomment-55069014 led me to believe that it would just remove the limit. If it truly does set a limit at "0" causing this sort of behaviour, then it should come with a big fat warning, and if it's a wanted behaviour it should not cause constant IO.

[behlendorf]

Thanks for pointing this out. We can look in to adding some reasonable limits to prevent this kind of thing.