rm / remove / delete a large file causes high load and irresponsiveness

[michaelxwang]

Here is a demonstration of the problem:

# ls -ls `pwd`/junk
101496218 -rw-r--r-- 1 root root 373720875008 Jan 14 21:52 /srv/mysqldata/data/backup/junk

# time rm junk

real    7m27.012s
user    0m0.000s
sys     0m4.348s

# while true; do uptime; sleep 5; done
...
 21:58:27 up  1:31,  4 users,  load average: 48.76, 24.15, 10.50
 21:58:32 up  1:31,  4 users,  load average: 48.94, 24.60, 10.72
 21:58:37 up  1:31,  4 users,  load average: 49.10, 25.03, 10.93
 21:58:42 up  1:31,  4 users,  load average: 49.26, 25.47, 11.15
 21:58:47 up  1:31,  4 users,  load average: 49.40, 25.89, 11.36
 21:58:52 up  1:32,  4 users,  load average: 49.52, 26.31, 11.58
 21:58:57 up  1:32,  4 users,  load average: 49.64, 26.72, 11.79
 21:59:02 up  1:32,  4 users,  load average: 46.15, 26.37, 11.76
...
 22:06:32 up  1:39,  4 users,  load average: 0.98, 8.03, 8.56

The top processes are things like:

top - 19:53:47 up 90 days,  1:05,  4 users,  load average: 26.10, 6.45, 2.69
Tasks: 546 total,   2 running, 543 sleeping,   1 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni, 98.0 id,  1.7 wa,  0.0 hi,  0.0 si,  0.2 st
KiB Mem:  25189913+total, 14417196+used, 10772716+free,   425504 buffers
KiB Swap:        0 total,        0 used,        0 free.  3173036 cached Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                    
 46003 root      20   0       0      0      0 R  59.7  0.0 324:05.24 txg_sync                                                                                                                                   
 45987 root       0 -20       0      0      0 S   7.6  0.0   5:07.18 z_fr_iss_1                                                                                                                                 
 45988 root       0 -20       0      0      0 S   7.3  0.0   5:04.00 z_fr_iss_2                                                                                                                                 
 45990 root       0 -20       0      0      0 S   7.3  0.0   5:12.88 z_fr_iss_4                                                                                                                                 
 45991 root       0 -20       0      0      0 S   7.3  0.0   5:10.03 z_fr_iss_5                                                                                                                                 
 45986 root       0 -20       0      0      0 S   7.0  0.0   5:05.78 z_fr_iss_0                                                                                                                                 
 45989 root       0 -20       0      0      0 S   7.0  0.0   5:11.09 z_fr_iss_3                                                                                                                                 
 45993 root       0 -20       0      0      0 S   7.0  0.0   5:12.80 z_fr_iss_7                                                                                                                                 
 45992 root       0 -20       0      0      0 S   5.6  0.0   5:06.98 z_fr_iss_6         

The zfs version I have is 0.6.5.4 (rebooted after installing the package):

# dpkg -l | grep zfs

ii  libzfs2                              0.6.5.4-1~trusty                    amd64        Native OpenZFS filesystem library for Linux
ii  mountall                             2.53-zfs1                           amd64        filesystem mounting tool
ii  ubuntu-zfs                           8~trusty                            amd64        Native ZFS filesystem metapackage for Ubuntu.
ii  zfs-dkms                             0.6.5.4-1~trusty                    amd64        Native OpenZFS filesystem kernel modules for Linux
ii  zfs-doc                              0.6.5.4-1~trusty                    amd64        Native OpenZFS filesystem documentation and examples.
ii  zfsutils                             0.6.5.4-1~trusty                    amd64        Native OpenZFS management utilities for Linux

The "junk" file is copied from a MySQL database file. I tried creating a large file with dd:

# dd if=/dev/zero of=junk2 bs=1024000 count=400000
# time rm junk2

real    0m0.002s
user    0m0.000s
sys     0m0.000s

but as seen, it is removed without issue. Obviously dd a file and cp a file are different.

I originally encountered the problem when dropping a large table (300GB) in MySQL database, and found this is due to the removal of the database file, so rm a large file is more basic issue.

[kernelOfTruth]

@michaelxwang please file the issue at: https://github.com/zfsonlinux/zfs/issues

@FransUrbo sorry if I understand the distinction of posting issues in the zfs repo wrong ?

[michaelxwang]

OK, I filed under openzfs/zfs#4227, thanks. I read the FAQ and thought that I should file here, but I read it again, it is for package issues not for packaged zfs issues.