txg_sync Call Trace

[poiuty]

Debian 8, kernel 3.16, zfs v0.6.5.2-2

Oct 22 13:06:02 ns3014809 kernel: [185803.449000] txg_sync        D ffff88103b23d808     0   981      2 0x00000000
Oct 22 13:06:02 ns3014809 kernel: [185803.449552]  ffff88103b23d3b0 0000000000000046 0000000000012f00 ffff88102f6b3fd8
Oct 22 13:06:02 ns3014809 kernel: [185803.450126]  0000000000012f00 ffff88103b23d3b0 ffff88107fc137b0 ffff8807d4cad530
Oct 22 13:06:02 ns3014809 kernel: [185803.450690]  ffff8807d4cad570 0000000000000001 ffff88103ef06000 0000000000000000
Oct 22 13:06:02 ns3014809 kernel: [185803.451234] Call Trace:
Oct 22 13:06:02 ns3014809 kernel: [185803.451801]  [<ffffffff8150e139>] ? io_schedule+0x99/0x120
Oct 22 13:06:02 ns3014809 kernel: [185803.452374]  [<ffffffffa07d3760>] ? cv_wait_common+0x90/0x100 [spl]
Oct 22 13:06:02 ns3014809 kernel: [185803.452925]  [<ffffffff810a7a40>] ? prepare_to_wait_event+0xf0/0xf0
Oct 22 13:06:02 ns3014809 kernel: [185803.453535]  [<ffffffffa0a3279b>] ? zio_wait+0x10b/0x1e0 [zfs]
Oct 22 13:06:02 ns3014809 kernel: [185803.454031]  [<ffffffffa09c07ca>] ? dsl_pool_sync+0xaa/0x460 [zfs]
Oct 22 13:06:02 ns3014809 kernel: [185803.454572]  [<ffffffffa09da6b6>] ? spa_sync+0x366/0xb30 [zfs]
Oct 22 13:06:02 ns3014809 kernel: [185803.455165]  [<ffffffffa09ec131>] ? txg_sync_thread+0x3d1/0x680 [zfs]
Oct 22 13:06:02 ns3014809 kernel: [185803.455772]  [<ffffffffa09ebd60>] ? txg_quiesce_thread+0x3e0/0x3e0 [zfs]
Oct 22 13:06:02 ns3014809 kernel: [185803.456362]  [<ffffffffa07cedeb>] ? thread_generic_wrapper+0x6b/0x80 [spl]
Oct 22 13:06:02 ns3014809 kernel: [185803.456946]  [<ffffffffa07ced80>] ? __thread_exit+0x20/0x20 [spl]
Oct 22 13:06:02 ns3014809 kernel: [185803.457535]  [<ffffffff81087f7d>] ? kthread+0xbd/0xe0
Oct 22 13:06:02 ns3014809 kernel: [185803.458137]  [<ffffffff81087ec0>] ? kthread_create_on_node+0x180/0x180
Oct 22 13:06:02 ns3014809 kernel: [185803.458753]  [<ffffffff815115d8>] ? ret_from_fork+0x58/0x90
Oct 22 13:06:02 ns3014809 kernel: [185803.459369]  [<ffffffff81087ec0>] ? kthread_create_on_node+0x180/0x180

[behlendorf]

@poiuty did the system hang or was it just slow servicing this I/O?

[poiuty]

@behlendorf slow I/O
I set echo 0 > /sys/module/spl/parameters/spl_taskq_thread_dynamic and ~17 days no Call Trace
#3955

[behlendorf]

@poiuty thanks for the follow up. My suggestion would be to update to 0.6.5.3 which makes spl_taskq_thread_dynamic=0 the default. The Debian repos should have been updated with that release already. Or if you want to leave dynamic taskqs enabled you can cherry pick the fix from master openzfs/spl@f5f2b87. Closing as resolved.

[poiuty]

@behlendorf, try to update from apt-get repo (debian 8), but no 0.6.5.3

apt-cache show zfs-dkms
Package: zfs-dkms
Source: zfs-linux
Version: 0.6.5.2-2
Architecture: all
Maintainer: Debian ZFS on Linux maintainers <pkg-zfsonlinux-devel@lists.alioth.debian.org>
Installed-Size: 7562
Pre-Depends: spl-dkms
Depends: dkms (>> 2.1.1.2-5), lsb-release, debconf (>= 0.5) | debconf-2.0, file
Recommends: zfsutils
Conflicts: lzfs, lzfs-dkms, zfs-modules-source
Replaces: lzfs, lzfs-dkms, zfs-modules-source
Provides: lustre-backend-fs, lzfs, lzfs-dkms, zfs-modules, zfs-modules-source
Homepage: http://www.zfsonlinux.org/
Priority: optional
Section: kernel
Filename: pool/main/z/zfs-linux/zfs-dkms_0.6.5.2-2_all.deb
Size: 1059562
SHA256: d2328eafb27c0387f0433314deb887f2f443d21e5878c5915f152f797d24937d
SHA1: 358e30a9dbd3d5fb0441a87105847f1bcbd20f58
MD5sum: 00cae5405b8aeab37d7a75b7f2512538
Description: Native ZFS filesystem kernel modules for Linux
 The Z file system is a pooled filesystem designed for maximum data
 integrity, supporting data snapshots, multiple copies, and data
 checksums.
 .
 This DKMS package includes the SPA, DMU, ZVOL, and ZPL components of ZFS.
Description-md5: f981018c4f7da79a8639358929aed351