Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

hung task on zpool import #65

Closed
nedbass opened this issue Sep 21, 2010 · 3 comments
Closed

hung task on zpool import #65

nedbass opened this issue Sep 21, 2010 · 3 comments

Comments

@nedbass
Copy link
Contributor

nedbass commented Sep 21, 2010

Saw this after a failed attempt to destroy a dataset and a reboot (see #66). After reboot, tried to import the volume with "zpool import lustre-zeno1 -d /dev/disk/zpool",
which hung. Had built a Lustre filesystem on the dataset and filled it to 100%.

2010-09-21 10:23:22 INFO: task l2arc_feed:14433 blocked for more than 120 seconds.
2010-09-21 10:23:22 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2010-09-21 10:23:22 l2arc_feed    D ffffffffffffffff     0 14433      2 0x00000000
2010-09-21 10:23:22  ffff8802d9cf3d30 0000000000000046 ffff8802d9cf3ca0 ffffffff8107c452
2010-09-21 10:23:22  ffff88063274c000 ffff8802d9cf3cc0 ffff8802d9cf3d50 ffffffff814d5757
2010-09-21 10:23:22  ffff8802e7bee670 ffff8802d9cf3fd8 00000000000103b8 ffff8802e7bee670
2010-09-21 10:23:22 Call Trace:
2010-09-21 10:23:22  [] ? del_timer_sync+0x22/0x30
2010-09-21 10:23:22  [] ? schedule_timeout+0x197/0x2f0
2010-09-21 10:23:22  [] __mutex_lock_slowpath+0x117/0x1a0
2010-09-21 10:23:22  [] ? autoremove_wake_function+0x0/0x40
2010-09-21 10:23:22  [] ? l2arc_feed_thread+0x0/0x840 [zfs]
2010-09-21 10:23:22  [] mutex_lock+0x2b/0x50
2010-09-21 10:23:22  [] ? l2arc_feed_thread+0x0/0x840 [zfs]
2010-09-21 10:23:22  [] l2arc_feed_thread+0xf3/0x840 [zfs]
2010-09-21 10:23:22  [] ? enqueue_entity+0x287/0x320
2010-09-21 10:23:22  [] ? enqueue_task+0x5c/0x70
2010-09-21 10:23:22  [] ? l2arc_feed_thread+0x0/0x840 [zfs]
2010-09-21 10:23:22  [] ? l2arc_feed_thread+0x0/0x840 [zfs]
2010-09-21 10:23:22  [] thread_generic_wrapper+0x68/0x80 [spl]
2010-09-21 10:23:22  [] ? thread_generic_wrapper+0x0/0x80 [spl]
2010-09-21 10:23:22  [] kthread+0x96/0xa0
2010-09-21 10:23:22  [] child_rip+0xa/0x20
2010-09-21 10:23:22  [] ? kthread+0x0/0xa0
2010-09-21 10:23:22  [] ? child_rip+0x0/0x20

The zpool was layed out like this.

  pool: lustre-zeno2
 state: ONLINE
 scan: none requested
config:

        NAME           STATE     READ WRITE CKSUM
        lustre-zeno2   ONLINE       0     0     0
          -0           ONLINE       0     0     0
            B16-part1  ONLINE       0     0     0
            A16-part1  ONLINE       0     0     0
            C16-part1  ONLINE       0     0     0
            D16-part1  ONLINE       0     0     0
            E16-part1  ONLINE       0     0     0
            F16-part1  ONLINE       0     0     0
            G16-part1  ONLINE       0     0     0
            H16-part1  ONLINE       0     0     0
            I16-part1  ONLINE       0     0     0
            J16-part1  ONLINE       0     0     0
          -1           ONLINE       0     0     0
            B17-part1  ONLINE       0     0     0
            A17-part1  ONLINE       0     0     0
            C17-part1  ONLINE       0     0     0
            D17-part1  ONLINE       0     0     0
            E17-part1  ONLINE       0     0     0
            F17-part1  ONLINE       0     0     0
            G17-part1  ONLINE       0     0     0
            H17-part1  ONLINE       0     0     0
            I17-part1  ONLINE       0     0     0
            J17-part1  ONLINE       0     0     0
          -2           ONLINE       0     0     0
            B18-part1  ONLINE       0     0     0
            A18-part1  ONLINE       0     0     0
            C18-part1  ONLINE       0     0     0
            D18-part1  ONLINE       0     0     0
            E18-part1  ONLINE       0     0     0
            F18-part1  ONLINE       0     0     0
            G18-part1  ONLINE       0     0     0
            H18-part1  ONLINE       0     0     0
            I18-part1  ONLINE       0     0     0
            J18-part1  ONLINE       0     0     0
          -3           ONLINE       0     0     0
            B19-part1  ONLINE       0     0     0
            A19-part1  ONLINE       0     0     0
            C19-part1  ONLINE       0     0     0
            D19-part1  ONLINE       0     0     0
            E19-part1  ONLINE       0     0     0
            F19-part1  ONLINE       0     0     0
            G19-part1  ONLINE       0     0     0
            H19-part1  ONLINE       0     0     0
            I19-part1  ONLINE       0     0     0
            J19-part1  ONLINE       0     0     0
          -4           ONLINE       0     0     0
            B20-part1  ONLINE       0     0     0
            A20-part1  ONLINE       0     0     0
            C20-part1  ONLINE       0     0     0
            D20-part1  ONLINE       0     0     0
            E20-part1  ONLINE       0     0     0
            F20-part1  ONLINE       0     0     0
            G20-part1  ONLINE       0     0     0
            H20-part1  ONLINE       0     0     0
            I20-part1  ONLINE       0     0     0
            J20-part1  ONLINE       0     0     0
          -5           ONLINE       0     0     0
            B21-part1  ONLINE       0     0     0
            A21-part1  ONLINE       0     0     0
            C21-part1  ONLINE       0     0     0
            D21-part1  ONLINE       0     0     0
            E21-part1  ONLINE       0     0     0
            F21-part1  ONLINE       0     0     0
            G21-part1  ONLINE       0     0     0
            H21-part1  ONLINE       0     0     0
            I21-part1  ONLINE       0     0     0
            J21-part1  ONLINE       0     0     0
          -6           ONLINE       0     0     0
            B22-part1  ONLINE       0     0     0
            A22-part1  ONLINE       0     0     0
            C22-part1  ONLINE       0     0     0
            D22-part1  ONLINE       0     0     0
            E22-part1  ONLINE       0     0     0
            F22-part1  ONLINE       0     0     0
            G22-part1  ONLINE       0     0     0
            H22-part1  ONLINE       0     0     0
            I22-part1  ONLINE       0     0     0
            J22-part1  ONLINE       0     0     0
        logs
          B23-part1    ONLINE       0     0     0
          C23-part1    ONLINE       0     0     0
        cache
          B23-part1    UNAVAIL      0     0     0
          D23-part1    ONLINE       0     0     0
          E23-part1    ONLINE       0     0     0
@behlendorf
Copy link
Contributor

I've got a hunch this is related to the l2arc devices we've added in and have started testing with. In particular I think we're likely stuck on the l2arc_dev_mtx mutex, but we'ld need to resolve l2arc_feed_thread+0xf3 to be sure. We might also be stuck on the l2arc_feed_thr_lock mutex but I think that's less likely.

This might be related to the pool being misconfigured. If you look above you'll see that disk B23 is part of both the 'logs' and 'cache' vdev. I'm very surprised the zpool command allowed you to do that.

@nedbass
Copy link
Contributor Author

nedbass commented Sep 21, 2010 

# zeno5 /root > zpool create -f lustre-zeno5 raidz2 A8 B8 C8 D8 E8 F8 G8 H8 I8 J8 raidz2 A9 B9 C9 D9 E9 F9 G9 H9 I9 J9 raidz2 A10 B10 C10 D10 E10 F10 G10 H10 I10 J10 raidz2 A11 B11 C11 D11 E11 F11 G11 H11 I11 J11 raidz2 A12 B12 C12 D12 E12 F12 G12 H12 I12 J12 raidz2 A13 B13 C13 D13 E13 F13 G13 H13 I13 J13 raidz2 A14 B14 C14 D14 E14 F14 G14 H14 I14 J14 log G15 H15 cache G15 I15 J15
cannot create 'lustre-zeno5': one or more vdevs refer to the same device, or one of
the devices is part of an active md or lvm device

Indeed it does not allow it. Perhaps something got corrupted or there was an error importing the zpool. The above layout was not from one of the the nodes that hung. I'm not sure if those nodes also had the duplicate disk problem, as their zpools are no longer accessible. I'll do more testing to see if this can be reproduced.

@behlendorf
Copy link
Contributor

I can't reproduce this, and we haven't seen it since. I'm closing this bug due to a lack of information, we'll open a new one if we see this failure again.

dajhorn referenced this issue in zfsonlinux/pkg-zfs Dec 14, 2011
@behlendorf
Replace struct spl_task with struct taskq_ent
046a70c 
The spl_task structure was renamed to taskq_ent, and all of
its fields were renamed to have a prefix of 'tqent' rather
than 't'. This was to align with the naming convention which
the ZFS code assumes.  Previously these fields were private
so the name never mattered.

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #65
dajhorn referenced this issue in zfsonlinux/pkg-zfs Dec 14, 2011
@behlendorf
Replace tq_work_list and tq_threads in taskq_t
2c02b71 
To lay the ground work for introducing the taskq_dispatch_prealloc()
interface, the tq_work_list and tq_threads fields had to be replaced
with new alternatives in the taskq_t structure.

The tq_threads field was replaced with tq_thread_list. Rather than
storing the pointers to the taskq's kernel threads in an array, they are
now stored as a list. In addition to laying the ground work for the
taskq_dispatch_prealloc() interface, this change could also enable taskq
threads to be dynamically created and destroyed as threads can now be
added and removed to this list relatively easily.

The tq_work_list field was replaced with tq_active_list. Instead of
keeping a list of taskq_ent_t's which are currently being serviced, a
list of taskq_threads currently servicing a taskq_ent_t is kept. This
frees up the taskq_ent_t's tqent_list field when it is being serviced
(i.e. now when a taskq_ent_t is being serviced, it's tqent_list field
will be empty).

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #65
dajhorn referenced this issue in zfsonlinux/pkg-zfs Dec 14, 2011
@behlendorf
Add Test: "Single task queue, recursive dispatch"
ac1e5b6 
Added another splat taskq test to ensure tasks can be recursively
submitted to a single task queue without issue. When the
taskq_dispatch_prealloc() interface is introduced, this use case
can potentially cause a deadlock if a taskq_ent_t is dispatched
while its tqent_list field is not empty. This _should_ never be
a problem with the existing taskq_dispatch() interface.

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #65
dajhorn referenced this issue in zfsonlinux/pkg-zfs Dec 14, 2011
@behlendorf
Implement taskq_dispatch_prealloc() interface
44217f7 
This patch implements the taskq_dispatch_prealloc() interface which
was introduced by the following illumos-gate commit.  It allows for
a preallocated taskq_ent_t to be used when dispatching items to a
taskq.  This eliminates a memory allocation which helps minimize
lock contention in the taskq when dispatching functions.

    commit 5aeb94743e3be0c51e86f73096334611ae3a058e
    Author: Garrett D'Amore <garrett@nexenta.com>
    Date:   Wed Jul 27 07:13:44 2011 -0700

    734 taskq_dispatch_prealloc() desired
    943 zio_interrupt ends up calling taskq_dispatch with TQ_SLEEP

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #65
dajhorn referenced this issue in zfsonlinux/pkg-zfs Dec 14, 2011
@behlendorf
Exercise new taskq interface in splat-taskq tests
699d5ee 
The splat-taskq test functions were slightly modified to exercise
the new taskq interface in addition to the old interface.  If the
old interface passes each of its tests, the new interface is
exercised.  Both sub tests (old interface and new interface) must
pass for each test as a whole to pass.

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #65
dajhorn referenced this issue in zfsonlinux/pkg-zfs Sep 7, 2012
@behlendorf
Update a comment to reflect new taskq internals
05b8f50 
As of the removal of the taskq work list made in commit:

    commit 2c02b71
    Author: Prakash Surya <surya1@llnl.gov>
    Date:   Mon Dec 5 17:32:48 2011 -0800

        Replace tq_work_list and tq_threads in taskq_t

        To lay the ground work for introducing the taskq_dispatch_prealloc()
        interface, the tq_work_list and tq_threads fields had to be replaced
        with new alternatives in the taskq_t structure.

the comment above taskq_wait_check has been incorrect. This change is an
attempt at bringing that description more in line with the current
implementation. Essentially, references to the old task work list had to
be updated to reference the new taskq thread active list.

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue #65
akatrevorjay pushed a commit to akatrevorjay/zfs that referenced this issue Dec 12, 2012
@dajhorn
Merge pull request openzfs#65 from wrouesnel/plymouth-patch
ff830b9 
Added call to hide Plymouth when error shell is launched.
jkryl referenced this issue in mayadata-io/cstor Feb 27, 2018
@vishnuitta
[cstor#21] thread to update txg every 10 mins (#65)
30c7c70 
[cstor#21] thread to update txg every 10 mins
jkryl referenced this issue in mayadata-io/cstor Mar 15, 2018
assertion in debug mode when destroying zvol (#65)
e909978 
Signed-off-by: Jan Kryl <jan.kryl@cloudbyte.com>
jkryl referenced this issue in mayadata-io/cstor Mar 15, 2018
assertion in debug mode when destroying zvol (#65)
7444b88 
Signed-off-by: Jan Kryl <jan.kryl@cloudbyte.com>
vishnuitta referenced this issue in vishnuitta/zfs Mar 21, 2018
@vishnuitta
assertion in debug mode when destroying zvol (mayadata-io#65)
4c11d30 
Signed-off-by: Jan Kryl <jan.kryl@cloudbyte.com>
richardelling pushed a commit to richardelling/zfs that referenced this issue Oct 15, 2018
@vishnuitta
[cstor#21] thread to update txg every 10 mins (openzfs#65)
3fef28e 
[cstor#21] thread to update txg every 10 mins
richardelling pushed a commit to richardelling/zfs that referenced this issue Oct 15, 2018
assertion in debug mode when destroying zvol (openzfs#65)
f7df8b0 
Signed-off-by: Jan Kryl <jan.kryl@cloudbyte.com>
tonynguien pushed a commit to tonynguien/zfs that referenced this issue Dec 21, 2021
@grwilson
DOSE-836 zoa.conf multiple disk zettacache support (openzfs#65)
808f37e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@behlendorf @nedbass