iterate_supers_type() spinning #790
Comments
|
wtf. (posting to track this) |
|
Just an update... arc_adapt has still not learned the error of its ways. It has taken up permanent residence on one of my CPU cores (see below). Any suggestions about how to find out what it is doing will be gratefully received. |
|
A further update... after another random load spike, the server became completely unresponsive yesterday morning. The stack trace from the kernel was revealing, in that it had nothing to do with ZFS. It was complaining about the Linux md driver, which supports the software RAID1 array I use to contain my OS install. So, quite possibly, the load spike issue is not relevant, and the question is simply why is arc_adapt spinning? Also, since I rebooted the server, arc_adapt has already returned to the top of the CPU utilisation list i.e. the problem resurfaced in under 24 hours. |
|
Kernel version is 3.2.0-25-generic (from Ubuntu Precise) |
|
My best guess for what's going on here is that your working meta data set size exceeds the arc_prune 4 37149 arc_meta_used 4 2064458416 arc_meta_limit 4 536870912 arc_meta_max 4 2066623440 First I'd try reverting to the default |
|
Thank you for taking the time to look into this bizarre issue. I have removed my user-specified zfs_arc_meta_limit. However, I also moved to CentOS 6, because I simply could not tolerate the instability of SATA drivers provided in Ubuntu. This has been an issue in the past, and it was disappointing to discover that it had resurfaced. Also, the constant goalpost moving of ever-newer kernel versions was not helping matters. In summary... Good news: Everything seems stable and arc_adapt is not spinning Bad news: I have changed two variables, which is bad experimentation |
|
Oh the irony! After I posted my last update, I checked dmesg, and found this. Note that it does not appear to have caused any instability. |
|
That's good news. As for the stack you posted that's a known issue which is being worked and it's harmless. In your exact case it resulted in a network packet getting dropped but TCP will resend so there was no harm done. |
|
Ok so I'm pretty sure I just ran into this on my Ubuntu fileserver running kernel 3.2.0-26-generic #41-Ubuntu SMP. I'm running the ZFS daily PPA, which I update regularly. I had the same problem with arc_adapt spinning at 100% utilizing, and it looks like it was probably to do with arc_meta_limit (which I've left at defaults). After reboot (with all the usual things running) the meta_usage is sitting close to the limit: I'm guessing I should just bump up arc_meta_limit, although I don't actually know what it does. |
|
The arc_meta_limit attempts to put a bound on the amount of cached meta data in the ARC. Basically this is anything that's not bulk file data but still needs to be cached. Things like inodes. Anyway, the limit is there to attempt to keep a reasonable balanve between the file data in this meta data. If your hitting this limit regularly you likely have a large working set of files and it may make sense to increase it. |
|
Ok I just had this bug lock up my server again. It happened right as I ran Unison against a fairly large directory full of large files. The behavior was the same - arc_adapt started spinning at 100+% in top, and filled up the memory until it hit the swap at which point the server became unresponsive. I have Is there something I can do to help debug this, since it doesn't seem reasonable that metadata operations should lock up the entire server - I'd expect simply to have slower file lists, data searches. |
|
I'm currently seeing this on a system, Ubuntu 12.04 64 3.2.0-27-generic, zfs 0.6.0.65. It occurred after a rsync and snapshot. The system has 2 CPUs, 2GB ram, 16G USB flash l2arc. I'll leave it spinning for a couple of days if anyone wants me to run anything. Looking at a "perf" callgraph it seems to be spinning inside iterate_supers_type(), not even calling zpl_prune_sb() from what I can tell (pasted below). It's as if the fs_supers list has become looped. In the 3.3 kernel the commit dabe0dc194d5d56d379a8994fff47392744b6491 looks like it fixes that kind of problem - maybe that's the issue? (I think there are some related previous commits too). |
|
Interesting, the additional profiling makes this look like a duplicate of #861 which was recently filed. Somehow we're getting caught on the sb_lock while iterating over the super blocks. The kernel commit you referenced certainly looks relevant we'll need to review it carefully. In the meanwhile, if you need a work around you could #undef HAVE_SHRINK in zfs_config.h and rebuild zfs until we exactly determine what's going on here. |
|
Looking at my system it seems that the bug occurred while unmounting a .zfs/snapshot dir (it's still mounted but nothing using it), matching #861. I think there's still a bug in iterate_supers_type() in head kernels. In generic_shutdown_super() it calls hlist_del_init() on s_instances, but don't think there's anything preventing it being the current iterator in iterate_supers_type(). Possibly deleting s_instances needs to be moved to __put_super(), like s_list. (And other changes made to things that look at s_instances being empty). |
|
I'm still experiencing this issue. It starts at overnight, when I leave the computer turned on, but no user is logged in. |
|
This will be fixed with proper page cache integration. Until then I'd suggest making your .zfs snapshot directory invisible so it's less likely to be traversed by processes walking the file system. This will cut down on the number of mount/umounts and make the issue less likely. |
|
Is this issue fixed? |
|
@Rudd-O No, but you can likely work around it by building zfs with HAVE_SHRINK undefined. |
The iterate_supers_type() function which was introduced in the 3.0 kernel was supposed to provide a safe way to call an arbitrary function on all super blocks of a specific type. Unfortunately, because a list_head was used a bug was introduced which made it possible for iterate_supers_type() to get stuck spinning on a super block which was just deactivated. The bug was fixed in the 3.3 kernel by converting the list_head to an hlist_node. However, to resolve the issue for existing 3.0 - 3.2 kernels we detect when a list_head is used. Then to prevent the spinning from occurring the .next pointer is set to the fs_supers list_head which ensures the iterate_supers_type() function will always terminate. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#1045 Issue openzfs#861 Issue openzfs#790
The iterate_supers_type() function which was introduced in the 3.0 kernel was supposed to provide a safe way to call an arbitrary function on all super blocks of a specific type. Unfortunately, because a list_head was used a bug was introduced which made it possible for iterate_supers_type() to get stuck spinning on a super block which was just deactivated. This can occur because when the list head is removed from the fs_supers list it is reinitialized to point to itself. If the iterate_supers_type() function happened to be processing the removed list_head it will get stuck spinning on that list_head. The bug was fixed in the 3.3 kernel by converting the list_head to an hlist_node. However, to resolve the issue for existing 3.0 - 3.2 kernels we detect when a list_head is used. Then to prevent the spinning from occurring the .next pointer is set to the fs_supers list_head which ensures the iterate_supers_type() function will always terminate. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#1045 Closes openzfs#861 Closes openzfs#790
Bumps [bytes](https://github.com/tokio-rs/bytes) from 1.3.0 to 1.4.0. - [Release notes](https://github.com/tokio-rs/bytes/releases) - [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md) - [Commits](tokio-rs/bytes@v1.3.0...v1.4.0) --- updated-dependencies: - dependency-name: bytes dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
I'm running ZFS 0.6.0.65 (kernel module) on Ubuntu 12.04 64-bit.
With this version of ZFS, and previous versions, I have the same issue, which I shall try to describe...
A few hours after booting the server (usually the next day), the load average reported will be insanely high (3600 is my personal best) for a period of time.
This situation resolves itself without any intervention from me. However, it leaves behind an arc_adapt process, which consumes 100% of one core, until I reboot the server.
Having read that arc_adapt doesn't have much responsibility these days, I am at a loss to explain this behaviour.
Annoyingly, I am unable to attach gdb or strace to the arc_adapt kernel space process, in order to further diagnose the issue.
Here is some further information:
RAM: 8GB
ZFS configuration: RAIDZ2 comprised of 8x2TB and 8x1TB drives
The text was updated successfully, but these errors were encountered: