ZFS 0.6.5.3 servers hang trying to get mutexes #4106
Comments
|
@jdavidlists your definitely on the right track, we need to identify why the mutex isn't being released. If you could post back traces from all the processes we may be able to spot the deadlock by looking at the stacks. If the system is still responsive to sysrqs,, |
|
OK, that worked, but it seems like it didn't grab the stacks of all processes. There are 626 processes listed under the CPUs' "runnable tasks" list but only 142 stack traces: That's not a good ratio, but we'll see what we get. That's still a lot of data, so rather than spam a huge amount of unformatted info, it might be helpful to thin it down. It seems safe to ignore the 113 nfsd's sitting in schedule_timeout(). Next, it seems safe to omit any stack trace that doesn't touch either the zfs or spl modules. That gets us down to just 4 nfsd processes of potential interest. Here they are: One of these, pid 2373 (in zfs_zget), is not like the others (in spl_kmem_cache_alloc). The first process to throw a "120 seconds" warning was kswapd again, with the exact same stack trace as my original message, followed by txg_sync (also same trace as original message), and 8 nfsd's. 6 of the nfsd's have the same stack trace (varying pid): Note that one of the 6 processes that reported "120 seconds blocked" with the above trace is 2320, which appears in the (subsequent) sysrq output with a different trace. So it did eventually make at least some progress. The other two nfsd traces for "blocked" processes were: and this one, which is similar to but not the same as the other six in zil_commit: Perhaps all of these things have a particular mutex (or condition variable) in common? The ZFS debug log was also enabled, although not too useful as it wrapped around and overwrote itself before we detected the problem and saved it, but it is from the "dead" time. It's 59147 lines, 32 of which are not "getting." They are: So, my understand is very limited, but the situation is something like: somebody took a mutex and then asked for memory and didn't get it, got put on hold, and everybody else piled up behind him waiting for the mutex? The machine doesn't seem low on memory, but it would match the total lack of any visible disk activity. (I.e. if it was waiting for I/O I would expect to see horrifying things in iostat -x or zpool iostat, but all that's there in this state are zeroes.) Is any of this helpful? The raw data is still available if this is over-trimmed, but it's a giant wall of text that doesn't seem to have much content. If there's another relevant process, it's likely the sysrq didn't catch it. Where do we go from here? Whatever we can do to get more information, we'll do. Starting with gathering more info about memory usage the next time this crops up. Thanks! |
|
Just for the sake of completeness, the 113 nfsd processes I left out all look like this: So they appear to be idle, which is why they got left out. |
|
Looking at the code involved here, it at least looks like there is significant possibility for mutex deadlock between zfs_zinactive() and anything else that uses ZFS_OBJ_HOLD_ENTER(). zfs_zinactive() is clearly being called essentially asynchronously by the overall virtual memory system as part of reclaiming memory (and comments in zfs_vnops.c assert that it may kick off IO of its own), but at the same time zfs_zget() and other users of ZFS_OBJ_HOLD_ENTER() do IO themselves via eg sa_buf_hold(), which can allocate memory and can clearly kick off memory pressure and wind up waiting for it (which is what is seen in the stack traces). (This doesn't explain all of the mutex waits in the traces, eg dmu_zfetch().) It strikes me as potentially significant that NFS is one of the few cases where you can have requests for an inode show up essentially out of the blue. For local IO, active inodes are tied to open files, directory lookups, and so on, and so will presumably always be held busy and non-evictable. However, NFS requests are purely directed against inodes and there is nothing to hold those active inodes busy on the server between requests. Is it possible that there's some sort of scenario where zfs_zget() is trying to revive an inode for NFS just as it's being evicted for memory pressure? |
|
I have a lockup on my laptop that seems related to this. after rebooting i retrieved several things from pstore. Here are a few of the stacktraces that seem related to locking (the others are mostly watchdog). I was doing a lot of diskIO at the time but this machine is not an NFS server (only client but not actively using the mounts). I have also been getting several lockups on my server too but have not managed to get any stacktraces from it. |
|
@jdavidlists thank you for stacks and detailed analysis. I'm fairly sure @siebenmann is on the right track here with a ZFS_OBJ_HOLD_ENTER() deadlock although the back traces don't appear to explicitly show this. They're just blocked waiting for memory. But this exact issue is something both Linux and FreeBSD have struggled with due to differences in the VM and VFS subsystems across platforms. We should investigate discarding the current locking scheme we inherited from illumos and replacing it with something optimized for Linux. Besides being a place which can result in deadlocks such as this it's a needless potential point of serialization which can negatively impact metadata performance. @jdavidlists you may be able to significantly reduce the likelihood of hitting this issue and help us verify that this is the root cause by increasing the number of locks in the array. Can you please make the following small change to ZFS. As I said it's not a proper fix but it should help. I suspect you're more likely to encounter this issue than much due to the large number of nfsd threads you're running makes it far more likely. diff --git a/include/sys/zfs_vfsops.h b/include/sys/zfs_vfsops.h
index 28407c6..dcd5311 100644
--- a/include/sys/zfs_vfsops.h
+++ b/include/sys/zfs_vfsops.h
@@ -112,7 +112,7 @@ typedef struct zfs_sb {
uint64_t z_groupquota_obj;
uint64_t z_replay_eof; /* New end of file - replay only */
sa_attr_type_t *z_attr_table; /* SA attr mapping->id */
-#define ZFS_OBJ_MTX_SZ 256
+#define ZFS_OBJ_MTX_SZ 4096
kmutex_t *z_hold_mtx; /* znode hold locks */
} zfs_sb_t;
|
|
As much as it hurts to disappoint you, I'm using the Launchpad packages. Building from source should be doable, but it'll take awhile to figure out how to do that. Also, on the mailing list there has been some discussion and investigation that suggests that the source of the memory pressure may be the sudden allocation of many millions of znode and dnode structures, if that information is in any way helpful. |
|
@behlendorf if I'm reading the code and traces correctly, kswapd has called zfs_zinactive(), which is probably trying to get the mutex, while zfs_zget() and company probably hold it and are trying to allocate memory. However, now that I look at the mm code it doesn't look like they're explicitly blocked waiting for kswapd; instead they're down in shrinker code flailing away and apparently not making much progress. (I think @jdavidlists has mentioned on the mailing list that kswapd et al have used a surprisingly large amount of CPU time, which might be a symptom that they're spinning without being able to make much forward progress.) |
|
@siebenmann that would make a lot of sense. To help avoid potential deadlocks like this PF_TRANS is set on all the nfsd threads as they handle requests thereby effectively disabling direct filesystem reclaim. That means we're going to be heavily dependent on kswapd, or another user process, to perform any filesystem reclaim. If kswapd or a different process can't make progress because it ends up blocked on the mutex in zfs_zinactive() that would go a long way towards explaining this. Plus the more znodes we need to evict the more elements in the hash array will need to be locked which makes this more likely. And I'm to go out on a limb and guess that this system sole job is as a nfs server so there's very little else in memory which can be discarded. Another possible easy mitigation might be to increase |
|
All of these servers are indeed NFS and nothing else, with twice-daily "zfs send" backups. The memory requirements of NFS on its own are vanishingly small, so nearly all of the RAM is available to ZFS. Unfortunately we tried zfs_arc_sys_free earlier today and it appeared to make the problem much worse. After setting zfs_arc_sys_free to 2G on these machines, this problem has happened several times. In fact, while I was typing this, it happened to a 16G machine that still had zfs_arc_sys_free set to 2G. The machine showed 750M of RAM available, so it was and was rapidly dying. After setting zfs_arc_sys_free down to 512M the system recovered for a little while. With zfs_arc_sys_free, machines were dying of apparent memory starvation with 1.5-1.8G free RAM. That, plus shrinking zfs_arc_sys_free allowing a gasping machine to recover, does make it seem like ARC that is the limiting factor. CPU usage looks like this: arc_reclaim is at the top by a huge margin (approaching 40% of all the CPU time used since boot after awhile), with arc_prune (1 per CPU) and kswapd trailing distantly behind, but still well ahead of any other single process. (And, when the stack is viewed while kswapd is busy, it's in ARC functions.) What it "feels like" is that 16-20G of RAM is so vastly inadequate for a 1TB pool that after a point it can't evict fast enough to stay afloat. Which seems wrong, and definitely was not the case on 0.6.3. If only we could figure out what changed! |
|
The symptoms you describe apply to me too. Im pretty sure it is related to ARC shrinking. I have a pretty old server and only 4GB ram. whenever I tried to emerge chromium (needs ~3GB to link) it would deadlock and eventually the watchdog would reboot it. A few times I have caught it happening with htop already open (once it starts anything at all that hits the disk will die so only things that need no disk work). arc and txg processes are the top CPU usages for me too. |
|
Just found one of the servers in the about-to-die state. It has arc_sys_free set to default (256M on this 16G machine) and was just sitting with about 170-200M free. This machine happens to run Debian, not Ubuntu, and is using SLAB, not SLUB, so I was able to fire up slabtop and watch what was happening on a per-slab basis without any misattribution. zio_buf_16384 (as a standin for ARC on the list) was at the top of the list, where it belongs, but zfs_znode_cache and dnode_t were right behind it. Basically, zio_buf_16384 was giving up RAM to zfs_znode_cache and dnode_t; it would free trying to get back to 256M and they would slurp up the difference just as fast. As far as I could tell, nothing was evicting dnodes or znodes. Every update their numbers were bigger. If anything was evicting them, it wasn't visible using CPU time in top and it wasn't evicting them fast enough to keep up with the allocation of new ones. As an experiment I tried "echo 3 | sudo tee /proc/sys/vm/drop_caches" (based on a mailing list suggestion from @siebenmann ) and wow, it dropped all the caches. 10 seconds later the machine had 15G of free RAM. Much more effect than I would have guessed. So the next step was to wait for "the problem" to happen in earnest. It soon did, on an Ubuntu machine. On that machine, signal_cache (read: zfs_znode_cache) at about 3.5G had totally eclipsed zio_buf_16384 at 2.9G and dnode_t was right behind it at 2.7G. What we have started taking as the red flag of a "hard" failure is txg_sync using consistent CPU and making no progress. At that point, ZFS writes hang and nfsd's start choking and it's time to reboot. This machine was in that state. Interestingly, writing to /proc/sys/vm/drop_caches at that point also stalled, and no memory was freed. It would have been good to get the stack trace for that to see whether it was inside ARC code, but I panicked and rebooted. The reboot also stalled and the machine had to be reset. So it seems like there may be two separate issues interacting: (1) Unbounded growth / lack of eviction in znode/dnode slabs pushing out ARC until it's too small to function. (The rising CPU usage of reclaim processes being a desperate effort to recycle a smaller and smaller space faster and faster.) So the next step is to try setting up a per-minute cron job on 2-3 of the machines that drops caches if there is <1G free and see if those machines become stable (and intermittently very slow). That should help establish (1). (2) may be trickier unless we can generate the magic stack trace that catches somebody red handed holding a mutex while they're sleeping on the job. Perhaps if we master building from source we'll be able to turn the traces into line numbers. Until we get the source to build to try @behlendorf 's suggestion, is this the right track? |
|
I wish I could comment on a comment: @behlendorf I think EDIT: This follow-up is really meant for whomever is likely to try using this as a "fix". You'll want to use 4096 instead of 4093. |
|
@jdavidlists I think your results from trying to drop all caches during a lockup are what you'd expect, since the mutex deadlock appears to involve locks used when dropping inodes (that's what zfs_zinactive() is trying to do). It's possible that dropping just the page cache (via 'echo 1 >/proc/sys/vm/drop_caches') wouldn't hit the deadlock and would free up enough memory to unlock things; it might be worth trying. On the issue of the mutex deadlock, my view is that we have a strong idea of where it happens, although we don't know why zfs_zget() is stuck in the memory system. Since zfs_zget() et al do IO, it seems like a bad idea in general for them to lock out zfs_zinactive() and thus kswapd for the entire duration of this (potentially quite long) IO et al. My gut says that something has to give here, either zfs_zinactive() abandoning attempts to evict inodes that zfs_zget() is working on or zfs_zget() somehow not locking this stuff for so long and dealing with zfs_zinactive() disappearing stuff while it's running. (The locking code is the same in upstream Illumos, but clearly the Linux and Illumos contexts are different in some way that causes issues in Linux and not in Illumos.) |
|
@dweeezil whoops, your right. I've updated by comment and updated it to 4096. |
Add a zfs_object_mutex_size module option to facilitate resizing the the per-dataset znode mutex array. Increasing this value may help make the deadlock described in openzfs#4106 less common, but this is not a proper fix. This patch is primarily to aid debugging and analysis. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#4106
|
@siebenmann nice summary of the problem. I'm pretty much in agreement with your analysis, but I'd like to add that the reason this doesn't happen under illumos is their memory reclaim behavior is very different. This really isn't a case which can occur for them. The recent ARC restructuring and improvements we pulled in from illumos are also the most likely reason this has become more common. Specifically the code paths fpr how the ARC recycles its buffers have changed. What I'd like to do in figure out a way to eliminate this lock entirely. Realistically that will require some careful thought and can't be done quickly. However, what we can do right now it make it relatively easy to increase ZFS_OBJ_MTX_SZ through a module option. I've proposed pull request #4124 to do this. While this isn't a proper fix it may be a reasonable band-aid for the short term while we address this properly. @jdavidlists if you do end up building from source please apply this patch. Then you will be able adjust the size of this array without needing to rebuild by adjusting the @siebenmann @dweeezil I'd love to hear any thoughts you may have on how we might eliminate these locks entirely. Alternately, I do have some thoughts on how to optimize the current data structure to prevent this but frankly I'd rather just be rid of it if possible! |
Add a zfs_object_mutex_size module option to facilitate resizing the the per-dataset znode mutex array. Increasing this value may help make the deadlock described in openzfs#4106 less common, but this is not a proper fix. This patch is primarily to aid debugging and analysis. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#4106
|
@behlendorf I'm reviewing #3225 and related commits including d6bd8ea to remind myself of all the issues related to Assuming it says, for now, it does seem it could be converted to a hash table or something else to prevent collisions. Also, I wonder if |
|
OK, here's an update. First things first, the per-minute check-and-drop_caches script seems very effective. No crashes on any of the five affected machines since deploying it. It really wasn't too hard to download and build the ZFS & SPL sources on a test Ubuntu VM, including the patch referenced by @behlendorf (which did apply to the 0.6.5-release branch, as I'm not brave enough to touch master). That built and installed and loaded and let me create a zpool, all on the test machine. Of course, the problem only happens on production machines. Is there a relatively straightforward way to set this up to switch back and forth on a production machine so I can swap in the new ones, reboot, try it, and if it blows up, revert back to the official packages? Can I just swap these kernel modules: $ lsmod | fgrep zfs and reboot? The userland tools should be the same, right? The other question is that the kernel is different on the production machine: Linux fs6 3.13.0-73-generic #116-Ubuntu SMP Fri Dec 4 15:31:30 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux versus the test machine: Linux ubu 3.19.0-42-generic #48~14.04.1-Ubuntu SMP Fri Dec 18 10:24:49 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux Despite they are both Ubuntu 14.04.3. If I remember right, installing ZFS from the official packages changed the kernel? Either way does this mean the build must happen on the production server itself (ugh!) to make sure everything is exactly the same? Or will doing the "standard" Ubuntu ZFS setup on the test machine put the same kernel in place and then I can build them offline and move them over? Tricky waters, but I'm making headway! Thanks! |
|
Well, a quick follow-up to my own question, installing the ubuntu-zfs packages on the test machine do not change the kernel. So I'm confused as to why the kernels are not the same on the same version of Ubuntu LTS, but it looks like the test machine somehow pulled in a Vivid backport all by itself. Which was easy enough to deal with, so the hanging question is: Now that the right kernel is there, is building an alternate set of kernel modules on the test server and swapping them in & out (with reboot) on the production server enough to pick up the changes in the patch this is meant to test? Thanks! |
|
@dweeezil as I understand it the locking is there to serialize access to a specific object number during create and destroy. Normally this would be done with a lock in the znode but in the create and destroy case the object doesn't yet exist so it needs some external mechanism for this. My first thought was to leverage the Linux VFS but these same functions are used in ZIL replay which happens without aid of the VFS so that might be a problem. Ideas welcome. @jdavidlists yes, swapping in the new set of kernel modules should be all you need to do. |
|
@behlendorf That was my impression as well; can't lock something when the "thing" doesn't yet exist. I'm kind of surprised illumos is only using a 64-entry array for this. The entries must be locked for very very short periods of time. |
|
@behlendorf OK, your patch is up and running on one server with zfs_object_mutex_size=4096. It sounds like the hope is this will result in lower CPU usage due to less mutex contention (because there are more of them), resulting in lower memory usage due to more effective eviction, resulting in a lack of hangs. Is that about right? Is there anything specific to watch for, other than lack of hangs? |
|
@jdavidlists yes the basic idea is that by increasing the number of mutex it's less likely that you'll run in to contention. Mind you it's still not impossible, just less likely so deadlocks are still possible. But this may mitigate the issue until we can work up a patch to eliminate them entirely. @dweeezil yes I always found that surprising too. But again this is probably far less likely on their platform because of how memory reclaim is handled. As for using |
|
@behlendorf Couldn't we use a |
|
@behlendorf Your patch does seem pretty effective. The one machine it's been applied to is making very efficient use of all available memory (down to 150M - 300M "free"), has not hung, and is using significantly less CPU on all fronts, especially in arc_reclaim. Its memory usage also shows the sort of give-and-take I would expect, with ARC usage going up and down, the data/metadata ratio fluctuating over time, and the znode/dnode slabs not always pegged at 100% and growing. We'll apply this patch to the other Ubuntu machine today then start trying to build it for the Debian servers over the next couple of days. What is it about our workload that is provoking the underlying issue so consistently compared to other uses? Is the large number of nfsd processes creating all the lock contention? (The sort of massive client-side parallelism that necessitates that can't be recreated in our test environment and hence probably why we only see this in production.) It'd be great if we could be more help in developing the real solution, but if I spend a year reading the source code, then I'd probably be caught up enough to understand most of the discussion happening now. :-P In any case, thanks very much for your help and this patch; if anything changes with it or further data emerges as we try it on more machines, I will certainly update this issue. Alternatively, if you come up with a lockless approach or other "permanent" fix you would like us to test, please tag me and I'll see what we can do. These are production servers, but we can try to schedule something for one machine during off hours now that we know how to do source builds. |
|
@jdavidlists I think the summary from @siebenmann above pretty much tells the story. NFS effectively performs a "first open" on every IO request and each of those requests use the relatively small per-filesystem object lock table to be used. I think the more interesting question is why your particular workload seems to be more prone to trigger deadlocks than anyone else's (NFS workload). |
Add a zfs_object_mutex_size module option to facilitate resizing the the per-dataset znode mutex array. Increasing this value may help make the deadlock described in openzfs#4106 less common, but this is not a proper fix. This patch is primarily to aid debugging and analysis. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#4106
|
The test rig is as follows:
Here's a high-level overview based on observing the system: The workload is almost 100% metadata due to the lots of small files. Arc size will quickly overshoot due to metadata and the arc reclaim thread starts trying to free stuff. Eventually, it seems to pick a header with waiters and lead to a deadlock. The reclaim thread can run for quite awhile before this happens and, if the This jives with @jdavidlists observations: He mentioned seeing the arc reclaim thread consume lots of CPU which was a precursor to the deadlock. Unfortunately, it also shows a pathological condition which could occur on any system given a sufficiently large workload. All it takes is enough un-evictable metadata to cause the whole (zfs) system to grind to a halt. In fact, even with the waiter skipping, the system can be live-locked (not everything is in D state) in which the arc reclaim thread runs perpetually trying to free metadata while other zfs tasks are blocked. |
|
@behlendorf This change is staged on all five machines, so as soon as one of them reboots for any reason, it'll take effect. @dweeezil The arcstat for waiter skips will be very interesting. Once implemented, in the livelock scenario, would that number just start skyrocketing? |
|
I ran quite a few stress tests today with the openzfs/spl@08fbc02 and without the waiter skipping code and it seems to be holding up very well. I've not been able to deadlock it a single time, although in this extreme test, the reclaim thread can consume lots of CPU. AFAICT, this is a good fix. |
Add a zfs_object_mutex_size module option to facilitate resizing the the per-dataset znode mutex array. Increasing this value may help make the deadlock described in openzfs#4106 less common, but this is not a proper fix. This patch is primarily to aid debugging and analysis. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#4106
The zfs_znode_hold_enter() / zfs_znode_hold_exit() functions are used to serialize access to a znode and its SA buffer while the object is being created or destroyed. This kind of locking would normally reside in the znode itself but in this case that's impossible because the znode and SA buffer may not yet exist. Therefore the locking is handled externally with an array of mutexs and AVLs trees which contain per-object locks. In zfs_znode_hold_enter() a per-object lock is created as needed, inserted in to the correct AVL tree and finally the per-object lock is held. In zfs_znode_hold_exit() the process is reversed. The per-object lock is released, removed from the AVL tree and destroyed if there are no waiters. This scheme has two important properties: 1) No memory allocations are performed while holding one of the z_hold_locks. This ensures evict(), which can be called from direct memory reclaim, will never block waiting on a z_hold_locks which just happens to have hashed to the same index. 2) All locks used to serialize access to an object are per-object and never shared. This minimizes lock contention without creating a large number of dedicated locks. On the downside it does require znode_lock_t structures to be frequently allocated and freed. However, because these are backed by a kmem cache and very short lived this cost is minimal. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#4106
Add a zfs_object_mutex_size module option to facilitate resizing the the per-dataset znode mutex array. Increasing this value may help make the deadlock described in openzfs#4106 less common, but this is not a proper fix. This patch is primarily to aid debugging and analysis. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#4106
The zfs_znode_hold_enter() / zfs_znode_hold_exit() functions are used to serialize access to a znode and its SA buffer while the object is being created or destroyed. This kind of locking would normally reside in the znode itself but in this case that's impossible because the znode and SA buffer may not yet exist. Therefore the locking is handled externally with an array of mutexs and AVLs trees which contain per-object locks. In zfs_znode_hold_enter() a per-object lock is created as needed, inserted in to the correct AVL tree and finally the per-object lock is held. In zfs_znode_hold_exit() the process is reversed. The per-object lock is released, removed from the AVL tree and destroyed if there are no waiters. This scheme has two important properties: 1) No memory allocations are performed while holding one of the z_hold_locks. This ensures evict(), which can be called from direct memory reclaim, will never block waiting on a z_hold_locks which just happens to have hashed to the same index. 2) All locks used to serialize access to an object are per-object and never shared. This minimizes lock contention without creating a large number of dedicated locks. On the downside it does require znode_lock_t structures to be frequently allocated and freed. However, because these are backed by a kmem cache and very short lived this cost is minimal. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#4106
If a thread is holding mutex when doing cv_destroy, it might end up waiting a thread in cv_wait. The waiter would wake up trying to aquire the same mutex and cause deadlock. We solve this by move the mutex_enter to the bottom of cv_wait, so that the waiter will release the cv first, allowing cv_destroy to succeed and have a chance to free the mutex. This would create race condition on the cv_mutex. We use xchg to set and check it to ensure we won't be harmed by the race. This would result in the cv_mutex debugging becomes best-effort. Also, the change reveals a race, which was unlikely before, where we call mutex_destroy while test threads are still holding the mutex. We use kthread_stop to make sure the threads are exit before mutex_destroy. Signed-off-by: Chunwei Chen <tuxoko@gmail.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Tim Chase <tim@chase2k.com> Issue openzfs/zfs#4166 Issue openzfs/zfs#4106
Add a zfs_object_mutex_size module option to facilitate resizing the the per-dataset znode mutex array. Increasing this value may help make the deadlock described in openzfs#4106 less common, but this is not a proper fix. This patch is primarily to aid debugging and analysis. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Tim Chase <tim@chase2k.com> Issue openzfs#4106
The zfs_znode_hold_enter() / zfs_znode_hold_exit() functions are used to serialize access to a znode and its SA buffer while the object is being created or destroyed. This kind of locking would normally reside in the znode itself but in this case that's impossible because the znode and SA buffer may not yet exist. Therefore the locking is handled externally with an array of mutexs and AVLs trees which contain per-object locks. In zfs_znode_hold_enter() a per-object lock is created as needed, inserted in to the correct AVL tree and finally the per-object lock is held. In zfs_znode_hold_exit() the process is reversed. The per-object lock is released, removed from the AVL tree and destroyed if there are no waiters. This scheme has two important properties: 1) No memory allocations are performed while holding one of the z_hold_locks. This ensures evict(), which can be called from direct memory reclaim, will never block waiting on a z_hold_locks which just happens to have hashed to the same index. 2) All locks used to serialize access to an object are per-object and never shared. This minimizes lock contention without creating a large number of dedicated locks. On the downside it does require znode_lock_t structures to be frequently allocated and freed. However, because these are backed by a kmem cache and very short lived this cost is minimal. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#4106
|
OK, the proposed patches to resolve this issue have been merged to the master branch. The first two prevent a deadlock where c96c36f Fix zsb->z_hold_mtx deadlock The second deadlock uncovered in this issue was resolved by @tuxoko who implemented @grwilson suggestion. Similar to illumos openzfs/spl@e843553 Don't hold mutex until release cv in cv_wait Thanks everyone! |
|
Thanks for all your hard work everyone! It is much appreciated! |
Add a zfs_object_mutex_size module option to facilitate resizing the the per-dataset znode mutex array. Increasing this value may help make the deadlock described in openzfs#4106 less common, but this is not a proper fix. This patch is primarily to aid debugging and analysis. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Tim Chase <tim@chase2k.com> Issue openzfs#4106
The zfs_znode_hold_enter() / zfs_znode_hold_exit() functions are used to serialize access to a znode and its SA buffer while the object is being created or destroyed. This kind of locking would normally reside in the znode itself but in this case that's impossible because the znode and SA buffer may not yet exist. Therefore the locking is handled externally with an array of mutexs and AVLs trees which contain per-object locks. In zfs_znode_hold_enter() a per-object lock is created as needed, inserted in to the correct AVL tree and finally the per-object lock is held. In zfs_znode_hold_exit() the process is reversed. The per-object lock is released, removed from the AVL tree and destroyed if there are no waiters. This scheme has two important properties: 1) No memory allocations are performed while holding one of the z_hold_locks. This ensures evict(), which can be called from direct memory reclaim, will never block waiting on a z_hold_locks which just happens to have hashed to the same index. 2) All locks used to serialize access to an object are per-object and never shared. This minimizes lock contention without creating a large number of dedicated locks. On the downside it does require znode_lock_t structures to be frequently allocated and freed. However, because these are backed by a kmem cache and very short lived this cost is minimal. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#4106
If a thread is holding mutex when doing cv_destroy, it might end up waiting a thread in cv_wait. The waiter would wake up trying to aquire the same mutex and cause deadlock. We solve this by move the mutex_enter to the bottom of cv_wait, so that the waiter will release the cv first, allowing cv_destroy to succeed and have a chance to free the mutex. This would create race condition on the cv_mutex. We use xchg to set and check it to ensure we won't be harmed by the race. This would result in the cv_mutex debugging becomes best-effort. Also, the change reveals a race, which was unlikely before, where we call mutex_destroy while test threads are still holding the mutex. We use kthread_stop to make sure the threads are exit before mutex_destroy. Signed-off-by: Chunwei Chen <tuxoko@gmail.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Tim Chase <tim@chase2k.com> Issue openzfs/zfs#4166 Issue openzfs/zfs#4106
Add a zfs_object_mutex_size module option to facilitate resizing the the per-dataset znode mutex array. Increasing this value may help make the deadlock described in #4106 less common, but this is not a proper fix. This patch is primarily to aid debugging and analysis. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Tim Chase <tim@chase2k.com> Issue #4106
The zfs_znode_hold_enter() / zfs_znode_hold_exit() functions are used to serialize access to a znode and its SA buffer while the object is being created or destroyed. This kind of locking would normally reside in the znode itself but in this case that's impossible because the znode and SA buffer may not yet exist. Therefore the locking is handled externally with an array of mutexs and AVLs trees which contain per-object locks. In zfs_znode_hold_enter() a per-object lock is created as needed, inserted in to the correct AVL tree and finally the per-object lock is held. In zfs_znode_hold_exit() the process is reversed. The per-object lock is released, removed from the AVL tree and destroyed if there are no waiters. This scheme has two important properties: 1) No memory allocations are performed while holding one of the z_hold_locks. This ensures evict(), which can be called from direct memory reclaim, will never block waiting on a z_hold_locks which just happens to have hashed to the same index. 2) All locks used to serialize access to an object are per-object and never shared. This minimizes lock contention without creating a large number of dedicated locks. On the downside it does require znode_lock_t structures to be frequently allocated and freed. However, because these are backed by a kmem cache and very short lived this cost is minimal. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue #4106
If a thread is holding mutex when doing cv_destroy, it might end up waiting a thread in cv_wait. The waiter would wake up trying to aquire the same mutex and cause deadlock. We solve this by move the mutex_enter to the bottom of cv_wait, so that the waiter will release the cv first, allowing cv_destroy to succeed and have a chance to free the mutex. This would create race condition on the cv_mutex. We use xchg to set and check it to ensure we won't be harmed by the race. This would result in the cv_mutex debugging becomes best-effort. Also, the change reveals a race, which was unlikely before, where we call mutex_destroy while test threads are still holding the mutex. We use kthread_stop to make sure the threads are exit before mutex_destroy. Signed-off-by: Chunwei Chen <tuxoko@gmail.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Tim Chase <tim@chase2k.com> Issue openzfs/zfs#4166 Issue openzfs/zfs#4106
Add a zfs_object_mutex_size module option to facilitate resizing the the per-dataset znode mutex array. Increasing this value may help make the deadlock described in openzfs#4106 less common, but this is not a proper fix. This patch is primarily to aid debugging and analysis. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Tim Chase <tim@chase2k.com> Issue openzfs#4106
The zfs_znode_hold_enter() / zfs_znode_hold_exit() functions are used to serialize access to a znode and its SA buffer while the object is being created or destroyed. This kind of locking would normally reside in the znode itself but in this case that's impossible because the znode and SA buffer may not yet exist. Therefore the locking is handled externally with an array of mutexs and AVLs trees which contain per-object locks. In zfs_znode_hold_enter() a per-object lock is created as needed, inserted in to the correct AVL tree and finally the per-object lock is held. In zfs_znode_hold_exit() the process is reversed. The per-object lock is released, removed from the AVL tree and destroyed if there are no waiters. This scheme has two important properties: 1) No memory allocations are performed while holding one of the z_hold_locks. This ensures evict(), which can be called from direct memory reclaim, will never block waiting on a z_hold_locks which just happens to have hashed to the same index. 2) All locks used to serialize access to an object are per-object and never shared. This minimizes lock contention without creating a large number of dedicated locks. On the downside it does require znode_lock_t structures to be frequently allocated and freed. However, because these are backed by a kmem cache and very short lived this cost is minimal. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#4106
After upgrading several of our NFS fileservers to zfs 0.6.5.3-1~trusty on Ubuntu 14.04.3 LTS with kernel 3.13.0-71-generic, a combination which did fine in all testing, we are suddenly seeing hangs in the ZFS code that cascade down to NFS and render the whole server unresponsive.
Once this happens, any access to the pool -- read or write -- will hang. iostat shows no disk activity of any kind on the zpool devices.
This has happened on multiple servers. All hardware is happy and healthy and tests OK, and prior to this update, everything was glassy smooth for months.
This problem most frequently happens while there's a "zfs send" in progress, but it can happen at other times as well, and not every send provokes it. Overall, the busier a server is, the more likely it is to happen.
Here are some stack traces for stuck processes in dmesg:
A z_wr_int:
A txg_sync:
An nfsd process (these form the bulk of any hang as that's what these servers do):
And, from a different incident, kswapd:
What these always seem to have in common is that they're almost always spinning on a mutex (the txg_sync trace above being a rare exception). So far, we haven't been able to identify what process is holding/losing the mutex.
What can we do to help resolve this?
The text was updated successfully, but these errors were encountered: