txg_sync, z_null_iss and txg_quiesce freezes #3409
Comments
|
Please also post some information on your harddrive setup (mirror ? raidz ? SSD ? even LVM ?), what models ? NCQ enabled/working ? Memory is obviously 8 GiB. Intel ? AMD ? NO swap NO L2arc since you're posting on paste.ubuntu.com - it's ubuntu ? (or according to your profile - Gentoo =) ) what kernel version ? ~amd64 ? (64bit ?) |
|
I'm doing a raidz consisting of 4tb (3 x 1tb + 2 x 0.5tb). No SSDs or LVM. NCQ enabled ( Correct, no swap enabled. According go edit: 64 bit gentoo I used the ubuntu pastebin because of a 512kb limit at pastebin.
I have also tried Linux 3.17 and the issue is still there. I'm using them compiled into kernel - not as modules. |
|
|
|
How full is the pool ? that should also show fragmentation |
|
Sometime between the April 23rd commits and May13th, something has happened that has made my issue even worse and I am now seeing signs like @hallabro is mentioning. I compiled ZFS & SPL from source last night and when my backups run I end up with My machine has 16GB of RAM, but it didn't exceed 5.9GB of usage. Running My ARC graphs are currently showing ~129K cache hits/sec and 18K misses/sec while the ARC is nearly empty. Once again, the system is a Dell R515 with 16GB of RAM. The pool is a single raidz2 pool made of of 7 2TB SATA drives. Compression is enabled and set to lz4. atime is off. The OS is Fedora 20 with the 3.17.8-200.fc20.x86_64 kernel. I've uploaded debugging information as requested in #3303 and #3235 to https://cloud.passcal.nmt.edu/index.php/s/gHacCaxkl8VCI6z. It is currently still running if you need me to do anything else. |
|
@angstymeat A shot in the dark: check |
|
@angstymeat Another idea: |
|
zpool events doesn't list
|
|
@angstymeat I figured the events were a longshot. In any case, the txg history output quite clearly shows the problem. The txgs are performing exactly zero IO and are occurring about every 150 microseconds. Ouch! No wonder the system has recorded over 5 billion txgs. |
|
@angstymeat You didn't by chance change the value of EDIT: Maybe |
|
No, all of my module parameters should be the defaults.
|
|
@angstymeat Just to be sure, could you please post all of them: |
|
And |
|
The module parameters are: and |
|
@angstymeat Very interesting. The system appears to have backed itself into a corner rather nicely by beating |
|
I've seen this same thing once before actually. We should probably give some thought on how to tweak that dmu_tx_dirty_throttle case so this can't happen. |
|
I can reproduce this in about 10 minutes. I've just changed |
|
@dweeezil other OpenZFS platforms might not be able to trigger this as easily because they never allow |
|
@behlendorf Agreed, it was more of a hypothetical assertion. We should certainly investigate some safeguards in the throttle to prevent this from happening. |
|
I'm wondering if I might have done something stupid and wasted your time with this. Last night I played around with applying #3308 and after I started my backup I saw the same thing I reported today, with Now, I'm wondering if I made a mistake and missed making a new initramfs after I recompiled so I was still running the code with #3308 applied. Currently, everything appears to be working normally, neglecting my issues with #3303. Would changing |
|
@angstymeat yes, setting |
|
@angstymeat The latest version of #3308 would effectively be a no-op so it shouldn't be entering into the equation. Raising |
|
Ok. I've tried it again with I'm going to run it with I will say that it seems like my memory is filling a lot slower with the commits from the last week or so in place. |
|
Running for about 75 minutes and all of a sudden |
|
The rsync process that was running the mail backup died with a socket error from hanging for so long, and the other processes dropped back down to normal CPU times. |
|
I'm running an unconventional setup, but I think I've just encountered this same issue. I've run through all of the diagnostic steps here with similar results, so I'm now rebooting with zfs_arc_min set to 4G. I use dedup & compression on one of three zpools, but this issue caused all IO on all pools to stall. (Interestingly, I could ls /pool and see a directory, but ls /pool/folder stalled -- presumably the former was cached). zfs iostat 1 also listed a small amount writes (but no reads) to one of the pools at probably an average of 1-2 MB/s -- but nothing on the system should have been causing them. (The unconventional setup in use is essentially: zpools /local and /remote on separate virtual disks, zvol /local/inner, zpool /storage with compress+dedup on the zvol, periodic zfs send/recv of snapshots between /local and /remote. The exact semantics and reasoning is likely unimportant for this issue, but I can provide it upon request.) |
|
@dewiniaid in the end - @angstymeat 's issues were fixed by running a fairly up-to-date zfs stack and an NUMA-aware per-superblock shrinker ( 90947b2 ) you're by chance running a NUMA setup ? |
|
All the related fixes have been merged to master if you want to try that. |
|
It turns out that I was running a NUMA setup within the VM -- which is odd, because the actual physical host only has one of the two sockets actually filled. Not entirely certain how that happened. That may explain some messaging I saw about possible memory deadlocks then. I just reconfigured the VM to not use NUMA (it shouldn't be anyways!), so I'll see if that improves things. (I'm running off the Ubuntu PPA currently -- which looks like it's about 2 months behind in terms of last being updated.) |
|
i think we hit the same issue here.
cat /etc/redhat-releaseCentOS Linux release 7.2.1511 (Core) rpm -q -a|grep -i zfszfs-0.6.5.8-1.el7.centos.x86_64 top - 10:52:58 up 7 days, 21:57, 2 users, load average: 12,35, 12,43, 11,60 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND zpool list -vNAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT zpool status -vpool: zfspool errors: No known data errors |
|
slabinfo http://pastebin.com/31ye2RUT cat /proc/spl/kstat/zfs/dmu_tx |
|
don`t know if this helps, but in #zfsonlinux i was told to collect some data with "perf record" , so i issued "perf record -ags". can provide datafile if this helps |
|
the issue starts to appear more and more often here, so i`d call this a real bummer. i think zfs is not "production ready" with this, as regular reboots being needed to resolve this. echo 1|3 >/proc/sys/vm/drop_caches does not work, btw - at least not when this issue is already occuring. echoing to that procfs file on my system hangs and never returns. i set zfs_arc_min to 1GB now and a background task with echo 3 >drop_caches ever five minutes - to see if this makes a difference NOTE: the above tuning seems to have cured the problem |
|
I'm pretty sure I just walked into this as well. My system started running low on memory, txg_sync, and txg_quiesce were both pegged at 50-60% (8 CPU system) for 40-50 minutes. I tried drop_caches a couple of times to no noticeable effect, but I noticed my arc size had dropped to 437M and the target size was down to 138M. Setting zfs_arc_min to 1G brought things back to life almost instantaneously. |
|
does that mean you can interactively "heal" the problem when it`s occurring just by setting zfs_arc_min via sysfs? |
|
I only have one data point at the moment, but in my case yes. My load average was over 100 when I did it and it immediately started dropping back. The system was serving a lot of interactive users when it started running into problems so might be a little bit hard for me to reproduce. Basically I think new user processes had chewed through all of the available system memory (I confirmed this) and so ZFS started squashing down the ARC to very small values. It then ended up working incredible hard trying to work within the memory constraint. I can't reproduce the exact conditions on demand, but I have another system running the same versions of everything and I'll try making a little bit of IO load then start claiming as much system memory as I can and see what happens. |
|
Considering the background, why don't we raise the minimum size for ARC (zfs_arc_min) to at least 1 GiB when available RAM equals 3 or more RAM ? (e.g. @devZer0 's RAM size is 3.5 GiB) Smaller systems might need more careful adjustment of that setting ... the comments I collected regarding this and similar issue was: |
|
Thanks for the recommendation, worked for me: Added to the file Before you copy the parameters, the server has 16GB of RAM |
|
Ideally we want to make the default minimum as small as possible, but no smaller. This ensures the system can reclaim as much memory as possible when it's absolutely needed. And it makes it possible to use ZFS on small low memory systems. @ianabc @joelzamboni if you have a test system it would be great if you could try increasing |
|
I have some test systems, but I'll need to think about how to reproduce the
load which preceded the problem. The system I was working with had 32GB of
memory, but I have some smaller systems I should be able to play with.
…On Thu, Mar 2, 2017 at 2:36 PM, Brian Behlendorf ***@***.***> wrote:
Ideally we want to make the default minimum as small as possible, but no
smaller. This ensures the system can reclaim as much memory as possible
when it's absolutely needed. And it makes it possible to use ZFS on small
low memory systems.
@ianabc <https://github.com/ianabc> @joelzamboni
<https://github.com/joelzamboni> if you have a test system it would be
great if you could try increasing zfs_arc_min to say 64M or 128M and see
if that's enough to resolve the issue.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3409 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABvy2L8E79-8y6pH1INGeb32lN09D5oyks5rh0SEgaJpZM4EZPB9>
.
|
|
I reproduced the problem when I tried to migrate to zfs and started copying images to zvol on Gentoo (kernel 4.9, zfs 0.7.0). Server has 16 GiB of memory and it chokes after 22GiB of data on 750 GiB partition. Increasing limits and periodically dropping cache seems to work but the latter is annoying. Most annoying is that system seems idle - no disk/cpu activity beyond background radiation level, swap empty etc. FWIW clearing pagecache seems to be the most important thing... |
|
@uzytkownik could you try setting the module option |
|
@behlendorf No, at least not during operation is running. I'm rebooting the system to check. I have an additional data point - I have zfs on lvm on mdraid. When I tried to pvmove an unrelated partition but it got stuck/hanged as well. Is it possible that Linux scheduler is in some weird state for whatever reason? |
|
No. I run into the same problem after reboot. |
|
I could reproduce this issue on my system, too. Setting zfs_arc_min to 1G resolves the problem. Even during a txg_sync freeze and stuck rsync this immediately fixed it. |
|
I experienced the same issue on a large (multi-TB) zfs send/recv to mirror my pool to a new pool featuring the new native encryption. Setting min arc size per comments above immediately unblocked I/O and CPU. |
|
@cytrinox @stewartadam would it be possible for either of you to experiment a little to determine if smaller values of |
|
I'm in the middle of a multi-TB send/recv and just changed the zfs_arc_min from 1G to 128M and it's been progressing fine for the last ~4 hours... but not sure if having it prior set to 1G affects the results. |
|
Just experienced the same issue running zts on a VM (vanilla zfs 0.7.1 from the F26 repos). Froze up on test |
|
Looks like the original issue was fixed in 1b8951b3 so in v0.7.0-rc1 |
|
I suspect that I'm experiencing same or very similar problem on module v0.7.3-1, ZFS pool version 5000, ZFS filesystem version 5. As soon as I hit rsync copy or VM backup system load shoots up so high that It will froze whole machine, I will get txg_sync (and more) blocked for more than 120s and copy speed will fall down to KB/s. Sometimes I have to do hard reboot to even get server back, sometimes it responds in short bursts to console. This problem started to appear slowly few months ago when I noticed occasionally slowdowns during copy and now (from week ago) I can't copy anything from pool without freezing server. I already tested copy/backups with all VMs turn of so It's not caused by load from them. Also sometimes txg_sync hang messages will pop up also during totally quiet period when server is not doing anything which confuses me most. Thanks to anybody who could try to help as I'm right now in very bad situation with backups. EDIT: task trace: |
|
Hey, just run into exactly the same issue, I'm just doing a zfs receive on one side:
and on the other side:
It happened reproducable twice with the same snapshot after about 39 GB (35 actually on receiving side) txg_sync, txg_quiesce and receive_writer are all running at 100%,60%,50 % cpu load respectivly. The pool is created with ashift=12 , it's running on a single SSD because of the compression feature. it's creating a lot of transactions and nothing is moving forward. Call Trace: ZFS 0.7.8 The pool was created with 4.9.31 and ZFS 0.7.0-rc4 -> I will now try to recreate it with 4.12.5 and 0.7.8 Drop_caches or increading arc_min did nothing Update: this seems to be somehow connected to the snapshot - or the target disk , because I have already copied this one snapshot multiple times to other servers without an issue - same kernel and same zfs version! Update 2: also produceable with kernel 4.12.5 and 4.16.3 and zfs 7.8.0 I now watched very closly, first it starts to print out 0 KB/s for some time, then there's quick bursts of ~600 MB/s and then again 0 MB/s, while zpool iostat shows ~30 MB/s until it reaches the 35 GB mark -> then it's dead ... Interestingly I can get over this part with the resume (zfs -t xxx send) functionality - that one stalled at ~41 GB transferred Update 3: now tried without the -c and -p flags -> ran through like it should, could there be a bug in that code part? |
|
Given this goes back to 2015 and I've hit what I think is it in 0.7.8, can anyone else confirm this was ever resolved or are we all doing the same dumb thing to trigger it? The default zfs_arc_min appears to still be 0 (if that was the cause?). |
|
@behlendorf I'm about to have to do a large transfer again so will likely repro this. Is there any data you'd like me to collect during such a hang, or should I just test for the min size that unblocks the transfer? |
|
This issue appears to have suffered sufficient creep that it might be better to close it and to re-open more specific PRs for anything still outstanding. The original issues appeared to have been related to ARC collapse and were apparently solved by increasing @jkalousek The non-zero I'll note that as of the 0.7.2 release, the minimum ARC size was changed from 32MiB to the larger of 32MiB or 1/32 the system memory. |
and others
I have an issue very similiar to #3235 where txg_sync, z_null_iss/0 and txg_quiesce are stuck and consuming CPU. Any processes trying to access some files inside the filesystem will get stuck as well.
I cannot restart the system but have to turn the power off.
Some information and stack traces:
slabinfohttp://pastebin.com/JLyqJKJAkmem/slabhttp://pastebin.com/sRY3BWQnarcstatshttp://pastebin.com/T6tm8702meminfohttp://pastebin.com/qDrDipkVSysRq-thttp://paste.ubuntu.com/11109410/SysRq-whttp://pastebin.com/qn8qqKiBWorth mentioning is also that I do use dmcrypt on the block devices, in case it makes any difference.
The freeze occured shortly after firing up rtorrent to start seeding a torrent.
I have no problems with "normal" IO (moving and opening files), however.
The text was updated successfully, but these errors were encountered: