unable to handle kernel paging request ... in zio_compress_data #8600
Comments
|
As a workaround, does disabling ABD result in stable operation? |
|
@richardelling thank you for your feedback. |
|
This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions. |
|
We have seen the same problem on a few hosts over the past couple of weeks on zfs 0.8.4: |
|
Fwiw, we have crash dumps, and we did some trawling around the stack to find the Re setting In the below case, dereferencing |
|
@cperl82 Do you still have the crash dump? Can you please do |
|
Yes, I have 4 different crash dumps saved, all seem to be the same issue (and, I expect before too long, I'll have some more). |
|
Do you have any recommendations for any settings/tunables that would assist with debugging this issue? |
|
It would be great if small reproducer existed... From the abd above and the fact the address is no longer mapped, I assume there is some kind of race causing use after free (this abd only borrows the buffer). From the crash dumps, one can inspect the zio object, and try to get to the owner: |
|
Something common to the crashes we see here, and in another past situation is the use of RAIDZ2 on top of a perc RAID controller in with each disk a single-disk RAID 0. Is there any known frailty perhaps with RAIDZ2 and the code paths referenced in these stacks? I am going to take another look at the dumps and see if I can answer your question about the zio_t owner. |
|
Sorry I haven't had time to come back to this. I still have the crash dump I was referencing earlier, so I'll use that for continuity. But, we've also experienced the same crash on zfs-2.0.3 (we were hoping we could just upgrade our way out of this). I had to do some more trawling around on the stack to find the zio_t, but I think I got it, as evidenced by the fact that it's So, following your example to get to the Obviously Any thoughts about what else to look at? |
|
Unsure if this is helpful or not... 0xdead000000000200 = LIST_POISON2 e.g. in Linux node->prev will be LIST_POISON2 instead of NULL |
|
So that (LIST_POISON2) seems to bolster the use after free argument. FWIW, when I did bt -a on the vmcore, I noticed an identical stack to this crashing one on another CPU, and %rdi pointed to the same (unmapped) address that caused the kernel page fault exception, which made me wonder if there might be a race between two threads referencing the same buffer. (%rdi is presumably the abd_iter.iter_mapaddr passed as a the first param to zio_compress_zeroed_cb in both cases. ) |
|
Actually, I think my last comment was slightly wrong. I don't believe this zio came through |
|
I just wanted to check-in and see if the data provided by cperl82 is sufficient @ironMann If not, please let us know what other data we can provide. |
|
@msd-nn I was just curious where you ended up with this issue. Did you find a resolution? |
|
@behlendorf Hi Brian, I was hoping I might be able to get your take on this. We are seeing these panics, repeatedly, every few weeks on our production Postgreql boxes, which use raidz2. Is there enough data for you guys to analyze here? If not, I'd love to try to get you whatever you need or help in any way I can. |
|
Looking at the code, not really being familiar with it, I get a feeling it might be related to prematurely releasing a lwb / some kind of race with the ZIL during dmu_sync, but I am probably talking out of school here. If I followed the dump properly, it looks like we are in LWB_STATE_ISSUED for whatever that's worth. |
|
Is there any hope of this issue being addressed? We had to stop using ZFS in production for a major part of our infrastructure due to the instability, which is really a shame and something I'd like to rectify. If you need something from me, I am more than happy to assist with fixing the issue. |
i can answer instead: we didn't have this problem in the last months fortunately, instead we had multiple appearences of #9741 (which seems to have been fixed now upstream, but maybe we can't benefit as we are not going to migrate to 2.x releases anytime soon and probably 0.8.x is kind of EOL) |
|
I find a very similar stack on a different core in this crash dump, with a zio at a separate address than the zio in the crashed thread and an zio.io_abd at a different address who shares the same abd->abd_u.abd_linear.abd_buf address that is unexpectedly unmapped in the crashed thread. That appears to be the same abd_linear->abd_buf address used by the two separate processes which is the unmapped address causing the panic. Is this expected, due to the borrowing of the buffer that you mentioned? I interpreted it at the zio would borrow the abd buffer, but wasn't sure if it's expected that two separate abds would be referencing the same linear buffer. I wondered if it's perhaps a locking issue. @ironMann |
Let me ask you, on the panicking systems, were you using Postgresql? If so, what version please? |
i can't tell 100%, but as far as i know there are only backups, no actual application working directly on these ZFS Volumes. btw, my statement about not moving to OpenZFS 2.x soon is wrong now, as 2.x started to appear in the repository, we decided to give it a go |
|
I may have just reproduced this deliberately on 2.1.2. I was running fsstress from LTP on a pool while a pair of scripts in a loop in the BG changed recordsize between 128k and 1M and compression between lz4 and zstd-3 at random intervals up to 30 seconds apart, trying to reproduce a different bug, and then it went bang. |
|
Very interesting! Any sense of whether it corresponds to the recordsize change or the compression change? It might be interesting to see if you can repro consistently while using only one of those changes. |
|
Absolutely no data at the moment, but when I'm not trying to polish off a couple things, I'll come back to it and see. (Well, that and I'm waiting for my kASAN kernel package to finish building for the VM.) Simple enough to reproduce, though, at least in my n=1 experiment. Watch I come back to it and it never happens again. (I also have a nice core dump laying around, though in my recent experience crash has a bad habit of obeying its name on loading dumps.) |
|
I've catched the same issue a day ago, not sure what useful info I can provide, but so far: System info: Kernel: ZFS: zpool: syslog (it resides on non-zfs volume): |
|
and happened for me again, on the same system: |
|
Happened for me too. |
|
happens quite often, I don't know exact case to repeat it, but I need to |
|
Update for others seeking solution: after upgrade to Ubuntu 22.04, things are rather stable - I probably seen issue of this sort once, but not even sure about once - server keeps running fine, under the same type of load (MySQL DB replica). kernel zfs |
System information
Describe the problem you're observing
After updating from zfs 0.6.5.9 to 0.7.12, our system became unstable and crashed about once per day. See example kernel logs below.
The crashes happened during high write IO. We tried an update to current zfs 0.7.13 which did not solve the problem.
Reverting to zfs 0.6.5.11 (and kernel 3.10.0-693.21.1.el7.x86_64 because of known incompatibility of zfs 0.6.5.x to recent RHEL kernels), the system became stable again. We can exclude the different kernel version as a reason, because we also tried an older kernel together with zfs 0.7.13 and saw the same crashes happen.
In all cases, zio_compress_data appears as on of the last function calls in the kernel trace. "unable to handle kernel paging request" is raised either from
or
so we guess that the crashes are caused by dereferencing of invalid pointers in connection with scattered ABD. (Scattered) ABDs did not exist in 0.6.5.x.
Interestingly, we operate a second server with identical hardware and zpool configuration. On this server, we upgraded to zfs 0.7.12 without problems (uptime is 32 days and > 300 TiB were written to that zpool without problems since). The two servers differ in data stored on the servers (backups/snapshots of Oracle and/or SAP HANA databases written via NFS in both cases), the problematic server has slightly higher degree of fragmentation and more zfs sub-volumes.
Some hardware info:
Single socket Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
128 GB of ECC RAM
zpool consists of storage LUNs from a storage appliance, 128TiB per LUN, attached via multipath FC
4 SAS SSDs as ZIL and L2ARC
some zpool infos
the "crashing" one:
the "good" one:
We use lz4 compression on both pools, compression ratio is 2.5x in average.
Describe how to reproduce the problem
Heavy write IO on a large zpool using zfs 0.7.x
Include any warning/errors/backtraces from the system logs
example 1 (crash in zio_compress_zeroed_cb)
example 2 (crash in lz4_compress_zfs):
The text was updated successfully, but these errors were encountered: