Consistent General Protection Fault; zfs_rangelock_enter() -> avl_nearest() -> avl_walk() #10593
Comments
|
I can report a similar problem, also with Plex in a Proxmox 6.2 container. ECC RAM, healthy mirrored pool, etc. [386809.776800] CPU: 2 PID: 20182 Comm: Plex Media Serv Tainted: P O 5.4.44-2-pve #1 |
|
Also seeing this issue, but on Ubuntu 18.04.5 LTS with 0.8.4. I also had this issue with 0.8.3, but did not have it with 7.x. I also have Plex running as well and while the effects aren't as bad (stopping containers, etc) it does effectively zombie the Plex server. |
|
I've seen these same stack traces in Ubuntu 16.04 (zfs 0.7.13) and in a Debian 10 lxc container (Debian 10 host/zfs 0.8.4). |
|
Also seeing this inside a container running on proxmox (running Plex): |
|
I'm going to try to move plex to a VM (with NFS) rather than an lxc container to see if it resolves the issue. My guess is this is related to ZFS usage inside an LXC container. Plex probably just is accessing the filesystem in such a way as to trigger it more consistently. |
|
Your comment reminded me to post my workaround... As indicated by the stacktraces, this appears to be an issue in the ZFS fsync() implementation. Disabling ZFS synchronous writes for the affected dataset (i.e. "zfs set sync=disabled somedataset") resolves the issue. I've been running this way for a month without the issue reappearing. I presume the primary Plex usage for fsync() is its SQLite databases, so doing this undermines Plex's data consistency expectations, and isn't something to be done lightly. In my case, its an isolated dataset just for Plex and I'm not concerned about Plex corruption or even complete data loss. |
|
I wonder if this related to this: #10440 ? If so, would that mean that this is an issue with zfs and SQLite in general? |
|
Could be. It may not really be sqlite specific, but just that the I/O
pattern sqlite generally has in use is triggering it. Which means it could
happen from other apps / access patterns as well. Thanks for linking
though...that issue seems to have more investigation into the root cause
and work that's happening on it.
…On Fri, Oct 9, 2020 at 11:56 AM 4oo4 ***@***.***> wrote:
I wonder if this related to this: #10440
<#10440> ?
If so, would that mean that this is an issue with zfs and SQLite in
general?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#10593 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAQ2PCW4CMW6JHRUAQY5FP3SJ4XD3ANCNFSM4O7PXVHA>
.
|
|
I am still having this problem in kernels 5.8 and 5.9. Setting sync=disable does not solve the issue for me. Did anyone found a workaround? Thanks |
|
For me, moving Plex to a VM instead of a container fixed it. I haven't seen the issue since. All of Plex's data (sqlite and such) I'm using inside the container's filesystem (which is ironic since that virtual disk is a ZFS volume...so you'd think it would have the same issues). The actual video is NFS shared into the container. My guess is that the problem has something to do with ZFS being used by a containerized app with a fast R/W pattern. Not sure why, but there's that. :) |
|
I can provide a data point on the container aspect -- my Plex instance is running outside of a container and exhibits the issue at hand. |
|
Interesting. Perhaps my using a ZFS volume vs a dataset is a factor then? Or, coincidence I guess. :) |
|
If you wanted to test whether zvol is the issue @bdaroz you could make a zvol with, e.g. ext4 on the volume, then use that ext4 filesystem for plex data. If that still exhibits the issue...then I guess it's not a zvol vs dataset issue. If that fixes it...then there's another data point that points to the issue being dataset specific. I don't know the ZFS code base at all so am not sure if that stack trace is more general or specific to datasets vs volumes. |
|
Actually, poking around in the code more (though I am DEFINITELY not well versed with it), this looks like it's dealing with any commit through the ZIL. I believe the zil is used for zvols like it is for datasets (write ahead log for synchronous writes). So, if there is any difference in behavior then it's possibly due to synchronous writes in the filesystem on top of the zvol not triggering triggering a corresponding synchronous write on the zvol (or some change in access pattern at least). Regardless of all that...the actual GP fault is in That file's had a lot of changes since the 0.8.4 release: https://github.com/openzfs/zfs/commits/master/module/avl/avl.c. So...maybe there has been a fix since this bug was filed and it's just not released. This is all speculation though. :) |
|
Oh. The newer code references here: https://illumos.org/man/9f/avl. Which explicitly calls out: So, yeah, maybe mt related if the locking isn't properly protecting the tree. So, that might indicate an issue in code related to locking with |
|
I recently hit very similar issue. The way it happened seems to be like this. Line 801 in 85703f6 When there's two callers doing range lock on the same offset with size 0. Line 415 in 85703f6 This uninitialized Line 326 in 85703f6 Line 342 in 85703f6 |
|
If anyone here can repro this consistently, can you please try this patch. |
|
hi, i have the same issue(*), i'm running the ubuntu-tree in Version 0.8.4 (1ubuntu11). What Version is your patch for? Are you able to generate a patch against 0.8.4? regards, (*) Not really the same, i'm using a zfs set for a medium plex installation, which contains tons of metadata, which is regulary scanned and updated. Thr Stacktrace is the same as this bug. |
|
@fbuescher |
|
does not work. I'm usually using the build from https://launchpad.net/ubuntu/+source/zfs-linux. dpkg-source: info: extracting zfs-linux in zfs-linux-0.8.4 after that, i applied your patch (worked without any fuzzy). compiled, installed, rebooted. after zpool import -al, gives the following error: This pool uses the following feature(s) not supported by this system: removed your patch, recompiled, installed, rebootet, everything worked as before. |
|
@tuxoko I'm testing the patch now, on 0.8.6 on Debian Buster. Unfortunately it only occurs randomly for me, so I'll just try to keep an eye on it to see if it recurs. Cheers |
|
@4oo4 @fbuescher |
|
I'm trying to test your patch with /var/lib/dkms/zfs/0.8.6/build/module/zfs/zfs_vnops.c: In function ‘zfs_get_data’:
/var/lib/dkms/zfs/0.8.6/build/module/zfs/zfs_vnops.c:1068:3: error: implicit declaration of function ‘zfs_zrele_async’; did you mean ‘zfs_iput_async’? [-Werror=implicit-function-declaration]
zfs_zrele_async(zp);
^~~~~~~~~~~~~~~
zfs_iput_async
/var/lib/dkms/zfs/0.8.6/build/module/zfs/zfs_vnops.c:1073:40: warning: format ‘%lu’ expects argument of type ‘long unsigned int’, but argument 3 has type ‘uint64_t’ {aka ‘long long unsigned int’} [-Wformat=]
printk("%s: gen mismatch, expected=%lu got=%lu\n", __func__, gen, zp_gen);
~~^ ~~~
%llu
/var/lib/dkms/zfs/0.8.6/build/module/zfs/zfs_vnops.c:1073:48: warning: format ‘%lu’ expects argument of type ‘long unsigned int’, but argument 4 has type ‘uint64_t’ {aka ‘long long unsigned int’} [-Wformat=]
printk("%s: gen mismatch, expected=%lu got=%lu\n", __func__, gen, zp_gen);
~~^ ~~~~~~
%llu
CC [M] /var/lib/dkms/zfs/0.8.6/build/module/zfs/zio_checksum.o
CC [M] /var/lib/dkms/zfs/0.8.6/build/module/zfs/zio_compress.o
CC [M] /var/lib/dkms/zfs/0.8.6/build/module/zfs/zio_crypt.o
CC [M] /var/lib/dkms/zfs/0.8.6/build/module/zfs/zio_inject.o
CC [M] /var/lib/dkms/zfs/0.8.6/build/module/zfs/zle.o
cc1: some warnings being treated as errors
make[7]: *** [/usr/src/linux-headers-4.19.0-13-common/scripts/Makefile.build:308: /var/lib/dkms/zfs/0.8.6/build/module/zfs/zfs_vnops.o] Error 1
make[7]: *** Waiting for unfinished jobs....
make[6]: *** [/usr/src/linux-headers-4.19.0-13-common/scripts/Makefile.build:549: /var/lib/dkms/zfs/0.8.6/build/module/zfs] Error 2
make[5]: *** [/usr/src/linux-headers-4.19.0-13-common/Makefile:1565: _module_/var/lib/dkms/zfs/0.8.6/build/module] Error 2
make[4]: *** [Makefile:146: sub-make] Error 2
make[3]: *** [Makefile:8: all] Error 2
make[3]: Leaving directory '/usr/src/linux-headers-4.19.0-13-amd64'
make[2]: *** [Makefile:30: modules] Error 2
make[2]: Leaving directory '/var/lib/dkms/zfs/0.8.6/build/module'
make[1]: *** [Makefile:843: all-recursive] Error 1
make[1]: Leaving directory '/var/lib/dkms/zfs/0.8.6/build'
make: *** [Makefile:712: all] Error 2Cheers |
|
@4oo4 |
|
@tuxoko Oh weird, I never really thought about that, that is interesting that gcc can do that. After correcting that, I'm still getting some dkms build errors, but different ones this time: /var/lib/dkms/zfs/0.8.6/build/module/zfs/zfs_vnops.c: In function ‘zfs_get_data’:
/var/lib/dkms/zfs/0.8.6/build/module/zfs/zfs_vnops.c:1068:18: error: passing argument 1 of ‘zfs_iput_async’ from incompatible pointer type [-Werror=incompatible-pointer-types]
zfs_iput_async(zp);
^~
/var/lib/dkms/zfs/0.8.6/build/module/zfs/zfs_vnops.c:992:30: note: expected ‘struct inode *’ but argument is of type ‘znode_t *’ {aka ‘struct znode *’}
zfs_iput_async(struct inode *ip)
~~~~~~~~~~~~~~^~
/var/lib/dkms/zfs/0.8.6/build/module/zfs/zfs_vnops.c:1075:18: error: passing argument 1 of ‘zfs_iput_async’ from incompatible pointer type [-Werror=incompatible-pointer-types]
zfs_iput_async(zp);
^~
/var/lib/dkms/zfs/0.8.6/build/module/zfs/zfs_vnops.c:992:30: note: expected ‘struct inode *’ but argument is of type ‘znode_t *’ {aka ‘struct znode *’}
zfs_iput_async(struct inode *ip)
~~~~~~~~~~~~~~^~
CC [M] /var/lib/dkms/zfs/0.8.6/build/module/zfs/zil.o
CC [M] /var/lib/dkms/zfs/0.8.6/build/module/zfs/zio.o
CC [M] /var/lib/dkms/zfs/0.8.6/build/module/zfs/zio_checksum.o
CC [M] /var/lib/dkms/zfs/0.8.6/build/module/zfs/zio_compress.o
CC [M] /var/lib/dkms/zfs/0.8.6/build/module/zfs/zio_crypt.o
CC [M] /var/lib/dkms/zfs/0.8.6/build/module/zfs/zio_inject.o
CC [M] /var/lib/dkms/zfs/0.8.6/build/module/zfs/zle.o
CC [M] /var/lib/dkms/zfs/0.8.6/build/module/zfs/zpl_ctldir.o
CC [M] /var/lib/dkms/zfs/0.8.6/build/module/zfs/zpl_export.o
cc1: some warnings being treated as errors
make[7]: *** [/usr/src/linux-headers-4.19.0-13-common/scripts/Makefile.build:308: /var/lib/dkms/zfs/0.8.6/build/module/zfs/zfs_vnops.o] Error 1
make[7]: *** Waiting for unfinished jobs....
make[6]: *** [/usr/src/linux-headers-4.19.0-13-common/scripts/Makefile.build:549: /var/lib/dkms/zfs/0.8.6/build/module/zfs] Error 2
make[5]: *** [/usr/src/linux-headers-4.19.0-13-common/Makefile:1565: _module_/var/lib/dkms/zfs/0.8.6/build/module] Error 2
make[4]: *** [Makefile:146: sub-make] Error 2
make[3]: *** [Makefile:8: all] Error 2
make[3]: Leaving directory '/usr/src/linux-headers-4.19.0-13-amd64'
make[2]: *** [Makefile:30: modules] Error 2
make[2]: Leaving directory '/var/lib/dkms/zfs/0.8.6/build/module'
make[1]: *** [Makefile:843: all-recursive] Error 1
make[1]: Leaving directory '/var/lib/dkms/zfs/0.8.6/build'
make: *** [Makefile:712: all] Error 2I'm guessing there's a few more tweaks that need to be done? Also is this commit related to this issue at all? EDIT: Managed to figure this out, after looking through some other examples of zfs_iput_async(ZTOI(zp));After that it builds properly. Will test that out and see what happens from here. Thanks |
|
@tuxoko Have been running with this patch for about 2 weeks now, I have not experienced a deadlock since. It finally put something in dmesg too. I'm also looking at upgrading to 2.0.3 sometime soon, is there a 2.x patch for this ready to go? Otherwise I can keep using 0.8.6 for a while if you want me to keep testing that. Cheers |
|
@4oo4 |
If TX_WRITE is create on a file, and the file is later deleted and a new directory is created on the same object id, it is possible that when zil_commit happens, zfs_get_data will be called on the new directory. This may result in panic as it tries to do range lock. This patch fixes this issue by record the generation number during zfs_log_write, so zfs_get_data can check if the object is valid. Signed-off-by: Chunwei Chen <david.chen@nutanix.com> Closes openzfs#10593 Change-Id: I6258f045ce5875d9f7acd29bef52b73a7679808e
If TX_WRITE is create on a file, and the file is later deleted and a new directory is created on the same object id, it is possible that when zil_commit happens, zfs_get_data will be called on the new directory. This may result in panic as it tries to do range lock. This patch fixes this issue by record the generation number during zfs_log_write, so zfs_get_data can check if the object is valid. Signed-off-by: Chunwei Chen <david.chen@nutanix.com> Closes openzfs#10593
If TX_WRITE is create on a file, and the file is later deleted and a new directory is created on the same object id, it is possible that when zil_commit happens, zfs_get_data will be called on the new directory. This may result in panic as it tries to do range lock. This patch fixes this issue by record the generation number during zfs_log_write, so zfs_get_data can check if the object is valid. Signed-off-by: Chunwei Chen <david.chen@nutanix.com> Closes openzfs#10593 Change-Id: I07307002ad3e0a7de577bab487dc11c447645a83
If TX_WRITE is create on a file, and the file is later deleted and a new directory is created on the same object id, it is possible that when zil_commit happens, zfs_get_data will be called on the new directory. This may result in panic as it tries to do range lock. This patch fixes this issue by record the generation number during zfs_log_write, so zfs_get_data can check if the object is valid. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Chunwei Chen <david.chen@nutanix.com> Closes #10593 Closes #11682
|
Will this resolve the issue reported in #10642? |
If TX_WRITE is create on a file, and the file is later deleted and a new directory is created on the same object id, it is possible that when zil_commit happens, zfs_get_data will be called on the new directory. This may result in panic as it tries to do range lock. This patch fixes this issue by record the generation number during zfs_log_write, so zfs_get_data can check if the object is valid. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Chunwei Chen <david.chen@nutanix.com> Closes openzfs#10593 Closes openzfs#11682
If TX_WRITE is create on a file, and the file is later deleted and a new directory is created on the same object id, it is possible that when zil_commit happens, zfs_get_data will be called on the new directory. This may result in panic as it tries to do range lock. This patch fixes this issue by record the generation number during zfs_log_write, so zfs_get_data can check if the object is valid. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Chunwei Chen <david.chen@nutanix.com> Closes openzfs#10593 Closes openzfs#11682
If TX_WRITE is create on a file, and the file is later deleted and a new directory is created on the same object id, it is possible that when zil_commit happens, zfs_get_data will be called on the new directory. This may result in panic as it tries to do range lock. This patch fixes this issue by record the generation number during zfs_log_write, so zfs_get_data can check if the object is valid. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Chunwei Chen <david.chen@nutanix.com> Closes openzfs#10593 Closes openzfs#11682
If TX_WRITE is create on a file, and the file is later deleted and a new directory is created on the same object id, it is possible that when zil_commit happens, zfs_get_data will be called on the new directory. This may result in panic as it tries to do range lock. This patch fixes this issue by record the generation number during zfs_log_write, so zfs_get_data can check if the object is valid. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Chunwei Chen <david.chen@nutanix.com> Closes openzfs#10593 Closes openzfs#11682
If TX_WRITE is create on a file, and the file is later deleted and a new directory is created on the same object id, it is possible that when zil_commit happens, zfs_get_data will be called on the new directory. This may result in panic as it tries to do range lock. This patch fixes this issue by record the generation number during zfs_log_write, so zfs_get_data can check if the object is valid. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Chunwei Chen <david.chen@nutanix.com> Closes openzfs#10593 Closes openzfs#11682
If TX_WRITE is create on a file, and the file is later deleted and a new directory is created on the same object id, it is possible that when zil_commit happens, zfs_get_data will be called on the new directory. This may result in panic as it tries to do range lock. This patch fixes this issue by record the generation number during zfs_log_write, so zfs_get_data can check if the object is valid. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Chunwei Chen <david.chen@nutanix.com> Closes #10593 Closes #11682
If TX_WRITE is create on a file, and the file is later deleted and a new directory is created on the same object id, it is possible that when zil_commit happens, zfs_get_data will be called on the new directory. This may result in panic as it tries to do range lock. This patch fixes this issue by record the generation number during zfs_log_write, so zfs_get_data can check if the object is valid. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Chunwei Chen <david.chen@nutanix.com> Closes #10593 Closes #11682
System information
Describe the problem you're observing
I'm experiencing a consistent general protection fault after upgrade from 0.7.x (Proxmox 5.4.x) to 0.8.4 (Proxmox 6.2). This fault looks similar to issue #7873, but not exactly the same. The result of this fault is a complete hang of most processes running within the Linux Container that uses the associated ZFS volume, along with various other unresponsive host processes. I cannot regain use of the container until the entire Proxmox host is (hard) rebooted. The normal reboot process fails to proceed due to the hung LXC.
Describe how to reproduce the problem
ZFS is my root (rpool), two JBOD drives mirrored, with several Linux Containers using their own ZFS volumes.
Bug can be encountered by using the system normally; I can make it maybe two days before the issue forces me to reboot the machine.
I believe Proxmox has a scheduled scrub that - while running - increases the chance of occurrence. However, daily use also seems to cause the same issue. This has been consistently triggered by the
Plex Media Serverprocess for me; I have not seen any other processes noted in the errors, but I do not see how this can be a Plex issue (all fingers point to a ZFS issue).I use ECC memory and have run memtest86+ recently to verify that memory is good.
Drives are healthy, pool is as well.
Include any warning/errors/backtraces from the system logs
The text was updated successfully, but these errors were encountered: