kernel: PANIC at dmu_objset.c:1844:do_userquota_cacheflush() #7753
Comments
|
Thanks for reporting this. Just FYI quota accounting in ZFS is always enabled, it's only the enforcement which can be enabled/disabled. What's a little surprising about this failure is the error reported by the VERIFY is Let's get @tcaputi's thoughts on this as well since I suspect it is encryption related. |
|
I agree with Brian. We've made a few fixes since the version that you have came out. I would suggest installing the newest version, rebooting / reinserting the kernel module, and seeing if the problem persists. If it does, reply here and we'll take a closer look. |
|
I will do that in the next few days and report back :) Just for the record, to be sure: upgrading & possibly loading it with my old kernel later on will currently work without any corruption? |
|
None of this code is currently in a stable release, and we make no guarantees about code that is just on master. That being said, I am fairly certain that we have done nothing to break the on-disk format of encryption since ba863d0 was released and I dont think any patches we have made could cause permanent errors that couldn't be undone by reverting to an older zs version. If you are feeling concerned I would recommend making a backup of any data you can't afford to lose. |
|
This may go without saying. But I'd just add to @tcaputi's comment that if you plan to rollback to the older version do not run |
|
Okay, understood :) Will do some testing & report back! |
|
I've switched to 3d503a7 and now I got at shutdown. Funny thing is, there were more logs that didn't make it to my log file, I'm adding a screen photo. |
|
I'll take a look on Monday, but we haven't seen anything like this in a long time.... Were the filesystems created by |
|
No, they were created from scratch. I was loosely following this wiki entry: |
|
@phryneas We are having a hard time getting NixOS up and running to do any tests. Do you have a nix configuration file that we can use to get started? |
|
@tcaputi https://gist.github.com/phryneas/7ef5eb97e35d973eb1e7c2a8c9590153 I can also give you access to a completely set up hetzner cloud VM (including virtual console, snapshots, rollback etc.) - you'd have to contact me via mail for that. See my contact info at https://phryneas.de PS: to clarify this: the hetzner cloud vm is set up quite similar to my system, but I haven't been able to reproduce that behaviour there yet :/ |
|
I think I have located the problem after spending some time with the affected machine. The problem only seems to effect one dataset, which is has the |
|
I am now able to reproduce the problem. It only happens with sync=disabled and when doing a readonly remount on the filesystem before umounting. I should be able to get a fix sometime today. |
Currently, when unmounting a filesystem, ZFS will only wait for a txg sync if the dataset is dirty and not readonly. However, this can be problematic in cases where a dataset is remounted readonly immediately before being unmounted, which often happens when the system is being shut down. Since encrypted datasets require that all I/O is completed before the dataset is disowned, this issue causes problems when write I/Os leak into the txgs after the dataset is disowned. This patch simply enforces that all dirty datasets should wait for a txg sync before umount completes. Fixes: openzfs#7753 Signed-off-by: Tom Caputi <tcaputi@datto.com>
Currently, when unmounting a filesystem, ZFS will only wait for a txg sync if the dataset is dirty and not readonly. However, this can be problematic in cases where a dataset is remounted readonly immediately before being unmounted, which often happens when the system is being shut down. Since encrypted datasets require that all I/O is completed before the dataset is disowned, this issue causes problems when write I/Os leak into the txgs after the dataset is disowned, which can happen when sync=disabled. This patch simply enforces that all dirty datasets should wait for a txg sync before umount completes. Fixes: openzfs#7753 Signed-off-by: Tom Caputi <tcaputi@datto.com>
|
Pull request has been opened with a fix. @phryneas Would you mind checking if this solves your issue? |
Currently, when unmounting a filesystem, ZFS will only wait for a txg sync if the dataset is dirty and not readonly. However, this can be problematic in cases where a dataset is remounted readonly immediately before being unmounted, which often happens when the system is being shut down. Since encrypted datasets require that all I/O is completed before the dataset is disowned, this issue causes problems when write I/Os leak into the txgs after the dataset is disowned, which can happen when sync=disabled. This patch simply enforces that all dirty datasets should wait for a txg sync before umount completes. Fixes: openzfs#7753 Signed-off-by: Tom Caputi <tcaputi@datto.com>
|
Will do this evening - thanks for all the work you've put into this so far! |
|
I posted this on the PR, but it seems like the work isn't completely done.... Although my reproducer is fixed, your VM is still having problems. I will continue investigating tomorrow. |
Currently, when unmounting a filesystem, ZFS will only wait for a txg sync if the dataset is dirty and not readonly. However, this can be problematic in cases where a dataset is remounted readonly immediately before being unmounted, which often happens when the system is being shut down. Since encrypted datasets require that all I/O is completed before the dataset is disowned, this issue causes problems when write I/Os leak into the txgs after the dataset is disowned, which can happen when sync=disabled. While looking into fixes for this issue, it was discovered that dsl_dataset_is_dirty() does not return B_TRUE when the dataset has been removed from the txg dirty datasets list, but has not actually been processed yet. Furthermore, the implementation is comletely different from dmu_objset_is_dirty(), adding to the confusion. Rather than relying on this function, this patch forces the umount code path (and the remount readonly code path) to always perform a txg sync on read-write datasets and removes the function altogether. Fixes: openzfs#7753 Signed-off-by: Tom Caputi <tcaputi@datto.com>
|
@phryneas I just pushed a commit to the PR that I think will fix the issue. I will have time to test it tomorrow, which should hopefully be the last time I need your VM's. Thank you for your patience in resolving this issue. |
|
Cool! I'll give this a test on my bare metal machine (that initially had the problem) tomorrow :) |
Currently, when unmounting a filesystem, ZFS will only wait for a txg sync if the dataset is dirty and not readonly. However, this can be problematic in cases where a dataset is remounted readonly immediately before being unmounted, which often happens when the system is being shut down. Since encrypted datasets require that all I/O is completed before the dataset is disowned, this issue causes problems when write I/Os leak into the txgs after the dataset is disowned, which can happen when sync=disabled. While looking into fixes for this issue, it was discovered that dsl_dataset_is_dirty() does not return B_TRUE when the dataset has been removed from the txg dirty datasets list, but has not actually been processed yet. Furthermore, the implementation is comletely different from dmu_objset_is_dirty(), adding to the confusion. Rather than relying on this function, this patch forces the umount code path (and the remount readonly code path) to always perform a txg sync on read-write datasets and removes the function altogether. Fixes: openzfs#7753 Signed-off-by: Tom Caputi <tcaputi@datto.com>
|
@phryneas I tested the patch against your VM 10 times without issue. Let me know if you have more problems on your bare metal machine. |
|
I've given it a few reboots and so far it shut down nicely every time. Seems to work 👍 |
Currently, when unmounting a filesystem, ZFS will only wait for a txg sync if the dataset is dirty and not readonly. However, this can be problematic in cases where a dataset is remounted readonly immediately before being unmounted, which often happens when the system is being shut down. Since encrypted datasets require that all I/O is completed before the dataset is disowned, this issue causes problems when write I/Os leak into the txgs after the dataset is disowned, which can happen when sync=disabled. While looking into fixes for this issue, it was discovered that dsl_dataset_is_dirty() does not return B_TRUE when the dataset has been removed from the txg dirty datasets list, but has not actually been processed yet. Furthermore, the implementation is comletely different from dmu_objset_is_dirty(), adding to the confusion. Rather than relying on this function, this patch forces the umount code path (and the remount readonly code path) to always perform a txg sync on read-write datasets and removes the function altogether. Fixes: openzfs#7753 Signed-off-by: Tom Caputi <tcaputi@datto.com>
|
I'll take down the VMs, now that this is merged. Again, thanks for your time and hard work @tcaputi 🥇 |
|
Thanks for letting me use the VM's to debug the issue. |
System information
Declaration of complete build process: https://github.com/NixOS/nixpkgs/blob/d6c6c7fcec6dbd2b8ab14f0b35d56c7733872baa/pkgs/os-specific/linux/zfs/default.nix
Describe the problem you're observing
On shutdown, the system hangs indefinitely after trying to unmount zfs shares.
According to logs, the kernel panics.
Describe how to reproduce the problem
Simply shut down - happens every time.
Include any warning/errors/backtraces from the system logs
I have not enabled quota on any volume:
If I can provide any additional data, please tell me.
The text was updated successfully, but these errors were encountered: