ZFS corruption related to snapshots post-2.0.x upgrade #12014
Comments
jgoerzen
added
Status: Triage Needed
|
Two other interesting tidbits... When I do the reboot after this issue occurs, the mounting of the individual zfs datasets is S L O W. Several seconds each, and that normally just flies by. After scrubbing, it is back to normal speed of mounting. The datasets that have snapshot issues vary with each one. Sometimes it's just one, sometimes many. But var is almost always included. (Though its parent, which has almost no activity ever, also is from time to time, so that's odd.) |
|
Same symptoms here, more or less. See also issue #11688. |
|
I also have the symptom with the corrupted snapshots, without kernel panics so far. So far it only affected my Debian system with Linux 5.10 and zfs 2.0.3 (I've turned the server off for today, I can check the exact versions tomorrow). Also, while the system has the 2.0.3 zfs utils + module, the pool is still left on 0.8.6 format. I wasn't able to execute On the corrupted system, after I got the mail from ZED, I manually ran a scrub at first, after which the I've rebooted the server into an Ubuntu 20.10 live with zfs 0.8.4-1ubuntu11 The errors didn't seem to affect the data on the zvols (all 4 affected snapshots are of zvols). The zvols are used as disks for VMs with ext4 on them. I have two other Ubuntu 21.04 based Systems with zfs-2.0.2-1ubuntu5 which are not affected until now. However, they have their pools already upgraded to 2. All are snapshotted with sanoid and have the datasets encrypted.
EDIT:
EDIT 2:
|
|
I'm seeing this too, on Ubuntu 21.04, also using zfs encryption I have znapzend running, and it makes a lot of snapshots. Sometimes, some of them are bad, and can't be used (for example, attempting to send them to a replica destination fails). I now use the In the most recent case (this morning) I had something like 4300 errors (many more than I'd seen previously). There are no block-level errors (read/write/cksum). They're cleared after destroying the affected snapshots and scrubbing (and maybe a reboot, depending on .. day?) Warning! Speculation below:
|
|
@jgoerzen Can you
In my case, #11688 (which you already reference), I've discovered that rebooting "heals" the snapshot -- at least using the patchset I mentioned there |
|
I'll be glad to. Unfortunately, I rebooted the machine yesterday, so I expect it will be about a week before the problem recurs. It is interesting to see the discussion today in #11688. The unique factor about the machine that doesn't work for me is that I have encryption enabled. It wouldn't surprise me to see the same thing here, but I will of course wait for it to recur and let you know. |
|
Hello @aerusso, The problem recurred over the weekend and I noticed it this morning. Unfortunately, the incident that caused it had already expired out of the It should be noted that my hourly snap/send stuff runs at 17 minutes past the hour, so that may explain this timestamp correlation. zpool status reported: Unfortunately I forgot to attempt to do a So I think that answers the question. After a reboot but before a scrub, the |
|
I have similar symptoms, on an encrypted single-ssd ubuntu 21.04 boot pool, using stock zfs from ubuntu's repos. Deleting the affected snapshots and scrubbing previously cleared the errors, but on reoccurence, repeated scrubbing (without deleting them) caused a deadlock. My system has ECC memory, so it's probably not RAM related.
|
|
@cbreak-black Was there a system restart between the occurrence of the corrupted snapshot and the problems? Restarting has "fixed" this symptom for me (though you will need to scrub twice for the message to disappear, I believe). I have a suspicion that this may be a version of #10737 , which has an MR under way there. The behavior I am experiencing could be explained by that bug (syncoid starts many I'm holding off on trying to bisect this issue (at least) until testing that MR. (And all the above is conjecture!) |
|
@aerusso No, without a restart I got into the scrub-hang, and had to restart hard. Afterwards, the scrub finished, and several of the errors vanished. The rest of the errors vanished after deleting the snapshots and scrubbing again. |
|
Can I join the club too? #10019 |
|
@InsanePrawn I can't seem to find commit 4d5b4a33d in any repository I know of (and neither can github, apparently, either). However, in your report you say this was a "recent git master" and the commit I'm currently betting on being guilty is da92d5c, which was committed in November of the previous year, so I can't use your data point to rule out my theory! Also, it sounds like you didn't have any good way to reproduce the error --- however, you were using a test pool. Compared to my reproduction strategy (which is just, turn my computer on and browse the web, check mail, etc.) it might be easier to narrow in on a test case (or might have been easier a year and a half ago, when this was all fresh). Anyway, if you have any scripts or ideas of what you were doing that caused this besides "snapshots being created and deleted every couple minutes", it might be useful too. (I already tried lots of snapshot creations and deletions during fio on several datasets in a VM). |
|
Yeah, idk why I didn't go look for the commit in my issue - luckily for us, that server (and pool; it does say yolo, but it's my private server's root pool. it's just that i won't cry much if it breaks; originally due to then unreleased crypto) and the git repo on it still exist. Looks like 4d5b4a33d was two systemd-generator commits by me after 610eec4 |
|
FWIW the dataset the issue appeared on was an empty filesystem (maybe a single small file inside) dataset that had snapshots (without actual fs activity) taken in quick intervals (somewhere between 30s and 5m intervals) in parallel with a few (5-15) other similarly empty datasets. The pool is a raidz2 on 3.5" spinning SATA disks. Edit: Turns out the dataset also still exists, the defective snapshot however does not anymore. I doubt that's helpful? |
|
@InsanePrawn Does running the zrepl workload reproduce the bug on 2.0.5 (or another recent release?) I don't think the snapshot is terribly important --- unless you're able to really dig into it with zdb (which I have not developed sufficient expertise to do). Rather, I think it's the workload, hardware setup, and (possibly, but I don't understand the mechanism at all) the dataset itself. Encryption also is a common theme, but that might just affect the presentation (i.e., there's no MAC to fail in the unencrypted, unauthenticated, case). Getting at |
|
I've since added redundancy to my pool (it's now a mirror with two devices), and disabled autotrim. The snapshot corruption still happens. Still don't know what is causing it. And I also don't know if the corruption happens when creating the snapshot, and only later gets discovered (when I try to zfs send the snapshots), or if snapshots get corrupted some time in between creation and sending. |
|
@cbreak-black Can you enable the all-debug.sh ZEDlet, and put the temporary directory somewhere permanent (i.e., not the default of This will get the output of I'll repeat this here: if anyone gets me a reliable reproducer on a new pool, I have no doubt we'll be able to solve this in short order. |
|
Just mentioning here that we saw this on TrueNAS 12.0-U5 with OpenZFS 2.0.5 as well -- see #11688 (comment) for our story. |
|
Since I don't see anyone mentioning it here yet, #11679 contains a number of stories about the ARC getting confused when encryption is involved and, in a very similar looking illumos bug linked from there, eating data at least once. |
|
@gamanakis Nope, I'm not using raw (-w). |
|
it's present in v2.1.1 as well: |
|
@aerusso you wrote that da92d5c may be the cause of this issue. My workstation at work panics after a couple of days and I need to reset it. Could you provide a branch of 2.1.1 with this commit reverted (as revert causes merge conflicts I can't fix myself) so I could test if the machine no longer crashes? |
|
@phreaker0 Unfortunately, the bug that da92d5c introduced (#10737) was fixed by #12299, which I believe is present in all maintained branches now. It does not fix #11688, (which I suspect is the same as this bug). I'm currently running 0.8.6 on Linux 5.4.y, and am hoping to wait out this bug (I don't have a lot of time right now, or for the foreseeable future). But, If you have a reliable reproducer (or a whole lot of time) you could bisect while running 5.4 (or some other pre-5.10 kernel). I can help anyone who wants to do that. If we can find the guilty commit, I have no doubt this can be resolved. |
@aerusso I have done What am I missing? |
|
First of all: it is a complete coincidence that I happened to read this message (so if I don't respond to something like this in the future, it's almost certainly just because I didn't see it at all). To get the zedlet working, you need that script in There should be a bunch of scripts in there already (as symlinks). Even better would be to add an appropriate symlink for Afterwards, you'll need to restart the ZED. Something like As a side note, I'd like to add that getting a reliable reproducer using a pair of VMs would be incredible for this bug. I am not aware of any reproducer that doesn't rely on two pieces of hardware connected by a physical network using someone's production system. That said, performing the bisection after making this reproducer will be another herculean task, since you'll need check everything between zfs-0.8.6 and zfs-2.0.0. This will have to be done on Linux 5.1 or earlier, since the common merge-base is 78fac8d from 2019, and The estimate git gave me is 9 steps, plus that commit. I can provide high-level assistance doing that git bisect if someone has a reliable reproducer, but I would not be surprised if that bisection takes a month of run time, given how long it took for me to personally experience the errors. Furthermore, you'd have to be aware that, during that bisection, you'd be running through other, possible very serious, buggy versions. |
|
Thank you Antonio, I understand your concerns about the reproducibility of the bug. VM N.1then continuosly running script that simulates user activity, by:
and tring to not exceed 100 GB of available space Aside this, I started to manually sync (unencrypted send) VM N.1 zfs encrypted dataset to VM N.1 zfs unencrypted dataset. I soon will schedule this with crontab VM N.2 |
Yes, I concur with your statement, based on my own experience (currently zfs 2.2.6) |
|
I put together a script to potentially mitigate the mixed raw/non-raw send issues, although I see that there are folks here who are having the snapshot corruption bug even without the mixed sends... But this script does the following:
🐢 Obviously this will be much slower since it's at the file level, not the block level
I also welcome any thoughts on this as a possible alternative to avoid mixed raw sends! https://gist.github.com/pypeaday/57a8a0ec18f7db797540174512b0c4eb |
Hi How does the raw send works? I now have the following: source = encrypted Now i can mount zfs filesystems on the target if needed. When i do raw sends can i still mount filesystems on the target? I believe its double necrypted this way? am i correct? |
It's not double-encrypted. But if you haven't been doing raw sends, you can't just switch, you have to rebuild the target from scratch. A raw send just sends the blocks as-is, so if they're compressed and/or encrypted on the source, they're sent as-is to the target, where they remain in the same compressed and/or encrypted state. The nice thing about raw sends is that you don't have to unlock the target to do the send, which is good for cloud storage or other environments you don't want your data to ever be accessible to anyone but you. To mount datasets on the target, you would use the same process as mounting on the source, including the same passphrase. If you don't do a raw send, the block is decrypted and uncompressed before its sent, then written to the target using the target pool's settings, which may be different - different compression algorithm, different encryption key, etc. So when sending non-raw streams, the target must be unlocked before it can receive. My preference has always been raw sends, so it's serendipitous that raw sends appear to be a workaround for these problems. |
Thank you for the info. Perfect i am going to try raw sends. Does this way also support incremental? And what is i have a snapshot mounted on the target machine and syncoid is starting a raw send? Is it not going to corrupt the data? |
Yes. Just do a raw send to build the target initially, then subsequent incremental sends can be raw.
In that respect, incremental using raw works just like incremental without it. It will roll the target back to the starting snapshot then update the target to the new incremental snapshot. Any data that was written on the target in between those source snapshots will be lost. But the rollback/update steps are atomic, so from the application layer's perspective, they both happen instantly after the send/receive transfer is complete. No file system corruption will occur. I.e., if it's working for you now, it'll work the same using raw. |
hmmm, I fully deleted the target filesystem. Now i want to syncoid to target and get this error: cannot receive new filesystem stream: incompatible embedded data stream feature with encrypted receive |
When i am using these options its working --compress=zstd-fast --sendoptions="wLecp" --no-sync-snap --no-clone-handling --no-privilege-elevation |
And i see when using raw send on the target i have to load-key for every zfs filesystem manualy (that i want to mount) but this is no problem. I am curious now that i am using raw sends the issue will return? |
Just a single observation point here, but I also use syncoid in pull mode with |
|
Good morning, To date I have reproduced file system corruption at least twice, sending data via There are literally tens of thousands of lines of information but I have tried to summarize the results of the most recent test at https://github.com/HankB/provoke_ZFS_corruption/blob/main/X86_trial_2.md#2024-12-22-early-am This test was using Debian 12 (stable) and ZFS 2.2.6 from This issue is getting a little long and if desired, I could start another issue or continue discussion on the Thanks! |
|
This is incredible, @HankB. Very good job! My suggestion is to try to bisect the commit that creates (or exposes!) the buggy behavior. The first step to that is getting the reproducer working with Linux kernel 5.1 (or earlier), and demonstrating that, with that kernel, 2.2.6 continues to reproduce the bug. Debian buster (oldoldstable) may be the easiest way to do that. After that, you'll want to demonstrate that 78fac8d does NOT reproduce the bug. Git bisect will then take about 9 more compile/run cycles to isolate the guilty commit. Another prong of attack should be to get this reproducer working in a VM so that it can be run as part of the CI pipeline, and catch these kinds of bugs before they hit stable branches. Just again, this is really incredible! I'll be looking at the repository to understand the details that I suspect are very important. |
That seems pretty sensible. I'm also thinking of performing this test on an old (but reliable) host that has proper ECC RAM. The way this problem cascades makes me wonder if some access pattern provokes something like a Rowhammer effect (though a wild pointer or buffer overrun seems more likely.) First step is to reproduce the issue with the present configuration on the upgraded motherboard. Then I can proceed to bisect. |
|
That is awesome! As a reminder, the first issues appeared with the 0.8.x to 2.0 transition, so bisecting that might be the way to get at the original root cause, which was never fully identified, afaik. |
|
@jimsalterjrs Your expertise could come in handy here! |
There isn't much I can offer here; this is probably more up @allanjude 's alley than mine. The one thing I'll note here is that some folks think this could be triggered by syncoid and sanoid walking the tree of datasets rather than using ZFS recursion. They do that for good reason--native ZFS recursion gets REAL screwy when you add and remove datasets from a tree you've already interacted with using ZFS native recursion!--but you have the option of NOT letting them walk the tree, if you'd like to further bisect. In sanoid, use recursive=zfs instead of recursive=yes if you want to use ZFS native recursion. In syncoid, instead of using the -r argument, use --sendoptions=R. Note that you WILL discover why my tools walk the tree manually if you decide to do this, and end up adding or removing datasets after you've begun. I tend to forget the exact details of what triggers which errors and undesired behaviors when... But it doesn't take long to find out the same way I did. BUT, if you understand those limitations and want to do things that way--either in general, or to bisect this particular problem from a new direction--that's how you'd go about it. |
|
Good evening, Poking around here I find the link above. Pointers to instructions that work with Buster and 0.8.6 are going to be very helpful in my progress. Specifically [whinge] Thanks! In other news, I'm working on a different platform (X8DTL). Good: Builds ZFS a lot quicker and disk operations are a lot faster. Bad news: It took several days to produce the corruption errors. :-/ |
|
The Specifically: are you running on bare metal or a VM? If a VM, I'd snapshot the VM image and then do the build/install/test. Rollback afterwards && repeat. In that case, you can just
You reproduced this on ZoL 2.2.6 and what kernel version? If it's a much earlier Linux version, it might just be that the bug takes longer to reproduce. |
|
Many thanks for the quick reply. Bare metal. And I agree, it makes sense to reinstall and start with a clean slate for each test. At the moment I'm working through
That was a fully up to date Debian 12 install (since repaved with Buster) but would have been running It's the same version as the low resource (J1900) host. Might be related to that or might just be luck of the draw. It just means that any negative (e.g. no corruption) tests will need to run for about a week before I consider them conclusive. |
So is that a good trade then? Or would you be better off to take longer to build ZFS and be able to reproduce the corruption faster? |
It would be best to be able to run on both hosts, but the low resource host is presently malfunctioning. The other host is solid. |
@robn link from issue 13755 to some more recent point in this issue 12014? |
|
I Now use a different approach with sanoid/syncoid. I use sendoptions to recusive send the snapshots. Now i don't have errors on the enrypted pool. Its now running 6 days without errors. before i always had errors within 5 days. I will keep you informed if it keeps running without errors. |
|
Guys, I can confirm. If i use the following i get encryption errors within the 5 days. syncoid --recursive But when i use it like this the errors are not happening. syncoid --sendoptions="R" I am using it now to send it in raw mode. So now the errors are not happening anymore. |
|
Just throwing this out there for those of you doing automated tests (@HankB and @Germano0) and are trying to reduce the amount of time it takes to reproduce the issue: |
Agreed. The script I wrote eventually goes 5 deep for nested data sets with a total of 35 in my present test.
And writes files throughout the pool (using I loop each pass with 750s sleep in between with the syncoid command and modify script in their own loops. Since the syncoid and modify script take different times to complete, that results in un-synchronized overlap between the two. I'm open to other suggestions, but in the interest of reproducible results, I can't make too many changes at this point. I'd suggest that someone interested in additional strategies try them either on available H/W or VMs and this can reduce the overall time to get an answer. best, Edit: I'm adding a daily scrub to the testing process (manually invoked.) It occurs to me that the corruption could be happening and just not detected. |
and others
System information
Describe the problem you're observing
Since upgrading to 2.0.x and enabling crypto, every week or so, I start to have issues with my zfs send/receive-based backups. Upon investigating, I will see output like this:
Of note, the
<0xeb51>is sometimes a snapshot name; if Izfs destroythe snapshot, it is replaced by this tag.Bug #11688 implies that zfs destroy on the snapshot and then a scrub will fix it. For me, it did not. If I run a scrub without rebooting after seeing this kind of
zpool statusoutput, I get the following in very short order, and the scrub (and eventually much of the system) hangs:However I want to stress that this backtrace is not the original cause of the problem, and it only appears if I do a scrub without first rebooting.
After that panic, the scrub stalled -- and a second error appeared:
I have found the solution to this issue is to reboot into single-user mode and run a scrub. Sometimes it takes several scrubs, maybe even with some reboots in between, but eventually it will clear up the issue. If I reboot before scrubbing, I do not get the panic or the hung scrub.
I run this same version of ZoL on two other machines, one of which runs this same kernel version. What is unique about this machine?
I made a significant effort to rule out hardware issues, including running several memory tests and the built-in Dell diagnostics. I believe I have rules that out.
Describe how to reproduce the problem
I can't at will. I have to wait for a spell.
Include any warning/errors/backtraces from the system logs
See above
Potentially related bugs
arc_buf_destroyis in silent corruption for thousands files gives input/output error but cannot be detected with scrub - at least for openzfs 2.0.0 #11443. The behavior described there has some parallels to what I observe. I am uncertain from the discussion what that means for this.The text was updated successfully, but these errors were encountered: