WD WDx0EFAX drive unable to resilver #10214
Comments
|
Thanks for reporting. Can you show the |
|
Here are some more details, the WD drives may also have a firmware bug: https://blocksandfiles.com/2020/04/15/shingled-drives-have-non-shingled-zones-for-caching-writes/ |
Perticuklar the last part is important for people who want to resilver:
|
|
That last entry is mine. This is definitely a WD firmware bug. 3 different drives showing the same errors at more or less the same points in the resilvering process is pretty damning. I strongly suggest that people DO NOT attempt to replicate my attempts to force a drive into the array to see what happens next. I'm still cleaning up the mess. ZFS kicked it out for good reason. WD are (of course) continuing to deny there's a problem or that they have firmware issues. |
|
If you wish to see the drive's internal error log, use smartcl --xall (or smartctl -x) The extended GP log will not show up with the --all (-a) flag |
|
|
It's worth noting that I usually fill a brand new drive with pseudorandom data, and then verify that it reads back correctly, before zeroing it and using it. I believe I did this, and of course I didn't know to TRIM it afterwards. |
|
The write error count, as reported by zpool status, keeps dropping to lower numbers. I think it's probably overflowing at 65.5K errors, is that correct? |
|
Trimming the drive won't make any difference. If anything a Trimmed or virgin drive exhibits more errors than a zero-filled one. (The advantage of having 3 such drives is you can verify this behaviour and know it's not "a bad drive" - and that it's not "excess vibration", as WD tried to paint it, by putting the drives on individual foam blocks out of the chassis...) |
|
Given all the IDNF errors that it's getting right now, I probably didn't test the drive and zero it. Failing at 28% likely corresponds to the amount that I previously wrote. |
|
I zeroed the partition, and tried doing the replace / resilver again. It's failed in the same way, at about 26%. The errors are still IDNF. |
|
I would take the drive out of the array and see if it passes a 4 pass data destructive badblocks test. This should take roughly 40 to 60 hours to complete on a 4TB drive. |
|
I filled the partition with pseudorandom data, and then tried the replace/resilver again. (In case the drive is doing something like a TRIM when you simply zero it.) But that didn't help, it's failed at 27%, spewing write errors as before. I will now do a four pass destructive badblocks test on the full drive, no partitions. |
|
Read/write speeds were similar, starting the pass at 180 MB/s, and gradually dropping to 80 MB/s by the end of the pass. |
|
How about vibrations? If the pool was idling, that could explain the success of badblocks. |
|
The pool is in a Supermicro server, so I doubt vibration has anything to do with it. Older WD Reds have no issues. I bought a new Seagate IronWolf ST4000VN008-2DR166 for comparison. Running through identical steps, first is a long self-test, then I'll fill it with pseudorandom data and read it back, then I'll zero it and create a labelled GPT partition, before trying to resilver as before. |
|
The ZFS resilver process is not completely sequential, and I think the random part is what brings the SMR WD RED to its knees: the drive firmware evidently has issues writing on different CMR/SMR bands, causing tremendous latency and, finally, a dropped drive. A pure sequential fill workload should not put this kind of stress on a SMR driver, resulting in no apparent problems and even showing quite good results (ie: the 180-80 MB/s transfer above). |
|
In this instance it's a drive firmware issue and I think ZFS is doing the right thing. I've been seeing this too on 3 different drives - it's not a vibration issue. I verified that by isolating the drives on blocks of foam. Based on Manfred Berger's 2015 OpenZFS Paris SMR presentation and some reading of SMR documentation, I believe the IDNF is a "relatively normal" error for SMR drives that occurs as the head seeks to EOD in SMR zones (just past end of written data) when switching between SMR zones (there are a number of open zones on a DM-SMR drive) and should be masked by the drive - except that WD screwed up. WD have just issued a new blog update: https://blog.westerndigital.com/wd-red-nas-drives/ (2020/4/22) and given a list of SMR drives, with more details coming. I've been dealing with WD over this issue for a few weeks and I suspect their sudden change of stance has something to do with my pointing out that there have been a number of consumer protection agency complaints filed around the world over the last week since stories appeared about them putting DM-SMR drives into the channel(*) whilst blowing off user complaints and passing on a UK legal opinion yesterday that in a british court they'd most likely to be found to have have acted with misfeasance, not just negligently (IANAL, but a friend is a magistrate) along with an observation that the continued gaslighting of customers they've been posting is likely to compound the penalties in judgements against them, especially as Seagate issued a statement that SMR and NAS don't mix: https://arstechnica.com/information-technology/2020/04/seagate-says-network-attached-storage-and-smr-dont-mix/, their own documentation says much the same thing in various locations http://zonedstorage.io/introduction/smr/ and Manfred Berger stated it in his presentation too - https://www.youtube.com/watch?v=a2lnMxMUxyc - around the 13 minute mark. Under the law in most countries, if you advertise something as a NAS/RAID drive and it won't work as a NAS/RAID drive, the that's a very bad thing (unfit for the purpose for which it was marketed) - especially if you change it without notice from a device that worked perfectly to something that doesn't work - or when it does work, it may take a 90-98% impact in performance from the previous device if you don't treat it "just right" (RAIDZ3 resilvers going from 12-20 hours on 19*2TB drives out to 8-10 DAYS, and scrubs failing to complete or taking 2 weeks - all on idle vdevs) - if you then issue the kinds of statements that are in the rest of that WD blog, the courts really won't like it - and when it's being punted as a home/SOHO device then consumer protection laws kick in falrly hard. (*) https://arstechnica.com/gadgets/2020/04/caveat-emptor-smr-disks-are-being-submarined-into-unexpected-channels/ |
|
Finished resilvering my brand new IronWolf ST4000VN008. No surprises whatsoever. |
|
Scrub completed without any errors. |
|
After the resilver and the scrub, I then took the new IronWolf offline, and did a sequential copy to the Red EFAX, using dd. I was unable do a During the resilver, the EFAX threw a couple of write errors, which from the kernel log, appeared to be the same IDNF error. It's worth noting that in all the previous configurations, the EFAX was on an IBM SAS2008 card, but now it is the only device on the motherboard's SATA controller, due to a shortage of cables. Finally, I did a scrub, and the scrub has completed without any errors. This is hardly a nice solution to getting an EFAX resilvered, as it's time-consuming, and it costs you one drive worth of redundancy. But for anyone who is desperate enough to try, I can say that it does work. (Try to limit the amount of writes done to the pool during this process, in case the final resilver fails as before.) |
|
FYI TOSHIBA MN04 HDD (includes RV sensor) seems to have similar problem. I've tried to replace MD04 series HDD (TOSHIBA) by MN04 series HDD, and got many read/write errors. I bought 2 MN04 drives and got similar result for both drives. No smart error is recorded. I haven't tried to find new firmware yet. |
|
I've also found this issue with a WD Blue WD20SPZX, which is a 2.5" 2TB drive that I have in a laptop. One day I attempted a transfer of ~500 GB of files to an NTFS partition under Windows, which got slower and slower until I killed the transfer. After rebooting to Ubuntu, it then also generated lengthy timeouts with zfs. This was a near new drive, so I thought I must have a faulty SATA controller. I mirrored the zpool with an external USB3 drive, and then dropped it out of the mirror. Now that I know about this issue with WD drives, I can confirm the internal drive is behaving just like the WD40EFAX, it seems to not be faulty, but it won't resilver into the mirror. It eventually throws thousands of write errors, until zfs faults it. |
|
I have 4 WDC_WD30EFRX installed in HP Microserver Gen8 connected to motherboard sata controller configured in AHCI mode. At first there were 2 drives in ZFS Mirror configuration and everything was fine. After few months I added 2 more drives and upgraded mirror to RAID10. Since then I'm getting random write errors, most likely during heavy write load. Scrub finishes without any error. No SMART errors.
|
|
I'm surprised to be seeing those IDNF errors alongside WD30EFRX. WD are maintaining that these are not SMR. See e.g. https://www.westerndigital.com/products/internal-drives/wd-red-hdd Do you have any other drive model in that array? I have three WD40EFRX Red NAS drives in mine, and they have performed without issue. All with firmware 82.00A82. I was surprised to find that the WD40EFAX drive also had firmware 82.00A82, which leads me to wonder whether the DM-SMR functionality has been in the firmware all along, but they only just enabled it through the configuration data in those particular drives. Can you find any difference between your old and new drives? Try |
|
Hi, I have only WD30EFRX in this RAID10 array. 2 older drives (few years old) are:
looks like its older hardware rev. and the lastest firmware is 80.00A80 and the other 2 installed recently are
The output of **hdparm -I /dev/sd[e-h] |grep TRIM ** is empty, so this is not SMR model. I have more drives in different arrays and I never had a single issue with them, but those are connected to HBA LSI controller. I thing I noticed after posting previous comment that 2 older drives had built-in APM timer value set to 300s (checked with idle3-tools and set to disabled) and the other 2 had APM disabled by default. I don't know what is the origin of this issue but it affects randomly one or all drives in this array |
HP Microserver Gen8 gave me various troubles with high capacity drives until I disabled AHCI and switched to IDE/Legacy mode. Give it a try. |
What kind of troubles? It worked fine with 2x3TB but throws errors with 4x3TB... I checked differences between old and new drives but there's nothing that matter... |
BIOS recognition issue and slow/unpredictable performance. After switching to IDE all issues disappeared. |
|
See https://forums.unraid.net/topic/72428-solved-clearing-synology-drive/ It's possible your EFRX drive has corrupted sectors that can't be read. (Power loss during writing?) Maybe try zeroing the whole drive, and see if that solves it. |
|
@darrenfreeman My issue is write not read related. Besides when this happen it affect many/random drives at once. |
|
Do you have It looks like the same error that we see on the SMR drives, but yours aren't known to be SMR. Can you try zeroing the whole drive anyway, and see if it throws those errors? |
|
@darrenfreeman Yes, all zpools have ashift =12
I cant zero any drive, it contains important data. Besides, I think that it's pointless. As I mentioned before this error appears randomly on 1-4 drives in this zpool and random sectores in the same moment. Sometimes its drive 1, another time 1+3, another 1,2,3 or 1,2,4 so this can't be just single drive issue. Untill I expanded this zpool from 2 to 4 drives this error never happend. I think it has something to do with 4 drives working together, maybe SATA controller or RAID10 data distribution issue... |
|
This has all the hallmarks of a power supply problem - especially as it only happened once you added more drives Replace your PSU and see if the problems continue. Pay close attention to the wattage of the unit. |
|
@Stoatwblr I don't think so. Total number of drives hasn't changed, just configuration. It was
now it's
I have 4 more drives (3x SDD + 1xHDD) connected to different SATA controller working without any problem since the beginning. Putting everything together: |
|
Replace the PSU anyway - It is the common factor |
|
It could be a SATA issue. Some SATA controllers are notorious for problems. One bad drive can throw errors on other drives, and it can throw other good drives out of your array. (I've had this happen.) You could try moving one drive out of each mirror onto the LSI HBA, just to see if those drives stop having issues. |
|
@darrenfreeman I'll do that but I have to buy new 8i LSI controller coz the one I have in my server is 4i only with all ports in use. |
|
@QBANIN again, before buying a new HBA: try setting your SATA port to IDE mode (rather than AHCI). |
|
@shodanshok Will do. Before switching to IDE I disabled Write Cache (in bios) for built-in controller. I'm runinng bonnie++ HDD stress test since few hours and so far no error appeared. Is it possible that this issue is somehow related to buggy Write Cache handling by ZFS? BTW: I don't see any downsides of disabling Write Cache. Sequential write speed to this RAID10 zpool is >200MB/s . Can't test random write speed but is not as much important because this is typical storage pool for long term data uploaded via single FTP/SMB connection or backup software. |
|
@QBANIN the disk own writeback cache can be useful when issuing random writes without invoking ZFS has excellent support for disk caching - so no, it is really difficult (if not impossible) than it is ZFS fault. Theoretically, disabling disk caching can void problem only if the DRAM cache itself is broken, which is a very remot possibility if happening on multiple disks at the same time. But as suggested by @Stoatwblr it can be the power supply having problem and corrupting in-cache data (I saw something similar with an assembled NAS). As suggested, re-enable the writeback cache and try to:
|
|
@shodanshok I can't replace power supply but will test IDE legacy mode. |
|
I was suggesting that you swap a couple of drives between the LSI HBA and the SATA controller. Same number of ports, but see if the problem follows the drive, or the controller. |
You were right about IDE mode. The issue is gone after switching from AHCI to IDE. :D So far 9 days without any write error. |
|
@QBANIN glad to hear that ;) The AHCI mode of these small HP Microservers Gen8 seems really problematic... |
|
@shodanshok That's why I'm going to switch to some 8i LSI controller anyway :) |
|
Since I had read errors on occasions with good HDDs, I replaced power supply and my problem is gone. Even if I didn't have any other issues but ZFS read failure, it seems the cause was faulty power supply. |
|
It happened again few days after my last comment. Switched to LSI 9207-8i and... it happened again. It looks like this issue is affecting only WD RED 3,5" (CMR) drives from 1/4 up to 4/4 at once (4 this time). Remaining 2x HDD (WD blue 2,5" and Seagate 2,5") and 3x SSD drives work fine.
After zpool clear RAID1 (in fact it's raid 10) everything works like before. Scrub show no issue. |
|
Closing. From the discussion it sounds like issues reported here were caused by hardware. |
System information
Describe the problem you're observing
I purchased a brand new Western Digital WD40EFAX. Then after I purchased it, this article came out, saying they are likely not suitable for use with zfs, because they fail to resilver:
https://blocksandfiles.com/2020/04/14/wd-red-nas-drives-shingled-magnetic-recording/
So to see if this is the case, I did a zpool replace of a known good drive for this new drive. And I can confirm that this drive is unable to resilver. It has effectively hanged at 27%, IOPS fell to zero, and the write error count is going up.
I will be returning this drive as not fit for purpose. I'll leave it trying to resilver for a few days, and so if anyone wants me to do further investigation, I can do this before returning it.
Note that issue #10182 also refers to performance issues caused by this model being SMR masquerading as CMR. The situation now seems far worse than that.
Describe how to reproduce the problem
zpool replace onto a Western Digital WD40EFAX.
Include any warning/errors/backtraces from the system logs
The text was updated successfully, but these errors were encountered: