WD30EZRX and aggressive head parking

For many years Western Digital has been flip-flopping their stance on the issue with aggressive (and excessive) parking of the drive’s heads, especially in the case of their Green series drives (including their “Enterprise” drives with the “-GP” suffix). WD advertises this as a feature called “IntelliPark” — a feature that, in all honesty, appears to do nothing useful in desktop or server systems.

To end-users and administrators, this behaviour manifests itself in two ways:

  1. Degraded performance for occasional I/O, due to the drive having to unlock and move its actuator arms back over the platters,
  2. A rapidly increasing Load_Cycle_Count (SMART attribute 193 / 0xC1). Internet users often refer to this as “LCC”.

It was brought to my attention via an Internet forum post that WD may have done away with excessive load cycling on newer Green drives, and also did not require use of WD’s Idle3 utility (more on that in a moment). I was ecstatic; the Green drives run at a lower temperature, and for a disk that’s being used to store backups (e.g. not regularly accessed) but kept spinning at all times (lots of reasons for this), the Green series seemed like a good choice.

So last week I bought a WD30EZRX to see what the behaviour was.

Sadly, the behaviour on the WD30EZRX is identical to that of the WD20EARS drive which came out many years ago: the drive is incredibly aggressive about parking its heads, and now there’s no reliable way to disable this behaviour.

Per the ATA specification IDENTIFY command, the WD30EZRX drive doesn’t support APM (see line 26 below); why that matters will soon become evident:

protocol              ATA/ATAPI-8 SATA 3.x
device model          WDC WD30EZRX-00MMMB0
firmware revision     80.00A80
serial number         WD-WCAWZ1930893
WWN                   50014ee2b179476a
cylinders             16383
heads                 16
sectors/track         63
sector size           logical 512, physical 4096, offset 0
LBA supported         268435455 sectors
LBA48 supported       5860533168 sectors
PIO supported         PIO4
DMA supported         WDMA2 UDMA6

Feature                      Support  Enabled   Value           Vendor
read ahead                     yes      yes
write cache                    yes      yes
flush cache                    yes      yes
overlap                        no
Tagged Command Queuing (TCQ)   no       no
Native Command Queuing (NCQ)   yes              32 tags
SMART                          yes      yes
microcode download             yes      yes
security                       yes      no
power management               yes      yes
advanced power management      no       no
automatic acoustic management  no       no
media status notification      no       no
power-up in Standby            yes      no
write-read-verify              no       no
unload                         no       no
free-fall                      no       no
data set management (TRIM)     no

SMART attributes:

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   253   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   163   163   021    Pre-fail  Always       -       8833
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       10
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       0
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       8
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       7
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       15
194 Temperature_Celsius     0x0022   124   121   000    Old_age   Always       -       28
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

A count of 15 doesn’t look that bad, does it? Yeah well, watch this.

The below sequence of commands prints the current date/time, followed by reading 4KBytes of data from the drive (LBAs 0 to 7, given logical sector size), sleeps 10 seconds, then checks SMART statistics and prints out the Load_Cycle_Count attribute and its raw value, rinse lather repeat indefinitely:

# while true; do echo -n "`date` -- " ; dd if=/dev/ada3 of=/dev/null bs=4k count=1 2>/dev/null ; sleep 10 ; smartctl -a /dev/ada3 | awk '/Load_Cycle_Count/ { print $2 ": " $NF }' ; done
Wed May 30 00:18:17 PDT 2012 -- Load_Cycle_Count: 28
Wed May 30 00:18:28 PDT 2012 -- Load_Cycle_Count: 29
Wed May 30 00:18:38 PDT 2012 -- Load_Cycle_Count: 30
Wed May 30 00:18:49 PDT 2012 -- Load_Cycle_Count: 31
Wed May 30 00:18:59 PDT 2012 -- Load_Cycle_Count: 31
Wed May 30 00:19:10 PDT 2012 -- Load_Cycle_Count: 32
Wed May 30 00:19:20 PDT 2012 -- Load_Cycle_Count: 33
Wed May 30 00:19:31 PDT 2012 -- Load_Cycle_Count: 34
Wed May 30 00:19:41 PDT 2012 -- Load_Cycle_Count: 35
Wed May 30 00:19:52 PDT 2012 -- Load_Cycle_Count: 36
Wed May 30 00:20:02 PDT 2012 -- Load_Cycle_Count: 37
Wed May 30 00:20:13 PDT 2012 -- Load_Cycle_Count: 38
Wed May 30 00:20:23 PDT 2012 -- Load_Cycle_Count: 39
Wed May 30 00:20:34 PDT 2012 -- ^C

Every time the drive load cycles, of course, there is an audible noise issued as well (the sound of the actuator arm having to be moved, thus heads unparked). So as I stated above in item #1, the drive will perform badly (initial I/O will stall/delay while the heads are being unparked), while Load_Cycle_Count increments.

If we try a shorter sleep interval (5 seconds), the issue doesn’t happen, which further proves that there is an “idle timeout” internal to the drive (not that this is anything new) and it’s probably set somewhere between 6-9 seconds:

# while true; do echo -n "`date` -- " ; dd if=/dev/ada3 of=/dev/null bs=4k count=1 2>/dev/null ; sleep 5 ; smartctl -a /dev/ada3 | awk '/Load_Cycle_Count/ { print $2 ": " $NF }' ; done
Wed May 30 00:20:55 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:21:01 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:21:06 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:21:11 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:21:16 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:21:21 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:21:26 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:21:32 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:21:37 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:21:42 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:21:47 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:21:52 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:21:57 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:22:02 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:22:07 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:22:12 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:22:18 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:22:23 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:22:28 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:22:33 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:22:38 PDT 2012 -- ^C

I should also note here that the load cycle count can/will increment when pulling SMART statistics after the drive has gone into idle state. The reason: SMART statistics are stored in the HPA region of the drive, which means for it to read statistics it has to unpark the heads. A beautiful catch-22…

And now some links to WD’s own justification/stance on the “LCC matter”:

Things to note about these documents, and in general, because none of it is made apparent:

  1. Neither document mentions the WD30EZRX, nor its brother/sister drives of smaller capacities or less cache (such as the WD25EZRX, WD20EARX, WD15EARX, WD10EARX, WD10EZRX, or the WD20EARS). However, some WD forum users are referring folks with those models of drives to the above documents, which could result in catastrophic problems (keep reading),
  2. Within the first document you’ll find preposterous recommendations, such as to disable or adjust syslog (the idea being that disabling logging won’t unpark the drive heads); this is WD’s horrible attempt at stating “try to not issue any drive I/O to the drive and let it sit idle as much as possible”. This approach is so moronic and wrong/blind that it’s unfathomable,
  3. Also within the first document you will see a recommended solution of disabling APM (Advanced Power Management) on the drive itself. However, the disable-APM advice is only sometimes applicable. Only some models of WD drives have a direct correlation between APM and the idle timeout feature, while others (such as the WD30EZRX) don’t support APM at all but still have the idle timeout feature. Furthermore, disabling APM also disables other internal features of the drive, which is a downside,
  4. Deep within the bowels of these documents you will find mention of a WD-specific utility which they call “Idle3”. This utility adjusts an internal “idle timeout” value within the drive itself (the config is probably stored in the HPA somewhere). However, this utility DOS-based, and often does not work with AHCI (and definitely will not work under RAID). Thus you are forced to not only reboot and deal with booting into MS-DOS or FreeDOS or whatever else you can find, but you’re forced to adjust your system BIOS to change your SATA controller’s option ROM mode to native SATA (or worse, PATA emulation) just to get this utility to work. Once you adjust the setting it stays (permanently, or until set back to defaults), but customers shouldn’t have to go through this rigmarole in the first place. Do not mistake this for TLER or SCT Error Recovery (same thing) — that is a completely unrelated feature and has no bearing on this issue.

So the last choice seems like the best, right? Wrong. A couple people have commented in that thread stating that their drive performance dropped horribly once they used the Idle3 utility to disable the aggressive idle timeout.

Furthermore, Western Digital’s own product support documentation states that this utility should not be used on this model of drive.

A review of the WD30EZRX back in 2011 indicated that the drive has a load/unload cycle count lifetime of 300,000 cycles. This came straight from WD’s own product documentation.

Furthermore, WD’s own forum is filled with people complaining about this problem, even on the WD30EZRX. One user has a drive that’s new/recent and is already reaching 284,000 cycles.

Needless to say, this WD30EZRX drive will be returned to its place of purchase for a full refund. Very, very disappointing.

I’ve discussed (ranted) at length with colleagues of mine how the storage industry is making a big mistake doing things like this. One pointed out that it’s basically impossible in this day and age to get a hard disk that just does what it’s supposed to: read/write. Instead, you have to reenact Riverdance just to get your disk working reliably without any nonsense in the way. The KISS principle has once again been ignored or forgotten.

Drive manufacturers continue adding new features that do nothing but make a mess for their consumers. This applies to more than just WD, and it’s incredibly depressing. In the case of WD, it’s quite obvious that the Green and Black drives use the exact same parts and thus are identical in mechanics, with the only difference being firmware.

With that in mind, take this into consideration: the WD2002FAEX (Caviar Black 2TB, 64MBytes cache) costs US$192.98, while the WD20EARX (Caviar Green 2TB, 64MBytes cache) costs US$116.00 (amounts taken on 2012/05/30). That’s a US$76.98 difference, per drive. This didn’t used to be the case though… until the whole Thailand flooding situation happened. It seems to me that as a result of an act of God someone is now using the opportunity to make serious cash where previously they were making much less. And I’m not the only one who noticed either. The world can blame the United States’ excessive focus on capitalism as the sole reason for this; it makes me sad.

And finally, if you think all the firmware versions are identical based entirely on the firmware string, you would be sadly mistaken. The below two Caviar Black 1TB drives are identical, yet they advertise completely different capabilities per the ATA specification IDENTIFY command. Note the lack of AAM on the 2nd drive, yet it’s quite clear they advertise the same firmware string:

Drive #1:

protocol              ATA/ATAPI-8 SATA 3.x
device model          WDC WD1002FAEX-00Z3A0
firmware revision     05.01D05
serial number         WD-WCATR1782965
WWN                   50014ee25a001e1c
cylinders             16383
heads                 16
sectors/track         63
sector size           logical 512, physical 512, offset 0
LBA supported         268435455 sectors
LBA48 supported       1953525168 sectors
PIO supported         PIO4
DMA supported         WDMA2 UDMA6

Feature                      Support  Enabled   Value           Vendor
read ahead                     yes      yes
write cache                    yes      yes
flush cache                    yes      yes
overlap                        no
Tagged Command Queuing (TCQ)   no       no
Native Command Queuing (NCQ)   yes              32 tags
SMART                          yes      yes
microcode download             yes      yes
security                       yes      no
power management               yes      yes
advanced power management      no       no
automatic acoustic management  yes      no      254/0xFE        128/0x80
media status notification      no       no
power-up in Standby            yes      no
write-read-verify              no       no
unload                         no       no
free-fall                      no       no
data set management (TRIM)     no

Drive #2:

protocol              ATA/ATAPI-8 SATA 3.x
device model          WDC WD1002FAEX-00Z3A0
firmware revision     05.01D05
serial number         WD-WCATR7016817
WWN                   50014ee25b1b1639
cylinders             16383
heads                 16
sectors/track         63
sector size           logical 512, physical 512, offset 0
LBA supported         268435455 sectors
LBA48 supported       1953525168 sectors
PIO supported         PIO4
DMA supported         WDMA2 UDMA6

Feature                      Support  Enabled   Value           Vendor
read ahead                     yes      yes
write cache                    yes      yes
flush cache                    yes      yes
overlap                        no
Tagged Command Queuing (TCQ)   no       no
Native Command Queuing (NCQ)   yes              32 tags
SMART                          yes      yes
microcode download             yes      yes
security                       yes      no
power management               yes      yes
advanced power management      no       no
automatic acoustic management  no       no
media status notification      no       no
power-up in Standby            yes      no
write-read-verify              no       no
unload                         no       no
free-fall                      no       no
data set management (TRIM)     no

I would love to have a long and detailed chat with present-day storage companies about all of the above. The KISS principle really needs to be put into effect here, and the sooner the better. Otherwise what we’re effectively doing is paying for firmware features — and ones we cannot disable or toggle. But then again I’m just a simple engineer — what do I know of such things, right?