Koitsu's Blog

How Intel’s SSD Toolbox Optimizer (TRIM) feature works

koitsu — Wed, 13 Feb 2013 00:05:49 +0000

There is a never-ending supply of people who don’t understand what TRIM is (or as it’s called per ATA specification, DATA SET MANAGEMENT) and how it’s used at the OS level.

How it works

The Optimizer feature creates a large number of what are called sparse files, which NTFS natively supports via the FSCTL_SET_SPARSE flag when calling DeviceIoControl(). For further information about NTFS and sparse files, see Microsoft’s documentation.

Before continuing on, please make sure you understand the concept of a sparse file. Sparse files are not files that are actively filled with data (i.e. not necessarily filled with zeros or some other content), but rather a files on the filesystem which have a predeclared size/length even though data has not been written to that allocated space. In fact, if the files were zeroed or had bytes written to them, not only would the Optimizer take significantly longer to run but would also add excessive wear/tear on NAND pages for no good reason (given what the Optimizer does during its subsequent steps — keep reading).

The files are stored at the root of the filesystem and are named --=={Intel_Trim_File_X}==--.bin, where X is an incrementing integer starting at 0. The files are not marked hidden and are untouchable (non-readable/non-writeable, due to how Windows’ default file locking model works) by processes other than the Optimizer.

The number of sparse files created depends upon how much free space is available on the filesystem; the Optimizer tends to leave exactly 1GByte (1,073,741,824 bytes) untouched while being run. Proof when run on an Intel 510 120GB SSD on Windows XP:

C:\>DIR --*
 Volume in drive C has no label.
 Volume Serial Number is 3442-346D

 Directory of C:\

02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_0}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_1}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_10}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_11}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_12}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_13}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_14}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_15}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_16}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_17}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_18}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_19}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_2}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_20}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_21}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_22}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_23}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_24}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_25}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_26}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_27}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_28}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_29}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_3}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_30}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_31}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_32}==--.bin
02/06/2013  16:01       291,819,520 --=={Intel_Trim_File_33}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_4}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_5}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_6}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_7}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_8}==--.bin
02/06/2013  16:01     3,221,225,472 --=={Intel_Trim_File_9}==--.bin
              34 File(s) 106,592,260,096 bytes
               0 Dir(s)   1,073,741,824 bytes free

It’s important to understand that with filesystems, not all data is contiguous (linear); NTFS cluster size or Allocation Unit Size (also often erroneously called “block size”) defines the size of a piece of data at the filesystem level. For example, if your NTFS cluster size is 16KBytes (16,384 bytes), a file that contains 300 bytes would take up 1 cluster (16,384 bytes) on the filesystem, while a file that contains 48,291 bytes would take up 3 clusters (49,152 bytes) on that same filesystem.

But as I said, the important part is to understand that these “clusters” do not have to be contiguous or stored in linear order; they can quite literally be scattered all over the disk (and this is what defragmentation utilities solve). Non-contiguous data only impacts performance with classic mechanical HDDs (MHDDs) due to seek time (which SSDs virtually lack).

How does TRIM fit into this?

So why does the non-contiguous nature of a “cluster” matter when it comes to the Optimizer? Because at the ATA level, TRIM only operates on LBA ranges — it has no knowledge about filesystems. For further details about TRIM, please see specification T13/2015-D revision 3 (working draft) or newer.

In turn, what the Optimizer does after allocating all these sparse files is to calculate the linear LBA ranges which those sparse files actually make up, then proceed to issue the raw DATA SET MANAGEMENT command to the controller for said LBA ranges, which in turn make it to the SSD. The SSD’s flash transition layer (a.k.a. FTL) then knows that LBA regions X to Y do not contain data that the filesystem/OS cares about any longer, and can be removed from (or marked unused within) the FTL map, and subsequently are considered by the SSD as “available for use”.

The sparse files are in effect used to allocate space (at the filesystem level) that can then be safely TRIM’d by the Optimizer itself.

Once all the TRIM operations have completed, the sparse files are deleted.

More about TRIM, its internal behaviour, and performance

Some folks have asked me in the past if the TRIM causes the underlying data at those LBAs to be zeroed, or if the data is simply left alone in a NAND page somewhere, waiting to be overwritten via the standard erase-program methodology of NAND flash.

The answer is: the behaviour depends on the implementation chosen by the SSD vendor. Table 35 of T13/2015-D revision 3 (working draft) specification defines what the methods/models are. Vendors are supposed to disclose the method via the ATA IDENTIFY command (0xec). Rather than explain it verbosely, I’ll just include a picture of the chart itself:

http://koitsu.files.wordpress.com/2013/02/trim_behaviour.png

From this we can see that the available methods of operation are:

No TRIM support
TRIM’d LBAs, when subsequently read, may return random/indeterminate data
TRIM’d LBAs, when subsequently read, return a consistent (static) value (e.g. 0xff)
TRIM’d LBAs, when subsequently read, return zero (i.e. 0×00)

The behaviour, as stated before, can be determined per the respective bits in the results of an ATA IDENTIFY command. Whether the SSD vendor chooses to implement that part of the spec is up to them (older SSDs may not, and even newer SSDs may not), and whether or not the utilities and OS you use that issue IDENTIFY to the device support showing you those bits is also up to the programmer and vendor, respectively.

Regardless of behaviour type, the important part is that the FTL map no longer considers those LBAs (or ranges of LBAs) as “used” by the filesystem.

TRIM is important for this sole reason: the more LBA entries (or LBA ranges) the FTL map contains, generally speaking, the worse the SSD performs.

Without TRIM, the SSD’s FTL map, over time, will lack any indication of available LBA ranges. This in turn forces garbage collection (a.k.a. GC) to be used instead, which is significantly less efficient than the OS (or application) telling the drive “I’m done with these LBAs”. GC has the major downside of halting all I/O to the drive while its internal calculations/analysis are performed (else risk marking an LBA or LBA region as unused at the same time as the filesystem/OS requesting use of that LBA for a legitimate file). The GC implementation and its behaviour varies between drive models and drive vendors, including some vendors going as far as performing actual filesystem-level analysis on the drive itself (something I feel is extremely risky).

Regardless of how efficient GC is today, the fact of the matter remains that performance is significantly worse relying entirely on GC. This is exactly why Intel’s Optimizer feature is so important for OSes which lack native TRIM support.

Furthermore, this is why you should not issue a “full format” (i.e. writing zeros or random data across the entire drive or a partition), sometimes (erroneously) called a “low-level format”, with an SSD. Unlike a classic mechanical HDD, if this is done on an SSD, the FTL map would almost certainly become full immediately afterwards (depending on if the formatting utility chooses to issue TRIM commands to the underlying device after writing zeros or random data to each LBA — in which case, there was no point in writing data there in the first place!).

For the SSD equivalent of a “full format”, the ATA SECURITY ERASE UNIT command (0xf4) should be used. This command resets the FTL map, as well as some other internal bits within the SSD. Intel’s SSD Toolbox “Secure Erase” feature does exactly this. The same can be accomplished within other OSes too.

So what about OSes that support TRIM natively? How does that work? Do I need the Optimizer?

There are many present-day OSes that support TRIM natively. Every Windows OS after Windows Vista has supported TRIM operation natively, while older OSes (like Windows XP) do not have support for it.

An OS having support for TRIM means that the OS will issue TRIM commands to the underlying device on certain I/O operations — the most common and widely used operation is file deletion. It is unknown (to me) whether or not Windows issues TRIM operations when doing other things, such as pruning (shrinking) the page file.

On FreeBSD, for example, TRIM is implemented via what’s called BIO_DELETE, which is used by the kernel for multiple things (not just file deletions). Because FreeBSD is open-source and the developers/engineers can be communicated with directly, knowing what uses BIO_DELETE is possible. With Windows, assumptions have to be made (and I would rather not make those), but it’s well-established that file deletions result in use of TRIM.

A Windows OS that supports TRIM natively does not require the Optimizer to be run. I cannot stress this point enough. In fact, depending on the behaviour of the TRIM implementation (see above), this could actually cause excessive wear/tear on the NAND flash if done repeatedly. Which leads me to…

A short story

In January I dealt with a report from a user stating his Intel 520-series SSD had “crapped out on him”, insisting it was caused by “a firmware bug” reported by some random Internet user (note the username) affecting only 520-series drives that were issued ATA STANDBY or ATA SLEEP commands by the OS, or when the drive was reattached on the SATA bus.

As I began to question his prognosis, he admitted that he was running Windows 7 or 8, did not let his drive go into standby/sleep, and did not excessively remove the drive from the bus. Further pressure revealed that he had scheduled the Intel SSD Optimizer function to run every day at 0300 — and found that his drive “had issues” around that time, repeatedly. Thus, he concluded on his own behalf that he was, somehow, “being bit by this [unsubstantiated] firmware bug”.

I was prompt to lecture him about this; he actually may have killed his own SSD due to his own misunderstanding/negligence of what TRIM is and how it works. I would have be able to determine this if he had provided SMART attributes for the SSD in question, but has since failed to provide them.

This misunderstanding and lack of education is heavily perpetuated by the “PC enthusiast” community, combined by badly-written documentation by vendors, neither of which are that individual’s fault of course. This just goes to show how someone’s lack of understanding of something can actually cause them grief.

An opinionated footnote

Over the past 15 years or so, there has been a growing trend where “PC enthusiasts” have been deemed reliable sources of information when it comes to technology. For every 5000 forum warriors there’s one person (probably an engineer) who actually understands the technology being discussed. While I fully admit SSDs are a fairly new technology that’s rapidly evolving, these “enthusiasts” remain a dominant source of reference material. I say that with as much modesty as possible.

It’s very important to remember that there is a world of difference between an engineer and an enthusiast: one works with the technology (almost certainly professionally) at a very low level, tends to be older (mid-30s and above), and often does not have the time to dedicate to providing information to the masses — while the other is rarely more than an I’ve-got-money-burning-a-hole-in-my-pocket all-I-care-about-is-how-many-frames-per-second-I-get consumer. Technology companies (especially all the big name brands) love the latter because they keep the cash rolling in.

I gladly acknowledge the importance of keeping the market flowing, but the proliferation of misinformation coming from such individuals is extremely damaging, especially to those of us who are engineers — we end up spending more time debunking, breaking down, and having to defend facts than if we were to simply provide good information from the get-go. But what people find on the Internet today is often speculative and en masse, and that disappoints me.

Bizarre Japanese news

koitsu — Wed, 23 Jan 2013 09:06:00 +0000

I came across this one today and had to post it. Parts that continue to make me LOL are bolded.

A man has been arrested on suspicion of biting his adult son to death to get rid of a “snake haunting him”.

TV Asahi reported that Katsumi Nagaya, 53, seriously injured his son, Takuya, 23, on Friday.

He was reported to have head-butted and bitten him after Takuya began writhing around claiming to be a serpent.

The son was declared dead after being taken to hospital.

The suspect reportedly told police he had attacked his son to remove the snake from his mind.

It’s a sad story, but also hilarious given its bizarre nature. The snake is long; seven miles…

FreeBSD, csup/cvsup, portsnap, and svn

koitsu — Wed, 02 Jan 2013 22:00:10 +0000

The subject of FreeBSD and it’s migration to SVN has come up on the mailing lists repeatedly over the past 6 months. The biggest problem seems to be the lack of communication and transparency of this change, combined with the recent security breach incident being used as a stepping stone to justify the situation (despite that incident having absolutely nothing to do with the transition of CVS->SVN). Peter Wemm has at least shed some light on the need for the migration itself; but it still leaves end users out in the cold (so-to-speak).

The most common question I’ve seen posted by end users is “how do I migrate?” and “is csup/cvsup broken?” The answers which have been provided by both the developer and user community have been abysmal at best, mainly because nobody actually bothers to take the time to explain the complexities and nuances involved in the migration. The official FreeBSD documentation is overwhelming and doesn’t take into account some of these nuances (nor should it, IMO). The migration itself is actually easy, but there are many catches along the way which are certain to bite even the most educated of administrators. I’ll also use this opportunity to explain why use of portsnap should be avoided if at all possible.

To try and keep it simple, I’ll outline each of the common setups people have, and how exactly to migrate from each one. I should be very clear with my intentions here:

By “migrate” I do not mean “converting your existing data on the filesystem to svn” — instead, all procedures are based upon the “start fresh, delete old cruft, and go from there” concept. It is not worth fiddling around with something so key/important to FreeBSD; think about it.
I strongly advocate management of both src and ports entirely via SVN. If you follow -RELEASE branches and do not care about src at all, I still recommend you use SVN to handle ports rather than portsnap (more on that later).
The SVN checkout URL I use for src is for what has historically been called RELENG_9 (i.e. 9.x-STABLE, 9.x-PRERELEASE, 9.x-RCx, etc.), not -RELEASE branches. I do not follow -RELEASE branches nor do I care about them; -RELEASE branches only contain fixes for security issues, and every single -RELEASE since the 5.x days has had too many general bugs that warrant immediate fixing.

Below is how I migrated my own systems, starting with a fresh OS installation. The following should be performed as root:

rm -fr /usr/ports
rm -fr /var/db/sup/* /usr/sup
portsnap fetch
portsnap extract
cd /usr/ports/devel/subversion
make install
cd /
rm -fr /var/db/portsnap/*
rm -fr /usr/ports
mkdir /usr/ports
svn checkout svn://svn.freebsd.org/ports/head /usr/ports
rm -fr /usr/src
mkdir /usr/src
svn checkout svn://svn.freebsd.org/base/stable/9 /usr/src
chmod 700 /root/.subversion
...Manually remove any SUP-related variables from /etc/make.conf...

Now to explain this utter madness.

Line 1 deletes the existing copy of /usr/ports from the filesystem
Line 2 deletes the CVS repository databases associated with src and ports; these are (were) managed by csup and cvsup themselves
Expanding on line 2: lots of people don’t know about these databases. Quite simply what’s in /usr/{src,ports} does not necessarily reflect what CVS thinks is in /usr/{src,ports}, and this is how/why there can be mismatches when going from one branch to another. Surprise! This is what bites a lot of people when switching between tags (i.e. going from RELENG_9_0 to RELENG_9 or vice-versa) or versions in general. So to make it clear: csup uses /var/db/sup for its database directory, while cvsup uses /usr/sup
Lines 3-4 use portsnap to fetch a copy of the /usr/ports tree. Why do we do this? Because we need to install subversion from ports
You may be asking: why can’t you just use pkg_add -r subversion? The reason is explained here by the port maintainer (read slowly, it’s easy to miss). As of this writing, the subversion package on the official FTP/package servers is old (October 13th) and pulls in an absurd number of dependencies when it shouldn’t. So we can’t use the package, thus we must build it from source ourselves. If/when this gets addressed by the maintainer and/or the package build cluster admins, yes, this procedure will become significantly simpler
Lines 5-6 build and install the subversion port itself, along with all of its dependencies. Please do not change any of the subversion options! The defaults as of this writing are as minimal as possible
At this point we have a working subversion (svn) binary that we can use
Line 7 changes the cwd to /, since we’ll soon be nuking /usr/ports (yes you read that correctly)
Line 8 deletes the portsnap database/tarball. This is the one of many reasons I cannot stand portsnap; it quite literally keeps an entire copy (tarball) of the /usr/ports tree laying around, by design. I wonder how many people with small /var partitions have gotten bit by this. portsnap also pulls down INDEX-x files for releases which aren’t relevant to your version of FreeBSD (i.e. it will pull down INDEX-7 on a 9.x system). portsnap’s ports tarball also lacks .svn/ directories (starting to get the picture?). All depressingly moronic
Lines 9-10 delete /usr/ports and recreate the directory
Line 11 checks out a copy of /usr/ports from the official FreeBSD SVN servers
Lines 12-14 are the same as 9-11 but for /usr/src and src
Please use svn:// and not http:// or https:// URI schemes — see below for why!
Line 15 fixes (what I consider to be) a security concern I have over /root/.subversion being globally readable. Not that I change anything in /root/.suvbersion/{config,servers} but some people might. This is the result of root’s default umask (0022), but I’m surprised svn doesn’t issue a chmod 0700 on the directory itself creates
Line 16 should be self-explanatory

It’s like I said — a simple procedure which is greatly complicated by many nuances that aren’t documented anywhere or explained by most.

For upgrading a system which already has a copy of /usr/ports, run what’s shown on line 2, then continue onwards starting at line 5. You can skip line 8 too, assuming portsnap has never been used prior.

Some other related topics or answers to questions I’ve seen asked (or asked myself!):

To update your ports tree, you can do any of the following commands — they are all effectively the same:

cd /usr/ports && make update
cd /usr/ports && svn update
cd /usr/ports && svn checkout
svn update svn://svn.freebsd.org/ports/head /usr/ports
svn checkout svn://svn.freebsd.org/ports/head /usr/ports

To update your src tree, you can do any of the following commands — they are all effectively the same (note that “make update” can no longer be used):

cd /usr/src && svn update
cd /usr/src && svn checkout
svn update svn://svn.freebsd.org/base/stable/9 /usr/src
svn checkout svn://svn.freebsd.org/base/stable/9 /usr/src

For the ports INDEX-x file itself, just use the not-so-well-documented “fetchindex” target. However, since the security breach, the generation of this file on the official FreeBSD servers has broken many times (3 or 4 so far). This has been complained about repeatedly by users; in that case, the only choice is to generate it yourself, which takes a very long time and often emits lovely warnings. Anyway, using fetchindex is simple:

cd /usr/ports && make fetchindex

I do not recommend changing the URI scheme from svn:// to http:// or https://. There have been repeated complaints of issues/problems with the HTTP and HTTP/SSL services hosted at svn.freebsd.org, by both users and committers (so many that I do not care to reference them all). Use native svn:// and be done with it; don’t cause yourself headaches using the other schemes. If you absolutely must switch the URI scheme and can deal with the annoyance, follow this procedure.
svn will never be added to the base system due to its large number of dependencies. There may also be licensing concerns with regards to some of those dependencies. I am one of the few people who wants to see the “base system” concept done away with entirely (everything should just be a package/port/whatever and managed that way), but the push-back I’ve received over the years has been from developers insisting that the base system must be kept to cater to commercial vendors that use FreeBSD (such as Juniper, NetApp, Citrix, etc.). The disconnect here is hilarious, and I doubt it will ever be fully addressed.
cvs will eventually be removed from the base system. I myself use cvs for my own projects; I imagine this won’t impact most people, but it does impact me. I’ll be forced to install devel/cvs from ports.
pkgng — if anyone mentions it to you — has absolutely no bearing/relevancy to the subject matter being discussed in this blog post. If someone tells you to install pkgng or switch to it to solve any of this, they are horribly confused and wrong. And besides, this right here is a good reason to avoid pkgng. Be sure to read the entire thread; also as of this writing there have been no follow-ups to it. Remarkable.

Instructions for building TomatoUSB Toastman RT-N firmwares

koitsu — Sat, 22 Dec 2012 03:45:45 +0000

Back in September/October I had the “pleasure” of dealing with trying to build TomatoUSB (specifically Toastman’s RT-N releases) on a Linux system I have (thank god for VMware Workstation!). I was given all sorts of reference materials from folks on the linksysinfo.org forum, except I kept running into all sorts of problems. I tried other Linux distributions, other releases of the same distribution, etc. and the failures all seemed to differ.

I figured it would be worthwhile to document exactly how I got the firmware to build and what the necessary steps are as of the date of this blog post.

Prerequisites:

Linux system running a 32-bit (i386/x86) kernel. I was not able to get 64-bit (amd64/x64) working (details in a moment). Distributions I can confirm work are Ubuntu 11.04, Ubuntu 11.10, Debian 6.0.6, and Debian 6.0.7.
- Reason: the toolchain binaries stored in the official git repository were built on a Linux 2.6.8 32-bit system
- The failure message you’ll receive if using a 64-bit system is something like mipsel-linux-uclibc-gcc: No such file or directory. The message is misleading to anyone not highly technical and familiar with multiple architectures; I imagine most users would have great difficult figuring this one out. This is not a library-related problem; this is the kernel or ELF loader not having support for 32-bit binaries (I did not care to figure out which)
- The proper way to solve this is to rebuild the toolchain from source, except I cannot find any documentation stating how to do that correctly
Around 20GBytes of disk space (really!), which should suffice for the Linux/base install and all of TomatoUSB
Root-level access. This is needed because symlinks need to made within the /opt directory
sudo must be installed in advance (the build system uses it)

Step-by-step instructions are below. Please note that these must be done as a user, and not as root! Building the firmware as root is not supported!

sudo apt-get install libncurses5 libncurses5-dev m4 bison flex libstdc++6-4.4-dev
sudo apt-get install g++-4.4 g++ libtool sqlite gcc g++ binutils patch bzip2 flex
sudo apt-get install bison make gettext unzip zlib1g-dev libc6 gperf sudo
sudo apt-get install automake automake1.7 automake1.9 git-core

cd ~
git clone git://repo.or.cz/tomato.git
cp -pR tomato tomato.orig

sudo mkdir -m 0755 -p /opt
sudo ln -s ~/tomato/tools/brcm /opt/brcm

cd tomato
git checkout Toastman-RT-N

export PATH="$PATH:/opt/brcm/hndtools-mipsel-linux/bin:/opt/brcm/hndtools-mipsel-uclibc/bin"

cd release/src-rt

make V1={versionstring1 V2={versionstring2} {buildtype}

Please note during the build you will see quite a large number of Makefile errors which are ignored (e.g. “Error 1 (ignored)”), and an even larger number of warnings during compiling. It absolutely blows my mind that something as critical as an embedded firmware is maintained in such a sloppy manner; I am one of those coders who uses -Wall -Werror and for a damn good reason.

If the compile completes successfully, you’ll end up with a firmware in the release/src-rt/image directory.

Now for some explanation of what the {versionstringX} and {buildtype} placeholders are:

The two {versionstringX} placeholders can be anything you want; they're used to create the actual firmware filename. For example, to make an "official-looking" Toastman firmware file called:

tomato-K26USB-1.28.0500.5MIPSR2Toastman-RT-N-Ext.trx

...you would do:

make V1=0500.5 V2=Toastman-RT-N r2m

{buildtype} is rather lengthy. Rather than explain all the possibilities here, it's a lot easier to just direct you to run make help in the release/src-rt directory and read the results for yourself. But just for posterity, here's the list as of this writing:

a            A build (standard)
b            B build (standard minus SSH)
c            C build (standard minus CIFS)
d            D build (standard minus Samba server)
m            M build (standard plus extra utilities and NTFS support)
nc           NC build (standard plus VPN, extra utilities, NTFS support and NOCAT)
ncm          NCS build (no USB support, NOCAT)
ncs          NCS build (std + NOCAT)
m            Ext - (standard plus extra utilities and NTFS support)
e            E build (standard plus VPN, extra utilities and NTFS support)
i            I build (IPv6 with no USB support minus CIFS and RIPv1/2)
s            S build (no USB support)
f            F build (no USB support minus CIFS and RIPv1/2)
fs           FS build (no USB support minus CIFS, JFFS, Zebra and RIPv1/2)
v            V build (VPN with no USB support)
vs           VS build (Small VPN with no USB support)
r2a          MIPS Release 2 A build (standard)
r2b          MIPS Release 2 B build (standard minus SSH)
r2c          MIPS Release 2 C build (standard minus CIFS)
r2d          MIPS Release 2 D build (standard minus Samba server)
r2e          MIPS Release 2 E build (standard plus VPN and extras)
r2f          MIPS Release 2 F build (no USB support minus CIFS, RIPv1/2)
r2fs         MIPS Release 2 FS build (no USB support minus JFFS, CIFS, RIPv1/2, Zebra)
r2i          MIPS Release 2 I build (IPv6 with no USB support minus CIFS and RIPv1/2)
r2m          MIPS Release 2 M build (standard plus extras)
r2nc         MIPS Release 2 NC build (standard plus VPN, extras and NOCAT)
r2ncm        MIPS Release 2 NCM build (no USB support, plus NOCAT)
r2ncs        MIPS Release 2 NCS build (standard plus NOCAT)
r2s          MIPS Release 2 S build (no USB support)
r2t          MIPS Release 2 T build (no USB support with extras)
r2u          MIPS Release 2 U build (no USB support plus VPN with extras)
r2v          MIPS Release 2 V build (VPN with no USB support)
r2vs         MIPS Release 2 VS build (Small VPN with no USB support)
       Linksys E-Series with 60k Nvram
n60m         Linksys E-series build (60KB NVRAM, standard plus extras)
n60e         Linksys E-series build (60KB NVRAM, standard plus VPN and extras)
n60nc        Linksys E-series build (60KB NVRAM, standard plus extras, plus VPN, plus NOCAT)
n60s         Linksys E-series build (60KB NVRAM, no USB support)
n60v         Linksys E-series build (60KB NVRAM, VPN with no USB support)
       Linksys E-Series with 64k Nvram/8MB Flash(E900/E1200v2/E1500)
n64m         Linksys E-series build (64KB NVRAM, standard plus extras)
n64e         Linksys E-series build (64KB NVRAM, standard plus VPN and extras)
n64nc        Linksys E-series build (64KB NVRAM, standard plus extras, plus VPN, plus NOCAT)
n64s         Linksys E-series build (64KB NVRAM, no USB support)
n64v         Linksys E-series build (64KB NVRAM, VPN with no USB support)
       Linksys E-Series with 4MB Flash(E1000v2/v2.1/E1200v1)
e1000v2i     Linksys E1000v2-v2.1/Cisco M10v2 MiniIPv6
e1000v2f     Linksys E1000v2-v2.1/Cisco M10v2 Mini
e1200v1i     Linksys E1200v1 MiniIPv6
e1200v1f     Linksys E1200v1 Mini
       Netgear WNR3500L v2 builds (128MB Flash)
v2e          Netgear WNR3500Lv2 build

If you wish to adjust Linux 2.6 kernel features, you will need to edit release/src-rt/linux/linux-2.6/config_base b/release/src-rt/linux/linux-2.6/config_base manually. Do not use make menuconfig! There are known adjustments applied manually during the build stage that may break if you do that.

If you want to add/remove features from netfilter (related to iptables), you will need to edit release/src/router/iptables/extensions/Makefile manually.

Here's an example where both of these files needed to be edited to add support for the IPv4-only xt_string.ko module (iptables -m string support).

I hope this helps folks in some way/shape/form, as I found the entire process to be quite badly documented in general.

Recent FreeBSD.org “cluster server” security breach

koitsu — Sat, 17 Nov 2012 17:39:23 +0000

For the past 4-5 days, users of FreeBSD have been unable to get any updates from servers worldwide. Specifically, this affected the following services:

csup and cvsup (for src, ports, doc, etc.)
portsnap (for ports only)
freebsd-update (for binary updates/security updates, if not using csup/cvsup)
Native CVS (note: intentionally using cached copy from Google)
GNATS (official FreeBSD problem reporting/bug tracking system)

All that was being stated publicly by FreeBSD Project members was that there was “an issue”, and then later this turned into “maintenance”:

Some others also began to wonder about how the outage was being handled, since nobody had updated the freebsd.org home page (until the aforementioned), updated the official RSS feed, posted anything on Twitter, or (most important of all) sent anything to the official freebsd-announce mailing list.

At the 4-day mark, I began wondering what kind of “maintenance” was unannounced and not done without some degree of prep-work. And it turns out my system administrator gut feeling was right — today the root cause was actually disclosed:

http://www.freebsd.org/news/2012-compromise.html

The omission of detail (particularly of the technical nature) in this report is astounding, resulting in almost more questions than answers. But worse, the report appears to be written in such a way to make it into a scapegoat for pushing portsnap and svn (instead of csup/cvsup), which makes absolutely no sense on multiple levels. Some actual FreeBSD Project members have questioned this as well and I applaud them for doing so.

First and foremost, the report seems to indicate the compromise provided the intruder a way to modify or replace packages on the master package server. If that were the case, the attacker then would have been able to modify relevant checksums/hashes of packages. portsnap just acts as a “ports tree snapshot” utility, it doesn’t download or verify each individual ports’ updates and do checksum validation (at least from the client-to-server perspective) — and even if it did, again, if the attacker had access to modify files used by the portsnap master to generate its update snapshot… You can see where I’m going with this.

Secondly, we do not know if this individual had root-level access or not, just that they could have potentially modified packages. So people citing concerns of csup/cvsup being susceptible to MITM has absolute zero relevancy to what actually transpired in this compromise. Unless the intruder had the ability to stop + start the portsnap mirroring daemon (or whatever it uses — there aren’t details) on the portsnap master server itself, or a way to stick a MITM daemon in the middle by siphoning packets to/from another daemon on the machine (or the 2nd machine mentioned in the report), then MITM has no bearing on this situation thus the concern cited is pure FUD with regards to this compromise. We FreeBSD users have an awful tendency to try and divert attention/focus to something else (perfect example) instead of zeroing in on the problem at hand; “how do I solve X” is never answered with “you do Y”, instead it’s “why are you trying to solve X” or “there’s nothing to solve” or “let me give you a one-liner that practically demeans you personally”.

Thirdly, I still cease to see how users using svn (instead of csup/cvsup or cvs natively) would have rectified this situation directly or indirectly. This is a bit long/heavy, but detail and history are needed to understand my as-usual KISS principle stance:

The problem with svn is that it isn’t in the base system. csup is in the base system, which means right out of the box FreeBSD users can immediately begin getting updates to whatever src branch they follow, and/or ports. Users have repeatedly asked what the replacement for csup/cvsup is, and all they’re told is “portsnap and svn”, with no further information given.

Now, anyone who knows me knows I’m generally against the “base system” concept — that is to say, included software should all be packages or equivalent (i.e. no more “OpenSSL ports” vs. “OpenSSL base”, “gcc ports” vs. “gcc base”, etc.). But as long as the “base system” concept is kept, then it needs to be respected and treated as important. And in fact, the security breach report in question even alludes to its importance. Quote:

“No part of the base FreeBSD system has been put at risk. At no point has the intruder modified any part of the FreeBSD base system software in any way.”

The problem with svn is that it pulls in a crapload of runtime dependencies. The biggest and most notorious is SQLite — which I have no problem with as software, but I’m focused on the dependency count here. However, APR is also required (certainly because Subversion is an Apache project, sigh), as is gettext (NLS/character set support), expat (XML support (why?)), and neon29 (needed for HTTPS support — OPTIONS doesn’t say this but believe me it is):

root@icarus:/usr/ports/devel/subversion # make run-depends-list
/usr/ports/databases/sqlite3
/usr/ports/devel/apr1
/usr/ports/devel/gettext
/usr/ports/textproc/expat2
/usr/ports/www/neon29

All of these things would need to be pulled into FreeBSD for svn to make it into the base system. And I’m certain, especially in the case of gettext, there would be licensing concerns.

But maybe a history lesson is needed here…

Older FreeBSD users may remember the days of cvsup, where that also wasn’t included with the base system (because of its dependency on the Modula-3 language). All of us got used to doing pkg_add -r cvsup-without-gui after our OS installation and considered the matter closed, although it pulled in ezm3 (and that was its only dependency). I was one of the few who was annoyed by this dependency, and the author used to repeatedly state that Modula-3 was absolutely needed for some of the design choices of cvsup. It greatly amused me when years later csup was created, which was cvsup written completely in C — as you can see, Modula-3 isn’t needed at all, so this was just another example of a software author making his belief system everyone else’s problem, rather than being pragmatic.

Here we are again, with history repeating itself, just in a different way. So why don’t I just pkg_add -r subversion and be done with it? Mainly because it pulls in 5 dependencies, which means 5 things I have to deal with/maintain when it comes to software upgrades and so on. I can accept gettext (since it’s used by lots of other software anyway), but the others I really don’t want to deal with.

Okay, so what choices do we have? Well, the svn port can be built with the STATIC option, which results in a statically-linked build. That sounds good to me — except I don’t want to deal with building the software. Why? Because building it (vs. running it) pulls in even more dependencies:

root@icarus:/usr/ports/devel/subversion # make all-depends-list
/usr/ports/devel/libtool
/usr/ports/databases/sqlite3
/usr/ports/textproc/expat2
/usr/ports/www/neon29
/usr/ports/devel/apr1
/usr/ports/devel/gettext
/usr/ports/devel/pkgconf
/usr/ports/lang/perl5.12
/usr/ports/lang/python27
/usr/ports/devel/automake
/usr/ports/devel/autoconf
/usr/ports/converters/libiconv
/usr/ports/devel/automake-wrapper
/usr/ports/devel/m4
/usr/ports/misc/help2man
/usr/ports/devel/gmake
/usr/ports/devel/autoconf-wrapper
/usr/ports/devel/p5-Locale-gettext

And that brings me to my final point: so why can’t the FreeBSD package/build cluster build a static version of svn for people to pkg_add -r? A wonderful question, and it’s one I’ve asked many times over. Technically there is no reason this can’t be done. Purely on a technical level, building that static package would be quite easy in the situation where any of the runtime dependencies needed to be updated. The maintainer would then simply increment PORTREVISION in devel/subversion/Makefile to act as an indicator that the package/port should be rebuilt due to underlying dependencies changing (and this is how the ports system maintainers already operate). So why can’t subversion-static and subversion16-static packages be made? Nobody seems to know, only that “you can do it yourself” (and I won’t for the reasons listed above — plus this is what the package building cluster at FreeBSD.org is for).

This is why I continue to stick with csup, and I imagine many other FreeBSD users do (and will) as well. When the cvsup mirrors go down sometime in 2013 (it’s estimated February), I will most likely moving over to Debian Linux. No, Linux doesn’t magically solve all of this (and I will greatly miss the pf firewalling stack), and believe me Debian has its share of issues, but this kind of clown-car seat-of-the-pants operation needs to end — at least for me. The irony, again for me, is that I moved from Linux to FreeBSD in 1997 solely because of clown-car syndrome.

Borderlands 2 performance tweaking

koitsu — Sun, 30 Sep 2012 05:39:24 +0000

I’ve had a couple people online ask me what all I’ve done to my Borderlands 2 .ini files to solve some annoyances and framerate-related issues within the game, as well as decrease the amount of pressure on the GPU. I thought I’d take the time to list off all the changes I use for my GTX 560 Ti + Core i7 2600K-based system running on Windows XP SP3, under 1920×1200.

Before I get started, I want to make a few things clear in advance:

I don’t like games which cause my video card to kick its fans up into high speed due to high GPU or graphics MCH stress; I prefer to hear the sounds of the game without the white noise of CPU or GPU fans masking things. Likewise, temperatures like 60C aren’t acceptable by me.
I prefer to play games with Vsync enabled, even with LCDs. I can see screen tearing quite easily, even with things like nVidia’s adaptive Vsync in use. Windows XP doesn’t make accomplishing proper Vsync in windowed mode very easy (for DirectX/Direct3D or GDI applications for that matter). Using Vsync has an additional benefit with regards to the previously-mentioned item: it does in fact cap the maximum framerate. Windows 7 is quite a different beast in this regard; I’m talking on XP.
At least on Windows XP, the game must be run in full-screen mode (not “windowed fullscreen” either) to get the smoothest performance with Vsync enabled. This condition isn’t specific to Borderlands 2 either; the same advice applies to Diablo 3.
Borderlands 2 on my system tended to run around 30-40fps in “heavy” areas (such as the starting area — and don’t ask me what’s “heavy” about it). With the below tweaks, I get 60fps pretty much all the time, barring some areas where there is obviously a lot of detail. So overall the feel of the game is much, much better.
Unlike forums and “enthusiast sites”, I will happily explain what all of these settings do, rather than just tell you “hey u set theiz i g3t 10fpsss lol kbye”.

That said, here are the settings I use (meaning all values that differ from the stock .ini defaults):

WillowEngine.ini –

[Engine.Engine]
DefaultPostProcessName=WillowEngineMaterials.RyanScenePostProcess
MipFadeInSpeed0=0
MipFadeOutSpeed0=0
MipFadeInSpeed1=0
MipFadeOutSpeed1=0

[SystemSettings]
DynamicShadows=false
DepthOfField=false
AmbientOcculsion=false
Bloom=false
bAllowLightShafts=false
UseVsync=true
Fullscreen=true
WindowedFullscreen=false
MaxAnisotropy=0
FXAA=false
ResX=1920
ResY=1200
FoliageDrawRadiusMultiplier=0.500000
ViewDistance=2
FramerateLocking=6

[FullScreenMovie]
; StartupMovies=2K_logo
; StartupMovies=Gearbox_logo
; StartupMovies=NVidia
; StartupMovies=Loading

WillowInput.ini –

[Engine.PlayerInput]
bEnableMouseSmoothing=false

Now let’s talk about each of the settings individually. I’ll provide reference materials for each of them (if applicable); read and learn!

WillowEngine.ini –

DefaultPostProcessName=WillowEngineMaterials.RyanScenePostProcess — disables “vignette” or pseudo-cel-shaded-with-black-borders look. Some people like it — I don’t, and I didn’t like it in Borderlands 1 either (Reference)
MipFade{In,Out}Speed{0,1} — when set to 0, effectively disables the “texture streaming” feature, meaning the game will “pop in” a texture when it loads (vs. show some half-ass blurry garbage in the meantime). And yes, I am quite aware of the bUseTextureStreaming variable; go right ahead and set that to false and see what happens (hint: don’t do it) (Reference)
DynamicShadows — handles (to some degree) support for polygon-based shadows generated by moving objects. Disabling this has a very big effect on performance, just like it did in Borderlands 1 (Reference)
DepthOfField — (Reference)
AmbientOcculsion — (Reference)
Bloom — I’ve never liked the way bloom looks and it often kills performance (especially when used with HDR) (Reference)
bAllowLightShafts — (Reference)
UseVsync — should be obvious
Fullscreen — should be obvious
WindowedFullscreen — should be obvious
MaxAnisotropy — a value of 0 disables anisotropy (Reference)
FXAA — (Reference)
Res{X,Y} — should be obvious
FoliageDrawRadiusMultiplier — a value of 0.5=Near (Reference)
ViewDistance — a value of 2=High (compared to 3=Ultra High)
FramerateLocking — a value of 6=Unlimited. I set this to 6 because UseVsync effectively provides a cap on the framerate while ensuring no screen tearing
StartupMovies — commented out because I absolutely abhor these things, even if they’re skippable. Note that this is different than setting bForceNoMovies=true — that disables all in-game movies/cinematics rather than just the worthless startup videos

Despite what you might read online, I do not recommend setting DynamicLights=false to gain performance. The gain is negligible and the results in-game are crap (especially during the stage loading screens where you’ll see a rotating gun; now you’ll see a rotating set of pitch back polygons). Keep DynamicLights=true!

WillowInput.ini –

bEnableMouseSmoothing — just Google this one yourself. For a lot of people, disabling this allows for a much more responsive mouse; I disable it by habit. And if someone tells you to set OneFrameThreadLag=false because it also helps: don’t!

Naturally you should adjust these to your own liking.

Finally, for those who play the game via Steam, disabling the horrible Borderlands 2 launcher is a snap: do a Properties on the game, choose the General tab, pick Set Launch Options…, and put the string -nolauncher in the provided field.

Using FreeBSD graid (GEOM RAID)

koitsu — Tue, 18 Sep 2012 12:17:19 +0000

Quite some time ago I moved my home system away from ZFS and over to gmirror (GEOM mirror). There were many reasons for this, all of which are outside of the scope of this post thus I will not go into them. Things were good with gmirror, until after rebuilding world somewhat recently I began seeing this message:

GEOM: ada1: the secondary GPT header is not in the last LBA.
GEOM: ada2: the secondary GPT header is not in the last LBA.
GEOM_MIRROR: Device mirror/gm0 launched (2/2).

My initial reaction was: “um, no shit, and there’s no way to solve that, so why are you bitching?”

This problem is a well-known (but not as much as it should be) design mistake on the part of gmirror: the last physical LBA (sector) of the provider is used to store the metadata. The issue with this is that GPT also uses the exact same design choice, so you end up with a conflict of whose data ends up in the last LBA of the drive: either gmirror’s metadata or the backup GPT header. You can’t have both, and the FreeBSD Handbook even says so. I will point out to readers that even FreeBSD kernel developers have commented on this awful design choice (and were subsequently shot down). Be sure to note that the kernel developer discussion took place in July 2009 — over 3 years ago.

This becomes a serious problem with the introduction of 4K-sector disks and SSDs, where the only way to achieve proper partition alignment is to use GPT (compared to classic MBR). If you think I’m kidding, poke around with gpart(8) and it’s -a and -b flags using the MBR scheme sometime — good luck aligning a partition to a proper 1MByte boundary.

The “socially propagated” workaround for this cat-and-mouse problem is to create partitions on each disk and mirror those. Meaning: instead of using gmirror to mirror ada1 and ada2, you would use gpart(8) to create GPT partitions on ada1 and ada2, respectively, then using gmirror to mirror ada1p1 and ada2p1.

However, what nobody bothers to mention is that there are major ramifications to this workaround. The biggest problem, in my opinion, is when a disk fails. No longer do you yank the old disk + put in a new one and watch the mirror get rebuilt. Instead, upon insertion of the new disk, the administrator now has to manually fuddle with gpart(8) to re-create the partition table (and it must be 100% identical to that of the working disk) then “poke” gmirror to get it to re-read the partition table. Only then will the array rebuild. Sorry, this is just not acceptable — it almost smells of ensuring job security (“yes boss, when a disk fails I need to be there to do some magic”). Completely unacceptable.

Given this problem, today I decided to take the plunge and play with graid(8). I mentioned wanting wanting to do so last year but never got around to it until now; with ada1 and ada2 being 4KByte sector disks, I had a lot more reason to consider it.

First and foremost, I want to make something very clear: users considering graid should make sure they are running RELENG_9 from source dated 2012/05/25 UTC or newer. There was a bug in the graid code where if a write failure happened (disk pulled, disk failure, etc.) the entire system could panic. This was fixed in RELENG_9 source dated roughly 2012/05/24 UTC. When 9.1-RELEASE comes out, this fix will obviously be in there too.

Enabling this capability was quite simple on my Supermicro X7SBA system (Intel ICH8R-based): I changed the PC BIOS from AHCI mode to RAID mode. Once I did that, the AHCI option ROM loaded (RAID mode uses AHCI), followed immediately by the RAID option ROM being loaded. Pressing Ctrl-I allowed me to define two disks as part of a RAID-1 array. Stripe size was not selectable (option ROM appeared to pick 128KBytes (131072 bytes) on its own — more on stripe sizes later, as there is a very confusing set of terminology used that makes things very easy to mess up). Great, seemed easy enough.

My kernel config included the GEOM_RAID option so I did not need to manually load geom_raid.ko myself. I let the system boot and sure enough the RAID array was tasted and a new device called /dev/raid/r0 was created automatically.

I should note that the GEOM_RAID layer also emit messages about something called Intel-cfe2a69f, which appears to be the internal reference name which graid relies on when referring to the array itself. The hexadecimal digits which come after the “Intel-” string appear to be randomly generated every time an array is created.

However, there was one very disheartening flaw: upon the array being discovered, according to graid status, the array began rebuilding for no reason. The array was labelled DEGRADED; disk ada1 was in perfect shape but ada2 was being rebuilt. Huh? This was a brand new array.

Rather than let 1TB of zeros get mirrored (there’s no point to that), I began fooling around with graid. The first thing I did was attempt to get the rebuild to stop. I did a graid list to look at what the Intel RAID BIOS had created on its own; this is where I found “Strip: 131072″ — note Strip is not the same thing as Stripesize.

I decided to take aggressive action and completely destroy the array I had just created via the native Intel RAID BIOS: graid delete Intel-cfe2a69f did the trick.

Next, I opted to create the array from within FreeBSD rather than via the RAID BIOS. These sorts of things need to work reliably from within the OS, so I figured I might as well test them out while I had the chance. I issued the following command:

graid label -s 131072 Intel data RAID1 ada1 ada2

The reason I explicitly specified -s 131072 is because graid defaults to a “strip” of 65536 (per the man page). I wanted to ensure the value was the same as what the Intel RAID BIOS would have picked.

On the console, messages from GEOM_RAID showed up, and I had a brand new array available — and this time, no array rebuilding. graid list showed everything identical, and graid status showed everything in an optimal state. The internal reference name was different (this time “Intel-1512a6e3″) but so what. Output:

# graid list
Geom name: Intel-1512a6e3
State: OPTIMAL
Metadata: Intel
Providers:
1. Name: raid/r0
   Mediasize: 1000204881920 (931G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   Subdisks: ada1 (ACTIVE), ada2 (ACTIVE)
   Dirty: Yes
   State: OPTIMAL
   Strip: 131072
   Components: 2
   Transformation: RAID1
   RAIDLevel: RAID1
   Label: data
Consumers:
1. Name: ada1
   Mediasize: 1000204886016 (931G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   ReadErrors: 0
   Subdisks: r0(data):0@0
   State: ACTIVE (ACTIVE)
2. Name: ada2
   Mediasize: 1000204886016 (931G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   ReadErrors: 0
   Subdisks: r0(data):1@0
   State: ACTIVE (ACTIVE)

# graid status
   Name   Status  Components
raid/r0  OPTIMAL  ada1 (ACTIVE (ACTIVE))
                  ada2 (ACTIVE (ACTIVE))

I then made a filesystem on /dev/raid/r0 by doing newfs -U /dev/raid/r0 and mounted it. I also added a relevant line to /etc/fstab.

I did some basic sequential I/O tests (reads and writes), to make sure there weren’t any performance problems since the ada1 and ada2 disks were 4KByte sector drives. Initial sequential write I/O was around 157MBytes/second, which is about right for these models of disks. Fantastic.

I then rebooted the system (always good to make sure things come up cleanly). Output from the kernel:

GEOM_RAID: Intel-1512a6e3: Array Intel-1512a6e3 created.
GEOM_RAID: Intel-1512a6e3: Disk ada1 state changed from NONE to ACTIVE.
GEOM_RAID: Intel-1512a6e3: Subdisk data:0-ada1 state changed from NONE to ACTIVE.
GEOM_RAID: Intel-1512a6e3: Disk ada2 state changed from NONE to ACTIVE.
GEOM_RAID: Intel-1512a6e3: Subdisk data:1-ada2 state changed from NONE to ACTIVE.
GEOM_RAID: Intel-1512a6e3: Array started.
GEOM_RAID: Intel-1512a6e3: Volume data state changed from STARTING to OPTIMAL.
GEOM_RAID: Intel-1512a6e3: Provider raid/r0 for volume data created.

Excellent.

I then began restoring data from my backup drive to the array. This is when things started to concern me.

Stability-wise everything was fine. Disk I/O performance was also fine (monitoring using gstat -I500ms). What concerned me was how the system “felt” during heavy I/O. During an rsync -avH /backups/mirror/ /array/ I noticed that any disk I/O (reading or writing) from/to the /array filesystem was extremely slow. Things would block (wait) for very long periods of time before moving forward. Things like mutt -y (which loads 19 mbox mailboxes, totalling around 30MBytes of disk total) took 8-9 full seconds. This behaviour is easily reproducible. But when the rsync wasn’t running things were speedy.

So basically interactivity/usability during heavy I/O seems worse than with gmirror, but the overall disk I/O speeds are identical. I’m not sure if this is the result of using BIOS-level RAID or not, nor do I have any good way to determine “where” the slowdown is happening within the kernel. I imagine there will be some which will claim this is due to the lack of disk I/O scheduler in FreeBSD (and don’t mention gsched — that thing causes more problems than solutions, for example how it causes sysinstall to segfault due to what it does to kern.geom.conftxt). I do not believe lack of disk I/O scheduler is the cause; I believe there is a different explanation.

I have not tested things like yanking disks while I/O is occurring and things of that nature.

Anyway, that’s my experience with graid as of this writing. Big thanks to Alexander Motin , Warner Losh , and the iXSystems folks for graid, as it’s something FreeBSD has needed for a very, very long time.

Hard disk drive heights and polypropylene storage containers

koitsu — Wed, 12 Sep 2012 03:07:44 +0000

Given how many hard disks I tend to have, I recently decided to purchase some anti-static polypropylene foam containers to store my drives.

Common retail outlets (Amazon and others) were selling these containers (which would hold between 10 and 24 disks) but for crazy prices: US$100, US$130, and US$150. And some other retail vendors want US$425.

The price was simply too steep; I wasn’t needing something brand new, used would be fine. So I turned to the one place where stuff like this tends to end up being sold for reasonable prices: eBay.

Before continuing I want to make something crystal clear: I am not shunning the eBay seller in any way. The seller was communicative, fair, and permitted me to return the products for a full refund. So please do not avoid purchasing things from this seller just because of my story — instead, you should be aware of what it is you’re actually getting (though I do wish the seller would update the product description to reflect reality).

Almost immediately I found exactly what I was looking for. Advertised as holding up to 25 3.5″ hard disks (quoting the product description: “They fit all 3.5″ hard drives”), they were perfect. Or so I thought…

I purchased 3 (only needed 2 but figured a spare might be useful). Shipping took about a week. I was thrilled once they arrived, but irritation ensued once I tried fitting drives in each slot: the majority of my drives wouldn’t fit properly.

It quickly became apparent to me what the issue was, and it was something that anyone unfamiliar with hard disk drives could have overlooked.

When it comes to 3.5″ hard disks, there are two flavours:

Those which are 26mm (~1.072 inches) in height — these tend to be the majority (~90% of the time). An example would be the Western Digital WD3201ABYS,
Those which are 20mm (~0.790 inches) in height — these are all single-platter and are intended to be used in thin/slim desktops, SFF PCs with limited height clearance, or certain models of 1U servers. An example would be the Maxtor STM3160815A.

Here’s a side-by-side photo of the above two drives so you can see the height differences:

http://koitsu.files.wordpress.com/2012/09/drive_comparison.jpg

Hard disk drives are always enclosed in anti-static bags for multitudes of reasons (static discharge possibly damaging the PCB, keeping dust off the interface connectors and PCB, and so on). Such bags tend to add between 1-2mm to a drive’s width and height. So effectively what you end up with is either 28mm or 22mm heights, respectively.

All 3 containers I received had slots which measured between 21mm (bottom of slot) and 23mm (top of slot) for drive height:

Insertion of a 26mm height drive required excessive force and would not fit all the way to the bottom of the slot — but worse, the foam dividers between slots would bend/warp, defeating the use of the slot next to it:

http://koitsu.files.wordpress.com/2012/09/drive_26mm_01.jpg

…while a 20mm height drive fit just fine (and with anti-static bag, fit snugly):

Based on what I can tell, these containers were originally used for storing and shipping 20mm height drives and were not designed for storing more commonly-found 26mm height drives.

Could I simply shove 26mm drives in and not care? Yes, but due to the bowing of the foam dividers this would effectively cut the total drive capacity in half (from 25 to 13). And the dividers would almost certainly (eventually) break/snap after being left stressed for long periods of time. Instead, a better choice would be to use the right tool for the job. :-)

Bottom line: vendors really need to go the extra mile and provide proper measurements when selling products for used with diverse products. It’s easy to overlook the fact that 6mm makes a world of difference.

Examining the WD10EFRX

koitsu — Thu, 16 Aug 2012 04:45:46 +0000

Today I received a WD10EFRX, otherwise known as the 1TB model of Western Digital’s new Caviar Red drives…

The colour red, for me, is associated with two things: stop/halt signs and blood — two things which should definitely not be associated with a mechanical hard disk. :-) I say that in jest, but still…

The new Red drives boast “full NAS compatibility” or “NASware”. This is just more marketing buzzword-speak for something that does not need such segregation. A hard disk should work in any system and not induce problems or inhibit I/O) while providing features that the WD Caviar Green drives have (lower temperatures, “IntelliPark”, etc.), but still providing good throughput (not as great as the WD Caviar Black, but supposedly better than the Green). It’s very important to remember that companies like Western Digital created this delineation in the first place, so it’s very ironic (and depressing) that they’ve now begun to market models of hard disks that “solve these problems” when they themselves created the problem to begin with. Phrased differently: MHDDs used to work in anything you put them in (desktops, servers, etc.), with zero repercussions or issues, and it was the MHDD manufacturers themselves who created these firmware-level “features” (issues). It’s almost like the drives of today are being advertised/marketed with the assumption that those of us who were alive 13 years ago would hopefully have forgotten about how hard disks used to Just Work(tm). Well, I’m one of many who hasn’t forgotten.

Anyway, I’ve wanted to get my hands on one of these drives since they were announced, because I wanted to see feature-wise and behaviour-wise if these drives behaved better than the Green drives (you can read my “review” of present-day Green drives and how firmware-level features in them that do nothing other than frustrate and cause major problems).

Specifically, I wanted to examine the following things:

Platter count and reliability
What ATA-level and firmware features the drive supported, and which features were enabled (or disabled) by default
If the infamous Load Cycle Count (LCC) issue existed on these drives
Very quick performance testing (I am not OCD over benchmarks)

So let’s get to it!

The drive I received had a manufacturing date of 2012/07/07, firmware 01.01A01. However, as indicated at the bottom of my previous WD30EZRX “review”, firmware version doesn’t necessarily mean anything — WD, like too many companies these days, often doesn’t increment/change the firmware version.

1. The first thing I noticed about the WD10EFRX was its weight. I could tell right off the bat, simply by holding it, that the WD10EFRX uses one platter (i.e. a single platter w/ 1TB density). One platter also means 2 heads. All of this amounts to one thing: less chance of failure compared to older 1TB drives which use multiple platters (e.g. 2x500GB platters, thus 4 heads). It’s safe to say that the WD20EFRX (2TB) model probably has 2 platters, the 3TB model has 3 platters, etc.. Again: this is fantastic.

2. Upon hooking up the drive to a FreeBSD 9.x box, we see:

ada4 at ahcich4 bus 0 scbus4 target 0 lun 0
ada4:  ATA-8 SATA 3.x device
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)

The drive appears like pretty much every drive on the market (and justifiably so): it advertises 512-byte sectors despite being a 4096-byte sector (“Advanced Format”) drive. Nothing surprising about this, but I did want to point it out; people often forget that all AF drives advertise 512-byte sectors to retain 100% compatibility with older OSes and storage controllers/HBAs.

Also note that the drives’ SATA PHY supports SATA600 capability (though my motherboard (Supermicro X7SBA) only supports up to SATA300). Good to know too.

3. I immediately took a snapshot of SMART attributes to get an idea of what things looked like from the factory. Firmware version, serial number, WWN, etc. are all listed here:

# smartctl -a /dev/ada4
smartctl 5.43 2012-06-30 r3573 [FreeBSD 9.1-PRERELEASE amd64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD10EFRX-68JCSN0
Serial Number:    WD-WMC1U4917461
LU WWN Device Id: 5 0014ee 6ad3e27fd
Firmware Version: 01.01A01
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Wed Aug 15 20:35:11 2012 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (12960) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 148) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x30bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   253   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   100   253   021    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       2
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       0
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       2
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       1
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       0
194 Temperature_Celsius     0x0022   115   110   000    Old_age   Always       -       28
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   253   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Let’s run that again but using the -x flag instead, which provides some additional detail, insights into the SCT log, the SMART GPLog regions, and if the drive supports SATA device counters or not:

# smartctl -x /dev/ada4
smartctl 5.43 2012-06-30 r3573 [FreeBSD 9.1-PRERELEASE amd64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD10EFRX-68JCSN0
Serial Number:    WD-WMC1U4917461
LU WWN Device Id: 5 0014ee 6ad3e27fd
Firmware Version: 01.01A01
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Wed Aug 15 20:40:07 2012 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
ATA Security is:  Disabled, NOT FROZEN [SEC1]

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (12960) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 148) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x30bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   100   253   051    -    0
  3 Spin_Up_Time            POS--K   100   253   021    -    0
  4 Start_Stop_Count        -O--CK   100   100   000    -    2
  5 Reallocated_Sector_Ct   PO--CK   200   200   140    -    0
  7 Seek_Error_Rate         -OSR-K   100   253   000    -    0
  9 Power_On_Hours          -O--CK   100   100   000    -    0
 10 Spin_Retry_Count        -O--CK   100   253   000    -    0
 11 Calibration_Retry_Count -O--CK   100   253   000    -    0
 12 Power_Cycle_Count       -O--CK   100   100   000    -    2
192 Power-Off_Retract_Count -O--CK   200   200   000    -    1
193 Load_Cycle_Count        -O--CK   200   200   000    -    0
194 Temperature_Celsius     -O---K   114   110   000    -    29
196 Reallocated_Event_Count -O--CK   200   200   000    -    0
197 Current_Pending_Sector  -O--CK   200   200   000    -    0
198 Offline_Uncorrectable   ----CK   100   253   000    -    0
199 UDMA_CRC_Error_Count    -O--CK   200   253   000    -    0
200 Multi_Zone_Error_Rate   ---R--   100   253   000    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
GP/S  Log at address 0x00 has    1 sectors [Log Directory]
SMART Log at address 0x01 has    1 sectors [Summary SMART error log]
SMART Log at address 0x02 has    5 sectors [Comprehensive SMART error log]
GP    Log at address 0x03 has    6 sectors [Ext. Comprehensive SMART error log]
SMART Log at address 0x06 has    1 sectors [SMART self-test log]
GP    Log at address 0x07 has    1 sectors [Extended self-test log]
SMART Log at address 0x09 has    1 sectors [Selective self-test log]
GP    Log at address 0x10 has    1 sectors [NCQ Command Error log]
GP    Log at address 0x11 has    1 sectors [SATA Phy Event Counters]
GP    Log at address 0x21 has    1 sectors [Write stream error log]
GP    Log at address 0x22 has    1 sectors [Read stream error log]
GP/S  Log at address 0x80 has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x81 has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x82 has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x83 has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x84 has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x85 has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x86 has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x87 has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x88 has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x89 has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x8a has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x8b has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x8c has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x8d has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x8e has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x8f has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x90 has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x91 has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x92 has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x93 has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x94 has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x95 has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x96 has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x97 has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x98 has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x99 has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x9a has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x9b has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x9c has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x9d has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x9e has   16 sectors [Host vendor specific log]
GP/S  Log at address 0x9f has   16 sectors [Host vendor specific log]
GP/S  Log at address 0xa0 has   16 sectors [Device vendor specific log]
GP/S  Log at address 0xa1 has   16 sectors [Device vendor specific log]
GP/S  Log at address 0xa2 has   16 sectors [Device vendor specific log]
GP/S  Log at address 0xa3 has   16 sectors [Device vendor specific log]
GP/S  Log at address 0xa4 has   16 sectors [Device vendor specific log]
GP/S  Log at address 0xa5 has   16 sectors [Device vendor specific log]
GP/S  Log at address 0xa6 has   16 sectors [Device vendor specific log]
GP/S  Log at address 0xa7 has   16 sectors [Device vendor specific log]
GP/S  Log at address 0xa8 has    1 sectors [Device vendor specific log]
GP/S  Log at address 0xa9 has    1 sectors [Device vendor specific log]
GP/S  Log at address 0xaa has    1 sectors [Device vendor specific log]
GP/S  Log at address 0xab has    1 sectors [Device vendor specific log]
GP/S  Log at address 0xac has    1 sectors [Device vendor specific log]
GP/S  Log at address 0xad has    1 sectors [Device vendor specific log]
GP/S  Log at address 0xae has    1 sectors [Device vendor specific log]
GP/S  Log at address 0xaf has    1 sectors [Device vendor specific log]
GP/S  Log at address 0xb0 has    1 sectors [Device vendor specific log]
GP/S  Log at address 0xb1 has    1 sectors [Device vendor specific log]
GP/S  Log at address 0xb2 has    1 sectors [Device vendor specific log]
GP/S  Log at address 0xb3 has    1 sectors [Device vendor specific log]
GP/S  Log at address 0xb4 has    1 sectors [Device vendor specific log]
GP/S  Log at address 0xb5 has    1 sectors [Device vendor specific log]
GP/S  Log at address 0xb6 has    1 sectors [Device vendor specific log]
GP/S  Log at address 0xb7 has    1 sectors [Device vendor specific log]
GP/S  Log at address 0xbd has    1 sectors [Device vendor specific log]
GP/S  Log at address 0xc0 has    1 sectors [Device vendor specific log]
GP    Log at address 0xc1 has   93 sectors [Device vendor specific log]
GP/S  Log at address 0xe0 has    1 sectors [SCT Command/Status]
GP/S  Log at address 0xe1 has    1 sectors [SCT Data Transfer]

SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       258 (0x0102)
SCT Support Level:                   1
Device State:                        Active (0)
Current Temperature:                    28 Celsius
Power Cycle Min/Max Temperature:     28/28 Celsius
Lifetime    Min/Max Temperature:     28/33 Celsius
Under/Over Temperature Limit Count:   0/0
SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/60 Celsius
Min/Max Temperature Limit:           -41/85 Celsius
Temperature History Size (Index):    478 (9)

Index    Estimated Time   Temperature Celsius
  10    2012-08-15 12:43     ?  -
 ...    ..(467 skipped).    ..  -
   0    2012-08-15 20:31     ?  -
   1    2012-08-15 20:32    33  **************
   2    2012-08-15 20:33    33  **************
   3    2012-08-15 20:34     ?  -
   4    2012-08-15 20:35    28  *********
 ...    ..(  4 skipped).    ..  *********
   9    2012-08-15 20:40    28  *********

SCT Error Recovery Control:
           Read:     70 (7.0 seconds)
          Write:     70 (7.0 seconds)

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x0008  2            0  Device-to-host non-data FIS retries
0x0009  2            1  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2            1  Device-to-host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within host-to-device FIS
0x000f  2            0  R_ERR response for host-to-device data FIS, CRC
0x0012  2            0  R_ERR response for host-to-device non-data FIS, CRC
0x8000  4          351  Vendor specific

Looks good to me.

4. Next comes the ATA capabilities and which features were enabled/disabled by default, as well as what features had been added or removed when compared to Green and Black drives:

# camcontrol identify ada4
pass4:  ATA-8 SATA 3.x device
pass4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)

protocol              ATA/ATAPI-8 SATA 3.x
device model          WDC WD10EFRX-68JCSN0
firmware revision     01.01A01
serial number         WD-WMC1U4917461
WWN                   50014ee6ad3e27fd
cylinders             16383
heads                 16
sectors/track         63
sector size           logical 512, physical 4096, offset 0
LBA supported         268435455 sectors
LBA48 supported       1953525168 sectors
PIO supported         PIO4
DMA supported         WDMA2 UDMA6

Feature                      Support  Enabled   Value           Vendor
read ahead                     yes      yes
write cache                    yes      yes
flush cache                    yes      yes
overlap                        no
Tagged Command Queuing (TCQ)   no       no
Native Command Queuing (NCQ)   yes              32 tags
SMART                          yes      yes
microcode download             yes      yes
security                       yes      no
power management               yes      yes
advanced power management      no       no
automatic acoustic management  no       no
media status notification      no       no
power-up in Standby            yes      no
write-read-verify              no       no
unload                         no       no
free-fall                      no       no
data set management (TRIM)     no

This is where the fun starts. Here’s what we can discern from the advertised capabilities when compared to the Green and Black drives:

Model	Advertised sector size	Physical sector size	APM (Advanced Power Management)	AAM (Automatic Acoustic Management)	PUIS (Power Up In Standby)	“IntelliPark”
WD10EFRX (Caviar Red, 1TB)	512	4096	not supported	not supported	supported, disabled by default	no
WD30EZRX (Caviar Green, 3TB)	512	4096	not supported	not supported	supported, disabled by default	yes
WD1002FAEX (Caviar Black, 1TB)	512	512	not supported	supported on some firmware revisions	supported, disabled by default	no

2012/10/02 update: Western Digital has removed any mention of IntelliPark from their WD Red product documentation. So it appears that the first set of product documentation distributed for the Red drives incorrectly stated the product had IntelliPark when in fact they do not.

As we get into the Load Cycle Count topic, it’s important to note that historically the only way to work around the LCC problem was to disable APM. But as you can see, in all of these drives — including the WD Red — there is no way to do that. So let’s hope the newest WD Red drives don’t suffer from the “LLC issue” (excessive head parking)!

5. At this point, I decided to do the “LCC test” in the exact same manner that I did with the WD30EZRX (Green 3TB) drive in my previous “review”: issue some read requests to the drive, sleep 10 seconds, pull SMART stats (looking at SMART attribute 193), and seeing if the counter increases. So let’s do it:

# while true; do echo -n "`date` -- " ; dd if=/dev/ada4 of=/dev/null bs=4k count=1 2>/dev/null ; sleep 10 ; smartctl -a /dev/ada4 | awk '/Load_Cycle_Count/ { print $2 ": " $NF }' ; done
Wed Aug 15 20:51:51 PDT 2012 -- Load_Cycle_Count: 0
Wed Aug 15 20:52:01 PDT 2012 -- Load_Cycle_Count: 0
Wed Aug 15 20:52:11 PDT 2012 -- Load_Cycle_Count: 0
Wed Aug 15 20:52:21 PDT 2012 -- Load_Cycle_Count: 0
Wed Aug 15 20:52:31 PDT 2012 -- Load_Cycle_Count: 0
Wed Aug 15 20:52:41 PDT 2012 -- Load_Cycle_Count: 0
Wed Aug 15 20:52:51 PDT 2012 -- Load_Cycle_Count: 0

I then did the same LCC test but with writes (e.g. dd if=/dev/zero of=/dev/ada4 bs=4k count=1). The behaviour was the same: LCC did not increment, and the drive did not exhibit clicking or any audible noises.

Thus, the WD10EFRX I received does not suffer from the LCC problem. That’s really good news. It appears Western Digital has been listening to its consumers.

6. Now it’s time for the benchmarks. It’s very important to understand that I do not perform extensive benchmarking — meaning, I do not care to go to great extremes or run a whole gamut of tests, workloads, etc.. I prefer to focus on drive features, reliability, and behaviour. If you’re wanting extensive benchmarks and opinions relating to such, please use your favourite search engine to find reviews of the drive.

For benchmarks I resorted to switching to Windows (specifically Windows XP SP3, 32-bit, AHCI enabled with Intel RST drivers (version 10.8.0.1003)) and used HD Tune Pro 5.00. Windows 7 would very likely have gotten slightly better performance, but again, that is not my focus. The results are below, and are left to the reader to interpret/judge for themselves:

http://jdc.koitsu.org/disks/wd10efrx/

Two things to note:

I did see better overall performance using FreeBSD (sequential throughput was about 12MBytes/sec higher), so your mileage may vary.
The temperature of the WD10EFRX did not exceed 35C at any point. Compare that to the Black, which in the same system idles around 42C, and extensive I/O gets the temperature up to around 48C. I feel it’s very important to stress that fact, and can almost certainly be attributed to the disk containing a single platter.

So given all of the above information, what’s my opinion?

The WD Red drives — at least the WD10EFRX — are excellent given what they’re advertised to do. This is the exact kind of drive I have been wanting for quite some time: lightweight, good temperatures, decent-to-excellent I/O, uses 1TB density platters, and whose firmware doesn’t get in the way (re: LLC issue).

I’m going to keep this drive and put it to use in my Windows workstation. I should invest in some WD20EFRX and WD30EFRX drives and examine them for any oddities, but right now money is somewhat tight so I can’t experiment as much as I’d like.

Asus Xonar DG driver bugs on XP

koitsu — Sat, 21 Jul 2012 07:58:20 +0000

EDIT: Interestingly enough — and I did report this to Asus — this issue went away after a system reboot. The nature of the problem still implies a driver-level bug, but why a reboot would fix it is still unknown. I’m still looking to figure out if there’s a way to reproduce it consistently/reliably so that Asus can track down the source of the problem and fix it permanently.

About a week ago I replaced use of my crappy Realtek ALC889 (Azalia) on-board audio with an actual sound card — more specifically, the Asus Xonar DG. This card is deemed “one of the best” by the general Internet community for gaming and general audio use, plus it’s inexpensive (US$25).

It didn’t take me long to start finding bugs in its drivers.

Thankfully the issue I’m about to describe doesn’t affect most software. Most programs these days tend to use DirectSound for audio playback, while this bug purely affects Windows MME playback, and only at rates less than 44kHz.

You might be inclined to ask “What on earth uses MME these days? It dates back to Windows 3.0!” but surprisingly a large number of non-gaming applications continue to use Windows MME as a very simple way to play back .wav files or other such things. A great example is the ever-popular Windows IRC client called mIRC. But other programs such as Audacity and even Winamp (when using the Nullsoft WaveOut Output plug-in, rather than DirectSound) can use MME. Even Windows XP Control Panel applets that come with the OS use MME. My point is that MME is still used throughout many applications, thus audio card vendors need to make sure their drivers are tested properly with both MME and DirectSound.

I also imagine this only affects XP, as Windows Vista and Windows 7 both have entirely new audio layers and MME support is “emulated” where DirectSound is used natively.

The bug itself only affects playback when a frequency rate less than 44kHz (e.g. 11kHz or 22kHz) is used. Mono vs. stereo has no bearing, nor does 16-bit vs. 8-bit. The end result is audio that is played at a base frequency which is lower than what it should be; most people would describe the sound as “sad”.

It’s very easy to notice and doesn’t require a sharp ear in the least. I have mIRC configured to play a “ding” noise when someone says my nickname on IRC, using C:\WINDOWS\Media\ding.wav as the source audio, which happens to be 22kHz 16-bit stereo.

I made a Youtube video showing this exact problem so that people can hear the difference themselves:

http://youtu.be/KHwjKLOS6o4

And for those wanting to reproduce the issue, the easiest method is to use Audacity:

Install Audacity
Open C:\WINDOWS\Media\ding.wav
Make sure that “MME” mode is selected, not “DirectSound”
Play the file and listen closely
Change from “MME” to “DirectSound”
Play the file and notice the difference (DirectSound is correct)

I’ve mailed Asus about this problem, including a step-by-step method of reproducing it (it’s quite easy), and will post some follow-ups when they’ve gotten back to me.

The bug itself is almost certainly the result of someone’s resampling frequency table being incorrect, or if used via a formula, the formula being wrong. I imagine the fix to be fairly simple, and I sure hope Asus doesn’t give me the standard Tier 1 Technical Support run-around (“have you tried rebooting your computer”, etc.).

Memories

koitsu — Thu, 21 Jun 2012 08:11:06 +0000

Tonight I decided to do something nostalgic (more than the usual) — try to find all the places I’ve lived at over the years, or at least the ones I can remember. There are some I can remember but can’t find (such as when I briefly lived in the graduate dorms at Stanford), thus decided not to list them.

It’s pretty miraculous that after all these years I can still remember (and find!) the places I’ve lived. Most of them I didn’t have addresses for; I simply remembered where things were. Makes me wonder how much of my brain is used just for all that.

I feel bad for not being able to find the exact location in San Mateo where I lived. Sadly the location was so generic that I can’t remember any landmarks. I even checked old resumes and rental applications, but nothing goes back that far (1998!). Bummer.

Searching for all these places brought back a lot of memories, especially about ex-girlfriends. I’m still friends with most of my roommates and all, but I don’t tend to keep in touch with past partners. So lots of memories, old emotions, things like that surfaced during all of this; some good, some bad.

Diablo 3 crashing due to fake item hotlinking

koitsu — Thu, 07 Jun 2012 13:43:26 +0000

EDIT: Blizzard has addressed this with Diablo 3 patch 1.0.2c (build 9991). Under “Bug Fixes”, the ChangeLog states: “Fixed several crashes which occurred when clicking on player-generated item links in the game client”.

It appears that some nefarious Diablo 3 players have found a way to crash another player’s client by sending a malformed “item hotlink”. Apparently individuals have found a way to exploit the hotlinking feature, allowing for non-existent items to appear (such as potions that have sockets on them, etc.). However, in some cases these fake items result in the client itself crashing when clicked upon.

Here are threads which are directly related to this issue:

Reddit also mentions the issue:

Beware fake weapon scams. You can link fake items in chat.

There is obviously no workaround other than to avoid clicking on ANY link in chat, whether it be private or public.

One such item which absolutely crashes the client (if the name is clicked on) is the following string:

|HItem:2,1236607151:-877003260:-362610042,-362610042,-362610042,-362610042,-362610042,-362610042,-362610042,1791308545,1791308545,1791308545,1791308545,1791308545,1791308545,-362610042,-362610042,810509133,810509133,810509133-1008238675,-1008238675,-1008238675,-1243748674,-1243748674,-1243748674:-1:0:-1:-1:-1:9:444:444:0:0:4:0:|h [{c:ffff00ff}Diablo's Claw{/c}]|h

It amazes me that Blizzard continues to use a hotlinking model that allows for players to manually insert such text, not to mention provide a player to actually send something across the wire that will get parsed into an invalid item. Was it simply too hard to use a syntax that involved a high-bit (e.g. not possible to input by a player) character sequence, and directly refer to an item number? Such an example would be:

<0xBEEFCAFEF00DFACE>263<0x00>

…where the hex strings 0xBEEFCAFEF00DFACE and 0×00 are literal binary, and the number 263 refers to item 263. Item colour, etc. would be part of the data structure associated with the item itself, not something a player could input.

For modifiable items (e.g. with sockets which are populated with variable gems), a comma-delimited syntax could be used — where the comma-delimited values only applied if the preceding item number actually had sockets in it, and assuming the sockets were filled with items that were denoted in the internal data structure as socketable. An example where item 92145 was filled with 2 socketable items (9801 and 7625, respectively):

<0xBEEFCAFEF00DFACE>92145,9801,7625<0x00>

This isn’t rocket science. This is “secure programming 101″; it’s a combination of sanitizing input, and not provide humans the ability to enter characters/data which could cause a problem. The |HItem:...|h stuff is a disgrace.

All Blizzard sites unreachable

koitsu — Sun, 03 Jun 2012 17:06:14 +0000

EDIT: As of 10:48 PDT, things seem back online (and reliably at that). I strongly doubt there will ever be mention of this outage publicly.

EDIT #2: Imagine that! Assuming you’re actually able to log in to Diablo 3 (not sure about other battle.net services), there is an incredibly (and obviously intentionally) vague message about “a problem”. I’ve included it here as a screenshot. I love the use of the phrase “the parties involved”; wouldn’t want to name any names and risk holding a provider responsible for the problem, would we? US politics, what a disgrace.

EDIT #3: mtr now shows what appears to be a great improvement; it looks to me like there was a network-level problem within AT&T’s own network near/around Salt Lake City that caused this. Of course, I would still love to know why my packets bound for the NS-WEST.CERF.NET nameserver go from the Bay Area to Los Angeles to Salt Lake City to Los Angeles to San Diego. Asymmetry doesn’t explain this either. Obviously AT&T has some kind of networking issue that they tried to route around (which caused this entire ordeal) but failed miserably. Wouldn’t be the first, 2nd, or even 10th time I’ve seen AT&T fail at this. Welcome to the Internet: it’s broken 24x7x365, no exaggeration.

Not sure when this issue began today, but I was able to confirm its existence around 09:45 PDT (that’s 9:45am Pacific).

None of Blizzard’s sites — including blizzard.com, battle.net, diablo3.com, as well as any subdomain under those — are resolving. I imagine this would affect any game or service Blizzard offers which uses DNS for resolution; hard-coded IP addresses (which I believe WoW uses for its server list) would not be affected, of course, but I believe WoW does use DNS for some portion of its authentication mechanism. In the case of Diablo 3, the behaviour is amusing — you might see a strange GDI-based “Checking for updates” box, then get the usual launcher dialog with no details in it, with “Play” greyed out. (Note to Blizzard: you should really improve the error handling in the D3 Launcher. The way this manifested itself, from a UI / end-user perspective, is absolutely atrocious!)

Blizzard’s Facebook page is filled with people complaining about the problem, while Blizzard’s Twitter page (as of 10:09 PDT) has no mention of any issue.

The root cause appears to be some sort of major issue within AT&T. Blizzard relies on two authoritative nameservers for all of their sites — specifically NS-WEST.CERF.NET and NS-EAST.CERF.NET (this is AT&T). Neither of these authoritative nameservers are responding. I have not the slightest idea on earth why Blizzard has only *two* authoritative nameservers, and both with the same company (AT&T); you would expect them to have a tertiary that wasn’t AT&T-responsible. I dunno who maintains Blizzard’s WHOIS and SOA data, but that’s a very, very bad choice on their part; DNS and Networking 101 says do not do this.

Below are some technical details which won’t mean much to your average player but to a system or network administrator will be more than sufficient evidence of the problem.

I cannot explain why packets destined for NS-WEST.CERF.NET’s IP address are going through Salt Lake City, Utah. This smells of an AT&T IP routing problem but I have no easy way to confirm that given the asymmetric nature of packet flow on the Internet.

$ dig @a.gtld-servers.net ns blizzard.com.

; <<>> DiG 9.8.2 <<>> @a.gtld-servers.net ns blizzard.com.
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11635
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 2
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;blizzard.com.                  IN      NS

;; AUTHORITY SECTION:
blizzard.com.           172800  IN      NS      ns-east.cerf.net.
blizzard.com.           172800  IN      NS      ns-west.cerf.net.

;; ADDITIONAL SECTION:
ns-east.cerf.net.       172800  IN      A       207.252.96.3
ns-west.cerf.net.       172800  IN      A       192.153.156.3

;; Query time: 108 msec
;; SERVER: 192.5.6.30#53(192.5.6.30)
;; WHEN: Sun Jun  3 10:01:43 2012
;; MSG SIZE  rcvd: 114


$ dig @207.252.96.3 ns blizzard.com.

; <<>> DiG 9.8.2 <<>> @207.252.96.3 ns blizzard.com.
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached


$ dig @192.153.156.3 ns blizzard.com.

; <<>> DiG 9.8.2 <<>> @192.153.156.3 ns blizzard.com.
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached


$ mtr 192.153.156.3
                                                  Packets               Pings
 Host                                           Loss%   Snt   Rcv  Last   Avg  Best  Wrst
 1. gw.home.lan                                  0.0%    21    21   0.3   0.2   0.2   0.3
 2. c-67-180-84-1.hsd1.ca.comcast.net           20.0%    20    16 4932. 4829. 4264. 5128.
 3. te-0-0-0-12-ur05.santaclara.ca.sfba.comcast  0.0%    20    20  97.8  19.0   8.7  97.8
 4. te-1-11-0-1-ar01.sfsutro.ca.sfba.comcast.ne  0.0%    20    20  44.1  24.0  13.0  45.9
 5. 68.86.91.229                                 0.0%    20    20  62.5  25.9  12.1  66.7
 6. pos-0-4-0-0-pe01.11greatoaks.ca.ibone.comca  0.0%    20    20  30.6  20.7  16.8  32.0
 7. 192.205.37.1                                 0.0%    20    20  18.4  29.2  16.1 118.3
 8. cr2.sffca.ip.att.net                         0.0%    20    20  44.4  49.7  39.7  84.7
 9. cr2.la2ca.ip.att.net                         0.0%    20    20  58.0  48.0  39.7  66.3
10. cr1.slkut.ip.att.net                         0.0%    20    20  43.6  50.4  39.3  83.1
11. cr2.slkut.ip.att.net                         0.0%    20    20  56.6  48.6  38.5  87.9
12. 12.123.238.133                               0.0%    20    20  40.7  46.9  37.7  97.1
13. ???


$ mtr 207.252.96.3
                                                  Packets               Pings
 Host                                           Loss%   Snt   Rcv  Last   Avg  Best  Wrst
 1. gw.home.lan                                  0.0%    62    62   0.3   0.3   0.2   0.4
 2. c-67-180-84-1.hsd1.ca.comcast.net            8.2%    61    56 4528. 5193. 4525. 6339.
 3. te-0-0-0-12-ur05.santaclara.ca.sfba.comcast  0.0%    61    61  10.4  15.7   8.0  70.7
 4. te-1-11-0-1-ar01.sfsutro.ca.sfba.comcast.ne  0.0%    61    61  19.9  23.6  10.3  67.2
 5. 68.86.90.93                                  0.0%    61    61  21.8  20.8  11.4  53.8
 6. pos-0-9-0-0-pe01.11greatoaks.ca.ibone.comca  0.0%    61    61  17.7  24.7  16.0  56.1
 7. 192.205.37.1                                 0.0%    61    61  74.2  28.0  15.8 140.6
 8. cr2.sffca.ip.att.net                         0.0%    61    61  45.2  49.6  39.3 104.1
 9. cr2.la2ca.ip.att.net                         0.0%    61    61  41.3  50.3  37.9 114.7
10. cr1.slkut.ip.att.net                         0.0%    61    61  54.1  48.3  38.6  80.8
11. cr2.slkut.ip.att.net                         0.0%    61    61  62.7  50.7  39.7  91.2
12. 12.123.238.137                               0.0%    61    61  39.2  50.6  37.5 183.9
13. ???
14. ???
15. ???
16. ???
17. ???
18. ???
19. n54ny401me3.ip.att.net                       0.0%    58    57  96.6 104.9  94.4 145.6
20. ???


route-views>show ip route 192.153.156.3
Routing entry for 192.153.156.0/24
  Known via "bgp 6447", distance 20, metric 0
  Tag 7018, type external
  Last update from 12.0.1.63 7w0d ago
  Routing Descriptor Blocks:
  * 12.0.1.63, from 12.0.1.63, 7w0d ago
      Route metric is 0, traffic share count is 1
      AS Hops 2
      Route tag 7018


route-views>show ip route 207.252.96.3
Routing entry for 207.252.96.0/24
  Known via "bgp 6447", distance 20, metric 0
  Tag 7018, type external
  Last update from 12.0.1.63 7w0d ago
  Routing Descriptor Blocks:
  * 12.0.1.63, from 12.0.1.63, 7w0d ago
      Route metric is 0, traffic share count is 1
      AS Hops 2
      Route tag 7018

WD30EZRX and aggressive head parking

koitsu — Wed, 30 May 2012 07:57:44 +0000

For many years Western Digital has been flip-flopping their stance on the issue with aggressive (and excessive) parking of the drive’s heads, especially in the case of their Green series drives (including their “Enterprise” drives with the “-GP” suffix). WD advertises this as a feature called “IntelliPark” — a feature that, in all honesty, appears to do nothing useful in desktop or server systems.

To end-users and administrators, this behaviour manifests itself in two ways:

Degraded performance for occasional I/O, due to the drive having to unlock and move its actuator arms back over the platters,
A rapidly increasing Load_Cycle_Count (SMART attribute 193 / 0xC1). Internet users often refer to this as “LCC”.

It was brought to my attention via an Internet forum post that WD may have done away with excessive load cycling on newer Green drives, and also did not require use of WD’s Idle3 utility (more on that in a moment). I was ecstatic; the Green drives run at a lower temperature, and for a disk that’s being used to store backups (e.g. not regularly accessed) but kept spinning at all times (lots of reasons for this), the Green series seemed like a good choice.

So last week I bought a WD30EZRX to see what the behaviour was.

Sadly, the behaviour on the WD30EZRX is identical to that of the WD20EARS drive which came out many years ago: the drive is incredibly aggressive about parking its heads, and now there’s no reliable way to disable this behaviour.

Per the ATA specification IDENTIFY command, the WD30EZRX drive doesn’t support APM (see line 26 below); why that matters will soon become evident:

protocol              ATA/ATAPI-8 SATA 3.x
device model          WDC WD30EZRX-00MMMB0
firmware revision     80.00A80
serial number         WD-WCAWZ1930893
WWN                   50014ee2b179476a
cylinders             16383
heads                 16
sectors/track         63
sector size           logical 512, physical 4096, offset 0
LBA supported         268435455 sectors
LBA48 supported       5860533168 sectors
PIO supported         PIO4
DMA supported         WDMA2 UDMA6

Feature                      Support  Enabled   Value           Vendor
read ahead                     yes      yes
write cache                    yes      yes
flush cache                    yes      yes
overlap                        no
Tagged Command Queuing (TCQ)   no       no
Native Command Queuing (NCQ)   yes              32 tags
SMART                          yes      yes
microcode download             yes      yes
security                       yes      no
power management               yes      yes
advanced power management      no       no
automatic acoustic management  no       no
media status notification      no       no
power-up in Standby            yes      no
write-read-verify              no       no
unload                         no       no
free-fall                      no       no
data set management (TRIM)     no

SMART attributes:

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   253   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   163   163   021    Pre-fail  Always       -       8833
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       10
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       0
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       8
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       7
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       15
194 Temperature_Celsius     0x0022   124   121   000    Old_age   Always       -       28
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

A count of 15 doesn’t look that bad, does it? Yeah well, watch this.

The below sequence of commands prints the current date/time, followed by reading 4KBytes of data from the drive (LBAs 0 to 3), sleeps 10 seconds, then checks SMART statistics and prints out the Load_Cycle_Count attribute and its raw value, rinse lather repeat indefinitely:

while true; do echo -n "`date` -- " ; dd if=/dev/ada3 of=/dev/null bs=4k count=1 2>/dev/null ; sleep 10 ; smartctl -a /dev/ada3 | awk '/Load_Cycle_Count/ { print $2 ": " $NF }' ; done
Wed May 30 00:18:17 PDT 2012 -- Load_Cycle_Count: 28
Wed May 30 00:18:28 PDT 2012 -- Load_Cycle_Count: 29
Wed May 30 00:18:38 PDT 2012 -- Load_Cycle_Count: 30
Wed May 30 00:18:49 PDT 2012 -- Load_Cycle_Count: 31
Wed May 30 00:18:59 PDT 2012 -- Load_Cycle_Count: 31
Wed May 30 00:19:10 PDT 2012 -- Load_Cycle_Count: 32
Wed May 30 00:19:20 PDT 2012 -- Load_Cycle_Count: 33
Wed May 30 00:19:31 PDT 2012 -- Load_Cycle_Count: 34
Wed May 30 00:19:41 PDT 2012 -- Load_Cycle_Count: 35
Wed May 30 00:19:52 PDT 2012 -- Load_Cycle_Count: 36
Wed May 30 00:20:02 PDT 2012 -- Load_Cycle_Count: 37
Wed May 30 00:20:13 PDT 2012 -- Load_Cycle_Count: 38
Wed May 30 00:20:23 PDT 2012 -- Load_Cycle_Count: 39
Wed May 30 00:20:34 PDT 2012 -- ^C

Every time the drive load cycles, of course, there is an audible noise issued as well (the sound of the actuator arm having to be moved, thus heads unparked). So as I stated above in item #1, the drive will perform badly (initial I/O will stall/delay while the heads are being unparked), while Load_Cycle_Count increments.

If we try a shorter sleep interval (5 seconds), the issue doesn’t happen, which further proves that there is an “idle timeout” internal to the drive (not that this is anything new) and it’s probably set somewhere between 6-9 seconds:

# while true; do echo -n "`date` -- " ; dd if=/dev/ada3 of=/dev/null bs=4k count=1 2>/dev/null ; sleep 5 ; smartctl -a /dev/ada3 | awk '/Load_Cycle_Count/ { print $2 ": " $NF }' ; done
Wed May 30 00:20:55 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:21:01 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:21:06 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:21:11 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:21:16 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:21:21 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:21:26 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:21:32 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:21:37 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:21:42 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:21:47 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:21:52 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:21:57 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:22:02 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:22:07 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:22:12 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:22:18 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:22:23 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:22:28 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:22:33 PDT 2012 -- Load_Cycle_Count: 40
Wed May 30 00:22:38 PDT 2012 -- ^C

I should also note here that the load cycle count can/will increment when pulling SMART statistics after the drive has gone into idle state. The reason: SMART statistics are stored in the HPA region of the drive, which means for it to read statistics it has to unpark the heads. A beautiful catch-22…

And now some links to WD’s own justification/stance on the “LCC matter”:

Things to note about these documents, and in general, because none of it is made apparent:

Neither document mentions the WD30EZRX, nor its brother/sister drives of smaller capacities or less cache (such as the WD25EZRX, WD20EARX, WD15EARX, WD10EARX, WD10EZRX, or the WD20EARS). However, some WD forum users are referring folks with those models of drives to the above documents, which could result in catastrophic problems (keep reading),
Within the first document you’ll find preposterous recommendations, such as to disable or adjust syslog (the idea being that disabling logging won’t unpark the drive heads); this is WD’s horrible attempt at stating “try to not issue any drive I/O to the drive and let it sit idle as much as possible”. This approach is so moronic and wrong/blind that it’s unfathomable,
Also within the first document you will see a recommended solution of disabling APM (Advanced Power Management) on the drive itself. However, the disable-APM advice is only sometimes applicable. Only some models of WD drives have a direct correlation between APM and the idle timeout feature, while others (such as the WD30EZRX) don’t support APM at all but still have the idle timeout feature. Furthermore, disabling APM also disables other internal features of the drive, which is a downside,
Deep within the bowels of these documents you will find mention of a WD-specific utility which they call “Idle3″. This utility adjusts an internal “idle timeout” value within the drive itself (the config is probably stored in the HPA somewhere). However, this utility DOS-based, and often does not work with AHCI (and definitely will not work under RAID). Thus you are forced to not only reboot and deal with booting into MS-DOS or FreeDOS or whatever else you can find, but you’re forced to adjust your system BIOS to change your SATA controller’s option ROM mode to native SATA (or worse, PATA emulation) just to get this utility to work. Once you adjust the setting it stays (permanently, or until set back to defaults), but customers shouldn’t have to go through this rigmarole in the first place. Do not mistake this for TLER or SCT Error Recovery (same thing) — that is a completely unrelated feature and has no bearing on this issue.

So the last choice seems like the best, right? Wrong. A couple people have commented in that thread stating that their drive performance dropped horribly once they used the Idle3 utility to disable the aggressive idle timeout.

Furthermore, Western Digital’s own product support documentation states that this utility should not be used on this model of drive.

A review of the WD30EZRX back in 2011 indicated that the drive has a load/unload cycle count lifetime of 300,000 cycles. This came straight from WD’s own product documentation.

Furthermore, WD’s own forum is filled with people complaining about this problem, even on the WD30EZRX. One user has a drive that’s new/recent and is already reaching 284,000 cycles.

Needless to say, this WD30EZRX drive will be returned to its place of purchase for a full refund. Very, very disappointing.

I’ve discussed (ranted) at length with colleagues of mine how the storage industry is making a big mistake doing things like this. One pointed out that it’s basically impossible in this day and age to get a hard disk that just does what it’s supposed to: read/write. Instead, you have to reenact Riverdance just to get your disk working reliably without any nonsense in the way. The KISS principle has once again been ignored or forgotten.

Drive manufacturers continue adding new features that do nothing but make a mess for their consumers. This applies to more than just WD, and it’s incredibly depressing. In the case of WD, it’s quite obvious that the Green and Black drives use the exact same parts and thus are identical in mechanics, with the only difference being firmware.

With that in mind, take this into consideration: the WD2002FAEX (Caviar Black 2TB, 64MBytes cache) costs US$192.98, while the WD20EARX (Caviar Green 2TB, 64MBytes cache) costs US$116.00 (amounts taken on 2012/05/30). That’s a US$76.98 difference, per drive. This didn’t used to be the case though… until the whole Thailand flooding situation happened. It seems to me that as a result of an act of God someone is now using the opportunity to make serious cash where previously they were making much less. And I’m not the only one who noticed either. The world can blame the United States’ excessive focus on capitalism as the sole reason for this; it makes me sad.

And finally, if you think all the firmware versions are identical based entirely on the firmware string, you would be sadly mistaken. The below two Caviar Black 1TB drives are identical, yet they advertise completely different capabilities per the ATA specification IDENTIFY command. Note the lack of AAM on the 2nd drive, yet it’s quite clear they advertise the same firmware string:

protocol              ATA/ATAPI-8 SATA 3.x
device model          WDC WD1002FAEX-00Z3A0
firmware revision     05.01D05
serial number         WD-WCATR1782965
WWN                   50014ee25a001e1c
cylinders             16383
heads                 16
sectors/track         63
sector size           logical 512, physical 512, offset 0
LBA supported         268435455 sectors
LBA48 supported       1953525168 sectors
PIO supported         PIO4
DMA supported         WDMA2 UDMA6

Feature                      Support  Enabled   Value           Vendor
read ahead                     yes      yes
write cache                    yes      yes
flush cache                    yes      yes
overlap                        no
Tagged Command Queuing (TCQ)   no       no
Native Command Queuing (NCQ)   yes              32 tags
SMART                          yes      yes
microcode download             yes      yes
security                       yes      no
power management               yes      yes
advanced power management      no       no
automatic acoustic management  yes      no      254/0xFE        128/0x80
media status notification      no       no
power-up in Standby            yes      no
write-read-verify              no       no
unload                         no       no
free-fall                      no       no
data set management (TRIM)     no

protocol              ATA/ATAPI-8 SATA 3.x
device model          WDC WD1002FAEX-00Z3A0
firmware revision     05.01D05
serial number         WD-WCATR7016817
WWN                   50014ee25b1b1639
cylinders             16383
heads                 16
sectors/track         63
sector size           logical 512, physical 512, offset 0
LBA supported         268435455 sectors
LBA48 supported       1953525168 sectors
PIO supported         PIO4
DMA supported         WDMA2 UDMA6

Feature                      Support  Enabled   Value           Vendor
read ahead                     yes      yes
write cache                    yes      yes
flush cache                    yes      yes
overlap                        no
Tagged Command Queuing (TCQ)   no       no
Native Command Queuing (NCQ)   yes              32 tags
SMART                          yes      yes
microcode download             yes      yes
security                       yes      no
power management               yes      yes
advanced power management      no       no
automatic acoustic management  no       no
media status notification      no       no
power-up in Standby            yes      no
write-read-verify              no       no
unload                         no       no
free-fall                      no       no
data set management (TRIM)     no

I would love to have a long and detailed chat with present-day storage companies about all of the above. The KISS principle really needs to be put into effect here, and the sooner the better. Otherwise what we’re effectively doing is paying for firmware features — and ones we cannot disable or toggle. But then again I’m just a simple engineer — what do I know of such things, right?

WD Sentinel DX4000 — unacceptable power design

koitsu — Tue, 03 Jan 2012 03:55:27 +0000

Update: It appears Western Digital has indirectly responded to my blog post, stating that the concerns over disk power draw on system power-up are incorrect. They state that the disks are spun up individually, with 2 second intervals between each disk. This is good to know. I would not have written my “scathing review” had Western Digital taken the time to document this behaviour somewhere, especially in their own product specification sheet. This is why NAS manufacturers need to be fully open with disclosure of how their products work, including low-level engineering details. This stuff plays a bigger role today than ever.

Western Digital has released a new SoHo NAS product, the WD Sentinel DX4000, which is driven by an Intel Atom mainboard and CPU, supports dual gigE (for failover, not trunking), and up to 4 disks in RAID-5. It’s 8x6x8 inches in size for the 4-disk version. The cost? US$1500 retail.

Remember: the product is intended for businesses or high-class end-users.

Now take a look at its AC power connectors: two circular female connectors with a male centre pin. You know, the common “wall wart” and “in-line brick” crap you’d see on consumer-grade products that results in cable clutter (and absolutely impossible to properly deal with when placed in a rack)? No, surely that couldn’t be, especially for a SoHo device of those physical dimensions…

I thought for a moment it might be a proprietary connector that required a proprietary cable that went directly to a 115V AC plug, so I downloaded the official User Manual.

Lo and behold, on page 7 of the manual (page 11 per PDF index), an in-line brick (transformer) is shown. You get only one (the 2nd connector is for failover/redundancy), and its power requirements are miniscule. Quoting page 9 of the manual (page 13 per PDF index):

Input voltage: 100V to 240V AC
Input frequency: 50 to 60 Hz
Output voltage: 19V DC, up to 6.32A

I’m a little surprised by those power requirements: 6.3A at 19V. The unit comes with 5400rpm drives — if you were to shove four 7200rpm drives in there, you almost certainly would be taxing that little brick of an AC adapter. Your drives would start falling off the bus and experiencing strange anomalies during high I/O. Let me explain:

The WD Digital Caviar Black WD2002FAEX draws 10.7 watts when reading/writing according to WD’s own specifications.

Let’s figure out the percentage of power use a single WD2002FAEX drive would require: (10.7W / (6.3A * 19V)) * 100 = 8.94%. 8.94% * 4 disks = 35.76%. Not so shabby — but remember, we also have no idea how much power the Intel Atom motherboard, CPU, RAM, and other things require. Let’s just assume 35.76% is acceptable.

Now consider drive spin-up, which draws significantly more power than during normal operation: almost 4x more. StorageReview’s explanation (which I’ve linked) does not take into consideration technology like SSU or PUIS, which we do know the WD2002FAEX supports. But let’s assume worst-case scenario, that the WD Sentinel DX4000 spins up all the drives simultaneously.

8.94% * 4 disks * 4x more power = 143.04%. Oh dear…

Considering that I cannot find a single white paper or technical document describing in detail the gains from using SSU/PUIS (only actual SATA-IO specifications for how to implement it as part of the ATA protocol), I cannot determine what the power savings are. However, to be fair, I do imagine that in a worst-case scenario the WD Sentinel DX4000 might spin up each drive with 5 seconds of delay between drives. That means the entire unit would have to wait 20 full seconds before becoming even remotely operational. That probably isn’t what happens in real life. :-)

I cannot believe that for a US$1500 product, Western Digital would choose this kind of configuration. Why is it so hard for companies to use internal transformers? Nobody in a SoHo or business-class environment wants in-line transformers or “wall warts”. Using an internal transformer/PSU would extend the depth of the device by 1-2 inches at most. RFI is not a concern given good shielding, and airflow wouldn’t be a problem assuming the PSU was placed along the outside edge and provided with ventilation slits (I’d be more worried about the drives overheating than the PSU!).

Consider that those in-line transformers are usually of sub-par quality, and are usually the first thing to die/go out on you.

What is it going to take for companies to this kind of cheap engineering? Even the Drobo FS uses the same configuration, although the DroboPro and DroboElite definitely got it right (not to mention costs US$250 less than the WD Sentintel DX4000).