Remember: the product is intended for businesses or high-class end-users.

Now take a look at its AC power connectors: two circular female connectors with a male centre pin. You know, the common “wall wart” and “in-line brick” crap you’d see on consumer-grade products that results in cable clutter (and absolutely impossible to properly deal with when placed in a rack)? No, surely that couldn’t be, especially for a SoHo device of those physical dimensions…

I thought for a moment it might be a proprietary connector that required a proprietary cable that went directly to a 115V AC plug, so I downloaded the official User Manual.

Lo and behold, on page 7 of the manual (page 11 per PDF index), an in-line brick (transformer) is shown. You get only one (the 2nd connector is for failover/redundancy), and its power requirements are miniscule. Quoting page 9 of the manual (page 13 per PDF index):

Input voltage: 100V to 240V AC
Input frequency: 50 to 60 Hz
Output voltage: 19V DC, up to 6.32A

I’m a little surprised by those power requirements: 6.3A at 19V. The unit comes with 5400rpm drives — if you were to shove four 7200rpm drives in there, you almost certainly would be taxing that little brick of an AC adapter. Your drives would start falling off the bus and experiencing strange anomalies during high I/O. Let me explain:

The WD Digital Caviar Black WD2002FAEX draws 10.7 watts when reading/writing according to WD’s own specifications.

Let’s figure out the percentage of power use a single WD2002FAEX drive would require: (10.7W / (6.3A * 19V)) * 100 = 8.94%. 8.94% * 4 disks = 35.76%. Not so shabby — but remember, we also have no idea how much power the Intel Atom motherboard, CPU, RAM, and other things require. Let’s just assume 35.76% is acceptable.

Now consider drive spin-up, which draws significantly more power than during normal operation: almost 4x more. StorageReview’s explanation (which I’ve linked) does not take into consideration technology like SSU or PUIS, which we do know the WD2002FAEX supports. But let’s assume worst-case scenario, that the WD Sentinel DX4000 spins up all the drives simultaneously.

8.94% * 4 disks * 4x more power = 143.04%. Oh dear…

Considering that I cannot find a single white paper or technical document describing in detail the gains from using SSU/PUIS (only actual SATA-IO specifications for how to implement it as part of the ATA protocol), I cannot determine what the power savings are. However, to be fair, I do imagine that in a worst-case scenario the WD Sentinel DX4000 might spin up each drive with 5 seconds of delay between drives. That means the entire unit would have to wait 20 full seconds before becoming even remotely operational. That probably isn’t what happens in real life. :-)

I cannot believe that for a US$1500 product, Western Digital would choose this kind of configuration. Why is it so hard for companies to use internal transformers? Nobody in a SoHo or business-class environment wants in-line transformers or “wall warts”. Using an internal transformer/PSU would extend the depth of the device by 1-2 inches at most. RFI is not a concern given good shielding, and airflow wouldn’t be a problem assuming the PSU was placed along the outside edge and provided with ventilation slits (I’d be more worried about the drives overheating than the PSU!).

Consider that those in-line transformers are usually of sub-par quality, and are usually the first thing to die/go out on you.

What is it going to take for companies to this kind of cheap engineering? Even the Drobo FS uses the same configuration, although the DroboPro and DroboElite definitely got it right (not to mention costs US$250 less than the WD Sentintel DX4000).

 Edit ports/security/unssh/Makefile
  Add delta 1.4 2011.10.18.22.35.18 pav
 Edit ports/security/unssh/distinfo
  Add delta 1.3 2011.10.18.22.35.18 pav
 Delete ports/security/unssh/files/extra-patch-unssh.sh.in

I immediately wondered what unssh was. Apparently it’s a shell script that modifies (removes lines from) your ~/.ssh/known_hosts file when run with the same arguments as ssh.

This immediately made me think “why is this even necessary when OpenSSH has framework to already accomplish this task”? Just edit your ~/.ssh/config and enter the following:

Host gw
StrictHostKeyChecking no

The only difference here is that this will never remove the outdated (“offending”) entry in ~/.ssh/known_hosts and you will continue to see the nasty man-in-the-middle warning every time you SSH to the host “gw”. However, you’ll be able to connect regardless of the warning.

There’s really no harm in ignoring the nastygram when connecting to a machine you know is going to change its SSH identity keys all the time. Really — the person SSH’ing there will already be aware of that (the fact they would go and find/use unssh is proof of that).

Sadly there is no -q equivalent in ssh_config(5), otherwise StrictHostKeyChecking no and Quiet yes would be a great combination for this exact situation.

• http://boingboing.net/2011/10/12/dennis-ritchie-1941-2011-computer-scientist-unix-co-creator-c-co-inventor.html
• https://plus.google.com/u/0/101960720994009339267/posts/ENuEDDYfvKP
• https://twitter.com/#!/packetslave/status/124270413927809024
• http://www.biobiochile.cl/2011/10/12/muere-dennis-ritchie-padre-del-lenguaje-de-programacion-c-y-el-sistema-operativo-unix.shtml
• http://nodo9.com/2011/10/12/fallecio-dennis-ritchie-cofundador-de-unix-creador-de-el-lenguaje-los-70-aos-de-edad/
• http://en.wikipedia.org/wiki/Dennis_Ritchie

If you’re a programmer and don’t know who Dennis Ritchie is, then you should have your programming license revoked permanently.

RIP, dmr. You will never be forgotten.

You still have to store the acquired data somewhere/somehow. And that’s where RRDTool, unfortunately and unjustifiably, comes into play. If you’re an administrator (or developer) that has to deal with systems monitoring and statistics data acquisition in an open-source world, there’s a very good chance (~90%) that you’ve had to painfully deal with this software.

My use of the word “deal” is a bit vague. “Deal” means actually deal with RRDTool, not “install it and use it blindly”. I’m referring to getting into the intricacies and annoyances of rrdcreate, rrdgraph, and rrddump — especially in the case where existing data is already stored.

You see, the problems with RRDTool are almost limitless. The frustration level is astounding; literally every system administrator I have spoken to about software that relies on RRDTool has spewed forth nothing but raw obscenities intermixed with questions/comments. A past colleague of mine once said “If I ever meet the author of RRD, Tobias Oetiker, I’m going to hurt him”. This is a sentiment I can relate to. The most common comments/questions I see are:

• What happened to my data?! That value wasn’t what I inserted into the database…
• Why do these graphs look wrong?
• Where are these NaN values coming from?
• What do you mean I’ll lose all my data when moving between i386 and amd64, or i386/amd64 and sparc?
• What do you mean I can’t easily add another data field to my CF without deleting my existing data?
• Why is this monitoring software that uses RRDTool chewing up so much disk I/O?
• Why are all my values, when dumped using “rrdtool dump”, shown in exponential format rather than something sane, and how do I get them to be sane?
• How do I get the grapher to stop using irrelevant units and show exactly what I want?
• What the hell? “Ordering CD from Amazon”?!

Then there’s the software dependency nightmare. RRDTool version 1.3 and later relies on an outrageous number of graphing and text/font libraries; on FreeBSD, ports/databases/rrdtool pulls in 51 dependencies, including some X11 bits. That’s 51 pieces of software just for what should be a “simple database with graphing and basic layout capabilities”. RRDTool version 1.2 (ports/databases/rrdtool12) only has 9 dependencies, almost all of which are image format libraries and a text rendering library (freetype2). System administrators often refuse to install RRDTool due to the horrible dependency count. I myself have stuck with RRDTool 1.2 for quite some time because of this. I also used to be the FreeBSD port maintainer for ports/databases/rrdtool12, fancy that.

And God forbid you encounter a bug in it — read the CHANGES file sometime. Some of the bugs are astonishing. And for additional amazement, read the mailing list.

Those of us who despise RRDTool want something that doesn’t mess about with your data, doesn’t result in a mesh of software dependency nightmares, performs well, and “just works” — all while trying to comply with the KISS principle as much as possible.

Except there isn’t anything that meets these simple criteria. Instead, what’s out there is a disgrace, often in other ways. Everything violates the concept of KISS as much as possible — why do I need an aircraft carrier if I’m only travelling a couple miles? FreeBSD administrators tend to apply KISS as much as possible; it’s less commonly-applied on Linux, for whatever reason.

What I’ve wanted for years can be described as follows:

A daemon with a reasonable memory footprint that polls hosts/devices using SNMP (version 2) and writes the acquired data to CSV format text files. No graphing is required; that can be done client-side (or with Timeplot if you prefer). Scalability is a must. Use of 64-bit SNMP counters should be applied wherever possible.

Sounds simple doesn’t it? Indeed — except in the open-source world, you won’t find it. What you will find are aircraft carriers, cobbled together with glue and sticks.

The closest thing I’ve found is a decent piece of software called rrdbot, which meets all of the above requirements (and guess what it uses for its SNMP client code? A copy of FreeBSD’s bsnmp library API) — except it uses RRDTool for storage. I’ve used rrdbot for years with success and reliability, but when it comes to adding a new data source, the travesty that is RRDTool kicks in. rrdbot itself is okay — the underlying code and framework could really use some cleaning up. But could it be modified to support CSV? Yes, and that is something I’ve begun to work on, but it’s extremely complex given how integrated RRD is within the software, all the way down to the configuration file format. It’s not a simple task.

Which brings me to an alternative that many Linux users have told me about over the years: collectd. Because “it supports CSV“.

Last night I spent 6-7 hours looking at it and doing my best to implement it. collectd’s SNMP support is a sad joke, both on a configuration level as well as a technical/implementation level. Let’s talk about it.

Firstly, the documentation for the SNMP plugin starts off with this amazing quote:

… SNMP is a widespread standard to provide management data from devices such as switches, routers, rack monitoring systems, uninterruptible power supplies (UPS), etc. While theoretically possible, collecting values from other computers via this protocol is discouraged in favor of collectd’s own protocol …

Yeah, why use an existing and admitted widespread standard? Who would ever want that!? Just let me know when collectd can run on our HP ProCurve switches, APC AP7900 RackPDUs, and MRV LX-series serial console units… Regardless of the idiotic justification for pushing a proprietary protocol over a standard one, I continued working with collectd.

Secondly, the configuration file format does not “play well” with SNMP. The official documentation doesn’t touch on design choice nuances, probably because they become apparent to any senior administrator attempting to accomplish said task. What am I talking about? Let’s examine these error lines:

[2011-09-13 17:02:43] snmp plugin: DataSet `counter' requires 1 values, but config talks about 6
[2011-09-13 17:02:43] read-function of plugin `snmp-machine.box.lan' failed. Will suspend it for 20 seconds.

These were from starting collectd with the following SNMP plugin configuration bits:

<Plugin snmp>
  <Data "tcp">
    Type "counter"
    Table false
    Instance ""
    Values "TCP-MIB::tcpActiveOpens.0"   \
           "TCP-MIB::tcpPassiveOpens.0"  \
           "TCP-MIB::tcpAttemptFails.0"  \
           "TCP-MIB::tcpEstabResets.0"   \
           "TCP-MIB::tcpCurrEstab.0"     \
           "TCP-MIB::tcpInErrs.0"
  </Data>
  <Host "machine.box.lan">
    Address "machine.box.lan"
    Version 2
    Community "public"
    Collect "tcp"
  </Host>
</Plugin>

I want to point out in advance that this configuration is not entirely correct nor should people try to use it. Specifically, the tcpCurrEstab MIB returns a GAUGE (i.e. a rate), not a COUNTER. I knew this in advance but was simply messing about with collectd’s SNMP support.

Anyway, the first thing that caught my attention was the badly-formatted error line relating to what plugin failed. The Host is not called snmp-machine.box.lan — the “snmp-” part comes from the collectd itself; it’s indicating “the snmp plugin”. Why this error line cannot be changed to “read-function of plugin snmp, host machine.box.lan failed” is beyond me. Please don’t tell me to fix it myself — I would much rather not look at the error handling code for collectd, despite being quite able to.

The dataset called “counter” comes from collectd’s own types.db file (which is, and I quote, “inspired by RRDtool’s data-source specification” — just shoot me now), which only permits 1 argument handed to the Data directive. For 6 arguments, one would need to make a new entry in types.db that allowed for such. But then if you added a new data argument, you would have to edit types.db to extend things, and any collectd.conf bits that used that type. Lather rinse repeat until you’re blue in the face.

To try and work around this absurdity I attempted to use the Type directive to specify multiple “counter” arguments, e.g.:

<Data "tcp">
  Type "counter"  \
       "counter"  \
       "counter"  \
       "counter"  \
       "counter"  \
       "counter"
  ...
</Data>

But was immediately shot down by syntactical limitations of the Type directive itself:

[2011-09-13 17:05:50] snmp plugin: `Type' needs exactly one string argument.

Wonderful. So what we have here is a glaringly obvious design choice limitation, by someone who wasn’t really thinking about what they were doing. The only “solution” is to give every single SNMP OID its own Data section. That isn’t acceptable for two reasons:

1. Data sections are defined per-Host-section, which means if you had 10 hosts and approximately 25 OIDs per host you wanted to monitor, that’s 250 blocks, and those are usually 10-15 lines each
2. Under the hood collectd does not aggregate multiple OIDs into a single SNMP GET statement (protocol-wise) — it issues one at a time. On the SNMP server side, this is extremely inefficient and rude

So really the entire situation could get solved if the configuration syntax ordeals were dealt with appropriately. Look at rrdbot’s method — it makes sense, it just has the evil of RRDTool associated with it:

[poll]
interval:            30
activeOpens.source:  snmp2c://public@machine.box.lan/tcpActiveOpens.0
passiveOpens.source: snmp2c://public@machine.box.lan/tcpPassiveOpens.0
attemptFails.source: snmp2c://public@machine.box.lan/tcpAttemptFails.0
estabResets.source:  snmp2c://public@machine.box.lan/tcpEstabResets.0
currEstab.source:    snmp2c://public@machine.box.lan/tcpCurrEstab.0
inErrs.source:       snmp2c://public@machine.box.lan/tcpInErrs.0

[create]
activeOpens.type:    COUNTER
activeOpens.min:     0
passiveOpens.type:   COUNTER
passiveOpens.min:    0
attemptFails.type:   COUNTER
attemptFails.min:    0
estabResets.type:    COUNTER
estabResets.min:     0
currEstab.type:      GAUGE
currEstab.min:       0
inErrs.type:         COUNTER
inErrs.min:          0
cf:                  AVERAGE

archive:   2/minute * 2 days,
           30/hour * 1 month,
           2/hour * 1 year

There are also multiple design problems with the collectd CSV plugin, not to mention a bad buffer size choice too.

Next, collectd on FreeBSD pulls in a bunch of third-party library dependencies. For example, to get network I/O statistics it relies on a library called libstatgrab even though collectd has getifaddrs(3) support natively. But if you want disk I/O statistics, you need libstatgrab because collectd doesn’t have the native code for obtaining such on the BSDs. And once libstatgrab is installed, it’ll use it for network I/O statistics as well. I can stomach this, but I’m not happy about it.

Linux folks have access to the almost all of this through /proc filesystem — which may not be the best place for such statistics, but implementation-wise a filesystem providing such is very much the true UNIX way, which in my opinion makes this method more UNIX-like than the BSDs.

FreeBSD offers nothing like Linux /proc. Systems data acquisition on FreeBSD involves either libc or syscall functions, which means there’s only one language that you can use: C. Calling command-line utilities is not an option — waste of CPU and memory resources due to excessive fork/exec, spawning of shells, etc. is unreasonable, and any administrator worth a quarter of his salary won’t permit it.

So on FreeBSD the best thing we’ve got is an SNMP daemon called bsnmpd(8). But most of the statistics a person might want aren’t provided, requiring one to install ports/net-mgmt/bsnmp-ucd, a shared library that makes available the more commonly-desired OIDs and MIBs for host statistics. FreeBSD bsnmpd is an alternate to net-snmp’s daemon, but much more bare-bones. The daemon has a minimal memory footprint as well. But bsnmpd can return questionable data for some OIDs.

collectd also makes use of a third-party dlopen(3) wrapper called libltdl that comes from its reliance on libtool. There are patches that address this awful choice, but once again, patches upon patches is the nightmare that FreeBSD users generally do not tolerate. I remember my Linux days vividly — patches atop patches atop patches, many of which conflicted with one another. No thanks.

It seems I’m not the only one who wants rrdtool’s graphing nonsense to die a horrible death. Of course, converting RRD data into JSON is probably a waste of time (my opinion is that the individual in the thread is trying to solve the problem at the wrong layer) — CSV would suffice, but I guess JSON would be “okay”. Anyone up for ASN.1? :-)

Finally, I want to talk a bit about my focus on CSV, because RRDTool advocates often point out how CSV does not scale when it comes to long-term data storage or trending. This is true — CSV doesn’t scale. It’s just a text log of comma-delimited values with timestamps. And given that client-side graphing software tends to lean towards CSV, network bandwidth quickly becomes a concern given that a year’s worth of data could result in a multi-megabyte CSV file. Here’s a write-up of mine on the matter and how to solve it efficiently:

One of the downsides to using CSV is that the file can become extremely large depending on how much data is within the file. This is something RRDTool solves by using RRAs; this is not something CSV can solve without implementing a CSV parser and creating “averaged” CSVs which can be read by the software. This might be a worthwhile enhancement, but is currently not implemented.

For example: if your polling interval is once a day, and you’re polling only 2 OIDs, your data file is going to be quite small: maybe 12KBytes for an entire year’s worth of data. That’s 2*24*30*12, or 17280 pieces of data (2 OIDs times 24 hours times 30 days times 12 months).

However, if your polling interval is once every minute, this gets much worse: 2*60*24*30*12, or 1036800 pieces of data (2 OIDs times 60 minutes per hour times 24 hours times 30 days times 12 months). That’s 60 times more data! 12000 bytes * 60 = 720,000 bytes (~700KBytes).

Even on a fast internet connection, 700KBytes of data might take a while. Plus there’s the parsing time by dygraphs, HTTP protocol overhead, and so on. And just as important is bandwidth usage; this could really hurt if you’re on a 95th-percentile connection.

So how do we solve both of these problems?

Web servers like Apache offer server-side gzip compression; in the Apache world this is called mod_deflate. The browser has to support this too, of course, but all present-day browsers do, as well as some previous-generation browsers. It’s been around for a while.

The HTTP server will compress the data sent to the client in real-time. What’s important to keep in mind is that CSV data is ASCII and very repetitious in nature — it compresses **extremely** well.

For example, with mid-level gzip compression (level 4), a 950KByte CSV file compresses down to around 9-10KBytes. Really! So as long as the web server environment is configured to use gzip compression when serving CSV files to the browser (when viewing graphs), and as long as the browser supports gzip compression, we have very little to worry about.

The trade-off is that the HTTP server may have a slightly higher load. As long as thousands of people aren’t pounding the graphing page all at the same time, this shouldn’t be a problem; most present-day processors are amazingly fast, and zlib is extremely optimised.

So use mod_deflate, make sure your webserver is configured to use it for CSV files (client should send an Accept-Encoding HTTP header with gzip/deflate provided, and the server should respond with a Content-Encoding header with gzip provided). There’s an old write-up at LinuxJournal which can help.

I guess for now I’ll stick with using rrdbot and gritting my teeth over RRDTool, until I can finish writing the necessary CSV code bits for rrdbot. Stef Walter, the author of rrdbot, seems to have gone MIA so I wouldn’t even be able to submit patches upstream and have them implemented in a timely manner.

Are we having fun yet? Yeah, me neither.

Well, it didn’t.

Things worked for about 4-5 days, or so I thought. Then suddenly one morning while doing backups (from the local hard disk to a USB-connected hard disk), the machine sporadically shut off and stayed off. I had no 3D applications/games running, so this worried me even more — plus the machine didn’t power back on. Manually powering the system on worked fine, where it came back up and remained up for about 5-10 minutes more before losing power (but this time, starting back up on its own).

At this point I knew the issue was either the motherboard or the PSU. It had to be. I disassembled my system and began inspecting all of the components to see if anything stood out. All the PCBs looked fine from what I could tell, nothing looked amiss or awry.

That is, until I found this…

What you see here is the motherboard-facing connector of a 24-pin ATX power extension cable, used between my Antec TP-650 (TruePower New) PSU and my Asus P5Q SE. Why? Because I use an Antec P180 case. Extension cables are needed for both the 24-pin ATX and 4-pin 12V CPU power connectors. (Note: still to this day it amazes me that Antec PSUs aren’t fully compatible, cable-length-wise, their own cases/enclosures!). The extension cable cost about US$10 and has been in use for 4-5 years.

In case you can’t tell: from the photo one of the pins is about 3/4ths gone (especially visible in the 3rd photo), while the other is intact but not looking good. I then found myself wondering what the 2 pins in question were for. Reading the official ATX 2.2 specification answered my question — here’s a GIF of the 24-pin ATX pin-out itself:

So the pins in question are two (2) +12V lines. It’s impossible for me to tell what devices on the motherboard use said pins, but it doesn’t take a genius to know that the effect this can have on a system is extremely bad. Admittedly the PC BIOS does monitor +12V (via a Nuvoton/Winbond W83667HG chip), however I never witnessed any oddities there.

I inspected the Asus P5Q SE motherboard: the ATX power connector on the board showed no discolouration, however both +12V pins were extremely tarnished (a dull, dark grey rather than silver).

I pondered whether the issue was caused by the PSU or caused by an issue on the motherboard. I still don’t know which could have truly caused this problem. Gut feeling says the PSU, but as I said, it’s very difficult to tell. Do I deem the motherboard reliable? No. Do I deem the PSU reliable? No.

I’d like to point out that had the 24-pin Molex connector been of black plastic rather than white, I wouldn’t have found this problem. All these new PSUs come with black plastic connectors rather than white, for the stupid reason that it’s more aesthetically pleasing. Seriously, how much time do you spend looking at your motherboard? PSU manufacturers should really use white connectors on everything solely for this exact reason. Otherwise, you end up having to destroy the connector anyway.

Lots of other people have had the same experience I have, though with different PSU brands and different motherboards. Some of my favourites:

• Two (2) damaged +12V pins (same as me!)
• Four (4) damaged +5V pins
• Four (4) damaged +5V pins
• Two (2) damaged +5V pins
• Three (3) damaged +3.3V pins
• Three (3) damaged +3.3V pins
• Melted SATA power connector and damaged DVD drive

Replacement parts are on their way. Too bad it’s so hard to find Socket 775 parts these days; I’m not jumping on the Socket 1366/1155/1156 bandwagon with Intel soon to be releasing their LGA 2011 (Socket R) stuff.

The new PSU hasn’t arrived yet, but as previously stated I did go about rebuilding the system to some degree (sans hardware replacements). More specifically, the following things were performed:

• Relocation of chassis/system atop my desk (vs. on the floor near my desk).
• Full OS disk format. System disk is an Intel 320-series 80GB SSD with no anomalies of any sort (verified analysing SMART attribute data). “Full format” means I booted a Vista recovery CD, used diskpart.exe to delete the old C: partition and recreate it (CREATE PARTITION PRIMARY ALIGN=1024, followed by FORMAT FS=NTFS LABEL="OS", ACTIVE, then used HxD to skim through LBAs on the SSD to ensure they all got zeroed), and reinstalled Windows XP, drivers, DirectX, etc…
• Re-did all system GUI settings, configuration adjustments that I like, and so on. This, combined with the above item, took most of the day.
• My secondary storage disk (D:), a Western Digital Caviar Black 1TB drive, was not formatted or adjusted in any way. SMART attribute data also verifies the disk is in perfect condition.
• Deleted my D:\Games\Steam directory. This is a key point because the game which was causing the reboots was Rift, despite the “Recover” option in-game finding nothing wrong (the option performs MD5 checksums of all game files and re-downloads any which don’t match what their server claims is healthy).

Much to my surprise, all games (particularly Rift) run completely fine now. No more random/sporadic system reboots.

Because of this, I’m inclined to believe the issue wasn’t PSU-related at all, but rather some sort of OS or kernel driver corruption may have occurred on my C: drive. Since I don’t perform MD5/SHA-1/SHA-256 checksums of all files on my OS drive, it’s impossible for me to determine if silent filesystem corruption may have occurred.

I don’t consider this issue solved; it’ll be a week or 2 before I feel certain that the problem is gone. If I had to take a guess, I’d say some sort of weird GPU driver corruption or DirectX installation corruption resulting in something extremely weird going on. Whatever it was, it was bad enough to cause my system to lose power rather than crash.

I wanted to document how I went about troubleshooting this issue and what I did to solve it. At the time of this writing I have no confirmation of what the problem part is, but over time I hope to figure it out.

The problem is directly related to GPU use — more specifically, 3D games. memtestG80 and memtestCL, which are used for testing VRAM/GDDR RAM on video cards (and to some degree stress-testing the GPU itself) works with flying colours for hours, yet within 1-15 minutes of loading any sort of 3D game the system abruptly loses power. memtest86 (for testing system RAM) also runs fine.

Very important things to note:

• My flat has no air conditioning. Yesterday it was around 92-93F outside (its been like this all week). Lack of airflow in my flat means that during the early evening my thermostat reads 85-86F (really!).
• The system is hooked up to a UPS. The UPS also has other (minor, low-power-draw) devices on it which do not lose power or go down. This means the UPS is not at fault.
• The Windows kernel is not crashing. There are no blue-screens, memory mini-dumps, or evidence of driver crashes in the Event Log.
• Voltages and temperatures for my MSI N560GTX-TI Twin Frozr II card are excellent; there’s no sign of any anomalies shortly before the system loses power.
• The same with my system CPU — well, temperature are higher than usual, but well within permitted parameters (e.g. I am not anywhere near TjunctionMax for my CPU). The same system has run fine for a few months now, barring bad RAM on my Gigabyte GTX 560 Ti card.
• I cleaned out the entire PC using a extremely reliable duster and there was no difference (aside from some lower temperatures).

With this information, I’m under the impression I have a piece of hardware (transistor, capacitor, who knows what) “somewhere” within the system that is flaking out intermittently; almost like a short. It’s probably power-related given the symptoms. If my BIOS setting was changed, the system would power down and stay down/off.

The first piece of hardware I’m choosing to replace is my PSU (an Antec TruePower New 650W). My logic is that the PSU may have some sort of internal damage due to the recent extreme heat. The replacement is a Corsair CMPSU-850AX. It should be here later this week.

I’m really hoping the issue turns out to be PSU-related, as the next part to replace is the mainboard. Sadly the Asus P5Q SE is no longer sold/manufactured, which means I’d need to go with another brand/model (probably a Supermicro C2SBX). The problem with switching mainboards is that I’d have to redo my entire Windows XP slipstreaming/driver integration process. Windows, making my life miserable as usual. :-)

Update: I attempted an “extreme” system stress test by running Prime95 in “blend” mode with 4 threads, Furmark 3D in “Xtreme burn-in” mode with PostFX enabled, and Winamp playing a series of MP3s — all simultaneously. This ran for about 30 full minutes. CPU core temps reached 58C, and GPU temps reached 70C… all with full stability. No shutdown.

One thing to note is that Furmark3D doesn’t actually test every single DirectX or video card feature; meaning, for example, Furmark3D is not the same thing as Rift or World of Warcraft. There’s also the possibility that (despite my drive looking heathly) there is something bad going on filesystem or disk-wise where there may be corrupted files of some sort (possibly even DirectX itself).

Before the PSU replacement gets here I’m inclined to rebuild the system (full format, OS reinstall, software reinstall) to see if that fixes it. If not, that’s just one more piece I can rule out.

Every time there’s a release I have to go through the annoyingly repetitious process of finding an Addon that addresses the lack of said context menu. With the release of Firefox 5, there are absolutely none which work with it — except one, which I’ll save for last.

Edit: It appears that QuickPrint has been updated to work with Firefox 5, however there’s no indication of that change in the ChangeLog or on the addons site. Very, very mysterious.

All the Addons which have historically worked are nothing but copies of one another, simply because the previous author chooses to neglect the responsibility of updating the Addon every time there’s a new Firefox release. See for yourself:

• Print Context Menu; last updated 2008/10/15; works with 1.5 to 3.6.x
• Print Preview; last updated 2009/03/09; works with 1.0.x to 3.0.x
• Print Preview; last updated 2009/10/10; works with 1.5 to 3.5.x
• Print/Print Preview; last updated 2010/01/13; works with 1.5 to 3.6.x
• Print; last updated 2011/02/15; works with 1.0 to 4.0.x
• QuickPrint; last updated 2011/02/17; works with 2.0 to 4.0.x

Starting to get the picture? And as promised, I saved the best (worst) for last:

• Print/Print Preview (Update); last updated 2011/05/27; works with 3.0 to 6.x.x

The sad part is this one does work with Firefox 5, but where it decides to insert its context menu items is a complete disgrace. It’s almost like nobody bothered testing it. Sure, the general web page context menu looks great, but what about if you try to print a page while some text is selected?

…oh dear. It’s back to basic UI design school for you, bucko! Who thought it would be a good idea to put the Print context menu before Copy/Select All, especially when the item is further down the list when text isn’t selected?

Is the order customisable? Nope, and I’m not too surprised by that either.

Why do I care so much about the ability to print directly from the context menu? Why not just go to File/Print? Well, part of the annoyance stems from this absolute monstrosity of a bug where printing without doing a Preview first results in most of the text of the printed web page turning into gobbledegook. Supposedly that issue is due to “badly-written Windows printer drivers”, but if you skim the Mozilla bug you’ll see that tons of different printers, all from different manufacturers, all do this — including present-day mainstream printers. Internet Explorer, Google Chrome, nor Opera behave just fine. Makes you wonder just what’s “badly-written” doesn’t it? The priority of that bug should be P0 given that all it does is result in internationally wasted paper and aneurysms.

I always think it’s interesting how the severity of an (unrelated) bug can lead one on a quest to find workarounds/solutions that only induce more annoyances, lather rinse repeat…

Sadly, however, starting sometime this week my card began to exhibit very strange problems — but only on 2D (or presumed to be 2D) surfaces. It’s not something easily noticeable in a 3D-based game, but playing something like Dwarfs!? made it apparent within seconds. Then I tried Two Worlds (which uses a 2D texture for its sky) and noticed occasional problems there too.

Later, I ran Misha’s Video Memory Tester application which confirmed that the likely source of the problem was bad GDDR5 RAM on the card, or a bad bus/path between the RAM and the GPU itself.

Here are some screen shots of Two Worlds, Dwarfs!?, and Video Memory Tester. In the latter case, you can even see some discoloured pixels in GDI, confirming the issue is with RAM on the video card and not system RAM:

I tried doing things like decreasing both the core and memory clock on the card to no avail. Temperatures appear to have no bearing on the failure; the card can be running extremely cool or extremely hot and the problem occurs with the same regularity. Using tools like Furmark worked fine, ATITool’s artefact tester is useless since it gives false positives (search the Web for validation of that statement), and EVGA’s artefact tester only works on EVGA-branded cards.

I’ve put in an order for an MSI N560GTX-TI Twin Frozr II to hold me over while the GV-N560OC-1GI is being RMA’d with Gigabyte.

It probably doesn’t help that these cards are slightly overclocked from their reference specs either; too bad nobody’s selling Gigabyte’s GV-N560UD-1GI, which is a GTX 560 Ti with reference specifications (no overclocking).

Misha’s Video Memory Tester is ultimately what convinced me the problem was with the GDDR5 RAM. All the errors shown indicate primarily a series of single-bit errors (e.g. I wrote $FF00 (%1111111100000000) but what I got back was $EF00 (%1110111100000000)), and some multi-bit errors. Since the error locations are mostly within the same region (I had to look at the error log to confirm this), this would appear (visually) as anomalies in a game (texture problems or random bits not being set in texture/RGB data) — or, in the case of GDI, certain pixels being mangled on the desktop.

http://mosax.sakura.ne.jp/fswiki.cgi?page=SCFH+DSF+Dev

The translated bullet items on the above page (as of 2011/04/29) read as follows, acting as a “to-do” list:

* Reorganise source code
* Fix audio/video de-sync issue pertaining to timing/timers
* Fix incompatibility with Skype
* Re-write in-line assembler code to work on x64/64-bit OSes
* Fix stencil buffer processing (currently lost when using software resizing, as a result of optimisation)
* Reduce OS/environment load when capturing via DWM (can be managed via DWMCAPsrc.zip (see main page for .zip file))
* Recreate the GUI entirely
* Make the GUI simpler to use

The actual github URL is here:

https://github.com/mosamosa/SCFH-DSF

I hope this will provide much evolution and, eventually, better support for Windows 7. For now, this guy is sticking with XP… :-)

I think one of these days I might have to mimic the above while doing SMART analysis and see what the drive claims is going bad. ;-)

I’ve been using an nVidia GeForce 9800GT for quite some time now, and prior to that a 9600GT. When going from my 9600GT to my 9800GT, I was disappointed by the extreme jump in temperature, as well as power use — but the major performance increase made it worthwhile. I was also quite fond of my 9800GT since it was one of the BFG cards which didn’t have a PCIe power connector (it was powered completely off the PCIe bus) and only took up the space of one physical slot.

The reason I stayed with my 9800GT was that none of the newer cards impressed me. Sure, performance-wise they were superior, but I kept reading horror stories about temperatures reaching 90-100C (and people justifying that this is normal), in addition to fans reaching 55-65dB while the card was under load. Why were consumers accepting this garbage?

When it comes to video cards, my focal points are noise, temperatures, power usage, then performance. When I read reviews, I always look for sections associated with those attributes first. I’ll often swap the positions of power use and performance too; depends on the overall impression I have of the card.

Also, I want to make something clear in advance: as of this writing I tend to stick to nVidia cards given my horrible experiences with ATI/AMD Radeon cards having 2D/GUI driver bugs (I spend far more time using my desktop than I do playing games). And I only buy cards with dual DVI connectors (I eventually plan on getting a 2nd monitor).

Early last week, Gigabyte announced RTM of their new nVidia GTX 560 Ti cards: the GV-N560OC-1GI and the GV-N560SO-1GI. For those unsure of the the differences between cards: the former is overclocked with core/shader/memory clocks of 900MHz/1800MHz/1002MHz, while the latter is SOC (super overclocked) with core/shader/memory clocks of 1000MHz/2000MHz/1145MHz. The nVidia reference specifications are 822MHz/1645MHz/1002MHz.

I started reading reviews of the SOC version and became ecstatic:

• Noise: idle = 30-32dB, load = 32-34dB.
• Temperatures: idle = 30-35C, load = 60-65C.
• Power usage: hard to judge, but looked lower than my 9800GT during idle, yet higher during load.
• Performance: would destroy my 9800GT. :-)

Was this for real? Most gamer-centric review sites tend to use chassis that contain absurd amounts of fans, and their reviewers don’t have ears as sensitive as mine; would I see similar numbers using my Antec P182?

I immediately ordered the GV-N560OC-1GI from NewEgg (since I couldn’t find any other online vendors who had them for sale at the time); I never take chances with SOC cards. It arrived Friday. Installation went fine, as my Antec TruePower 650W has dual PCIe power connectors on a single cable. The new card took up two physical slots compared to my 9800GT but that wasn’t a deal-breaker for me.

The first thing I noticed was the level of noise. It was quieter than my 9800GT, and as an indirect result I found that one of my case fans was making an audible clicking noise. Once I booted up and installed the latest nVidia drivers, I ran GPU-Z to check things out. My jaw hit the floor:

• Noise: idle = fan at 40% (1750rpm), load = fan at 60% (2450rpm). I couldn’t hear any difference between the two, which greatly impressed me. 9800GT: idle = 30%, load = 80-90% (and extremely loud).
• Temperatures: idle = 28-30C, load = 55-57C. 9800GT: idle = 55-60C, load = 72-75C.
• Power usage: idle = roughly 20W lower than the 9800GT, load = roughly 60W higher than the 9800GT. This is actually an improvement, since 90% of the time my workstation idles, so overall I’m probably saving power!
• Performance: I’m not going to review this attribute. The simple version is that the new card stomps all over the 9800GT in every way.
• Three-tier clock throttling — when at the desktop, the clock frequencies would drop to literally 50MHz/101MHz/68MHz, but when under load the frequencies would jump up to their maximum. There was also a mid-level tier in between the minimum and maximum, providing somewhat of a “mid-level” performance tier.
• More monitoring attributes available under GPU-Z — now I could see actual fan RPMs, VDDC, memory controller load, and shader clock rate.

I imagine the extreme temperature difference between these two cards can be attributed to going from 55nm to 40nm TSMC manufacturing, since the 9800GT has 754 million transistors while the GTX 560 Ti has 1950 million.

I’m an extremely happy camper. This card is amazing and is exactly what I was hoping it would be. Well worth the money!

Reviews of GTX 560 Ti cards:

• Anandtech – nVidia GTX 560 Ti reference card
• Bit-Tech – nVidia GTX 560 Ti reference card
• Guru3D – nVidia GTX 560 Ti reference card
• HEXUS – Gigabyte GV-N560SO-1GI
• MaximumPC – nVidia GTX 560 Ti reference card
• Overclockers Club – ASUS GTX560 TI DirectCUII TOP
• TechSpot – Gigabyte GV-N560SO-1GI
• TweakTown – Gigabyte GV-N560SO-1GI

Why? Too many reasons, but a few stood out from all the others.

The more time I spent there, the more I realised the service did absolutely nothing that couldn’t be accomplished through existing means. If people want to find me (nostalgia, chatting about old times, etc.), I’m incredibly easy to find on the web given what I do, what I’m associated with, where I live, and so on. And if you already communicate with me then you already know what works best (Email, IM, or phone call).

I came to the (somewhat narcissistic) conclusion that if I matter to someone or something is important enough, then “tracking me down” should be pretty easy (find my home page/blog/whatever and Email me) — as folks have for the past, oh, 15 years.

Facebook might make that whole process easier, but there something very uncomfortable about the model of “social networking” sites (including MySpace, Friendster, etc.). When someone does “dig you up” there’s a sudden feeling of excitement or amazement (“Wow! I haven’t heard of or talked to person in ages!”). This feeling continues for a while (varies on the person and the relationship you had with them), but as time progresses less and less actual interaction happens. Eventually the only communication that occurs is through status updates, links to random web sites, Youtube videos, or some Facebook application/game. And (for me any way), I start feeling disconnected from those who I just “reconnected” with. This even applies to people who I see in person on a daily basis (then ask yourself how that’s even possible).

The feelings I’m describing are almost like that of drugs — initial highs, with a gradual decline (increase in monotonous behaviour) until the cycle repeats. Communication starts feeling like that of a personal service: a few messages back and forth, then silence followed by occasional viewing of one another’s profiles. I always have something to say, but most other people don’t. I’m never bored, but maybe other people become so.

Or maybe, just maybe, I’m not cut out for this “social networking” thing.

$ curl -i -s -S <b>http://get.adobe.com/reader/</b>
HTTP/1.1 301 Moved Permanently
Server: Apache/2.0.63 (Unix)
<b>Location: http://get.adobe.com/reader/</b>
Keep-Alive: timeout=5, max=500
Content-Type: text/html; charset=iso-8859-1
Accept-Ranges: bytes
Connection: Keep-Alive
Date: Thu, 09 Sep 2010 04:47:10 GMT
Age: 248
Content-Length: 236

But every so often, you’ll get back something that works…

$ curl -i -s -S <b>http://get.adobe.com/reader/</b>
HTTP/1.1 302 Moved Temporarily
Date: Thu, 09 Sep 2010 04:47:11 GMT
Server: JRun Web Server
Content-Type: text/html; charset=UTF-8
Cache-Control: private, no-store, no-cache
Content-Language: en-US
Content-Language: en-US
<b>Location: /reader/otherversions/</b>
Set-Cookie: SETTINGS.LOCALE=en%5Fus;domain=.adobe.com;expires=Sat, 01-Sep-2040 04:47:11 GMT;path=/cfusion/
Set-Cookie: READER_HTTPREFERER=;domain=.adobe.com;path=/
Content-Length: 0
Connection: close

At Adobe, web services are hard. I wonder if they’ve been replacing their system administrators with monkeys.

$ date
Tue 31 Aug 2010 06:53:37 PDT
$ curl -v -i http://www.facebook.com/
* About to connect() to www.facebook.com port 80 (#0)
*   Trying 69.63.189.26... connected
* Connected to www.facebook.com (69.63.189.26) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.20.1 (amd64-portbld-freebsd8.1) libcurl/7.20.1 OpenSSL/0.9.8n zlib/1.2.3
> Host: www.facebook.com
> Accept: */*
>
* Closing connection #0
* Failure when receiving data from the peer
curl: (56) Failure when receiving data from the peer

Hmm, what’s happening on a TCP level?

# tcpdump -p -i em0 -l -n -s 8192 "port 80"
listening on em0, link-type EN10MB (Ethernet), capture size 65535 bytes
06:55:33.380449 IP 192.168.1.51.50324 > 69.63.189.26.80: Flags [S], seq 2661770843, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 2499401659 ecr 0], length 0
06:55:33.482251 IP 69.63.189.26.80 > 192.168.1.51.50324: Flags [S.], seq 1390727061, ack 2661770844, win 4380, options [mss 1460,nop,wscale 0,nop,nop,TS val 3362678839 ecr 2499401659,sackOK,eol], length 0
06:55:33.482285 IP 192.168.1.51.50324 > 69.63.189.26.80: Flags [.], ack 1, win 16471, options [nop,nop,TS val 2499401761 ecr 3362678839], length 0
06:55:33.482416 IP 192.168.1.51.50324 > 69.63.189.26.80: Flags [P.], ack 1, win 16471, options [nop,nop,TS val 2499401761 ecr 3362678839], length 148
06:55:33.585188 IP 69.63.189.26.80 > 192.168.1.51.50324: Flags [R.], seq 1, ack 149, win 4528, length 0

Oh, TCP RST + ACK. Nice.