Increased network latency to Europe servers — part 4

For the past couple weeks, the latency described in my previous blog posts has been somewhat tolerable. There have been times where it’s gotten pretty bad, but overall it’s been acceptable (averaging around 500ms). I’ve also seen a couple occasions where it’s dropped to under 200ms, which is ideal (pristine, even!), but it only lasts for a few minutes.

During the past 2-3 days when I’d see 800-1000ms latency, my mtrs would show absolutely no packet loss in Paris or Frankfurt. That indicates the problem (at that time) was somewhere between Frankfurt and the Blizzard server itself (which isn’t visible in any traceroute), or something on the server.

I decided to check the forums to see what others have been seeing… and sure enough, the problem continues. The official thread (for February) is listed first:

FreeBSD 7-STABLE installkernel broken

I rebuilt world/kernel today, and on installkernel, noticed it bailed out — nuking /boot/kernel/* in the process. How nice:

kern.bootfile: /boot/kernel/kernel -> /boot/kernel.old/kernel
mkdir -p /boot/kernel
install -p -m 555 -o  -g  kernel /boot/kernel
install: -g: Invalid argument
*** Error code 67

Stop in /usr/obj/usr/src/sys/GENERIC
*** Error code 1

The nuking of /boot/kernel/* is caused indirectly — as you can see, the install line fails due to variable expansions not happening, resulting in the variables being empty, which ends up botching the passed arguments. I must’ve spent 3 hours poking at this, including booting up VMWare and installing 7.1-RELEASE then csup’ing just to reproduce the same problem.

After much mailing list searching, I finally found the root cause:

http://lists.freebsd.org/pipermail/svn-src-stable-7/2009-February/000584.html

The only three words coming out of my mouth right now are “test your changes before you commit your shit”. Why are people committing without fully testing prior (that includes world and kernel, both build and install)? This is completely unacceptable. Mistakes happen, but this is not a “oops sorry” mistake, this is flat out negligence.