EDIT: Those interested in the upcoming release of FreeBSD 8.0 should read both the below, as well as my Testing out FreeBSD 8.0-RC2 post (which notes that many, but not all, of these problems have been fixed).
Yesterday I took the plunge and upgraded my home FreeBSD amd64 box from RELENG_7 to FreeBSD 8.0-RC1, which going forward I will refer to as RELENG_8 (yes, it has been tagged!). I did a complete reinstall, like I always do when migrating between major FreeBSD releases. Said box consists of a Supermicro X7SBA motherboard, Intel Core2Duo E8400 CPU, 4GB of RAM, and 3 SATA disks connected via the on-board Intel ICH9 + AHCI — one for the OS, and two in a ZFS mirror pool. Relevant dmesg information:
CPU: Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz (2992.52-MHz K8-class CPU)
real memory = 4294967296 (4096 MB)
avail memory = 4112478208 (3921 MB)
atapci0: <Intel ICH9 SATA300 controller> port 0x1c50-0x1c57,0x1c44-0x1c47,0x1c48-0x1c4f,0x1c40-0x1c43,0x18e0-0x18ff mem 0xdc000800-0xdc000fff irq 17 at device 31.2 on pci0
atapci0: [ITHREAD]
atapci0: AHCI called from vendor specific driver
atapci0: AHCI v1.20 controller with 6 3Gbps ports, PM supported
ata4: <ATA channel 2> on atapci0
ata4: [ITHREAD]
ata5: <ATA channel 3> on atapci0
ata5: [ITHREAD]
ata7: <ATA channel 5> on atapci0
ata7: [ITHREAD]
ad8: 190782MB <WDC WD2000JD-00HBB0 08.02D08> at ata4-master SATA150
ad10: 953869MB <WDC WD1001FALS-00J7B1 05.00K05> at ata5-master SATA300
ad14: 953869MB <WDC WD1001FALS-00J7B1 05.00K05> at ata7-master SATA300
And ZFS-related details:
pool: storage
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
storage ONLINE 0 0 0
mirror ONLINE 0 0 0
ad10 ONLINE 0 0 0
ad14 ONLINE 0 0 0
errors: No known data errors
The first thing worth noting is that I performed the installation entirely using the FreeBSD 8.0-RC1 amd64 memstick image on a SanDisk Cruzer Micro 2GB USB drive. If you’re going to try this, make sure you have a 2GB or larger USB drive, since the memstick image is larger than 1GB.
Writing the .img file to the USB drive required the use of another FreeBSD box (someone is going to have to address that fact eventually), and was achieved per Ken Smith’s original instructions:
dd if=8.0-RC1-amd64-memstick.img of=/dev/da0 bs=10240 conv=sync
The entire dd took about 4 minutes. I then booted it directly without any issues. Installation went as expected, with the exception of choosing a different installation medium than usual; there’s a new USB menu item near the bottom of the installation medium selection list. I should also note that I deleted the existing FreeBSD partition and re-created it during the sysinstall phase, and during the Boot Manager selection phase, I chose Standard like I always do. You’ll understand why I’ve noted these two things in a moment.
After rebooting + booting off the main OS hard disk, the first thing I saw which was different/anomalous was that I was being shown the F1/F5/F6 FreeBSD boot manager menu — as if I had selected BootMgr and not Standard. Options shown were F1 for FreeBSD, F5 for Floppy, and F6 for PXE (that’s a new one!). So there’s definitely a bug/regression somewhere with regards to the boot manager you choose; or maybe it was because I was installing from a USB drive? Not sure.
The FreeBSD box booted fine, but the following kernel message caught my eye:
GEOM: ad8s1: geometry does not match label (255h,63s != 16h,63s).
This indicated that what sysinstall wrote to the actual on-disk BSD label, as far as drive geometry went, didn’t match what GEOM expected — GEOM expecting 16 heads, the drive label containing 255 heads. I’m not using GPT (at least not knowingly; if sysinstall does it without telling you, then someone needs to work out where the actual problem lies). I first looked at gpart show ad8 and gpart show ad8s1 to see what it claimed:
# gpart show ad8
=> 63 390721905 ad8 MBR (186G)
63 390716802 1 freebsd [active] (186G)
390716865 5103 - free - (2.5M)
# gpart show ad8s1
=> 0 390716802 ad8s1 BSD (186G)
0 4194304 1 freebsd-ufs (2.0G)
4194304 16777216 2 freebsd-swap (8.0G)
20971520 33554432 4 freebsd-ufs (16G)
54525952 16777216 5 freebsd-ufs (8.0G)
71303168 319413634 6 freebsd-ufs (152G)
Next, and more conclusive, I used bsdlabel -e -A ad8s1 and this is what I got:
# /dev/ad8s1:
type: ESDI
disk: ad8s1
label:
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 24321
sectors/unit: 390721968
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0 # milliseconds
track-to-track seek: 0 # milliseconds
drivedata: 0
8 partitions:
# size offset fstype [fsize bsize bps/cpg]
a: 4194304 0 4.2BSD 0 0 0
b: 16777216 4194304 swap
c: 390716802 0 unused 0 0 # "raw" part, don't edit
d: 33554432 20971520 4.2BSD 0 0 0
e: 16777216 54525952 4.2BSD 0 0 0
f: 319413634 71303168 4.2BSD 0 0 0
So yes, sysinstall is definitely doing the Wrong Thing(tm) here. One of the biggest problems with this is that users cannot easily fix this — it requires booting a fixit or LiveFS image and editing the disk label, as well as users having to re-calculate sectors/cylinder and cylinders by hand. And no, I’m not talking out of my ass. Also note that the post to freebsd-current is dated March 2009 — 7 months ago. So this issue has existed for quite some time without proper attention.
Upon exiting the editor session inside of bsdlabel (using :q!), I was given the following error:
bsdlabel: partition c doesn't cover the whole unit!
bsdlabel: An incorrect partition c may cause problems for standard system utilities
bsdlabel: Class not found
re-edit the label? [y]: n
Whoa whoa whoa, even more insanity going on here! Also, what’s with the Class not found error? Wow, this is pretty jacked, and apparently I’m not the only one who’s noticed. Note this post is dated July 2009 — 3 months ago. The easy solution seems to be to use gpart(8) to create all of the slices… except it’s obvious no one has fixed sysinstall to do this. There’s also a patch mentioned which fixes the problem, but that obviously wasn’t committed before the RELENG_8 tagging, nor backported since.
But the root of the problem does appear to be sysinstall not doing the Right Thing(tm) any longer.
Next up was my attempt to fix the Boot Manager oddities. Historically, re-writing the boot blocks on a disk consisted of doing the appropriate equivalent of bsdlabel -B ad8s1. However, I was greeted with the exact same “Class not found” error as above. Hmmm… This doesn’t bode well. Presumably I can use gpart(8) to re-write the boot blocks, but given that GEOM is complaining about disk geometry errors, I don’t dare mess with it.
All of this is a pretty major bug. The kernel message is going to catch a lot of user attention if it’s not fixed by the time 8.0-RELEASE is announced. I plan on sending Robert Watson and Ken Smith an Email about the issue after I get done writing this.
At this point I decided to make appropriate modifications to /etc/rc.conf, specifically the addition of zfs_enable="yes", so that I could get access to my ZFS filesystems. After doing so, and running /etc/rc.d/hostid start then /etc/rc.d/zfs start, I was greeted with the usual ZFS kernel messages — but with an unexpected surprise:
ZFS NOTICE: system has less than 4GB and prefetch enable is not set... disabling.
Given how familiar I am with FreeBSD and ZFS at this point, this message caught me off-guard.
First of all, grammatically this sentence is confusing as hell — there is no “prefetch enable” tunable; the tunable is actually called vfs.zfs.prefetch_disable, and it defaults to 0 (off, e.g. prefetching enabled), so why would I exclusively enable something which is enabled by default? And why’s it getting disabled? Secondly, my system has 4GB of RAM installed… so what’s going on here?!
I dug around in the relevant CVS commit logs and found numerous changes to this file, specifically the message that was printed. Apparently the existing message was reviewed by 5 separate people (revision 1.23) to “Improve wording”.
3554 #ifdef _KERNEL
3555 if (TUNABLE_INT_FETCH("vfs.zfs.prefetch_disable", &zfs_prefetch_disable))
3556 prefetch_tunable_set = 1;
3557
...
3565 if ((((uint64_t)physmem * PAGESIZE) < (1ULL << 32)) &&
3566 prefetch_tunable_set == 0) {
3567 printf("ZFS NOTICE: system has less than 4GB and prefetch enable is not set"
3568 "... disabling.\n");
3569 zfs_prefetch_disable=1;
3570 }
3571 #endif
Ahh, now we have a much better idea of what’s going on. There are two reasons why this message got printed on my machine:
1) I had not done any tuning of /boot/loader.conf at this point, so vfs.zfs.prefetch_disable hadn’t been set. The above code basically says “if someone has administratively set vfs.zfs.prefetch_disable to something in loader.conf, set prefetch_tunable_set to 1″. You can set the tunable to whatever you want (enabled or disabled) and the message won’t get printed. If you don’t set the tunable, the following applies:
2) physmem is actually the amount of memory in pages that’s available to the kernel when it loads. The multiplication is actually hw.availpages * hw.pagesize. The 1ULL << 32 statement may look ugly but it’s a bitshift equivalent of 2^32, e.g. 4294967296.
Let’s work out the math:
# sysctl hw.pagesize hw.availpages
hw.pagesize: 4096
hw.availpages: 1046201
# expr 1046201 "*" 4096
4285239296
4285239296 is indeed less than 4294967296. Wait a minute… where’s that extra memory going?
Well, it’s going to two places on the X7SBA: 1) on-board video (which has an 8MB framebuffer), and 2) the AHCI BIOS which takes up an unknown amount of RAM, but I’d guess about 1-2MB. So let’s do the math:
# expr 4294967296 - 4285239296
9728000
With all of this information kept in mind, the kernel message really should be re-worded to say the following:
ZFS NOTICE: System has less than 4294967296 bytes (4GB) of usable memory,
ZFS NOTICE: and vfs.zfs.prefetch_disable has not explicitly been defined
ZFS NOTICE: in loader.conf. Setting vfs.zfs.prefetch_disable="1"...
I’m also questioning the logic behind why prefetching is disabled on systems with less than 4GB of available memory; I’d like to know what the reasoning is there. Is it in regards to stability? Performance? I don’t know. I can’t find an answer on the mailing lists either.
Finally, I found an unexpected oddity with the new tty/pty/pts code with regards to EOF. All other operating systems, including RELENG_7 and earlier, behave as follows when EOF is pressed on a terminal. This is regardless of shell, by the way:
bash$ cat
{press Control-D here}bash$
While on RELENG_8, the literal Control-D (^D) character is shown on-screen:
bash$ cat
{press Control-D here}^Dbash$
I’ve already mailed Ed Schouten about this, and he agrees it’s a bug which he’ll work on fixing, hopefully tonight.
Everything else past this point was peachy keen. No odd problems building ports, no system lock-ups or odd experiences, and so on. It’s all worked great so far. I’m looking forward to upgrading our production servers to RELENG_8 when it comes out.
