FreeBSD 7.2, ZFS, amd64, and vm.kmem tuning

I may no longer participate in the FreeBSD Project, but I still follow commits to RELENG_7 daily. That said, I decided to see what recently had been happening on the mailing lists, and was surprised to see some threads discussing ZFS still requiring tuning vm.kmem_size and vm.kmem_size_max:

What shocks me is that not a single person appears to have mentioned or noticed Alan Cox (alc@)’s commits back in January 2009 which allow the kernel to auto-configure these variables on boot, as well as increasing KVA space on amd64. This was mentioned in src/UPDATING:

20090207:
        ZFS users on amd64 machines with 4GB or more of RAM should
        reevaluate their need for setting vm.kmem_size_max and
        vm.kmem_size manually.  In fact, after recent changes to the
        kernel, the default value of vm.kmem_size is larger than the
        suggested manual setting in most ZFS/FreeBSD tuning guides.

The relevant commits are below. I’ve marked the important commits (that is to say, the comments that explain them) with (!!!):

These commits extend KVA space on RELENG_7 from 2GB (yes, on amd64) to 6GB. If users need something larger, at this time they’ll need to run CURRENT (which supports KVA up to 512GB).

Those who follow the RELENG_7 branch and use amd64 should rebuild world/kernel, and remove vm.kmem_size and vm.kmem_size_max tunings from /boot/loader.conf. I’d advocate this be done on systems with 2GB as well.

And yes, it works: Here’s one of our amd64 servers using ZFS, with no /boot/loader.conf tunings:

$ sysctl -a | grep vm.kmem_size
vm.kmem_size_scale: 3
vm.kmem_size_max: 3865468109
vm.kmem_size_min: 0
vm.kmem_size: 1378717696

$ sysctl -a | egrep 'kstat.zfs.misc.arcstats.(p|c|c_min|c_max|size):'
kstat.zfs.misc.arcstats.p: 327294579
kstat.zfs.misc.arcstats.c: 386455777
kstat.zfs.misc.arcstats.c_min: 43084928
kstat.zfs.misc.arcstats.c_max: 1034038272
kstat.zfs.misc.arcstats.size: 337688064

Community members may remember statements from select FreeBSD kernel developers and FreeBSD Core members claiming KVA expansion “would very likely never happen on RELENG_7” and instead were told to run CURRENT to be able to use more than 2GB of memory for ZFS on amd64. That statement made me furious; so Alan’s commits were quite a surprise, much like that of Santa Claus and Christmas gifts.

I strongly recommend those reliant on ZFS send Alan an Email thanking him for his efforts and taking the need for expansion and stability on RELENG_7 seriously. He can be reached at alc@freebsd.org. Oh one other thing: he isn’t the same Alan Cox that works for Red Hat / on the Linux kernel. Two separate people, same name. :-)

Someone should also see about updating the FreeBSD “ZFS Tuning” Wiki to reflect this wonderful change.

EDIT: Ivan’s taken care of editing the Wiki. Thanks Ivan! :-)