I may no longer participate in the FreeBSD Project, but I still follow commits to RELENG_7 daily. That said, I decided to see what recently had been happening on the mailing lists, and was surprised to see some threads discussing ZFS still requiring tuning vm.kmem_size and vm.kmem_size_max:
http://lists.freebsd.org/pipermail/freebsd-stable/2009-April/049307.html
http://lists.freebsd.org/pipermail/freebsd-stable/2009-April/049251.html
What shocks me is that not a single person appears to have mentioned or noticed Alan Cox (alc@)’s commits back in January 2009 which allow the kernel to auto-configure these variables on boot, as well as increasing KVA space on amd64. This was mentioned in src/UPDATING:
20090207:
ZFS users on amd64 machines with 4GB or more of RAM should
reevaluate their need for setting vm.kmem_size_max and
vm.kmem_size manually. In fact, after recent changes to the
kernel, the default value of vm.kmem_size is larger than the
suggested manual setting in most ZFS/FreeBSD tuning guides.
The relevant commits are below. I’ve marked the important commits (that is to say, the comments that explain them) with (!!!):
http://svn.freebsd.org/changeset/base/188291
http://svn.freebsd.org/changeset/base/187523
http://svn.freebsd.org/changeset/base/187522
http://svn.freebsd.org/changeset/base/187520
http://svn.freebsd.org/changeset/base/187485
http://svn.freebsd.org/changeset/base/187466 (!!!)
http://svn.freebsd.org/changeset/base/187465 (!!!)
http://svn.freebsd.org/changeset/base/187464
http://svn.freebsd.org/changeset/base/187458
http://svn.freebsd.org/changeset/base/187428
http://svn.freebsd.org/changeset/base/187425
http://svn.freebsd.org/changeset/base/187420 (!!!)
http://svn.freebsd.org/changeset/base/187419
http://svn.freebsd.org/changeset/base/187416
http://svn.freebsd.org/changeset/base/187414 (!!!)
http://svn.freebsd.org/changeset/base/187408 (!!!)
http://svn.freebsd.org/changeset/base/187407
http://svn.freebsd.org/changeset/base/187404 (!!!)
http://svn.freebsd.org/changeset/base/187400 (!!!)
These commits extend KVA space on RELENG_7 from 2GB (yes, on amd64) to 6GB. If users need something larger, at this time they’ll need to run CURRENT (which supports KVA up to 512GB).
Those who follow the RELENG_7 branch and use amd64 should rebuild world/kernel, and remove vm.kmem_size and vm.kmem_size_max tunings from /boot/loader.conf. I’d advocate this be done on systems with 2GB as well.
And yes, it works: Here’s one of our amd64 servers using ZFS, with no /boot/loader.conf tunings:
$ sysctl -a | grep vm.kmem_size vm.kmem_size_scale: 3 vm.kmem_size_max: 3865468109 vm.kmem_size_min: 0 vm.kmem_size: 1378717696 $ sysctl -a | egrep 'kstat.zfs.misc.arcstats.(p|c|c_min|c_max|size):' kstat.zfs.misc.arcstats.p: 327294579 kstat.zfs.misc.arcstats.c: 386455777 kstat.zfs.misc.arcstats.c_min: 43084928 kstat.zfs.misc.arcstats.c_max: 1034038272 kstat.zfs.misc.arcstats.size: 337688064
Community members may remember statements from select FreeBSD kernel developers and FreeBSD Core members claiming KVA expansion “would very likely never happen on RELENG_7″ and instead were told to run CURRENT to be able to use more than 2GB of memory for ZFS on amd64. That statement made me furious; so Alan’s commits were quite a surprise, much like that of Santa Claus and Christmas gifts.
I strongly recommend those reliant on ZFS send Alan an Email thanking him for his efforts and taking the need for expansion and stability on RELENG_7 seriously. He can be reached at alc@freebsd.org. Oh one other thing: he isn’t the same Alan Cox that works for Red Hat / on the Linux kernel. Two separate people, same name. :-)
Someone should also see about updating the FreeBSD “ZFS Tuning” Wiki to reflect this wonderful change.
EDIT: Ivan’s taken care of editing the Wiki. Thanks Ivan! :-)
