FreeBSD ZFS v13 MFC — force unmount is experimental — finale

EDIT: And just like I predicted, my blog is gradually beginning to see more and more hits from search engine queries for “force unmount is experimental” or “zfs force unmount”.

I’ve talked to Kip about this. Below is the Email response I got this morning:

On Sun, May 24, 2009 at 1:26 AM, Jeremy Chadwick wrote:
> On Sun, May 24, 2009 at 12:58:19AM -0700, Kip Macy wrote:
>> On Sun, May 24, 2009 at 12:57 AM, Kip Macy wrote:
>> > Doing a forced unmount means that we try to force data out to disk
>> > before shutting down. Most of consider this desirable.
>>
>> Most of us consider this desirable.
>
> A few questions:
>
> 1) Isn't this already being done during the "Syncing disks, vnodes
> remaining..." stage of kernel shutdown?

You are watching a force unmount in progress when you see that.

> 2) Doesn't "zfs unmount -a" or "zfs volfini" (particularly the latter)
> do this for us?

Evidently not. The force unmount flag was passed.

> 3) If the forced unmount is intended/by design, is the ZFS_LOG() really
> necessary?  Is this something specific to the FreeBSD adaptation of ZFS?
> I haven't ever seen this message on our hundreds of Solaris 10 machines
> using ZFS at work, which is why it caught my eye.

Prior to my recent changes there was a similar message - force unmount
was disallowed entirely. Problem with that though is that people with
ZFS root lose data sometimes.

This is FreeBSD specific.

-Kip

So it seems the quirk (not the message itself, but the commit to permit force unmount) is FreeBSD-centric. The call to ZFS_LOG(), IMHO, is superfluous and should be removed. Otherwise I expect to see more and more hits on my blog for this issue once 7.3-RELEASE is made available…

This Email does not give me “warm fuzzies” in regards to ZFS being used on a root filesystem, which indirectly means booting ZFS directly is also impacted. Given that ZFS is incredibly resilient about data integrity, this quirk does not sit well with me, especially given that OpenSolaris doesn’t behave this way.

For now, I’d urge people to be very careful about ZFS for a root filesystem on FreeBSD. I know some folks have used it reliably for quite some time, but Kip knows more about the inner-workings than I do.

FreeBSD ZFS v13 MFC — force unmount is experimental

As mentioned in previous blog post, ZFS version 13 has been MFC’d to RELENG_7. Things looked great even after upgrading zpool and zfs metadata… until I shut down the machine (e.g. reboot, shutdown -p now, etc.). A message in similar format is being spit out on VGA/serial console, once per ZFS pool/filesystem listed:

Force unmount is experimental - report any problems.

The code responsible for the message is here:

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c

Further investigative efforts turn up this, which indicate this is may be “by design”…

http://svn.freebsd.org/viewvc/base?view=revision&revision=192211

…but I’d like to know why FreeBSD is setting MS_FORCE in fflag when calling zfs_umount(), especially when the machine is being shut down.

Someone did report this problem back in November 2008 (when the code was slightly different), but no one responded. However please note that the message in question is output once per ZFS pool/filesystem listed, and has nothing to do with ZFS as a root filesystem or not:

http://www.mailinglistarchive.com/freebsd-current@freebsd.org/msg23424.html

I’ve mailed Kip about this, so we’ll see what transpires…

EDIT: And here’s the explanation.

Shuttle SG45H7 — Firewire bug in BIOS SG45U10O

Below is a copy of an Email I sent Shuttle yesterday, indicating that I had found a bug in their latest BIOS for the Shuttle SG45H7. The bug is described below.

From: Jeremy Chadwick
To: support@tw.shuttle.com
Subject: SG45H7 -- bug in BIOS version SG45U10O

Support,

There is a bug introduced in BIOS version SG45U10O (date 03/11/2009)
for the Shuttle SG45H7.  The bug is the following:

If the 1394 controller is Disabled in the BIOS, upon the next reboot,
the BIOS crashes/locks up (BIOS startup screen is never seen, POST never
happens).  Clearing the CMOS is the only way to get the machine usable
again.

Please see about fixing this.

If you need any other information from me (motherboard version, etc.)
just ask.

Thanks.

I received two responses from Shuttle within 24 hours. The first confirms the bug (nice QA!):

From: support S
To: Jeremy Chadwick
Subject: RE: SG45H7 -- bug in BIOS version SG45U10O

Dear Jeremy
Thank you choosing Shuttle.
Regarding your concern about SG45SH7 problem.
We found out the issue like what you described.
We are going to modify and release it as soon as we can.
Please feel free to let me know if you have any questions or concerns

Shuttle Inc.
Technical Support

And the second providing a beta BIOS — which I am not willing to try given that there are obviously other changes within (note the subrevision letter has gone from O to Q. What was P? Sorry, I don’t play roulette with my BIOSes):

From: support S
To: Jeremy Chadwick
Subject: RE: SG45H7 -- bug in BIOS version SG45U10O

Dear Jeremy
Thank you choosing Shuttle.
Attached is the BETA BIOS that can solve your problem, please try it.
Please feel free to let me know if you have any questions or concerns

Shuttle Inc.
Technical Support

Regardless, it’s good to see a company taking bug reports seriously. Now I’m left wondering if I should tell them about a bug in their ACPI DSDT which FreeBSD whines about upon boot-up. Hmm…

ZFS version 13 committed to RELENG_7

Looks like Kip Macy just MFC’d ZFS version 13 to RELENG_7. The src/UPDATING message is:

20090520
        Update ZFS to version 13. ZFS users will need to re-build
        kernel and world. Existing pools will continue to work
        without upgrade. If a pool is upgraded it will no longer be
        usable by older kernel revs. ZFS send / recv between
        pool version 6 and pool version 13 is not supported.

I’m not going to list off all the commits, but below is the commit message. I’ve bolded items which are of interest to me, and probably the rest of the world. ZFS boot support (including from raidz1 and raidz2 volumes) is probably the most notable of the bunch.

SVN rev 192498 on 2009-05-20 23:34:59Z by kmacy

MFC ZFS version 13. This includes the changes by pjd (see original message
below) as well as the following:

- the recurring deadlock was fixed by deferring vinactive to a dedicated thread

- zfs boot for all pool types now works
      Submitted by: dfr

- kmem now goes up to 512GB so arc is now limited by physmem

- the arc now experiences backpressure from the vm (which can be too
much - but this allows ZFS to work without any tunables on amd64)

- frequently recurring LOR in the ARC fixed

- zfs send coredump fix

- fixes for various PRs

Supported by: Barrett Lyon, BitGravity

Revision 185029 - (view) (annotate) - [select for diffs]
Modified Mon Nov 17 20:49:29 2008 UTC (6 months ago) by pjd
File length: 38244 byte(s)
Diff to previous 177698

Update ZFS from version 6 to 13 and bring some FreeBSD-specific changes.

This bring huge amount of changes, I'll enumerate only user-visible changes:

- Delegated Administration

       Allows regular users to perform ZFS operations, like file system
       creation, snapshot creation, etc.

- L2ARC

       Level 2 cache for ZFS - allows to use additional disks for cache.
       Huge performance improvements mostly for random read of mostly
       static content.

- slog

       Allow to use additional disks for ZFS Intent Log to speed up
       operations like fsync(2).

- vfs.zfs.super_owner

       Allows regular users to perform privileged operations on files stored
       on ZFS file systems owned by him. Very careful with this one.

- chflags(2)

       Not all the flags are supported. This still needs work.

- ZFSBoot

       Support to boot off of ZFS pool. Not finished, AFAIK.

       Submitted by:   dfr

- Snapshot properties

- New failure modes

       Before if write requested failed, system paniced. Now one
       can select from one of three failure modes:

       Before if write requested failed, system paniced. Now one
       can select from one of three failure modes:
       - panic - panic on write error
       - wait - wait for disk to reappear
       - continue - serve read requests if possible, block write requests

- Refquota, refreservation properties

       Just quota and reservation properties, but don't count space consumed
       by children file systems, clones and snapshots.

 - Sparse volumes

       ZVOLs that don't reserve space in the pool.

 - External attributes

       Compatible with extattr(2).

 - NFSv4-ACLs

       Not sure about the status, might not be complete yet.

       Submitted by:   trasz

 - Creation-time properties

 - Regression tests for zpool(8) command.

 Obtained from:        OpenSolaris

Existing users of ZFS on RELENG_7 will need to upgrade both world and kernel. Once in single-user mode, use zpool upgrade [pool] to upgrade the pool version, and zfs upgrade [fs] to upgrade the filesystem version. You can run either command without a pool or fs specified to see what version is currently in use on the disk(s). NOTE: These are not reversible, so consider yourself warned.

Big thanks to kmacy and others who were involved in making this MFC happen.