FreeBSD: "older" fsck version is incompatible with "newer" versions of UFS
While we were setting up an elabinelab for testing the new firewall, we used prebuilt full-disk images of 12.2-based boss and ops nodes and put those down on a couple of d430s. Upon booting, we got all kinds of cylinder group checksum errors from both kernels. After many (many) bad theories we discovered that the FreeBSD 10 version of fsck in the MFS will fix up bad summary information, which is actually metadata in a FreeBSD 12 filesystem.
I only know for sure that this is a problem between FreeBSD 10 and 12, I haven't tracked this down to see when the incompatibility really happened. Hence the vague "older" and "newer" in the title.
This needs to be fixed in anything that deals with FreeBSD filesystems. Possibly this is as simple as switching to a FreeBSD 12 version of fsck--assuming that version does not screw up filesystems for older versions of FreeBSD. This will affect the FreeBSD and Linux based MFSes as well as the blockstore code in FreeBSD.
We were here once before, with FreeBSD 4 and 5 I think. Didn't really have a satisfactory fix in that case. I think. That was many generations of Mike ago though.