use DKMS driver wrappers in standard images
Given that 1) we do kernel updates in our standard images 4 times a year, 2) I have a ton of OpenStack prebuilt images to update, and 3) each kernel update requires reinstalls of newer out-of-tree versions of certain drivers (due to the need to support newer hardware) -- I created a repository that packages those drivers as DKMS packages. A DKMS package is a deb or rpm package that contains the driver source and some build metadata. When a new kernel is installed, the DKMS build hooks will automatically rebuild the DKMS module packages against that kernel, and rebuild the initramfs if the driver should be in it.
The upside is obvious: user-installed kernels (whether distro kernel packages or custom installs) will get the latest out-of-tree drivers without any thought on the user's part. The downside: if an installed DKMS module source package is too old to compile against a newer kernel version, and there is no in-tree version already in that newer kernel version, then the build will fail. This is unlikely to happen as we upgrade kernels during security updates, since those are minor/patchlevel updates that don't break APIs. However, a user might install a much newer kernel that does break internal API compat. Consider the Emulab IPOD module (which uses internal netfilter APIs): this breaks every time the netfilter interface changes, and that seems to happen every 10-20 minor revisions. And of course, installing a new kernel now requires slightly more time, since all the DKMS modules must be built against the new version, and a new initramfs generated. That said, I think the upside outweighs the down, so currently, I think we should probably go ahead with this to ease our maintenance burden a bit.
For now, I am only going to install all this stuff on the OpenStack images, and we'll consider installing it in the standard Ubuntu and CentOS images in the next round of security updates.