linux_mfs

This is the build tree for the Linux-based Frisbee/admin/newnode MFS.

This can probably only be compiled on a Linux machine.

COMPILING
---------
- Install the fakeroot package (fakeroot.alioth.debian.org)
- Edit the file globals.mk and set the variable MFS_ARCH to i386 for
  a 32-bit build or x86_64 for a 64-bit build.  You can also select a
  specific buildroot defconfig -- e.g., '' for the default; 'min' for a
  minimal MFS; 'full' for an MFS packed with just about any tool you
  would ever need.
- Set TESTBED_PATH and TESTBEDOBJ_PATH in source/testbed/Makefile to the
  location of your Emulab source tree and your build tree respectively.
  Alternatively, you can create emulab-devel and emulab-devel-obj symlinks in
  source/testbed that point to these directories.
- Run 'make'.

ADDING/MODIFYING PACKAGES
-------------------------
We use buildroot for nearly all parts of the build.  The only exceptions
to this are the testbed client side sources (source/testbed), and the
building of the final initramfs (we take the buildroot initramfs, unpack
it, overlay atop it the skeleton we generated from the packages built in
the source/*/Makefile makefiles (except source/toolchain, of course,
which kicks off the buildroot build), and overlay atop that the
source/target_template skeleton FS).  We keep these things separate for
convenience; everything else is built by buildroot.

We use the strategy of a buildroot-external tree (see br2-external).
This directory essentially follows the recommended buildroot external
tree structure (see https://buildroot.org/downloads/manual/manual.html#customize-dir-structure).
We place per-arch and per-feature defconfig files in br2-external/configs/*_defconfig.
Buildroot discovers these; the top-level linux_mfs/Makefile selects
which to build based on the values of the MFS_ARCH and DEFCONFIG_POSTFIX
variables specified in linux_mfs/globals.mk (e.g., if you want an x86_64
'full' build, you would set MFS_ARCH=x86_64 and
DEFCONFIG_POSTFIX=full).  This external tree also builds several
packages that are not packaged by the maintainers (jove, tcsh, and shadow).

Each non-buildroot package has a directory under source/.  Inside that directory lives a GNU
Makefile that knows how to configure, build, and install the package.  This
makefile must define the following targets:

- extract: extract the source tarball into the build/ directory
- patch: apply any patches (usually stored in the same directory as the
  Makefile and end with '.patch').
- <packagename>: compile the package (e.g., 'make busybox' to build busybox).
- install: install the package into the target/ subdirectory
- clean: may call 'make clean' within the build tree for the package, or may
  just remove the build tree itself.

Look at the Makefiles for existing package directories under source/ for
examples.  The tricky part is getting things to compile properly using the
uClibc toolchain.  Usually you may need to play with the options you pass to
configure, but you may need to patch configure or build scripts if they assume
that you're not cross-compiling.

It's best to both develop and compile the MFS as a non-root user to avoid
accidentally installing something onto your build machine itself by mistake.
The fakeroot package is used to simulate root permission when creating the
MFS image, so there's no need to become root for that part of the process. 

Once you've created the makefile, edit the top-level Makefile and add the
package name to the MODULES variable.  If you're building a library that needs
to be installed in the cross compiler's library directory (like zlib or
openssl), add the necessary $(MAKE) lines to $(BUILDROOT_PATH)/toolchain-built
target.  For these packages, look at source/zlib/Makefile and source/openssl/Makefile
for examples of how to install the libraries where the toolchain can find them.

Source tarballs should be stored on boss, in
/usr/testbed/www/downloads/linux-mfs-sources. The existing Makefiles under
source/* will run 'scripts/get_source' to download the tarball if it isn't
already in the tarballs/ directory.

UPGRADING
---------

There are several typical ways and reasons to upgrade the MFS:

  * Update to a new major Buildroot release (e.g. 2022.02 -> 2024.02)
  * Update to a new minor Buildroot release (e.g. 2022.02.11 -> 2022.02.12)
  * Update to a new kernel version (e.g. 5.15.96 -> 5.15.109)

Upgrading should be done with the primary goal of keeping the MFS small, by
not enabling new features that are unnecessary.  For each variant, we
customize the buildroot `defconfig`, busybox, uclibc (except on powerpc64),
and of course the Linux kernel.  The kernel is typically 50% or more of the
build, mainly due to the large sets of ethernet and storage controllers that
must be supported.  Of the four config customizations per variant, the only
one that enables new features by default en masse is the Linux kernel.
Typically, you only need to enable a new buildroot config option if you want
to install a new package.  busybox and uclibc configurations only need to be
adjusted if a new buildroot package requires additional options enabled in
those configs.

A typical way to build a new kernel version by starting from an early
build's .config file is to copy your existing config to .config, and run
`make oldconfig`.  This make target will prompt you to confirm a value for
each new kernel option.  Unfortunately, buildroot does not (cannot) help us
here.  First, it does not have a wrapper target for kernel oldconfig
(e.g. linux-oldconfig).  This is reasonable, given that the typical
Buildroot usage is to store only a defconfig kernel config fragment
(meaning, a partial kernel config file containing only non-default options
and those values).  We use this method to store configuration; storing only
the non-default kernel config options makes it much easier to reason about
changes over time.  But if you try to use the non-default-options config
only, as .config for a new kernel build, the `make oldconfig` target can
only assume that the caller has "seen" those config options named in the
non-default-options-only kernel defconfig... and no others.  This is of
course not true; the caller has seen all the defaults from the build that
created the kernel defconfig, plus the modified options; but `make
oldconfig` only knows what the user has seen in the past by parsing a full
kernel config (with default values, included commented options).

The best way around this that we have discovered is to create a build tree
for the new version(s) (e.g., buildroot major/minor, kernel version),
configure it, and manually run a `make oldconfig` in the
`buildroot-*/output/build/linux-[^h]*` dir, after first copying in a full
kernel config from a previous build of this Linux MFS variant.  If you are
upgrading a previous build that was run on Flux Gitlab, you can just grab
the full kernel config from the variant's build artifacts
(metadata/kernel.config-full), and cp that into the build dir as .config,
and then run `make oldconfig` in the build dir.  The trick is to run `make
oldconfig` in the variant's target environment, so that if you are
cross-compiling for a different architecture, that is the calling
environment.

To do this, create a fresh build tree, but change the version numbers you
need (e.g., buildroot major/minor version in `linux_mfs/globals.mk`; kernel
version in the variant linux_mfs/br2-external/configs/*_defconfig` files as
necessary, via the `BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="..."` option) and
run

    make MFS_ARCH=x86_64 DEFCONFIG_POSTFIX=full toolchain-patch

This will get the build directory configured.  Then, get the kernel ready for configuration.  Run

    make MFS_ARCH=x86_64 DEFCONFIG_POSTFIX=full -C buildroot-* linux-configure

This will unfortunately install many of the host dependencies (e.g. gcc)
because configuring the kernel in certain ways requires the ability to build
C programs.  So you'll pause for some time here.  After that, you can run

    cp path/to/kernel.config-full buildroot-*/output/build/linux-5.15.96/.config
    make MFS_ARCH=x86_64 DEFCONFIG_POSTFIX=full -C buildroot-* linux-oldconfig

and answer the interactive prompts.  Then, to update your minimized kernel defconfig, run

    make MFS_ARCH=x86_64 DEFCONFIG_POSTFIX=full -C buildroot-* linux-update-defconfig

and check the diff of your
linux-mfs/br2-external/configs/x86_64_full/x86_64_full_defconfig to see what, if
anything, has changed; and capture (and when ready, commit) those changes.


NOTES
-----
- The default root password is 'root'.  You'll probably want to change it by
  running 'chroot target passwd root' from the top-level directory, set the
  password, then run 'cp target/etc/shadow
  source/target_template/template/etc/shadow'.

BUGS
----
- There are no docs (except this).
- The build system needs some work:
  - There are occasional problems building the toolchain.  These are due
    to weird bugs in the gcc build process that I haven't figured out a
    good fix for.
  - Can't pass '-j' to make; the toolchain won't build properly.  This
    is really a bug with the gcc build process.
  - There are some minor dependency issues in building some of the
    targets that cause other targets to be rebuilt unnecessarily.