|
|
Linux MFS Installation
|
|
|
======================
|
|
|
|
|
|
(For an overview of the design of the Linux MFS and Emulab grub2, you
|
|
|
should read section 2 of
|
|
|
https://gitlab.flux.utah.edu/emulab/emulab-devel/wikis/Gory-Details/Low-level-Boot-Process).
|
|
|
|
|
|
The current release of the MFS is based on
|
|
|
buildroot-2018.05-linux-4.14
|
|
|
(https://gitlab.flux.utah.edu/emulab/linux-mfs/tree/buildroot-2018.05-linux-4.14).
|
|
|
we maintain a fork of the grub2 repository containing our
|
|
|
modifications (https://gitlab.flux.utah.edu/emulab/emulab-grub2).
|
|
|
|
|
|
We provide MFS builds for x86_64, i386, aarch64, and ppc64le
|
|
|
architectures. However, we only provide grub2 binaries for x86,
|
|
|
because our netboot configuration for the aarch64 and ppc64le machines
|
|
|
we've supported in the past has relied on low-level firmware-based
|
|
|
bootloaders that themselves read (via PXE/TFTP) syslinux or grub
|
|
|
config files that then point to our MFSes.
|
|
|
|
|
|
The MFS is PXE-booted, and so consists of several pieces:
|
|
|
|
|
|
* Our `grub2pxe` binaries. These contact the Emulab `bootinfo` server
|
|
|
and are told what to boot: an MFS, or a partition on disk.
|
|
|
|
|
|
* Several different low-level grub2 configuration files that support
|
|
|
different kinds of physical hardware configurations (e.g.,
|
|
|
console=ttyS0|ttyS1|tty0, grub BIOS/native disk drivers, etc).
|
|
|
These files contain grub2 module init, console setup, and magic to
|
|
|
talk to the Emulab bootinfo server.
|
|
|
|
|
|
* Higher-level grub2 configuration files that describe the binaries
|
|
|
for each variant of the Linux MFS: the `frisbee` MFS (our
|
|
|
disk-loading environment, minimal size, only ); the admin MFS
|
|
|
(larger, contains more administrative tools); and the recovery MFS
|
|
|
(largest, rarely booted, contains even more tools). When the
|
|
|
bootinfo server tells the initial grub2pxe environment to boot a
|
|
|
specific variant of the Linux MFS, it gives grub2pxe a tftp-boot
|
|
|
path to the higher-level grub2 config file that specifies the
|
|
|
binaries to boot for that variant. These high-level files inherit
|
|
|
special environment variables from the low-level grub config
|
|
|
environment, such as console information.
|
|
|
|
|
|
* The MFS binaries themselves, per-architecture, and per-variant.
|
|
|
Some architectures (aarch64, ppc64le) only provide
|
|
|
one-size-fits-all MFS binaries, and contain all frisbee, admin,
|
|
|
and recovery tools.
|
|
|
|
|
|
Depending on your architecture, you will need all or some of these pieces.
|
|
|
|
|
|
Installation
|
|
|
------------
|
|
|
|
|
|
(Note: all commands below should be run on your `boss` node as a non-root uid that can write to `/tftpboot`. You may need to use `sudo` for anything that touches `/tftpboot` if your uid cannot write `/tftpboot`.)
|
|
|
|
|
|
1. Download and extract the `grub2pxe` binaries and low-level configuration. Run these commands on your `boss` node:
|
|
|
|
|
|
```
|
|
|
cd /tmp
|
|
|
wget https://www.emulab.net/downloads/linux-mfs/emulab-grub2-tftpboot-x86-master.tar.gz
|
|
|
tar -xzvf emulab-grub2-tftpboot-x86-master.tar.gz -C /tftpboot/ --strip-components 2
|
|
|
```
|
|
|
|
|
|
This will add several `grub2` dirs to `/tftpboot` on boss:
|
|
|
|
|
|
grub2.0
|
|
|
grub2pxe-bios-sio1
|
|
|
grub2pxe-bios-sio2
|
|
|
grub2pxe-bios-vga
|
|
|
grub2pxe-efi-sio1
|
|
|
grub2pxe-efi-sio2
|
|
|
grub2pxe-efi-vga
|
|
|
grub2pxe-native-sio1
|
|
|
grub2pxe-native-sio2
|
|
|
grub2pxe-native-vga
|
|
|
|
|
|
2. Download the high-level, per-variant grub2 config files; run these commands on your `boss` node.
|
|
|
|
|
|
```
|
|
|
cd /tmp
|
|
|
wget https://www.emulab.net/downloads/linux-mfs/emulab-grub2-tftpboot-mfs-configs.tar.gz
|
|
|
tar -xzvf emulab-grub2-tftpboot-mfs-configs.tar.gz -C /tftpboot/ --strip-components 2
|
|
|
```
|
|
|
|
|
|
This will add several dirs to `/tftpboot` on boss:
|
|
|
|
|
|
admin_linux
|
|
|
frisbee_linux
|
|
|
newnode_linux
|
|
|
recovery_linux
|
|
|
|
|
|
(Note that the "high-level" grub.cfg files in each of the above dirs
|
|
|
point to the kernel binaries in the admin_linux_kernel,
|
|
|
frisbee_linux_kernel, newnode_linux_kernel, and recovery_linux_kernel
|
|
|
dirs. We will create those as symlinks in the next step.)
|
|
|
|
|
|
3. Download the actual MFS kernel binaries.
|
|
|
|
|
|
In this dir, you will see a listing of the MFS binary tarballs, among
|
|
|
others. The MFS tarballs are named like
|
|
|
`buildroot-2018.05-linux-4.14-<arch>[-<variant>]`;
|
|
|
`buildroot-2018.05-linux-4.14` is the current release. As previously
|
|
|
noted, some architectures have only a single variant.)
|
|
|
|
|
|
You most likely want to use the x86_64 version as your admin, frisbee,
|
|
|
and newnode MFSes; and the x86_64-full variant as your recovery MFS.
|
|
|
Here are the instructions to do that; run these on your `boss` node.
|
|
|
(Note that unlike the grub2 tarballs, the MFS binaries unpack to a
|
|
|
single dir with the same basename as the tarball -- no `tftpboot/`
|
|
|
top-level dir.)
|
|
|
|
|
|
```
|
|
|
cd /tmp
|
|
|
wget https://www.emulab.net/downloads/linux-mfs/buildroot-2018.05-linux-4.14-x86_64.tar.gz
|
|
|
tar -xzvf buildroot-2018.05-linux-4.14-x86_64.tar.gz -C /tftpboot/
|
|
|
wget https://www.emulab.net/downloads/linux-mfs/buildroot-2018.05-linux-4.14-x86_64-full.tar.gz
|
|
|
tar -xzvf buildroot-2018.05-linux-4.14-x86_64-full.tar.gz -C /tftpboot/
|
|
|
```
|
|
|
|
|
|
(Note that each of the kernel binary dirs has a `metadata` subdir with
|
|
|
file listings, sizes, and packages; this may be useful to help
|
|
|
evaluate which variant of the MFS you want.)
|
|
|
|
|
|
4. Localize any MFSes you downloaded; run these commands on your
|
|
|
`boss` node. (Localization copies boss's root pubkey, various
|
|
|
certificates, and sshd host keys into the initramfs files, so that
|
|
|
you can interact with nodes running these MFSes as expected from
|
|
|
your `boss` node.)
|
|
|
|
|
|
```
|
|
|
cd /tftpboot/buildroot-2018.05-linux-4.14-x86_64
|
|
|
./localize_initramfs
|
|
|
cd /tftpboot/buildroot-2018.05-linux-4.14-x86_64-full
|
|
|
./localize_initramfs
|
|
|
```
|
|
|
|
|
|
5. Add the *_linux_kernel symlinks the high-level grub2 configs require; run these commands on your `boss` node.
|
|
|
|
|
|
(If you did this in an earlier install, you might only need to update
|
|
|
the symlinks if using a new version of the MFS binaries; otherwise, no
|
|
|
action is required. Similarly, if you've customized your grub2
|
|
|
configs, you'll want to skip this.)
|
|
|
|
|
|
```
|
|
|
cd /tftpboot
|
|
|
ln -s buildroot-2018.05-linux-4.14-x86_64 admin_linux_kernel
|
|
|
ln -s buildroot-2018.05-linux-4.14-x86_64 frisbee_linux_kernel
|
|
|
ln -s buildroot-2018.05-linux-4.14-x86_64 newnode_linux_kernel
|
|
|
ln -s buildroot-2018.05-linux-4.14-x86_64-full recovery_linux_kernel
|
|
|
```
|
|
|
|
|
|
Configuring
|
|
|
-----------
|
|
|
|
|
|
Now that you have your MFS binaries and grub2 config files all ready
|
|
|
to go, you need to tell your Emulab installation to use them for each
|
|
|
of your node types (at least for those that you want to run the Linux
|
|
|
MFS).
|
|
|
|
|
|
You will need four OS descriptors: `ADMIN-LINUX`, `FRISBEE-LINUX`,
|
|
|
`NEWNODE-LINUX`, and `RECOVERY-LINUX`. You can install these
|
|
|
descriptors via these commands:
|
|
|
|
|
|
```
|
|
|
XXXX
|
|
|
```
|
|
|
|
|
|
(Each descriptor points to the grub config files in
|
|
|
`/tftpboot/admin_linux`, and respectively to the others you just
|
|
|
downloaded above.)
|
|
|
|
|
|
Once you've installed the OS descriptors, you'll need to add
|
|
|
node_type_attributes for each node type. You can do this by browsing to
|
|
|
https://<boss-fqdn>/portal/show-nodetype.php?type=<node-type>
|
|
|
and clicking "Edit", if you've installed the new Portal interface; or
|
|
|
by using the Classic interface https://www.test1.powderwireless.net/editnodetype.php3?node_type=nuc8650
|
|
|
if not.
|
|
|
|
|
|
You need to change the `adminmfs_osid` to `ADMIN-LINUX`;
|
|
|
`diskloadmfs_osid` to `FRISBEE-LINUX`; and `recoverymfs_osid` to
|
|
|
`RECOVERY-LINUX`.
|
|
|
|
|
|
You also need to add the `pxe_boot_path` attribute; and set it to one
|
|
|
of the `grub2pxe` binaries in the binary/low-level grub dirs you
|
|
|
downloaded and extracted in Step 1. This attribute tells the Emulab
|
|
|
DHCP server to send this binary to PXE booting nodes of this type. At
|
|
|
this point, you may need to fiddle around a bit to see which grub2pxe
|
|
|
binary works on your node type. If possible, you should first use one
|
|
|
of the `/tftpboot/grub2pxe-bios-*/grub2pxe` binaries, since those use
|
|
|
the BIOS disk drivers and have the highest probability of success. If
|
|
|
those don't work, you can fall back to one of the
|
|
|
`/tftpboot/grub2pxe-native-*/grub2pxe` binaries, or one of the
|
|
|
`/tftpboot/grub2pxe-efi-*/grub2pxe` binaries if booting via UEFI.
|
|
|
|
|
|
You also need to choose the appropriate `grub2pxe` binary/config for
|
|
|
your node type's console configuration. If you have a serial console
|
|
|
on ttyS0/sio1/com1, you would choose `/tftpboot/grub2pxe-bios-sio1`,
|
|
|
or `/tftpboot/grub2pxe-bios-sio2` if using the second console line.
|
|
|
If you don't have serial console lines for this node type, you would
|
|
|
want to use `/tftpboot/grub2pxe-bios-vga`.
|
|
|
|
|
|
Most people will want to use `/tftpboot/grub2pxe-bios-sio1/grub2pxe`.
|
|
|
|
|
|
Once you've set this attribute, you need to rebuild your `boss` node's
|
|
|
DHCP config file and restart dhcpd:
|
|
|
|
|
|
```
|
|
|
sudo /usr/testbed/sbin/dhcpd_makeconf -ir
|
|
|
```
|
|
|
|
|
|
At this point, you should be able to use your newly-installed Linux
|
|
|
MFSes.
|
|
|
|
|
|
Bleeding-edge builds
|
|
|
--------------------
|
|
|
|
|
|
Both the Linux MFS and the Emulab version of grub2 that it requires
|
|
|
are built automatically by our Gitlab server's CI jobs. You can
|
|
|
browse the list of the latest MFS builds at https://gitlab.flux.utah.edu/emulab/linux-mfs/pipelines |