Commit 7151a940 authored by Mike Hibler's avatar Mike Hibler

First (actually second) cut at documentation for updating Emulab software

on node images.
parent 20acceb4
How to update the client-side Emulab software on a node and make new images.
This is still a bit ugly at the moment.
A. Things to understand up front:
1. The disk image I am talking about updating here, is one which has both
a FreeBSD partition and a Linux partition. So obviously, you will need
to update both parts before creating a new whole-disk image. By
convention FreeBSD is in DOS partition #1 and Linux in DOS partition #2.
2. Since you will need to boot both the BSD and Linux partitions as well
as the Emulab "admin MFS" (a scaled-down, PXE-loaded, memory filesystem
based FreeBSD system), it is best to understand how to interact with
the Emulab pxeboot program as that is the easiest way to get between
them. When pxeboot loads, it prompts the console for override input
before contacting boss for its default behavior:
Type a key for interactive mode (quick, quick!)
So hit the space bar (quick, quick!) and you go into interactive mode
where you can tell it to boot from the FreeBSD disk partition:
the linux partition:
of to boot the admin MFS:
(type "help" to get the complete list of commands). So below, when I
speak of "rebooting into Linux" or "rebooting into the admin MFS", this
is how I expect you to do it.
B. Now we can begin the process:
1. Make sure you have a testbed source and build trees in a filesystem
that is visible to a testbed node, either in your home directory or
/proj/emulab-ops. You will need a build tree for both BSD and Linux
(see ~hibler/obj for example).
2. Load up a node with the current image, set either to boot BSD or
Linux, you'll need to boot both eventually.
3. Login to the node, and fill in a little bit of missing source.
We don't distribute the prototype password files for BSD/Linux,
so you'll have to copy the "template" versions from the current node:
# when running linux:
sudo cp -p /etc/emulab/shadow <testbed-source-tree>/tmcd/linux/
# when running BSD:
sudo cp -p /etc/emulab/master.passwd <testbed-source-tree>/tmcd/freebsd/
The only thing special about these (and the reason we don't distribute ours)
is that they contain your site's node root password.
4. Go to your build directory and install new client binaries.
Paranoid guy that I am, I first backup directories that will be affected,
sudo cp -pr /etc /Oetc
sudo cp -pr /usr/local/etc/emulab /usr/local/etc/Oemulab
then do the install. For FreeBSD 4.x and Redhat 7.x systems you can
just do:
cd <build-tree-for-this-os>
sudo gmake client-install
and it will build and install all client-side binaries and scripts.
Due to various compiler and library related issues that we have not
yet resolved, the event agents in the event subdirectory, don't build
cleanly on newer FreeBSDs or Redhats. On these systems you can install
the most frequently changed stuff (mostly scripts) by doing:
cd <build-tree-for-this-os>/tmcd
cd {linux,freebsd} # whichever OS you are running
sudo gmake install
If you need to install the other binaries, contact us.
If you did the backup, you can then compare the original to the new:
sudo diff -r /Oetc /etc
sudo diff -r /usr/local/etc/Oemulab /usr/local/etc/emulab
The diffs can be significant however, so it may not tell you much of value.
5. Make sure everything works. Reboot the node once and make sure it comes
up ok with the new binaries/scripts.
6. Cleanup the filesystem prior to making the image. Login at the console
and do a shutdown to go to single-user mode. In single-user mode do:
# BSD paranoia: unmount all NFS filesystems, this will already be
# done for RHL
umount -h fs
cd /usr/local/etc/emulab
sudo ./prepare
7. Now you need to do the same (3-6) for the other OS on the disk.
So reboot the machine and tell pxeboot (see A2 above) to boot from
the other partition:
# wait for pxeboot prompt
part:N # N==1 for BSD, 2 for Linux
When it comes up in the OS, go do steps 3-6 again.
9. All done? Ok, now you can make the new images. I don't use the form
since you need to create three images: one whole disk, one each for
the individual partitions. You usually don't need the partition images,
but we'll make em anyway!
First step is to get into the admin MFS via pxeboot:
# wait for pxeboot prompt
This will boot into the MFS. Now you can ssh in as root from boss
and make the images:
cd /proj/emulab-ops/images
imagezip -o /dev/da0 FBSD47+RHL73-STD.ndz
imagezip -o -s 1 /dev/da0 FBSD47-STD.ndz
imagezip -o -s 2 /dev/da0 RHL73-STD.ndz
10. Move the new images into place. The only trick here is to make sure
frisbeed isn't currently serving up the image. This is another hack.
On boss:
cd /proj/emulab-ops/images
sudo cp -p FBSD47+RHL73-STD.ndz FBSD47-STD.ndz RHL73-STD.ndz /usr/testbed/imaegs/N/
cd /usr/testbed/images
sudo mv FBSD47+RHL73-STD.ndz FBSD47-STD.ndz RHL73-STD.ndz O/
sudo mv N/* .
Now the images are in place. If there is no currently active frisbeed
serving up that image, you are done. If there is an existing frisbeed,
it will still be serving up the old image since it has it open. You
need to kill that and start a new one. There are three processes
(threads) per frisbeed instance. If you just kill the first one (it'll
be the one that has used the most CPU), then you are done. The parent
frisbeelauncher will see that it has died and start a new one, which
will open the new image file. One catch: if the old frisbeed is actively
sending out the image (as opposed to sitting around idle waiting for a
client), you can really screw things. The new frisbeed will happily
take up where the old one left off, continuing to feed blocks to any
active client. Unfortunately, it will be feeding blocks from a
completely different image. There is currently no unique serial number
in an image that would enable us to detect this scenario.
11. As long as you still have your node allocated, you might as well test
the whole disk image. On boss just do:
os_load pc<XXX>
and it will reload the node, and bring it back up in what ever OS is
the default. Make sure it comes up, and then use your pxeboot prowess
to boot into the other OS and make sure it works.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment