customize-image.txt 13.2 KB
Newer Older
1 2
# Copyright (c) 2004, 2005 University of Utah and the Flux Group.
4 5
# All rights reserved.
6 7 8 9 10
Instructions for creating a site-customized image from a "generic" image
provided by Utah.  The basic procedure is:

  - load the image on a testbed machine
  - customize both FreeBSD and Linux filesystems
  - save customized image
12 13 14 15 16 17 18 19 20

We will provide you with a generic disk image.  This disk image is one
which has both a FreeBSD partition and a Linux partition.  You will need
to customize both.  By convention FreeBSD is in DOS partition #1 and Linux
in DOS partition #2.

You will need to have the Emulab network-booted, memory-filesystem-based,
FreeBSD systems (hereafter known as the MFSes).  If you have not done this
yet, do it now.  If you do not know what I am talking about, contact
22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

In particular, the customization of the disk partitions is done using a
node booted into the "admin" MFS.  If you have followed the Emulab setup
directions and have added nodes to your testbed to the point where they
are now in the "hwdown" experiment, they should already be in the admin MFS.
At this point you should be able to pick one and slogin as root from your
boss machine.

If the node does not respond to slogin and you have serial consoles hooked
up, connect to the console, reboot the node and wait for the Emulab pxeboot

	Type a key for interactive mode (quick, quick!)

So hit the space bar (quick, quick!) and you go into interactive mode
where you can tell it to boot from the admin MFS:


41 42 43 44
Another option is to use this command on your boss node:
	node_admin on pc1
(Where pc1 is replaced with the name of the actual testbed pc...)

45 46 47 48 49 50 51
Now on with the show.

A. Load the image on a testbed machine.

We will assume the machine is called "pc1" in the following directions.

* Put the Utah-provided generic image in /usr/testbed/images on your
  boss machine.  The generic image will be loaded, and the custom
  image saved, using this directory.
55 56 57

* Boot pc1 into the admin MFS as described above

58 59 60 61 62 63 64 65 66 67 68
* Figure out what your root disk is.  You can look through the boot
  time output of FreeBSD on pc1 to find it.  If you missed that output,
  you can ssh into pc1 and run "dmesg":

      sudo ssh pc1 dmesg

  If you have IDE disks, the disk you want will probably be "ad0".
  If you have SCSI disks, it will be "da0".  For SATA disks, it will
  probably be "ad4".  RAID controllers are variously things like: "ar",
  "aacd", "twed", depending on the controller you have.  If you cannot
  find anything in the output that looks like a disk, you may have an
  unsupported disk controller.  Contact if this
70 71 72 73 74 75 76 77 78 79 80 81 82 83
  happens (and be sure to have your "dmesg" output handy!)

  If your disk is anything other than "ad0", there are several files
  under both FreeBSD and Linux that will need to be "fixed".  The good
  news is that the post-disk-load customization pass should do this
  for you.  Just make sure that the Emulab database node_types table
  has the correct value of disktype for each node type.  But, you still
  need to know the disk type for the following steps, so lets set a
  shell variable:

      set DSK=<your-disk-here> # e.g. "ad0", "da0", "ad4"
      DSK=<your-disk-here>; export DSK

* Use 'imageunzip' to load the disk.
  If all has gone well, the node should be up and accessible via ssh.
86 87
  To copy the generic image onto the test machine disk, do the following
  from boss:

      sudo ssh pc1 imageunzip - /dev/$DSK \
	< /usr/testbed/images/FBSD410-RHL90-GENERIC.ndz

  Image loading should take anywhere from 45 seconds, to several minutes.

94 95
  If the ssh returns with "Killed" then imageunzip ran out of memory.
  By default, imageunzip will consume memory without bound for buffering
96 97 98
  of pending disk writes.  If imageunzip grows too big, the system will
  kill it.  In this case, retry the imageunzip with "-W <num-MB>" where
  <num-MB> is a number of MB maximum to use for disk buffering.
99 100 101
  Using about half of the available physical memory should be safe
  (e.g., if the machine are loading has 512MB of memory, try "-W 256").

102 103 104 105 106
Now you can begin customizing the FreeBSD and Linux partitions.

B. Customize FreeBSD:

First, login as root from boss and set that magic disk variable:

109 110
      sudo slogin pc1
      set DSK=<your-disk-here>  # you *will* be in csh here

and mount the FreeBSD filesystems on the disk:

114 115 116
      mount /dev/${DSK}s1a /mnt
      mount /dev/${DSK}s1e /mnt/var
      mount /dev/${DSK}s1f /mnt/usr
117 118 119 120 121 122 123

Now you can update the necessary files as follows.
The MFS has a much scaled-down set of binaries.  To get access to a more
full-featured system, you can run binaries from the disk image itself:

     ldconfig /mnt/usr/lib /mnt/usr/X11R6/lib /mnt/usr/local/lib
     set path=($path /mnt/sbin /mnt/bin /mnt/usr/sbin /mnt/usr/bin /mnt/usr/local/bin)
125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147

Now update the following files:

* /mnt/root/.ssh/authorized_keys

  Put in local boss root pub key.  Leave in Utah (Emulab) pub key if
  acceptable (if you want/need our help debugging).  Otherwise, remove it.

* /mnt/etc/localtime

  Copy the correct file over from /mnt/usr/share/zoneinfo

* /mnt/etc/master.passwd

  Change the root password.  The password needs to be changed in the
  etc/emulab subdirectory as well:

      chroot /mnt passwd root
      <set password>
      cp -p /mnt/etc/master.passwd /mnt/etc/emulab/

* /mnt/etc/ssh/ssh_host*

148 149 150
  We use the same host key for all images and all OSes.  If you correctly
  customized your MFSes, you have already generated a set of site-specific
  host keys, and you can copy them to the disk with:
151 152 153

      cp -p /etc/ssh/ssh_host* /mnt/etc/ssh/

154 155 156 157
  and then skip to the next bullet item.

  If you did NOT generate host keys for your MFSes, you can generate
  keys now with:
158 159 160 161 162

      ssh-keygen -t rsa1 -N "" -f /mnt/etc/ssh/ssh_host_key
      ssh-keygen -t rsa -N "" -f /mnt/etc/ssh/ssh_host_rsa_key
      ssh-keygen -t dsa -N "" -f /mnt/etc/ssh/ssh_host_dsa_key

163 164 165 166
  This installs them in the disk image, you will still have to go back and
  install these same keys in the sources for your frisbee/freebsd MFSes later
  using the updating instructions in the README file in the MFS tarball.
  So save the keys from /mnt/etc/ssh off somewhere.
167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191

* /mnt/etc/emulab/{client,emulab}.pem

  These should have been created on your boss node when you did the boss
  setup.  So from your boss node do:

      sudo scp -p /usr/testbed/etc/{client,emulab}.pem pc1:/mnt/etc/emulab/

That is it for FreeBSD.  Now remount the filesystems read-only so you
can still run binaries but don't accidentally clobber anything:

      cd /
      mount -u -o ro /mnt/usr
      mount -u -o ro /mnt/var
      mount -u -o ro /mnt

and move on to updating the Linux partition on the disk.

C. Customize Linux:

Mount the Linux filesystems (recall that the FreeBSD filesystems are
still mounted on /mnt, so we use another directory):

      mkdir /mnt2
      mount -t ext2fs /dev/${DSK}s2 /mnt2
193 194 195 196 197 198 199 200 201 202 203

Now you can update the necessary files as follows.
* /mnt2/root/.ssh/authorized_keys

  Copy over from the FreeBSD partition:

      cp -p /mnt/root/.ssh/authorized_keys /mnt2/root/.ssh/

* /mnt2/etc/shadow

204 205 206 207 208 209 210
  Using your favorite editor, copy in the password hash for root from
  the FreeBSD password file (/mnt/etc/master.passwd) to /mnt2/etc/shadow.
  The password hash is the second colon-separated field in the "root"
  password file line.  Note that these two files are not the same format
  just the password hash field of the file, so do NOT just copy the FreeBSD
  password file to /mnt2/etc/shadow.  Finally copy the newly modified file
  to the emulab subdirectory:
211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236

      cp -p /mnt2/etc/shadow /mnt2/etc/emulab/

* /mnt2/etc/localtime

  Copy the correct file over from /mnt2/usr/share/zoneinfo

* /mnt2/etc/ssh/ssh_host*

  Copy the host keys you created for FreeBSD above:

      cp -p /mnt/etc/ssh/ssh_host* /mnt2/etc/ssh/

* /etc/testbed/{client,emulab}.pem

  Copy over from the FreeBSD side:

      cp -p /mnt/etc/emulab/*.pem /mnt2/etc/emulab/

Now unmount the Linux filesystem:

      umount /mnt2

and fsck it for good luck.  Actually, not only good luck but also to reset
the time stamp that forces a periodic fsck:

      e2fsck /dev/${DSK}s2
238 239

240 241 242
D. Saving the customized image

   The "whole disk" image is the one used most frequently, but we also
243 244 245 246
   create single partition images of FreeBSD and Linux as well.  Put the
   images onto a test area at first, and install them permanently later
   (in step G.)

247 248
   From boss do:

       cd /proj/emulab-ops/images
250 251 252
       sudo ssh pc1 imagezip /dev/$DSK - > FBSD410+RHL90-STD.ndz
       sudo ssh pc1 imagezip -s 1 /dev/$DSK - > FBSD410-STD.ndz
       sudo ssh pc1 imagezip -s 2 /dev/$DSK - > RHL90-STD.ndz
253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314

E. Testing the image

   Now that you have saved a copy of your customization, you can test out
   the image and make sure it works.

   NOTE: you will need console access to the node at this point, either
   the VGA or the serial line depending on which version of pxeboot you
   are using.

   From your slogin session on the node do:


   and wait for the pxeboot prompt to appear on the console as described
   earlier.  When the "Type a key..." message appears, hit space and then
   at the prompt type:


   which tells pxeboot to boot from partition 1 (aka, the FreeBSD partition).
   The machine should proceed to boot all the way to a login prompt.  Watch
   for startup problems, like account setup or mount failures.  If all goes
   well, login to the console with the root password you set and reboot again.
   When you get the pxeboot prompt again, type space and then:


   to boot into the Linux partition.  Again watch for problems.  If
   everything worked, skip the next step and proceed to "Installing the
   images" below.  If there was a catastrophic failure, you can reboot
   the node into the admin MFS and reload the disk either with the image
   snapshot you made or, worst case, the generic image.  If you just
   need to make some minor changes, make them and proceed with the next

F. Recreating the image

   If you need to tweak either the FreeBSD or Linux partitions, you will
   need to save the image again.  Doing this properly involves cleaning up
   anything that the Emulab node self configuration might have done.
   While you are running either FreeBSD or Linux you do the following
   from the console:

       shutdown now
       <wait for single user root prompt>
       cd /usr/local/etc/emulab

   As the node reboots, catch the pxeboot prompt and boot into the admin
   MFS.  Go back to the "Saving the customized image" step (D).

G. Installing the images

   Once you have a working image, go back to your boss and do:

       mv /proj/emulab-ops/images/*.ndz /usr/testbed/images/

315 316 317
   and record these initial images in the Emulab database.

   The image tarball includes sample SQL INSERT commands in the
318 319 320
   setup-images.sql, setup-osids.sql and setup-o2i.sql files to make this
   easier.  Note however, that the database schema occasionally changes,
   so it is important to ensure that your DB matches the INSERT commands in
321 322 323 324 325 326 327
   those files before doing them.  The table formats for the file commands
   are listed at the beginning of the files.  Compare those formats to the
   current schema in your database by doing:

       mysql tbdb
       describe images;
       describe osids;
       describe osidtoimageid;

330 331 332
   setup-o2i.sql is just a template and will have to be modified to use.
   Tweak the other .sql files as necessary for schema changes, and apply
   the commands.
333 334 335 336

Dealing with SCSI disks (or RAID).

337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359
* For completeness here are the files that need to be changed if you have
  other than IDE disks:

      FreeBSD /etc/fstab: needs "ad0" changed to appropriate type
      Linux /etc/fstab: needs "hda" changed to appropriate type
      Linux /etc/lilo.conf: ditto, plus lilo must be rerun

  Again, you should not need to mess with these files, the post-disk-loader
  script will do it for you.

* For FreeBSD, we have four different kernels that are used for various
  purposes.  Only one of them is generic.  So you will need to build
  customized kernels in order to do "link delays", delay nodes and virtual
  nodes.  The image does include our versions of all these kernels, so
  try those first and maybe save yourself some work.  These kernels support:

      disks: ad, ar, da (with ahc, ahd controllers)
      network: xl, dc, fxp, em, nge

  Copy /kernel.100HZ over to /kernel on a node and try booting it.  If
  that works and all your network interfaces were found, just create a
  new image after copying /kernel.100HZ to /kernel (copy, don't move,
  /kernel.100HZ needs to exist).

361 362 363
  Otherwise, you need to build your own kernels.  You can look at output from
  the admin MFS (aka GENERIC) kernel if necessary.  Add the necessary driver
  to the various TESTBED configs in the kernel source, and rebuild and install
  the kernels in the image.  Mail for more info.

366 367 368 369
* For Linux, it is even more of a PITA.  We currently don't even have a
  generic Linux kernel in the image.  So if Linux doesn't boot in the image,
  you will have to configure/build a new kernel on some external machine.
  If you don't have a Linux machine to do this with, contact us.

[ This file is a copy of doc/customize-image.txt in the Emulab source tree. ]
[ Last updated 05/10/05 ]