customize-image.txt 16.5 KB
Newer Older
1
2
#
# EMULAB-COPYRIGHT
Mike Hibler's avatar
update    
Mike Hibler committed
3
# Copyright (c) 2004-2008 University of Utah and the Flux Group.
4
5
# All rights reserved.
#
Mike Hibler's avatar
Mike Hibler committed
6
7
8
9
10

I. Introduction

These are instructions for creating a site-customized image from a "generic"
image provided by Utah.  The basic procedure is:
11
12
13

  - load the image on a testbed machine
  - customize both FreeBSD and Linux filesystems
14
  - save customized image
15

Mike Hibler's avatar
update    
Mike Hibler committed
16
We will provide you with a generic disk image tarball.  This tarball will
Mike Hibler's avatar
Mike Hibler committed
17
include a multi-OS "full disk" combo image and one or more auxiliary "single
Mike Hibler's avatar
update    
Mike Hibler committed
18
19
20
21
partition" images.  The combo image is one which has both a FreeBSD partition
and a Linux partition.  You will need to customize both.  By convention
FreeBSD is in DOS partition #1 and Linux in DOS partition #2.

Mike Hibler's avatar
Mike Hibler committed
22
23
24
25
26

II. Which generic image do I need?

It depends largely on the hardware in your testbed and what advanced Emulab
features you might need.  The choices are:
Mike Hibler's avatar
update    
Mike Hibler committed
27

Mike Hibler's avatar
Mike Hibler committed
28
29
30
31
32
33
34
  * FBSD62+FC6-GENERIC.ndz
    Includes FreeBSD 6.2 and Fedora 6.
    Download http://www.emulab.net/downloads/generic-image.tar for this.

  * FBSD410+RHL90-GENERIC.ndz
    Includes FreeBSD 4.10 and RedHat 9.
    Download http://www.emulab.net/downloads/generic-image-old.tar for this.
Mike Hibler's avatar
update    
Mike Hibler committed
35
36
37
38
39

Generally, you will want to use the former as your default image that disks
are loaded with.  The reason the second image is included at all is that it
must be used for "virtual node" support, which right now is only available
in our modified FreeBSD 4.10 kernel.  Unfortunately, the 4.10 kernel, being
Mike Hibler's avatar
Mike Hibler committed
40
41
42
43
44
45
46
47
48
49
more than 5 years old, doesn't run on all modern hardware, due to lack of
drivers and/or buggy ACPI support.  So you may be stuck without vnodes for
awhile if your hardware is in the unsupported category.  There is also a
lesser dependency on FreeBSD 4.10 for the control net firewall support.

So, if you want/need vnodes or firewalls but your node hardware won't boot
FreeBSD 4.10, contact us and we'll figure something out.


III. Prerequisites
50
51
52

You will need to have the Emulab network-booted, memory-filesystem-based,
FreeBSD systems (hereafter known as the MFSes).  If you have not done this
Mike Hibler's avatar
Mike Hibler committed
53
54
55
56
57
58
59
60
61
62
63
64
yet, do it now.  For this you will need to download the MFS tarball:

	http://www.emulab.net/downloads/tftpboot-lastest.tar.gz

extract the README, and follow the directions there.

You will also need to have at least one test node integrated into your
testbed.  If you haven't done that yet, go back to the setup documentation
under "Adding Nodes" and add one!


IV. The Process
65

Mike Hibler's avatar
Mike Hibler committed
66
67
68
69
70
The customization of the disk partitions is done using a testbed node booted
into the "admin" MFS.  If you have followed the Emulab setup directions and
have added nodes to your testbed to the point where they are now in the
"hwdown" experiment, they should already be in the admin MFS.  At this point
you should be able to pick one and slogin as root from your boss machine.
71
72
73
74
75
76
77
78
79
80
81
82

If the node does not respond to slogin and you have serial consoles hooked
up, connect to the console, reboot the node and wait for the Emulab pxeboot
prompt:

	Type a key for interactive mode (quick, quick!)

So hit the space bar (quick, quick!) and you go into interactive mode
where you can tell it to boot from the admin MFS:

	loader:/tftpboot/freebsd

83
84
85
86
Another option is to use this command on your boss node:
	node_admin on pc1
(Where pc1 is replaced with the name of the actual testbed pc...)

87
88
89
90
91
92
93
Now on with the show.


A. Load the image on a testbed machine.

We will assume the machine is called "pc1" in the following directions.

94
* Put the Utah-provided generic image in /usr/testbed/images on your
95
  boss machine.  The generic image will be loaded, and the custom
96
  image saved, using this directory.
97
98
99

* Boot pc1 into the admin MFS as described above

100
101
102
103
104
105
106
107
108
109
110
* Figure out what your root disk is.  You can look through the boot
  time output of FreeBSD on pc1 to find it.  If you missed that output,
  you can ssh into pc1 and run "dmesg":

      sudo ssh pc1 dmesg

  If you have IDE disks, the disk you want will probably be "ad0".
  If you have SCSI disks, it will be "da0".  For SATA disks, it will
  probably be "ad4".  RAID controllers are variously things like: "ar",
  "aacd", "twed", depending on the controller you have.  If you cannot
  find anything in the output that looks like a disk, you may have an
111
  unsupported disk controller.  Contact testbed-ops@flux.utah.edu if this
112
113
114
115
116
117
118
119
120
121
122
  happens (and be sure to have your "dmesg" output handy!)

  If your disk is anything other than "ad0", there are several files
  under both FreeBSD and Linux that will need to be "fixed".  The good
  news is that the post-disk-load customization pass should do this
  for you.  Just make sure that the Emulab database node_types table
  has the correct value of disktype for each node type.  But, you still
  need to know the disk type for the following steps, so lets set a
  shell variable:

      set DSK=<your-disk-here> # e.g. "ad0", "da0", "ad4"
Mike Hibler's avatar
update    
Mike Hibler committed
123
124
125

  for csh, or:

126
127
      DSK=<your-disk-here>; export DSK

Mike Hibler's avatar
update    
Mike Hibler committed
128
129
  for sh or bash.

130
* Use 'imageunzip' to load the disk.
131
  If all has gone well, the node should be up and accessible via ssh.
Mike Hibler's avatar
update    
Mike Hibler committed
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
  To copy the generic image onto the test machine disk, first set some
  variables based on what image you are customizing:

  FBSD62+FC6 (csh):
     set FULL=FBSD62+FC6
     set BSD=FBSD62
     set LNX=FC6

  FBSD62+FC6 (sh):
     FULL=FBSD62+FC6; export FULL
     BSD=FBSD62; export BSD
     LNX=FC6; export LNX

  FBSD410+RHL90 (csh):
     set FULL=FBSD410+RHL90
     set BSD=FBSD410
     set LNX=RHL90

  FBSD410+RHL90 (sh):
     FULL=FBSD410+RHL90; export FULL
     BSD=FBSD410; export BSD
     LNX=RHL90; export LNX

  Now do the following on boss:
156

157
      sudo ssh pc1 imageunzip - /dev/$DSK \
Mike Hibler's avatar
update    
Mike Hibler committed
158
	< /usr/testbed/images/$FULL-GENERIC.ndz
159

160
  Image loading should take anywhere from 45 seconds, to several minutes.
161

162
163
  If the ssh returns with "Killed" then imageunzip ran out of memory.
  By default, imageunzip will consume memory without bound for buffering
164
165
166
  of pending disk writes.  If imageunzip grows too big, the system will
  kill it.  In this case, retry the imageunzip with "-W <num-MB>" where
  <num-MB> is a number of MB maximum to use for disk buffering.
167
168
169
  Using about half of the available physical memory should be safe
  (e.g., if the machine are loading has 512MB of memory, try "-W 256").

170
171
172
173
174
Now you can begin customizing the FreeBSD and Linux partitions.


B. Customize FreeBSD:

175
First, login as root from boss and set that magic disk variable:
176

177
178
      sudo slogin pc1
      set DSK=<your-disk-here>  # you *will* be in csh here
179

180
and mount the FreeBSD filesystems on the disk:
181

182
183
184
      mount /dev/${DSK}s1a /mnt
      mount /dev/${DSK}s1e /mnt/var
      mount /dev/${DSK}s1f /mnt/usr
185
186
187
188
189
190
191

Now you can update the necessary files as follows.
	
The MFS has a much scaled-down set of binaries.  To get access to a more
full-featured system, you can run binaries from the disk image itself:

     ldconfig /mnt/usr/lib /mnt/usr/X11R6/lib /mnt/usr/local/lib
192
     set path=($path /mnt/sbin /mnt/bin /mnt/usr/sbin /mnt/usr/bin /mnt/usr/local/bin)
193
194
195
196
197

Now update the following files:

* /mnt/etc/localtime

198
199
200
  Copy the correct file over from /mnt/usr/share/zoneinfo.  Note that these
  zoneinfo files have been updated to reflect the new (as of 2007) DST rules
  in the US.
201
202
203
204
205
206
207
208
209
210
211
212

* /mnt/etc/master.passwd

  Change the root password.  The password needs to be changed in the
  etc/emulab subdirectory as well:

      chroot /mnt passwd root
      <set password>
      cp -p /mnt/etc/master.passwd /mnt/etc/emulab/

* /mnt/etc/ssh/ssh_host*

213
214
215
  We use the same host key for all images and all OSes.  If you correctly
  customized your MFSes, you have already generated a set of site-specific
  host keys, and you can copy them to the disk with:
216
217
218

      cp -p /etc/ssh/ssh_host* /mnt/etc/ssh/

219
220
221
222
  and then skip to the next bullet item.

  If you did NOT generate host keys for your MFSes, you can generate
  keys now with:
223
224
225
226
227

      ssh-keygen -t rsa1 -N "" -f /mnt/etc/ssh/ssh_host_key
      ssh-keygen -t rsa -N "" -f /mnt/etc/ssh/ssh_host_rsa_key
      ssh-keygen -t dsa -N "" -f /mnt/etc/ssh/ssh_host_dsa_key

228
229
230
231
  This installs them in the disk image, you will still have to go back and
  install these same keys in the sources for your frisbee/freebsd MFSes later
  using the updating instructions in the README file in the MFS tarball.
  So save the keys from /mnt/etc/ssh off somewhere.
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256

* /mnt/etc/emulab/{client,emulab}.pem

  These should have been created on your boss node when you did the boss
  setup.  So from your boss node do:

      sudo scp -p /usr/testbed/etc/{client,emulab}.pem pc1:/mnt/etc/emulab/

That is it for FreeBSD.  Now remount the filesystems read-only so you
can still run binaries but don't accidentally clobber anything:

      cd /
      mount -u -o ro /mnt/usr
      mount -u -o ro /mnt/var
      mount -u -o ro /mnt

and move on to updating the Linux partition on the disk.


C. Customize Linux:

Mount the Linux filesystems (recall that the FreeBSD filesystems are
still mounted on /mnt, so we use another directory):

      mkdir /mnt2
257
      mount -t ext2fs /dev/${DSK}s2 /mnt2
258
259
260
261
262

Now you can update the necessary files as follows.
	
* /mnt2/etc/shadow

263
264
265
266
267
268
269
  Using your favorite editor, copy in the password hash for root from
  the FreeBSD password file (/mnt/etc/master.passwd) to /mnt2/etc/shadow.
  The password hash is the second colon-separated field in the "root"
  password file line.  Note that these two files are not the same format
  just the password hash field of the file, so do NOT just copy the FreeBSD
  password file to /mnt2/etc/shadow.  Finally copy the newly modified file
  to the emulab subdirectory:
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295

      cp -p /mnt2/etc/shadow /mnt2/etc/emulab/

* /mnt2/etc/localtime

  Copy the correct file over from /mnt2/usr/share/zoneinfo

* /mnt2/etc/ssh/ssh_host*

  Copy the host keys you created for FreeBSD above:

      cp -p /mnt/etc/ssh/ssh_host* /mnt2/etc/ssh/

* /etc/testbed/{client,emulab}.pem

  Copy over from the FreeBSD side:

      cp -p /mnt/etc/emulab/*.pem /mnt2/etc/emulab/

Now unmount the Linux filesystem:

      umount /mnt2

and fsck it for good luck.  Actually, not only good luck but also to reset
the time stamp that forces a periodic fsck:

Mike Hibler's avatar
Mike Hibler committed
296
      e2fsck -f -y /dev/${DSK}s2
297
298


299
300
301
D. Saving the customized image

   The "whole disk" image is the one used most frequently, but we also
302
303
304
305
   create single partition images of FreeBSD and Linux as well.  Put the
   images onto a test area at first, and install them permanently later
   (in step G.)

306
307
   From boss do:

308
       cd /proj/emulab-ops/images
Mike Hibler's avatar
update    
Mike Hibler committed
309
310
311
       sudo ssh pc1 imagezip /dev/$DSK - > $FULL-STD.ndz
       sudo ssh pc1 imagezip -s 1 /dev/$DSK - > $BSD-STD.ndz
       sudo ssh pc1 imagezip -s 2 /dev/$DSK - > $LNX-STD.ndz
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334


E. Testing the image

   Now that you have saved a copy of your customization, you can test out
   the image and make sure it works.

   NOTE: you will need console access to the node at this point, either
   the VGA or the serial line depending on which version of pxeboot you
   are using.

   From your slogin session on the node do:

       reboot

   and wait for the pxeboot prompt to appear on the console as described
   earlier.  When the "Type a key..." message appears, hit space and then
   at the prompt type:

       part:1

   which tells pxeboot to boot from partition 1 (aka, the FreeBSD partition).
   The machine should proceed to boot all the way to a login prompt.  Watch
335
336
337
338
339
340
341
342
343
344
345
346
   for startup problems, like account setup or mount failures.  If your
   kernel panics reporting unknown/missing bus errors, you may need to
   enable ACPI.  You can either do this directly in /boot/loader.conf by
   setting acpi_load="YES", or by invoking our slicefix script while in
   the admin-MFS:
   
       set SLICEFIX_ACPI=YES
       /usr/local/etc/emulab/slicefix -0 <disk>  # e.g. ad4, etc.
       
   If all goes well, login to the console with the root password you set
   and reboot again.  When you get the pxeboot prompt again, type space
   and then:
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382

       part:2

   to boot into the Linux partition.  Again watch for problems.  If
   everything worked, skip the next step and proceed to "Installing the
   images" below.  If there was a catastrophic failure, you can reboot
   the node into the admin MFS and reload the disk either with the image
   snapshot you made or, worst case, the generic image.  If you just
   need to make some minor changes, make them and proceed with the next
   step.


F. Recreating the image

   If you need to tweak either the FreeBSD or Linux partitions, you will
   need to save the image again.  Doing this properly involves cleaning up
   anything that the Emulab node self configuration might have done.
   While you are running either FreeBSD or Linux you do the following
   from the console:

       shutdown now
       <wait for single user root prompt>
       cd /usr/local/etc/emulab
       ./prepare
       reboot

   As the node reboots, catch the pxeboot prompt and boot into the admin
   MFS.  Go back to the "Saving the customized image" step (D).


G. Installing the images

   Once you have a working image, go back to your boss and do:

       mv /proj/emulab-ops/images/*.ndz /usr/testbed/images/

383
384
385
   The boss installation process should have created image (and OS)
   descriptors for these standard images.  You can verify via the web
   page by clicking on "List ImageIDs" on the left menu and ensuring
Mike Hibler's avatar
update    
Mike Hibler committed
386
387
   that FBSD-STD, RHL-STD, FBSD62-STD, FBSD410-STD, FC6-STD, and RHL90-STD
   are all shown.
388

Mike Hibler's avatar
update    
Mike Hibler committed
389
390
391
392
393
394
395
396
397
398
399
400
401
402
H. Additional images

   If you have chosen FBSD62+FC6-GENERIC as your default image, you may
   also want the FBSD410 image for firewall and vnode support (this is
   assuming that FreeBSD 4.10 will run on your hardware).  To do this,
   you need to customize one or both of:

       FBSD410-IPFW2-GENERIC.ndz
       FBSD410-GENERIC.ndz

   (Note that we do not use the combo image containing FBSD410 here, we use
   a "single partition" image with just BSD, which is smaller).  Now, before
   you free up the node you have been using to customize the standard image,
   is the time to customize these.
403

404
405
   To do this, make sure you node is in the admin MFS and then do (from
   your boss node):
406

407
408
      sudo ssh pc1 imageunzip -s 1 - /dev/$DSK \
	< /usr/testbed/images/FBSD410-IPFW2-GENERIC.ndz
409

410
411
412
413
414
   Note the "-s 1" argument to imageunzip.  This tells it that the image
   should be loaded in the first DOS partition rather than at the beginning
   of the disk.  This is why you should not attempt to customize the firewall
   image before you have done the standard image; the latter contains a DOS
   partition table necessary for this step.
415

416
417
418
419
420
421
422
423
424
425
426
427
428
   Now proceed to customize as in B. (Customize FreeBSD) above.  After
   customization (don't forget the final step of remounting the FSes readonly!)
   save the image from boss via:

       cd /proj/emulab/ops/images
       sudo ssh pc1 imagezip -s 1 /dev/$DSK - > FBSD410-IPFW2.ndz

   Finally, move it into place on boss:

       mv /proj/emulab-ops/images/*.ndz /usr/testbed/images/

   Again, an image descriptor (FW-IPFW2) should already exist in the database.
     
Mike Hibler's avatar
update    
Mike Hibler committed
429
430
431
432
433
434
435
436
437
438
439
440
   Repeat for the vnode kernel:

      sudo ssh pc1 imageunzip -s 1 - /dev/$DSK \
	< /usr/testbed/images/FBSD410-GENERIC.ndz

       cd /proj/emulab/ops/images
       sudo ssh pc1 imagezip -s 1 /dev/$DSK - > FBSD410-STD.ndz

       mv /proj/emulab-ops/images/*.ndz /usr/testbed/images/


I. Dealing with SCSI disks (or RAID) and unrecognized hardware.
441

442
443
444
445
* For completeness here are the files that need to be changed if you have
  other than IDE disks:

      FreeBSD /etc/fstab: needs "ad0" changed to appropriate type
Mike Hibler's avatar
update    
Mike Hibler committed
446
447
          ("ad4" for SATA, "da0" for SCSI, "aac", "twe" or various
            others for RAID)
448
      Linux /etc/fstab: needs "hda" changed to appropriate type
Mike Hibler's avatar
update    
Mike Hibler committed
449
          ("sda" for SATA or SCSI, ?? for RAID)
Mike Hibler's avatar
Mike Hibler committed
450
      Linux /etc/lilo.conf: ditto, plus /sbin/lilo must be rerun
451

452
  However, you should not need to mess with these files, the post-disk-loader
453
454
  script will do it for you.

455
456
457
* For FreeBSD, we have five different FreeBSD 4.10 kernels that are used
  for various purposes.  All of them are based on the GENERIC configuration.
  If the default kernel ("/kernel" aka "/kernel.100HZ") does not boot on your
Mike Hibler's avatar
Mike Hibler committed
458
  machine or your disk or Ethernet interfaces are not recognized, you will
Mike Hibler's avatar
update    
Mike Hibler committed
459
  have to move to the FreeBSD 6.2 based solution above.
460

461
462
463
* For Linux, it is even more of a PITA.  We currently don't even have a
  generic Linux kernel in the image.  So if Linux doesn't boot in the image,
  you will have to configure/build a new kernel on some external machine.
Mike Hibler's avatar
update    
Mike Hibler committed
464
465
466
  Since the default Linux image is either Fedora 6 with a 2.6 kernel or
  RedHat 9 with a 2.4 kernel, it is also possible that it just doesn't
  support newer hardware.  Contact us if this happens.
467

468
[ This file is a copy of doc/customize-image.txt in the Emulab source tree. ]
Mike Hibler's avatar
update    
Mike Hibler committed
469
[ Last updated 06/24/08 ]