customize-image.txt 11 KB
Newer Older
1
2
3
4
5
Instructions for creating a site-customized image from a "generic" image
provided by Utah.  The basic procedure is:

  - load the image on a testbed machine
  - customize both FreeBSD and Linux filesystems
6
  - save customized image
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

We will provide you with a generic disk image.  This disk image is one
which has both a FreeBSD partition and a Linux partition.  You will need
to customize both.  By convention FreeBSD is in DOS partition #1 and Linux
in DOS partition #2.

You will need to have the Emulab network-booted, memory-filesystem-based,
FreeBSD systems (hereafter known as the MFSes).  If you have not done this
yet, do it now.  If you do not know what I am talking about, contact
testbed-ops@emulab.net.

In particular, the customization of the disk partitions is done using a
node booted into the "admin" MFS.  If you have followed the Emulab setup
directions and have added nodes to your testbed to the point where they
are now in the "hwdown" experiment, they should already be in the admin MFS.
At this point you should be able to pick one and slogin as root from your
boss machine.

If the node does not respond to slogin and you have serial consoles hooked
up, connect to the console, reboot the node and wait for the Emulab pxeboot
prompt:

	Type a key for interactive mode (quick, quick!)

So hit the space bar (quick, quick!) and you go into interactive mode
where you can tell it to boot from the admin MFS:

	loader:/tftpboot/freebsd

Now on with the show.


A. Load the image on a testbed machine.

We will assume the machine is called "pc1" in the following directions.

43
* Put the Utah-provided generic image in /usr/testbed/images on your
Mike Hibler's avatar
Mike Hibler committed
44
  boss machine.  The generic image will be loaded, and the custom
45
  image saved, using this directory.
46
47
48

* Boot pc1 into the admin MFS as described above

49
50
51
52
* Use 'imageunzip' to load the disk.
  If all went well, the node should be up and accessible via ssh.
  To copy the generic image onto the test machine disk, do the following
  from boss:
53

54
      sudo ssh pc1 imageunzip - /dev/ad0 < /usr/testbed/images/FBSD47+RHL73-GENERIC.ndz
55

56
57
  (or use "/dev/da0" for SCSI).  Image loading should take anywhere from
  45 seconds, to several minutes.
58

59
60
61
62
63
64
65
  If the ssh returns with "Killed" then imageunzip ran out of memory.
  By default, imageunzip will consume memory without bound for buffering
  of pending disk writes.  In this case, retry the imageunzip with
  "-W <num-MB>" where <num-MB> is a number of MB maximum to use for
  disk buffering.  If the node you are loading has 512MB of memory, try
  a number like 300.

66
67
68
69
70
Now you can begin customizing the FreeBSD and Linux partitions.


B. Customize FreeBSD:

71
72
73
74
75
First, login as root from boss:

       sudo slogin pc1

and mount the FreeBSD filesystems on the node:
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199

      mount /dev/ad0s1a /mnt
      mount /dev/ad0s1e /mnt/var
      mount /dev/ad0s1f /mnt/usr

(replacing "ad0" with "da0" if you have a SCSI root disk).
Now you can update the necessary files as follows.
	
The MFS has a much scaled-down set of binaries.  To get access to a more
full-featured system, you can run binaries from the disk image itself:

     ldconfig /mnt/usr/lib /mnt/usr/X11R6/lib /mnt/usr/local/lib
     set path=($path /mnt/sbin /mnt/bin /mnt/usr/sbin /mnt/usr/bin)

Now update the following files:

* /mnt/root/.ssh/authorized_keys

  Put in local boss root pub key.  Leave in Utah (Emulab) pub key if
  acceptable (if you want/need our help debugging).  Otherwise, remove it.

* /mnt/etc/localtime

  Copy the correct file over from /mnt/usr/share/zoneinfo

* /mnt/etc/master.passwd

  Change the root password.  The password needs to be changed in the
  etc/emulab subdirectory as well:

      chroot /mnt passwd root
      <set password>
      cp -p /mnt/etc/master.passwd /mnt/etc/emulab/

* /mnt/etc/ssh/ssh_host*

  We use the same host key for all images and all OSes.  If you have
  already generate a set of site-specific host keys, copy them to the
  disk with:

      cp -p /etc/ssh/ssh_host* /mnt/etc/ssh/

  and skip to the next bullet item.  If you have NOT generated appropriate
  host keys, you can generate them with:

      ssh-keygen -t rsa1 -N "" -f /mnt/etc/ssh/ssh_host_key
      ssh-keygen -t rsa -N "" -f /mnt/etc/ssh/ssh_host_rsa_key
      ssh-keygen -t dsa -N "" -f /mnt/etc/ssh/ssh_host_dsa_key

  Note that you will have to go back and install these keys in the sources
  for your frisbee/freebsd MFSes later using the updating instructions in
  the README file in the MFS tarball.

* /mnt/etc/emulab/{client,emulab}.pem

  These should have been created on your boss node when you did the boss
  setup.  So from your boss node do:

      sudo scp -p /usr/testbed/etc/{client,emulab}.pem pc1:/mnt/etc/emulab/

That is it for FreeBSD.  Now remount the filesystems read-only so you
can still run binaries but don't accidentally clobber anything:

      cd /
      mount -u -o ro /mnt/usr
      mount -u -o ro /mnt/var
      mount -u -o ro /mnt

and move on to updating the Linux partition on the disk.


C. Customize Linux:

Mount the Linux filesystems (recall that the FreeBSD filesystems are
still mounted on /mnt, so we use another directory):

      mkdir /mnt2
      mount -t ext2fs /dev/ad0s2 /mnt2

(replacing "ad0" with "da0" if you have a SCSI root disk).
Now you can update the necessary files as follows.
	
* /mnt2/root/.ssh/authorized_keys

  Copy over from the FreeBSD partition:

      cp -p /mnt/root/.ssh/authorized_keys /mnt2/root/.ssh/

* /mnt2/etc/shadow

  Copy in the password hash for root from the FreeBSD password file
  (/mnt/etc/master.passwd) to /mnt2/etc/shadow.  Then copy that file to
   the emulab subdirectory:

      cp -p /mnt2/etc/shadow /mnt2/etc/emulab/

* /mnt2/etc/localtime

  Copy the correct file over from /mnt2/usr/share/zoneinfo

* /mnt2/etc/ssh/ssh_host*

  Copy the host keys you created for FreeBSD above:

      cp -p /mnt/etc/ssh/ssh_host* /mnt2/etc/ssh/

* /etc/testbed/{client,emulab}.pem

  Copy over from the FreeBSD side:

      cp -p /mnt/etc/emulab/*.pem /mnt2/etc/emulab/

Now unmount the Linux filesystem:

      umount /mnt2

and fsck it for good luck.  Actually, not only good luck but also to reset
the time stamp that forces a periodic fsck:

      e2fsck /dev/ad0s2

again, replace "ad" with "da" for SCSI.


200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
D. Saving the customized image

   The "whole disk" image is the one used most frequently, but we also
   create single partition images of FreeBSD and Linux as well.
   From boss do:

       cd /usr/testbed/images
       sudo ssh pc1 imagezip /dev/ad0 - > FBSD47+RHL73-STD.ndz
       sudo ssh pc1 imagezip -s 1 /dev/ad0 - > FBSD47-STD.ndz
       sudo ssh pc1 imagezip -s 2 /dev/ad0 - > RHL73-STD.ndz

   replacing "ad0" with "da0" for SCSI of course.


E. Testing the image

   Now that you have saved a copy of your customization, you can test out
   the image and make sure it works.

   NOTE: you will need console access to the node at this point, either
   the VGA or the serial line depending on which version of pxeboot you
   are using.

   From your slogin session on the node do:

       reboot

   and wait for the pxeboot prompt to appear on the console as described
   earlier.  When the "Type a key..." message appears, hit space and then
   at the prompt type:

       part:1

   which tells pxeboot to boot from partition 1 (aka, the FreeBSD partition).
   The machine should proceed to boot all the way to a login prompt.  Watch
   for startup problems, like account setup or mount failures.  If all goes
   well, login to the console with the root password you set and reboot again.
   When you get the pxeboot prompt again, type space and then:

       part:2

   to boot into the Linux partition.  Again watch for problems.  If
   everything worked, skip the next step and proceed to "Installing the
   images" below.  If there was a catastrophic failure, you can reboot
   the node into the admin MFS and reload the disk either with the image
   snapshot you made or, worst case, the generic image.  If you just
   need to make some minor changes, make them and proceed with the next
   step.


F. Recreating the image

   If you need to tweak either the FreeBSD or Linux partitions, you will
   need to save the image again.  Doing this properly involves cleaning up
   anything that the Emulab node self configuration might have done.
   While you are running either FreeBSD or Linux you do the following
   from the console:

       shutdown now
       <wait for single user root prompt>
       cd /usr/local/etc/emulab
       ./prepare
       reboot

   As the node reboots, catch the pxeboot prompt and boot into the admin
   MFS.  Go back to the "Saving the customized image" step (D).


G. Installing the images

   Once you have a working image, go back to your boss and do:

       mv /proj/emulab-ops/images/*.ndz /usr/testbed/images/

   and record these initial images in the Emulab database using the
   SQL INSERT commands in setup-images.sql and setup-osids.sql.
   As the database schema occasionally changes, it is important to
   ensure that your DB matches the INSERT commands before doing them.
   The table formats are listed at the beginning of those files.



[ XXX THIS ENDS THE INSTRUCTIONS.  THE REST OF THIS FILE IS SUPPLEMENTAL
  MATERIAL AND NEEDS SOME WORK. ]

Dealing with SCSI disks (or RAID).

287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321

* Since we have IDE disks, it takes a little more work to create an
  image on IDE disks for a site with SCSI disks.

* For FreeBSD, you need to make sure that the kernel includes support
  for the host adaptor in question.  You can look at output from the
  "newnode" (aka GENERIC) kernel if necessary.  If the HA is not already
  supported in our various TESTBED configs, add it and rebuild and install
  the kernels in the image.  Then, in the image, you need to change /etc/fstab
  to refer to "da" disks rather than "ad" disks.

* For Linux, it is even more of a PITA.  As with BSD, you need to configure
  our kernel with generic SCSI support and support for the HA.  Note that
  by default, Linux includes SCSI as a module, but to load a module into the
  Linux kernel at boot time, you need to have a little MFS (initrd) that
  contains the appropriate modules.  We just build a kernel with the SCSI
  stuff hardwired instead.  Anyway, once you have a kernel, modify /etc/fstab
  to talk about "sda" rather than "hda".  Now you also have to modify
  /etc/lilo.conf, again, to reference "sda" rather than "hda".  But, once
  you have done that, you cannot rerun lilo as it will complain about not
  having an sda disk; i.e., lilo looks at the config file to figure out
  what disk to write the new stuff to, and that file now says "sda" instead
  of the "hda" that you are constructing the image on!  Go ahead and create
  the image without re-running lilo.  When you boot Linux from the image for
  the first time, interrupt lilo (type anything once lilo starts loading the
  image) and then type:

	 linux root=/dev/sda2

  (assuming Linux is in partition 2 on the disk).  This will boot it up
  using SCSI as the root disk.  Now, assuming your kernel has the appropriate
  SCSI support, the machine will come up ok.  Then login as root and run
  "lilo" to fix things up for good (remember to create a new image afterward).