update-node.txt 7.5 KB
Newer Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
How to update the client-side Emulab software on a node and make new images.
This is still a bit ugly at the moment.

A. Things to understand up front:

1. The disk image I am talking about updating here, is one which has both
   a FreeBSD partition and a Linux partition.  So obviously, you will need
   to update both parts before creating a new whole-disk image.  By
   convention FreeBSD is in DOS partition #1 and Linux in DOS partition #2.

2. Since you will need to boot both the BSD and Linux partitions as well
   as the Emulab "admin MFS" (a scaled-down, PXE-loaded, memory filesystem
   based FreeBSD system), it is best to understand how to interact with
   the Emulab pxeboot program as that is the easiest way to get between
   them.  When pxeboot loads, it prompts the console for override input
   before contacting boss for its default behavior:

	Type a key for interactive mode (quick, quick!)

   So hit the space bar (quick, quick!) and you go into interactive mode
   where you can tell it to boot from the FreeBSD disk partition:
	part:1
   the linux partition:
	part:2
   of to boot the admin MFS:
	loader:/tftpboot/freebsd
   (type "help" to get the complete list of commands).  So below, when I
   speak of "rebooting into Linux" or "rebooting into the admin MFS", this
   is how I expect you to do it.

Mike Hibler's avatar
Update.    
Mike Hibler committed
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
3. To do an Emulab client software install, certain other software packages
   must be installed.  If you are updating a recent Emulab image, these
   should all be in place.  But if you are updating a really old image,
   or you are installing the software for the first time, you will need
   these things:

   - Elvin headers and libraries.  Check for the existance of "libvin4.a"
     in your library path (probably in /usr/local/lib).  If it doesn't
     exist, you will need to download:

       http://www.emulab.net/downloads/libelvin-4.0.3.tar.gz

     build, and install it.  Note that this is an older version of elvin,
     so you cannot just install a package or RPM.

   - Boost headers.  Check for the boost directory in the include directory
     path (probably in /usr/local/include or even /usr/include).
     For FreeBSD you can just install the package or port (version >= 1.30).
     For Linux, you may have a harder time.  The RedHat RPMs I have found
     only include the libraries, you need just the headers (everything we
     use is implemented as a template I think).  I think I just copied over
     the installed headers from a BSD box.

   - Perl.  On FreeBSD 5, perl is not installed by default.  Make sure you
     have a version of perl5 installed.

   - Ethtool.  On Linux, with certain NICs, you will need ethtool (instead
     of mii-tool) so that the Emulab software can change link speed/duplex.
     Just install an RPM.

61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91

B. Now we can begin the process:

1. Make sure you have a testbed source and build trees in a filesystem
   that is visible to a testbed node, either in your home directory or
   /proj/emulab-ops.  You will need a build tree for both BSD and Linux
   (see ~hibler/obj for example).

2. Load up a node with the current image, set either to boot BSD or
   Linux, you'll need to boot both eventually.

3. Login to the node, and fill in a little bit of missing source.
   We don't distribute the prototype password files for BSD/Linux,
   so you'll have to copy the "template" versions from the current node:

	# when running linux:
	sudo cp -p /etc/emulab/shadow <testbed-source-tree>/tmcd/linux/

	# when running BSD:
	sudo cp -p /etc/emulab/master.passwd <testbed-source-tree>/tmcd/freebsd/

   The only thing special about these (and the reason we don't distribute ours)
   is that they contain your site's node root password.

4. Go to your build directory and install new client binaries.
   Paranoid guy that I am, I first backup directories that will be affected,
   ala:

	sudo cp -pr /etc /Oetc
	sudo cp -pr /usr/local/etc/emulab /usr/local/etc/Oemulab

Mike Hibler's avatar
Update.    
Mike Hibler committed
92
93
   then do the install.  For FreeBSD 4, FreeBSD 5, RedHat 7 and RedHat 9
   systems you can just do:
94
95

	cd <build-tree-for-this-os>
Mike Hibler's avatar
Update.    
Mike Hibler committed
96
	gmake client
97

Mike Hibler's avatar
Update.    
Mike Hibler committed
98
99
100
101
   and it will build the necessary client-side binaries.  If something
   doesn't build, most likely it is because of a missing software
   package, see A3 above.  After successfully building, install the
   binaries and scripts with:
102

Mike Hibler's avatar
Update.    
Mike Hibler committed
103
	sudo gmake client-install
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162

   If you did the backup, you can then compare the original to the new:

	sudo diff -r /Oetc /etc
	sudo diff -r /usr/local/etc/Oemulab /usr/local/etc/emulab

   The diffs can be significant however, so it may not tell you much of value.

5. Make sure everything works.  Reboot the node once and make sure it comes
   up ok with the new binaries/scripts.

6. Cleanup the filesystem prior to making the image.  Login at the console
   and do a shutdown to go to single-user mode.  In single-user mode do:

	# BSD paranoia: unmount all NFS filesystems, this will already be
	# done for RHL
	umount -h fs

	cd /usr/local/etc/emulab
	sudo ./prepare

7. Now you need to do the same (3-6) for the other OS on the disk.
   So reboot the machine and tell pxeboot (see A2 above) to boot from
   the other partition:

	sync
	reboot

	# wait for pxeboot prompt
	part:N		# N==1 for BSD, 2 for Linux

   When it comes up in the OS, go do steps 3-6 again.

9. All done?  Ok, now you can make the new images.  I don't use the form
   since you need to create three images: one whole disk, one each for
   the individual partitions.  You usually don't need the partition images,
   but we'll make em anyway!

   First step is to get into the admin MFS via pxeboot:

	sync
	reboot

	# wait for pxeboot prompt
	loader:/tftpboot/freebsd

   This will boot into the MFS.  Now you can ssh in as root from boss
   and make the images:

	cd /proj/emulab-ops/images
	imagezip -o /dev/da0 FBSD47+RHL73-STD.ndz
	imagezip -o -s 1 /dev/da0 FBSD47-STD.ndz
	imagezip -o -s 2 /dev/da0 RHL73-STD.ndz

10. Move the new images into place.  The only trick here is to make sure
    frisbeed isn't currently serving up the image.  This is another hack.
    On boss:

	cd /proj/emulab-ops/images
Jay Lepreau's avatar
typo    
Jay Lepreau committed
163
	sudo cp -p FBSD47+RHL73-STD.ndz FBSD47-STD.ndz RHL73-STD.ndz /usr/testbed/images/N/
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192

	cd /usr/testbed/images
	sudo mv FBSD47+RHL73-STD.ndz FBSD47-STD.ndz RHL73-STD.ndz O/
	sudo mv N/* .

    Now the images are in place.  If there is no currently active frisbeed
    serving up that image, you are done.  If there is an existing frisbeed,
    it will still be serving up the old image since it has it open.  You
    need to kill that and start a new one.  There are three processes
    (threads) per frisbeed instance.  If you just kill the first one (it'll
    be the one that has used the most CPU), then you are done.  The parent
    frisbeelauncher will see that it has died and start a new one, which
    will open the new image file.  One catch: if the old frisbeed is actively
    sending out the image (as opposed to sitting around idle waiting for a
    client), you can really screw things.  The new frisbeed will happily
    take up where the old one left off, continuing to feed blocks to any
    active client.  Unfortunately, it will be feeding blocks from a
    completely different image.  There is currently no unique serial number
    in an image that would enable us to detect this scenario.

11. As long as you still have your node allocated, you might as well test
    the whole disk image.  On boss just do:

	os_load pc<XXX>

    and it will reload the node, and bring it back up in what ever OS is
    the default.  Make sure it comes up, and then use your pxeboot prowess
    to boot into the other OS and make sure it works.