setup.txt 15.8 KB
Newer Older
1
#####
2
##### Setting up the Utah Network Testbed software on a boss node
3
4
##### Tested on FreeBSD 4.3, FreeBSD 4.5, and FreeBSD 4.6
#####
5

6
7
8
Note: steps labeled "Local Only" are only required when setting up a testbed
with local nodes - they can be skipped in a widearea-only testbed.

9
10
11
12
13
14
15
##### Step 0

First of all, the machine should be configured correctly for the network it is
on, have the root password set, etc.

##### Step 1 - Package installation

16
17
Install the necessary packages. The following are necessary, and are
the default versions in the FreeBSD 4.3 "ports" collection:
18
19
20
21
22
23
24
25
26
27
28
29
(Ports marked with an 'L' are only needed if you have local nodes.)

  apache+mod_ssl-1.3.19+2.8.2 The Apache 1.3 webserver with SSL/TLS functionalit
  cvsupd-bin-16.1     A general network file distribution system optimized for C
  fping-2.2b1         Quickly ping N hosts w/o flooding the network
  gd-1.8.4_1          A graphics library for fast PNG creation
  gmake-3.79.1        GNU version of 'make' utility
  graphviz-1.7c       Graph Visualization Software from AT&T and Bell Labs
L isc-dhcp-2.0.5      ISC Dynamic Host Configuration Protocol client and server
  linuxthreads-2.1.3_2 POSIX pthreads implementation using rfork to generate ker
  mhash-0.8.9         Library provides an easy way to access strong hashes such
  mod_auth_mysql-2.20 Allows users to use MySQL databases for user authenticatio
Leigh B. Stoller's avatar
Leigh B. Stoller committed
30
  mod_php4-4.1.1      PHP4 module for Apache
31
32
  mysql-client-3.23.36 Multithreaded SQL database (client)
  mysql-server-3.23.36 Multithreaded SQL database (server)
33
  netpbm-9.12	      A toolkit for conversion of images between different formats
34
35
36
37
38
39
40
41
42
43
44
45
  otcl-1.0a6          MIT Object Tcl
  p5-DBI-1.15         The perl5 Database Interface.  Required for DBD::* modules
  p5-GD-1.32_2        A perl5 interface to Gd Graphics Library
  p5-Mysql-modules-1.2215 Perl5 modules for accessing MySQL databases
L p5-SNMP-4.2.0       A perl5 module for interfacing with the CMU SNMP library
L p5-SNMP_Session-0.83 A perl5 module for providing rudimentary access to SNMPv1
  p5-BSD-Resource	    BSD resource goo
  rpm-3.0.6_5         The Red Hat Package Manager
  swig-1.1p5_5        Simplified Wrapper and Interface Generator
  tcl-8.2.3           Tool Command Language
  tcl-sql-20000114_1  TCL module for accessing MySQL databases
L ucd-snmp-4.2        An extendable SNMP implimentation
46
  vcg-1.30	      A Visualization Tool for compiler graphs
47

Leigh B. Stoller's avatar
Leigh B. Stoller committed
48
49
50
IMPORTANT: The mhash package should be installed _before_ the mod_php4 port is
built. When building mod_php4, you will be given a menu to choose the various
features you want support for. Make sure to select support for mhash.
51

52
Note on TCL: Do NOT install tcl83; otcl, which is used by some testbed
53
54
55
scripts, requires tcl82. When you install the tcl-sql package, it will be put
in the library directory for the latest version of tcl you have installed, so
if you have tcl83 installed at the time, you will have tcl-sql support under
56
57
58
59
60
8.3.X, but not under 8.2.X (which testbed scripts use) Newer versions of
the FreeBSD port have a TCL_VERSION variable you can use to get around
this problem - Stick the line:
TCL_VERSION=tcl8.2 in /etc/make.conf
before building, and you should be fine.
61

62
63
64
For the most part, subsitituting newer versions of the packages should be OK,
but know that we haven't tested them... The one exception is version 3.23.47 
of the mysql packages. In fact, we recommend 3.23.47, as it fixes some
65
boot-time problems that we've had with 3.23.36.
66

67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
Finally, you will need to download and install elvin, which is the core of our
event system. You can download it from: http://elvin.dstc.edu.au/ . You'll
need to grab both the C 'libelvin' distribution and elvind . So far, we have
only tried releases in the 4.0 branch. Compile these two (libelvin first)
with the standard untar/configure/make procedure. When configuring libelvin,
be sure to give the '--enable-threads' option.

Many of these packages need to be started by RC scripts. First, the
mysql-client rc script needs to run before ANY testbed services are started!
You can ensure this by changing directories to /usr/local/etc/rc.d and renaming
'mysql-client.sh' to '1.mysql-client.sh'. Also, we use some different options
when staring mysqld, so remove the 'mysql-server.sh'. (We'll replace it in a
second.) Furthermore, dhcpd needs to start before proxydhcp, so rename
'dhcpd.sh' to '2.dhcpd.sh'. You will also need to install the '3.testbed.sh',
'cvsupd.sh', '2.elvind.sh', and '2.mysql-server.sh' scripts, which are in the
rc.d directory of the testbed tree - you can install them by doing a 'gmake
install' in that directory. Finally, remove the 'snmpd.sh' script, because
we don't need to be running it, and it may have security vulernabilities.

Notes for newer versions of FreeBSD: Newer versions of the mysql port install
the starup script as '00mysql-client.sh' - you're fine either leaving it with
that name, or renaming it as suggested above. Also, newer versions of the
dhcpd port install their startup script as 'isc-dhcpd.sh.sample' - just
90
rename this as suggested above.
91

92
93
##### Step 2 - LEDA

94
95
96
We rely on a software package call LEDA which we are not allowed to
distribute (source *or* binary), and is currently not being
distributed by the authors. As a result, we will distribute a binary
97
98
99
100
101
for the one program (assign) that requires linking with the LEDA
library. In the meantime we will be working on removing our dependence
on the LEDA library.

The program in question is assign (see the assign subdir). Please
102
103
104
105
contact us if you need to replace it. The default operation of
configure is to assume the existence of assign/assign.bin in the
source tree. However, you can override that by using the configure 
option --with-assignbinary=/path/to/assign/binary.
106
107
108

##### Step 3 - Testbed tree configuration/installation

109
110
It is best to add the following group before creating the following
directories. Basically, testbed software should be in the tbadmin
111
112
group. Run this command as root. (NOTE: The numeric gid used for the
tbadmin group must be greater than 100)
113

114
	pw groupadd tbadmin -g 101
115
	
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
Configure the testbed tree. Many of the configuration parameters are stored in
a 'defs' file. Make a copy of the defs-default file (found in the root of
the testbed tree,) and customize it for your site. The key variables
are:
WWWDEFS - The name of the defs file to use for the web pages. Instructions
	for this file are below
TBADMINGROUP - The name of the group you created above, probaby tbadmin
TBOPSEMAIL - An email address for the 'operations staff'. Copies of
	error messages, etc. will get sent to this address, and in some
	cases users are directed to this address if they have questions and/or
	problems.
TBLOGSEMAIL - An email address to which some logs (experiment
	creation/deletion, etc.) will get sent.
DELAYCAPACITY - Maximum delay capacity of any nodes. Set to 1 prevent the
	same node from acting as multiple delay nodes.
BOSSNODE - Fully-qualified hostname of the boss node
USERNODE - Fully-qualified hostname of the users node (the node to which
	users will log on to control their nodes, etc.)
FSNODE - Fully-qualified hostname of the NFS server. (In our setup, this is
	the same machine as the users node.)
OURDOMAIN - The domain in which this testbed lies
FSDIR_GROUPS - The _real_ pathname (no symbolic links, etc.) of the groups
	directory on the FSNODE. See the setup-ops.txt file for the
	instructions for creating this directory.
FSDIR_PROJ - Ditto, for the proj directory
FSDIR_USERS - Ditto, for the users (home) directory

Other variables can generally be left alone. Pass the name of this file to
configure with the --with-TBDEFS option.

You'll also need to make a defs file for the web system. This is found in
the www/ directory of the testbed source tree. Copy default-defs.php3 to
<name>-defs.php3, and put <name> into WWWDEFS in the main defs file. The
149
main variables to worry about are:
150
151
152
153
154
155
156
157
$WWWHOST, $TBAUTHDOMAIN - replace emulab.net with your domain
$THISHOMEBASE, $THISPROJECT - Use the name of your site
$TBMAIL* - These work like the mail addresses of the same names in the
	defs file above. The new addresses are the approval address, to which
	requests to start projects are sent, and the audit address, which is
	CCed on certain mail sent to users.
$TBMAINSITE - Set this to 0

158
159
160
Create the /usr/testbed directory. Make sure that this directory is writable to
the tbadmin group.

161
162
163
Now, build and install the software. For example, I have the testbed source in
~/testbed, and use the ~/tbobj directory to do my builds in. To use the
defs-ricci-emulab defs file in my home directory, I would do:
164
165

cd ~/tbobj
166
~/testbed/configure --with-TBDEFS=/users/ricci/testbed/defs-ricci-emulab
167
168
gmake
gmake boss-install
169
sudo gmake post-install
170
171

The 'post-install' target needs to be done as root, because certain scripts
172
need to be setuid root.
173

174
175
176
177
178
Also, you'll need to create the 'tbdb' database (described in the next step)
before you can do the install, because we check to make sure that the software
you're installing matches the running database. Obviously, this can't be done
until the database is running.

179
180
##### Step 4 - Database Creation

181
See the file setup-db.txt in this directory 
182

183
##### Step 5 - Directories to create, and other misc. filesystem changes
184

185
186
NFS - Make the machine an NFS server and client with the following in
/etc/rc.conf:
187
188
189
190
nfs_server_enable="YES"
nfs_server_flags="-u -t -n 16"
nfs_client_enable="YES"

191
192
You also need some cross mounts between bossnode and fs. For example, on
one of our boss nodes, we have the following in /etc/fstab
193

194
195
196
197
    fs.mini.emulab.net:/z/users     /users  nfs     rw      0       0
    fs.mini.emulab.net:/z/proj      /proj   nfs     rw      0       0
    fs.mini.emulab.net:/z/groups    /groups nfs     rw      0       0
    fs.mini.emulab.net:/var         /usr/testbed/usersvar nfs ro,soft,-b    0
198

199
Note that you will need exports on the fs node (see setup-ops.txt).
200

201
202
203
204
205
206
207
208
209
210
suidperl - In order for setuid perl scripts to work properly, you'll need to:
chmod u+s /usr/bin/suidperl

##### Step 6 - Services to set up

DNS zones - We don't have documentation for creating these yet. Best bet right
now is to ask Utah for a copy of theirs.
Stick:
named_enable="YES"
in /etc/rc.conf
211

Leigh B. Stoller's avatar
Leigh B. Stoller committed
212
213
214
215
216
apache - You should have installed apache with mod_ssl, and php4.
have an auto-generated config file that you can install by changing to
the apache subdir of your build tree and running 'gmake install'.

Some apache installations may expect to find their
217
218
219
220
221
222
223
224
config file at /usr/local/etc/apache/httpd.conf, rather than apache.conf (where
ours gets installed.) Just symlink apache.conf to httpd.conf if you have
trouble with this. Also our config file expects to find SSL certificates in:
/usr/local/etc/apache/ssl.crt/www.<sitename>.crt and
SSLCertificateKeyFile /usr/local/etc/apache/ssl.key/www.<sitename>.net.key
(where <sitename> is OURSITE from the configure defs file.) Make sure yours
go there, or edit the apache.conf file appropriately.

225
226
227
228
229
inetd - In FreeBSD, you need to prevent inetd from rate-limiting connections
(an attempt to defend against DOS attacks, but very annoying in a testbed
environment). Put the following in /etc/rc.conf:
inetd_flags="-wW -R 0"

230
Local  Only:
231
232
233
234
235
236
237
238
239
240
241
242
243
tftp - Should have the following line in /etc/inetd.conf
tftp    dgram   udp wait    nobody  /usr/libexec/tftpd  tftpd /tftpboot /proj
(make sure to HUP inetd)

ntpd: The boss node should be running ntpd. In FreeBSD, you can enable this with
the line
xntpd_enable="YES"
in /etc/rc.conf. Check out the ntpd man page for configuration information.

cvsupd - Minor changes to images can be distributed at boot time with cvsup.
See doc/newimage.txt for an overview of setting up a sup tree. Make sure to
copy over the old one (if it exists), and make sure cvsupd is running (there's
an example rc.d script in the rc.d/ directory of the testbed CVS tree.) Create
244
a group named 'root', with any gid. This is because cvsup uses the name of
245
246
247
248
the group, rather than its gid, to determine what group the file should belong
to. Since Linux uses 'root' instead of BSD's 'wheel', this is needed for the
Linux sup tree.

249
Local Only:
250
251
252
253
254
255
syslogd - Normally, sylogd on FreeBSD is run with the '-s' flag to prevent
logging to it over the network. We use network logging, so we need this
feature. Re-enable it by putting:
syslogd_flags=""
in /etc/rc.conf

256
Local Only:
257
258
259
260
261
262
263
264
265
dhcpd - Need to install the dhcpd config file. The old (deprecated) location was
/usr/site/bin/dhcp/dhcpd.conf. The new location (and the place you should
install it if you used the 'isc-dhcpd' port) is /usr/local/etc/dhcpd.conf .
After you've filled the nodes and interfaces tables, (described in the database
setup documentation) use the dhcpd_makeconf script, along with the template in
the dhcpd directory of the CVS repository, to generate the dhcpd.conf file.

##### Step 7 - Misc. Files and Services

266
Local Only:
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
SNMP MIBs - MIBs go in /usr/local/share/snmp/mibs. In addition to the ones
installed by the ucd-snmp package, you'll need MIBs for Cisco and Intel
switches. You can grab the Cisco MIBs from:
ftp.cisco.com/pub/mibs
The Intel ones can be found from the site for the 510T switches at:
http://www.intel.com/network/connectivity/products/exp510t.htm
If you have SNMP-controllable APC power controllers, grab the 'PowerNet MIB'
from:
http://www.apcc.com/tools/download/
Now, a step that involves some voodoo I don't quite understand: make sure that
/usr/local/share/snmp/mibs/.index exists (touch it if it doesn't), and chmod it
to 666. Now, do an snmpwalk of some device (eg. 'snmpwalk cisco1 public') -
this will force the .index file to get rebuilt. Suggestions of better ways to
rebuild this file are welcome!

282
SSH - You'll probably also want to add
283
PermitRootLogin yes
284
to /etc/ssh/sshd_config (and HUP sshd) so that you can log in as root remotely
285

286
287
/etc/syslog.conf needs entries for some of our own services. Example:
!bootinfo
288
*.*                     /usr/testbed/log/bootinfo.log
289
!tmcd
290
*.*                     /usr/testbed/log/tmcd.log
291
!capture
292
*.*                     /usr/testbed/log/tiplogs/capture.log
293
294
!dhcpd
*.*                     /usr/testbed/log/dhcpd.log
295
296
297
298
!capserver
*.*                     /usr/testbed/log/capserver.log
!frisbeed
*.*                     /usr/testbed/log/frisbeed.log
299
300
All of these logs should be created before you HUP syslogd or reboot - All of
them can be world-readable
301

302
cron jobs: We currently have two cron jobs running for the testbed. Both can be
303
304
run out of /etc/crontab
45	1	*	*	*	root	/usr/testbed/sbin/backup
305
*/5	*	*	*	*	root	/usr/testbed/sbin/node_status
306
307
Don't forget to HUP cron!

308
You may want a program to allow administrator-types to run stuff easily as root.
309
Here at Utah, we have two: su1 (developed locally) and sudo (installed from
310
FreeBSD ports) - don't forget to get it set up! Our strategy on boss was to
311
give everyone in the wheel group unrestricted sudo access with:
312
%wheel  ALL=(ALL) NOPASSWD: ALL
313

Mac Newbold's avatar
Mac Newbold committed
314
315
316
317
318
319
320
checkpass - in the testbed software:
    cd tbsetup/checkpass/cracklib,2.7
    make all
    make install
  Note that these steps depend on having the dictionaries 
  /usr/share/dict/{propernames,words} available (standard for FreeBSD).
  If they're in different places, edit the obvious makefile vars.
321

322

323
324
325
326
SSH keys - Generate an SSH key for root by running ssh-keygen. Put an empty
passphrase on it. You'll want to copy boss:/root/.ssh/identity.pub to
ops:/root/.ssh/authorized_keys so that boss has the appropriate access on ops.

327
328
329
330
Process accounting - We generally turn on process accounting, by putting
accounting_enable="YES"
in /etc/rc.conf .

331
##### Copying from an old boss node
332
333
334

If you're simply moving from one boss node to another, there are a few files
and trees you'll want to make sure to copy over:
335

336
/usr/testbed/images/
337
/tftpboot/ (a link to /usr/testbed/tftpboot)
338
339
340
341
342
/etc/namedb/
/etc/master.password
/etc/group
/usr/testbed/sup/
/usr/site/
343
344
/root/.ssh
/etc/ssh
345

346
Right before bringing the new boss node online (if copying from an old boss
347
node), make sure to have a copy of the latest versions of:
348
349
350
351
* The database
* The sup tree
* The dhcpd.conf file
* The DNS records
Robert Ricci's avatar
Robert Ricci committed
352
* The password file