setup.txt 13.8 KB
Newer Older
1
##### Setting up the Utah Network Testbed software on a boss node
2
##### Last updated  February 27, 2002
3 4 5 6 7 8 9 10 11
##### Tested on FreeBSD 4.3

##### Step 0

First of all, the machine should be configured correctly for the network it is
on, have the root password set, etc.

##### Step 1 - Package installation

12 13
Install the necessary packages. The following are necessary, and are
the default versions in the FreeBSD 4.3 "ports" collection:
14 15 16

apache+mod_ssl-1.3.19+2.8.2 The Apache 1.3 webserver with SSL/TLS functionality
cvsupd-bin-16.1     A general network file distribution system optimized for CV
17
fping-2.2b1         Quickly ping N hosts w/o flooding the network
18
gd-1.8.4_1          A graphics library for fast PNG creation
19
gmake-3.79.1        GNU version of 'make' utility
20
graphviz-1.7c       Graph Visualization Software from AT&T and Bell Labs
21
isc-dhcp-2.0.5      ISC Dynamic Host Configuration Protocol client and server c
22
linuxthreads-2.1.3_2 POSIX pthreads implementation using rfork to generate kern
23
mod_auth_mysql-2.20 Allows users to use MySQL databases for user authentication
24 25 26 27 28
mod_php3-3.0.18     PHP3 module for Apache
mysql-client-3.23.36 Multithreaded SQL database (client)
mysql-server-3.23.36 Multithreaded SQL database (server)
otcl-1.0a6          MIT Object Tcl
p5-DBI-1.15         The perl5 Database Interface.  Required for DBD::* modules
29
p5-GD-1.32_2        A perl5 interface to Gd Graphics Library
30 31 32
p5-Mysql-modules-1.2215 Perl5 modules for accessing MySQL databases
p5-SNMP-4.2.0       A perl5 module for interfacing with the CMU SNMP library
p5-SNMP_Session-0.83 A perl5 module for providing rudimentary access to SNMPv1 a
33
p5-BSD-Resource	    BSD resource goo
34
rpm-3.0.6_5         The Red Hat Package Manager
35
tcl-8.2.3           Tool Command Language
36 37 38
tcl-sql-20000114_1  TCL module for accessing MySQL databases
ucd-snmp-4.2        An extendable SNMP implimentation

39
Note on TCL: Do NOT install tcl83; otcl, which is used by some testbed
40 41 42 43
scripts, requires tcl82. When you install the tcl-sql package, it will be put
in the library directory for the latest version of tcl you have installed, so
if you have tcl83 installed at the time, you will have tcl-sql support under
8.3.X, but not under 8.2.X (which testbed scripts use)
44

45 46 47
For the most part, subsitituting newer versions of the packages should be OK,
but know that we haven't tested them... The one exception is version 3.23.47 
of the mysql packages. In fact, we recommend 3.23.47, as it fixes some
48
boot-time problems that we've had with 3.23.36.
49

50 51
##### Step 2 - LEDA

52 53 54
We rely on a software package call LEDA which we are not allowed to
distribute (source *or* binary), and is currently not being
distributed by the authors. As a result, we will distribute a binary
55 56 57 58 59
for the one program (assign) that requires linking with the LEDA
library. In the meantime we will be working on removing our dependence
on the LEDA library.

The program in question is assign (see the assign subdir). Please
60 61 62 63
contact us if you need to replace it. The default operation of
configure is to assume the existence of assign/assign.bin in the
source tree. However, you can override that by using the configure 
option --with-assignbinary=/path/to/assign/binary.
64 65 66

##### Step 3 - Testbed tree configuration/installation

67 68 69 70 71 72
It is best to add the following group before creating the following
directories. Basically, testbed software should be in the tbadmin
group. Run this command as root.

	pw groupadd tbadmin -g 99
	
73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117
Configure the testbed tree. Many of the configuration parameters are stored in
a 'defs' file. Make a copy of the defs-default file (found in the root of
the testbed tree,) and customize it for your site. The key variables
are:
WWWDEFS - The name of the defs file to use for the web pages. Instructions
	for this file are below
TBADMINGROUP - The name of the group you created above, probaby tbadmin
TBOPSEMAIL - An email address for the 'operations staff'. Copies of
	error messages, etc. will get sent to this address, and in some
	cases users are directed to this address if they have questions and/or
	problems.
TBLOGSEMAIL - An email address to which some logs (experiment
	creation/deletion, etc.) will get sent.
DELAYCAPACITY - Maximum delay capacity of any nodes. Set to 1 prevent the
	same node from acting as multiple delay nodes.
BOSSNODE - Fully-qualified hostname of the boss node
USERNODE - Fully-qualified hostname of the users node (the node to which
	users will log on to control their nodes, etc.)
FSNODE - Fully-qualified hostname of the NFS server. (In our setup, this is
	the same machine as the users node.)
OURDOMAIN - The domain in which this testbed lies
FSDIR_GROUPS - The _real_ pathname (no symbolic links, etc.) of the groups
	directory on the FSNODE. See the setup-ops.txt file for the
	instructions for creating this directory.
FSDIR_PROJ - Ditto, for the proj directory
FSDIR_USERS - Ditto, for the users (home) directory

Other variables can generally be left alone. Pass the name of this file to
configure with the --with-TBDEFS option.

You'll also need to make a defs file for the web system. This is found in
the www/ directory of the testbed source tree. Copy default-defs.php3 to
<name>-defs.php3, and put <name> into WWWDEFS in the main defs file. The
main variables to worray about are:
$WWWHOST, $TBAUTHDOMAIN - replace emulab.net with your domain
$THISHOMEBASE, $THISPROJECT - Use the name of your site
$TBMAIL* - These work like the mail addresses of the same names in the
	defs file above. The new addresses are the approval address, to which
	requests to start projects are sent, and the audit address, which is
	CCed on certain mail sent to users.
$TBMAINSITE - Set this to 0

Now, build and install the software. For example, I have the testbed source in
~/testbed, and use the ~/tbobj directory to do my builds in. To use the
defs-ricci-emulab defs file in my home directory, I would do:
118 119

cd ~/tbobj
120
~/testbed/configure --with-TBDEFS=/users/ricci/testbed/defs-ricci-emulab
121 122
gmake
gmake boss-install
123
sudo gmake post-install
124 125

The 'post-install' target needs to be done as root, because certain scripts
126
need to be setuid root.
127 128 129

##### Step 4 - Database Creation

130
See the file setup-db.txt in this directory 
131 132


133
##### Step 5 - Directories to create, and other misc. filesystem changes
134

135
Logs - Create /usr/testbed/log, and link /var/log/testbed to it
136

137 138
NFS - Make the machine an NFS server and client with the following in
/etc/rc.conf:
139 140 141 142
nfs_server_enable="YES"
nfs_server_flags="-u -t -n 16"
nfs_client_enable="YES"

143 144
You also need some cross mounts between bossnode and fs. For example, on
one of our boss nodes, we have the following in /etc/fstab
145

146 147 148 149
    fs.mini.emulab.net:/z/users     /users  nfs     rw      0       0
    fs.mini.emulab.net:/z/proj      /proj   nfs     rw      0       0
    fs.mini.emulab.net:/z/groups    /groups nfs     rw      0       0
    fs.mini.emulab.net:/var         /usr/testbed/usersvar nfs ro,soft,-b    0
150

151
Note that you will need exports on the fs node (see setup-ops.txt).
152

153 154 155 156 157 158 159 160 161 162
suidperl - In order for setuid perl scripts to work properly, you'll need to:
chmod u+s /usr/bin/suidperl

##### Step 6 - Services to set up

DNS zones - We don't have documentation for creating these yet. Best bet right
now is to ask Utah for a copy of theirs.
Stick:
named_enable="YES"
in /etc/rc.conf
163

164 165 166 167 168
inetd - In FreeBSD, you need to prevent inetd from rate-limiting connections
(an attempt to defend against DOS attacks, but very annoying in a testbed
environment). Put the following in /etc/rc.conf:
inetd_flags="-wW -R 0"

169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216
tftp - Should have the following line in /etc/inetd.conf
tftp    dgram   udp wait    nobody  /usr/libexec/tftpd  tftpd /tftpboot /proj
(make sure to HUP inetd)

ntpd: The boss node should be running ntpd. In FreeBSD, you can enable this with
the line
xntpd_enable="YES"
in /etc/rc.conf. Check out the ntpd man page for configuration information.

cvsupd - Minor changes to images can be distributed at boot time with cvsup.
See doc/newimage.txt for an overview of setting up a sup tree. Make sure to
copy over the old one (if it exists), and make sure cvsupd is running (there's
an example rc.d script in the rc.d/ directory of the testbed CVS tree.) Create
a group named 'root', with any gid. This is because cvsup uses the game of
the group, rather than its gid, to determine what group the file should belong
to. Since Linux uses 'root' instead of BSD's 'wheel', this is needed for the
Linux sup tree.

syslogd - Normally, sylogd on FreeBSD is run with the '-s' flag to prevent
logging to it over the network. We use network logging, so we need this
feature. Re-enable it by putting:
syslogd_flags=""
in /etc/rc.conf

dhcpd - Need to install the dhcpd config file. The old (deprecated) location was
/usr/site/bin/dhcp/dhcpd.conf. The new location (and the place you should
install it if you used the 'isc-dhcpd' port) is /usr/local/etc/dhcpd.conf .
After you've filled the nodes and interfaces tables, (described in the database
setup documentation) use the dhcpd_makeconf script, along with the template in
the dhcpd directory of the CVS repository, to generate the dhcpd.conf file.

##### Step 7 - Misc. Files and Services

SNMP MIBs - MIBs go in /usr/local/share/snmp/mibs. In addition to the ones
installed by the ucd-snmp package, you'll need MIBs for Cisco and Intel
switches. You can grab the Cisco MIBs from:
ftp.cisco.com/pub/mibs
The Intel ones can be found from the site for the 510T switches at:
http://www.intel.com/network/connectivity/products/exp510t.htm
If you have SNMP-controllable APC power controllers, grab the 'PowerNet MIB'
from:
http://www.apcc.com/tools/download/
Now, a step that involves some voodoo I don't quite understand: make sure that
/usr/local/share/snmp/mibs/.index exists (touch it if it doesn't), and chmod it
to 666. Now, do an snmpwalk of some device (eg. 'snmpwalk cisco1 public') -
this will force the .index file to get rebuilt. Suggestions of better ways to
rebuild this file are welcome!

217 218
SSH - If possible, grab the old machine's SSH host keys (from
/etc/ssh/ssh_host*) and HUP sshd. Also, get the root identity and known_hosts
219 220 221
files from the old machine (/root/.ssh/{identity,identity.pub,known_hosts}) -
Make sure to preserve file and directory permissions. You'll probably also want
to add
222
PermitRootLogin yes
223
to /etc/ssh/sshd_config (and HUP sshd) so that you can log in as root remotely
224

225 226
/etc/syslog.conf needs entries for some of our own services. Example:
!bootinfo
227
*.*                     /usr/testbed/log/bootinfo.log
228
!tmcd
229
*.*                     /usr/testbed/log/tmcd.log
230
!capture
231
*.*                     /usr/testbed/log/tiplogs/capture.log
232 233
!dhcpd
*.*                     /usr/testbed/log/dhcpd.log
234 235 236 237
!capserver
*.*                     /usr/testbed/log/capserver.log
!frisbeed
*.*                     /usr/testbed/log/frisbeed.log
238 239
All of these logs should be created before you HUP syslogd or reboot - All of
them can be world-readable
240 241 242 243 244

RC scripts - The mysql-client rc script needs to run before ANY testbed
services are started! The mysql server should also be started early in the
process.  boot process. You can ensure this by changing directories to
/usr/local/etc/rc.d and renaming 'mysql-client.sh' to '1.mysql-client.sh' and
245 246 247 248
'mysql-server.sh' to '2.mysql-server.sh'. Furthermore, dhcpd needs to start
before proxydhcp, so rename 'dhcpd.sh' to '2.dhcpd.sh'. You will also need to
install the '3.testbed.sh' and 'cvsupd.sh'  scripts (in the rc.d directory of
the testbed tree)
249

Robert Ricci's avatar
Robert Ricci committed
250 251
Apache - You should have installed apache with mod_ssl, and php3 (NOTE: Version
3.0.17 is known to have broken file uploading support. Use 3.0.16 or 3.0.18 -
252 253 254 255 256 257 258 259 260 261
newer versions are likely to work as well.) We have an auto-generated config
file that you can install by changing to the apache subdir of your build tree
and running 'gmake install'. Some apache installtions may expect to find their
config file at /usr/local/etc/apache/httpd.conf, rather than apache.conf (where
ours gets installed.) Just symlink apache.conf to httpd.conf if you have
trouble with this. Also our config file expects to find SSL certificates in:
/usr/local/etc/apache/ssl.crt/www.<sitename>.crt and
SSLCertificateKeyFile /usr/local/etc/apache/ssl.key/www.<sitename>.net.key
(where <sitename> is OURSITE from the configure defs file.) Make sure yours
go there, or edit the apache.conf file appropriately.
Robert Ricci's avatar
Robert Ricci committed
262

263 264 265
Cron jobs: We currently have two cron jobs running for the testbed. Both can be
run out of /etc/crontab
45	1	*	*	*	root	/usr/testbed/sbin/backup
266
*/5	*	*	*	*	root	/usr/testbed/sbin/node_status
267 268
Don't forget to HUP cron!

269
You may want a program to allow administrator-types to run stuff easily as root.
270
Here at Utah, we have two: su1 (developed locally) and sudo (installed from
271
FreeBSD ports) - don't forget to get it set up! Our strategy on boss was to
272
give everyone in the wheel group unrestricted sudo access with:
273
%wheel  ALL=(ALL) NOPASSWD: ALL
274

Mac Newbold's avatar
Mac Newbold committed
275 276 277 278 279 280 281
checkpass - in the testbed software:
    cd tbsetup/checkpass/cracklib,2.7
    make all
    make install
  Note that these steps depend on having the dictionaries 
  /usr/share/dict/{propernames,words} available (standard for FreeBSD).
  If they're in different places, edit the obvious makefile vars.
282

283

284 285 286 287
SSH keys - Generate an SSH key for root by running ssh-keygen. Put an empty
passphrase on it. You'll want to copy boss:/root/.ssh/identity.pub to
ops:/root/.ssh/authorized_keys so that boss has the appropriate access on ops.

288 289 290 291
Process accounting - We generally turn on process accounting, by putting
accounting_enable="YES"
in /etc/rc.conf .

292
##### Copying from an old boss node
293 294 295

If you're simply moving from one boss node to another, there are a few files
and trees you'll want to make sure to copy over:
296

297
/usr/testbed/images/
298
/tftpboot/ (a link to /usr/testbed/tftpboot)
299 300 301 302 303 304
/etc/namedb/
/etc/master.password
/etc/group
/usr/testbed/sup/
/usr/site/

305 306
Right before bringing the new boss node online (if copying from an old boss
node), make sure to have copy over the latest versions of:
307 308 309 310
* The database
* The sup tree
* The dhcpd.conf file
* The DNS records
Robert Ricci's avatar
Robert Ricci committed
311
* The password file