setup.txt 10.3 KB
Newer Older
1 2
##### Setting up the Utah Network Testbed software on a boss node
##### Last updated  January 18, 2001
3 4 5 6 7 8 9 10 11 12 13 14 15 16
##### Tested on FreeBSD 4.3

##### Step 0

First of all, the machine should be configured correctly for the network it is
on, have the root password set, etc.

##### Step 1 - Package installation

Install the necessary packages. The following are necessary, and available as
FreeBSD ports:

apache+mod_ssl-1.3.19+2.8.2 The Apache 1.3 webserver with SSL/TLS functionality
cvsupd-bin-16.1     A general network file distribution system optimized for CV
17
fping-2.2b1         Quickly ping N hosts w/o flooding the network
18 19
gmake-3.79.1        GNU version of 'make' utility
isc-dhcp-2.0.5      ISC Dynamic Host Configuration Protocol client and server c
20
linuxthreads-2.1.3_2 POSIX pthreads implementation using rfork to generate kern
21
mod_auth_mysql-2.20 Allows users to use MySQL databases for user authentication
22 23 24 25 26 27 28 29 30
mod_php3-3.0.18     PHP3 module for Apache
mysql-client-3.23.36 Multithreaded SQL database (client)
mysql-server-3.23.36 Multithreaded SQL database (server)
otcl-1.0a6          MIT Object Tcl
p5-DBI-1.15         The perl5 Database Interface.  Required for DBD::* modules
p5-Mysql-modules-1.2215 Perl5 modules for accessing MySQL databases
p5-SNMP-4.2.0       A perl5 module for interfacing with the CMU SNMP library
p5-SNMP_Session-0.83 A perl5 module for providing rudimentary access to SNMPv1 a
rpm-3.0.6_5         The Red Hat Package Manager
31
tcl-8.2.3           Tool Command Language
32 33 34
tcl-sql-20000114_1  TCL module for accessing MySQL databases
ucd-snmp-4.2        An extendable SNMP implimentation

35 36 37 38 39
Note on TCL: Do NOT install tcl83 - otcl, which is used by some testbed
scripts, requires tcl82. When you install the tcl-sql package, it will be put
in the library directory for the latest version of tcl you have installed, so
if you have tcl83 installed at the time, you will have tcl-sql support under
8.3.X, but not under 8.2.X (which testbed scripts use)
40

41 42 43 44 45 46 47 48 49 50 51 52
##### Step 2 - LEDA

Currently, the LEDA library is required to compile some testbed software (we
hope to remove this dependency at some point.) The simplest place to install it
is /usr/testbed/LEDA. If another location is used, be sure to use the
--with-LEDA=<dir> option to configure in the next step. The home page for LEDA
is at:

http://www.algorithmic-solutions.com/as_html/products/products.html

##### Step 3 - Testbed tree configuration/installation

53 54 55 56 57 58 59 60
It is best to add the following group before creating the following
directories. Basically, testbed software should be in the tbadmin
group. Run this command as root.

	pw groupadd tbadmin -g 99
	
Configure the testbed tree. For example, I have the testbed source in
~/testbed, and use the ~/tbobj directory to do my builds in.
61 62 63 64 65 66 67 68

cd ~/tbobj
~/testbed/configure
gmake
gmake boss-install
gmake post-install

The 'post-install' target needs to be done as root, because certain scripts
69
need to be setuid root.
70 71 72

##### Step 4 - Database Creation

73
See the file setup-db.txt in this directory 
74 75 76 77 78 79 80

##### Step 5 - Misc. Files and Services

SNMP MIBs - MIBs go in /usr/local/share/snmp/mibs. In addition to the ones
installed by the ucd-snmp package, you'll need MIBs for Cisco and Intel
switches. You can grab the Cisco MIBs from:
ftp.cisco.com/pub/mibs
81 82
The Intel ones can be found from the site for the 510T switches at:
http://www.intel.com/network/connectivity/products/exp510t.htm
83 84 85
If you have SNMP-controllable APC power controllers, grab the 'PowerNet MIB'
from:
http://www.apcc.com/tools/download/
86 87 88 89 90
Now, a step that involves some voodoo I don't quite understand: make sure that
/usr/local/share/snmp/mibs/.index exists (touch it if it doesn't), and chmod it
to 666. Now, do an snmpwalk of some device (eg. 'snmpwalk cisco1 public') -
this will force the .index file to get rebuilt. Suggestions of better ways to
rebuild this file are welcome!
91 92 93 94 95 96 97 98 99 100 101

DNS zones - Make sure to include the DNS configuration files from /etc/named/
Stick:
named-enable="YES"
in /etc/rc.conf

NFS - Make the machine an NFS server and client with the following in /etc/rc.conf:
nfs_server_enable="YES"
nfs_server_flags="-u -t -n 16"
nfs_client_enable="YES"

102 103 104 105 106 107 108 109 110 111
You also need some cross mounts between bossnode and fs. On bossnode:

	fs.mini.emulab.net:/z/users     /users  nfs     rw      0       0
	fs.mini.emulab.net:/z/proj      /proj   nfs     rw      0       0
	fs.mini.emulab.net:/z/groups    /groups nfs     rw      0       0
	fs.mini.emulab.net:/var         /usr/testbed/usersvar nfs ro,soft,-b    0

Note that you will need exports on fs (see setup-ops.txt).
	

112 113 114 115
tftp - Should have the following line in /etc/inetd.conf
tftp    dgram   udp wait    nobody  /usr/libexec/tftpd  tftpd /tftpboot /proj
(make sure to HUP inetd)

116 117 118 119 120
inetd - In FreeBSD, you need to prevent inetd from rate-limiting connections
(an attempt to defend against DOS attacks, but very annoying in a testbed
environment). Put the following in /etc/rc.conf:
inetd_flags="-wW -R 0"

121 122
SSH - If possible, grab the old machine's SSH host keys (from
/etc/ssh/ssh_host*) and HUP sshd. Also, get the root identity and known_hosts
123 124 125
files from the old machine (/root/.ssh/{identity,identity.pub,known_hosts}) -
Make sure to preserve file and directory permissions. You'll probably also want
to add
126
PermitRootLogin yes
127
to /etc/ssh/sshd_config (and HUP sshd) so that you can log in as root remotely
128

129 130
Grab the old /etc/master.passwd file, and run
'cd /etc && pwd_mkdb -p master.passwd'
131
Also grab the old /etc/groups file
132 133 134

/etc/syslog.conf needs entries for some of our own services. Example:
!bootinfo
135
*.*                     /usr/testbed/log/bootinfo.log
136
!tmcd
137
*.*                     /usr/testbed/log/tmcd.log
138
!capture
139
*.*                     /usr/testbed/log/tiplogs/capture.log
140 141
!dhcpd
*.*                     /usr/testbed/log/dhcpd.log
142 143
All of these logs should be created before you HUP syslogd or reboot - All of
them can be world-readable
144 145 146

DHCP - Need to install the dhcpd config file. The old (deprecated) location was
/usr/site/bin/dhcp/dhcpd.conf. The new location (and the place you should
147
install it if you used the 'isc-dhcpd' port) is /usr/local/etc/dhcpd.conf .
148 149 150
After you've filled the nodes and interfaces tables, (described in the database
setup documentation) use the dhcpd_makeconf script, along with the template in
the dhcpd directory of the CVS repository, to generate the dhcpd.conf file.
151 152 153 154 155

RC scripts - The mysql-client rc script needs to run before ANY testbed
services are started! The mysql server should also be started early in the
process.  boot process. You can ensure this by changing directories to
/usr/local/etc/rc.d and renaming 'mysql-client.sh' to '1.mysql-client.sh' and
156 157 158 159
'mysql-server.sh' to '2.mysql-server.sh'. Furthermore, dhcpd needs to start
before proxydhcp, so rename 'dhcpd.sh' to '2.dhcpd.sh'. You will also need to
install the '3.testbed.sh' and 'cvsupd.sh'  scripts (in the rc.d directory of
the testbed tree)
160 161 162

Logs - To avoid filling up /var, link /var/log/testbed to /usr/testbed/log

163 164 165 166 167 168 169 170
CVSUPD - Minor changes to images can be distributed at boot time with cvsup.
See doc/newimage.txt for an overview of setting up a sup tree. Make sure to
copy over the old one (if it exists), and make sure cvsupd is running (there's
an example rc.d script in the rc.d/ directory of the testbed CVS tree.) Create
a group named 'root', with any gid. This is because cvsup uses the game of
the group, rather than its gid, to determine what group the file should belong
to. Since Linux uses 'root' instead of BSD's 'wheel', this is needed for the
Linux sup tree.
171

Robert Ricci's avatar
Robert Ricci committed
172 173
Apache - You should have installed apache with mod_ssl, and php3 (NOTE: Version
3.0.17 is known to have broken file uploading support. Use 3.0.16 or 3.0.18 -
174 175 176 177 178 179 180 181 182 183
newer versions are likely to work as well.) We have an auto-generated config
file that you can install by changing to the apache subdir of your build tree
and running 'gmake install'. Some apache installtions may expect to find their
config file at /usr/local/etc/apache/httpd.conf, rather than apache.conf (where
ours gets installed.) Just symlink apache.conf to httpd.conf if you have
trouble with this. Also our config file expects to find SSL certificates in:
/usr/local/etc/apache/ssl.crt/www.<sitename>.crt and
SSLCertificateKeyFile /usr/local/etc/apache/ssl.key/www.<sitename>.net.key
(where <sitename> is OURSITE from the configure defs file.) Make sure yours
go there, or edit the apache.conf file appropriately.
Robert Ricci's avatar
Robert Ricci committed
184

185 186 187
Cron jobs: We currently have two cron jobs running for the testbed. Both can be
run out of /etc/crontab
45	1	*	*	*	root	/usr/testbed/sbin/backup
188
*/5	*	*	*	*	root	/usr/testbed/sbin/node_status
189 190
Don't forget to HUP cron!

Robert Ricci's avatar
Robert Ricci committed
191 192 193 194 195
ntpd: The boss node should be running ntpd. In FreeBSD, you can enable this with
the line
xntpd_enable="YES"
in /etc/rc.conf. Check out the ntpd man page for configuration information.

196
You may want a program to allow administrator-types to run stuff easily as root.
197
Here at Utah, we have two: su1 (developed locally) and sudo (installed from
198
FreeBSD ports) - don't forget to get it set up! Our strategy on boss was to
199
give everyone in the wheel group unrestricted sudo access with:
200
%wheel  ALL=(ALL) NOPASSWD: ALL
201

Mac Newbold's avatar
Mac Newbold committed
202 203 204 205 206 207 208
checkpass - in the testbed software:
    cd tbsetup/checkpass/cracklib,2.7
    make all
    make install
  Note that these steps depend on having the dictionaries 
  /usr/share/dict/{propernames,words} available (standard for FreeBSD).
  If they're in different places, edit the obvious makefile vars.
209

210 211 212 213 214 215
syslogd - Normally, sylogd on FreeBSD is run with the '-s' flag to prevent
logging to it over the network. We use network logging, so we need this
feature. Re-enable it by putting:
syslogd_flags=""
in /etc/rc.conf

216 217 218
suidperl - In order for setuid perl scripts to work properly, you'll need to:
chmod u+s /usr/bin/suidperl

219 220 221 222
SSH keys - Generate an SSH key for root by running ssh-keygen. Put an empty
passphrase on it. You'll want to copy boss:/root/.ssh/identity.pub to
ops:/root/.ssh/authorized_keys so that boss has the appropriate access on ops.

223 224 225 226 227 228 229 230 231 232 233 234
##### Step 6 - Stuff to copy from an old boss node

If you're simply moving from one boss node to another, there are a few files
and trees you'll want to make sure to copy over:
/usr/testbed/images/
/tftpboot/
/etc/namedb/
/etc/master.password
/etc/group
/usr/testbed/sup/
/usr/site/

235
##### Last-minute synching
236 237
Right before bringing the new boss node online (if copying from an old boss
node), make sure to have copy over the latest versions of:
238 239 240 241
* The database
* The sup tree
* The dhcpd.conf file
* The DNS records
Robert Ricci's avatar
Robert Ricci committed
242
* The password file