setup-ops.txt 8.98 KB
Newer Older
1 2
# Copyright (c) 2002-2006 University of Utah and the Flux Group.
4 5 6
# All rights reserved.

##### Setting up the Utah Network Testbed software
##### Most recently tested on FreeBSD 4.11.

##### Step 1 - OS installation and setup

Install FreeBSD on the machine you'll be using for your ops node, using the
standard FreeBSD installation process. When asked by the installer, it's best
to choose the 'Developer' distribution set - this gets you full sources. When
Leigh B. Stoller's avatar
Leigh B. Stoller committed
17 18 19 20
it asks if you want to install the ports collection, answer *no*.  Do not
install any packages at this time - you'll get a chance to later.  You'll
need to partition your filesystems so that you have the proper amount of
space for certain directories - see below for details.

Make sure that you have the network correctly configured.

24 25 26
The following directories will need to exist on partitions that have enough
space to hold them:

Leigh B. Stoller's avatar
Leigh B. Stoller committed
27 28 29 30
/usr	      - Needs space for the ports tree and a system object tree.
	                At least 10GB. Be sure to build with plenty of
			inodes (the ports tree itself uses about 200000, so
			be safe and build with at least a million).
Leigh B. Stoller's avatar
Leigh B. Stoller committed
31 32
/usr/testbed/ - Needs space for testbed software and logs. Several (3-4) GB
			should be enough.
33 34 35 36 37 38 39 40 41 42 43 44 45

Do *not* create any user accounts yet, and just log in as root for the time being.
Our software will create user accounts later, once you get boss set up. If you
already created any users, then delete them with the "pw" command and make sure
the home directories for them are removed as well. 

##### Step 1a - Shared filesystem setup for combined ops/fs machine.

If you have a separate 'fs' node, you should have already setup your shared
filesystems and can skip to Step 2.  If this node is also to be your file
server, you will also need:

46 47 48 49
/users/       - Needs space for user home directories. Amount of space required 
			depends on how many users you expect to have.
			Generally, though, we suggest that users store large
			files related to their projects in the /proj directory.
/proj/        - Needs space for project files. We recommend that this be larger
51 52 53 54
			than /users, to encourage people to store files here,
			which aids per-project accountability.
/groups/      - Needs enough space for files shared by the sub-groups of
			projects. These are primarily used by classes, if any.
55 56 57 58 59 60
/scratch/     - Optional, large filesystem of "scratch" space.  The intent
			is that this filesystem provides per-project space
			that is not "guaranteed" (for the Utah Emulab that
			means we do not back it up to tape).  If used,
			you would either set no quotas, or higher quotas
			than for /proj.
61 62 63 64 65
/share/       - Exported read-only to all nodes, we use it for providing to
			experimenters the source for the FreeBSD and Linux
			versions we run as well as common packages and RPMs.
			This could require anything from 1GB to 20GB+ depending
			on what you want to make available.
You may want to enforce quotas on the user-writable filesystems. This is the
68 69 70 71
main reason you'd want to keep them in separate filesystems (i.e., so people
can have different /users/ and /proj/ quotas.) If you do not think you will
ever use quotas, then you could make /users and /proj part of the same

73 74 75 76 77
As mentioned /scratch is optional.  If you are not providing "guarantees"
such as filesaved or RAIDed disk space and you are not using quotas, you
might as well just put all your space in /proj and not define FSDIR_SCRATCH
in the defs file.

78 79 80 81
Note also since /share is exported read-only, FreeBSD requires that it be on
a separate filesystem from anything that is exported read-write.  So while
/users, /proj and /groups can be on the same filesystem, /share cannot.

82 83 84
Make sure that, no matter how you decide to partition things up, you make
symlinks to the appropriate places. ie., if you make one big filesystem called
/z that has /users, /proj, and /groups in it, make sure you:
Leigh B. Stoller's avatar
Leigh B. Stoller committed
85 86 87 88

	ln -s /z/users /users
	ln -s /z/proj /proj
	... etc.
Leigh B. Stoller's avatar
Leigh B. Stoller committed

90 91
In other words, we assume the existence of /users, /proj, /group and /share
(but not /scratch).
Leigh B. Stoller's avatar
Leigh B. Stoller committed

##### Step 2 - Installing packages

95 96 97
To make sure that you're running with versions of software from the ports
collection that are known to work with our software, and to save you hours
of compile time, we provide pre-built binary packages of the ports required
98 99 100 101 102 103 104 105
by Emulab. 

(Do not let the names of the following tar files bother you, e.g.
"FreeBSD-4.10" or the apparent 20041102 date stamp.  These are indeed
the correct files to use with FreeBSD 4.11 and with the current (May 2005)
Emulab distribution.)

Download the packages tarball from:

Leigh B. Stoller's avatar
Leigh B. Stoller committed
108 109 110 111 112

(You can use the FreeBSD 'fetch' command to download the file.)

Now, untar this someplace (you need about 70MB of space, so don't use /tmp).
Let's say you untarred it into /usr/tmp. You would then run:
Leigh B. Stoller's avatar
Leigh B. Stoller committed
113 114

	env PKG_PATH=/usr/tmp/FreeBSD-4.10-20041102 pkg_add emulab-ops-1.4
115 116 117

Of course, if you untarred it somewhere else, put in the correct path.

Leigh B. Stoller's avatar
Leigh B. Stoller committed
118 119 120 121 122 123 124 125
Now you need to download a ports tree that corresponds to the above
packages.  We have run into many, many problems with versions of the
FreeBSD ports.  So, rather than using the /usr/ports tree that comes on the
FreeBSD installation media, we use one that we've tested against. You can
grab our 'approved' copy of the ports tree from:
Leigh B. Stoller's avatar
Leigh B. Stoller committed
126 127 128 129 130
Untar it, move it into place as /usr/ports (rename the old directory to
ports.old, or just remove it), and install whatever ports you want to make
ops feel like 'home' (like emacs, jove, or whatever). NOTE: You must
download and copy the ports tree into place, even if you do not intend to
install any packages yourself.

##### Step 3 - Unpacking and running configure

At this point, you'll need to make a 'defs' file - You will use the same
defs file on boss, ops and fs. See defs-example in the top level directory
136 137 138
as a starting point. You want to make sure each of the variables has a
definition that makes sense in your environment. Feel free to ask Utah if
something is not clear.

Leigh B. Stoller's avatar
Leigh B. Stoller committed
Unpack the testbed source, and run it's configure script. A good place to
141 142
unpack the source code is /usr/testbed/src/testbed. You will use the
--with-TBDEFS option to configure to give it the path to your defs file:

Leigh B. Stoller's avatar
Leigh B. Stoller committed
144 145 146 147 148 149 150
	mkdir -p /usr/testbed/src/testbed
	mkdir -p /usr/testbed/obj/testbed
	cd /usr/testbed/src/testbed
	tar ....
	cd /usr/testbed/obj/testbed
	/usr/testbed/src/testbed/configure \

Leigh B. Stoller's avatar
Leigh B. Stoller committed
152 153
Typically, you would store your defs file in the source tree along with 
the other defs files that came in the tarball.

##### Step 4 - Running the ops installation script

Leigh B. Stoller's avatar
Leigh B. Stoller committed
157 158
In the object tree you've configured (say, /usr/testbed/obj/testbed), there's
an 'install' subdirectory, with a script called 'ops-install'. Just run this
159 160 161 162 163 164 165 166 167 168 169 170
script as root (note the same package directory argument as above).

	cd install
	env PKG_PATH=/usr/tmp/FreeBSD-4.10-20041102 perl ops-install

It will take care of installing any additional ports, and doing various
other configuration of FreeBSD required to make it into an ops node. The
script is designed so that you can run it as many times as you want, and
it'll just skip stuff it's already done. If it fails, send the output to
Utah so that we can fix it up. If it succeeds, follow any other
instructions it may have. The script will tell you to reboot the machine,
but you can wait until after you do the next step to do so, if you want.

172 173
(You may have to set the executable bit on this script, since configure won't;
we'd like to get this fixed at some point.)

175 176 177 178
You should be aware that, among other things, this script sets up sendmail, and
sets up password-less 'sudo' for anyone in the 'wheel' group. If you don't want
these for security reasons, you can undo them after the installation script is

##### Step 5 - Installing from source

182 183
To install the actual testbed software, simply do a 'gmake ops-install' in your
object directory.

185 186 187
(Note: If you're logged in as root, /usr/local/bin, where gmake lives on
FreeBSD, may not be in your path.)

If you'll be using Cyclades or similar boards to use 'ops' as a serial console server,
189 190 191 192 193
you'll also want to do a 'gmake tipserv-install' in the object directory. In
addition, you'll need to build a custom kernel with the appropriate driver (you
may even need to get patches from Utah for some boards). You'll also need to
set up a few things (not yet automated), described in doc/adding_nodes.txt .

##### Step 6 - Setting up mailing lists

196 197 198
Optional. The ops-install script set up some mailing lists for the email
addresses you set up in the defs file, in /etc/mail/lists . If you want to run
these mailing lists off your ops node, you can put some people in them now.
These are just standard sendmail list files: a list of addresses and/or
archive files, one per line. If you chose to use archive files, give a path
to the file you'd like sendmail to stick all correspondence in, and create
this file.

##### Step 7 - Other miscellaneous things to set up

  [Nothing at this time]
207 208

Once you're done with all of this, reboot ops.