setup-fs.txt 10.2 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232
# Copyright (c) 2002-2005 University of Utah and the Flux Group.
# All rights reserved.

##### Setting up the Utah Network Testbed software
##### Most recently tested on FreeBSD 4.11.

##### Step 0 - The Goal

In the past we have combined the filesystem function with the ops/users
functions.  However, we recognize that people may have pre-existing, dedicated
(and usually very expensive :-) file server machine that they would like to
take advantage of.  Or for performance reasons, you may want to separate
the filesystem service from user activities on the ops node.

So we have made a start toward separating the two.  The general guiding
principle is to keep modifications to the fileserver box to a minimum,
since the ability to control the box may be greatly reduced; for example,
the fileserver might be a NetApp box or running MacOS X or Windows.
At a bare minimum however, we still require that you be able to manipulate
the access control lists (e.g., the BSD "exports" file) from the boss node.
Optionally, you may need to be able to monitor and manipulate disk quotas.

But that is all theory right now.  In the current state of affairs, the
fileserver box still has to run FreeBSD and is assumed to be dedicated to
Emulab use (e.g., the Emulab boss node assigns uid/gids itself, with no
attempt to synchronize with pre-existing uid/gids on the fileserver).

With this caveat in mind, we can begin.

##### Step 1 - OS installation and filesystem setup

Install FreeBSD on the machine you'll be using for your fs node, using the
standard FreeBSD installation process. When asked by the installer, it's best
to choose the 'Developer' distribution set - this gets you full sources. When
it asks if you want to install the ports collection, answer *no*.  Do not
install any packages at this time - you'll get a chance to later.  You'll
need to partition your filesystems so that you have the proper amount of
space for certain directories - see below for details.

Make sure that you have the network correctly configured.

The following directories will need to exist on partitions that have enough
space to hold them:

/usr	      - Needs space for the ports tree and a system object tree.
	                At least 10GB. Be sure to build with plenty of
			inodes (the ports tree itself uses about 200000, so
			be safe and build with at least a million).
/usr/testbed/ - Needs space for testbed software and logs. For a dedicated
			fileserver machine, this won't be very much, 100MB
			should be plenty.
/users/       - Needs space for user home directories. Amount of space required 
			depends on how many users you expect to have.
			Generally, though, we suggest that users store large
			files related to their projects in the /proj directory.
/proj/        - Needs space for project files. We recommend that this be larger
			than /users, to encourage people to store files here,
			which aids per-project accountability.
/groups/      - Needs enough space for files shared by the sub-groups of
			projects. These are primarily used by classes, if any.
/share/       - Exported read-only to all nodes, we use it for providing to
			experimenters the source for the FreeBSD and Linux
			versions we run as well as common packages and RPMs.
			This could require anything from 1GB to 20GB+ depending
			on what you want to make available.
You may want to enforce quotas on the user-writable filesystems. This is the
main reason you'd want to keep them in separate filesystems (i.e., so people
can have different /users/ and /proj/ quotas.) If you do not think you will
ever use quotas, then you could make /users and /proj part of the same

Note also since /share is exported read-only, FreeBSD requires that it be on
a separate filesystem from anything that is exported read-write.  So while
/users, /proj and /groups can be on the same filesystem, /share cannot.

Make sure that, no matter how you decide to partition things up, you make
symlinks to the appropriate places. ie., if you make one big filesystem called
/z that has /users, /proj, and /groups in it, make sure you:

	ln -s /z/users /users
	ln -s /z/proj /proj
	... etc.

In other words, we assume the existence of /users, /proj, /group and /share.

Do *not* create any user accounts, Emulab does not require that its users
have login accounts on the fileserver.  For the purposes of this setup, just
log in as root.  You can manually add login accounts for Emulab admins later
if you desire.

This would be a good time to reiterate that Emulab currently assumes that it
has complete control of the uid/gid namespace.  However, it starts assigning
uids at 10000 and gids at 6000, so pre-existing accounts with values lower
than those should be ok.  Other accounts will have to be removed or manually
synchronized with Emulab later.

##### Step 2 - Installing packages

To make sure that you're running with versions of software from the ports
collection that are known to work with our software, and to save you hours
of compile time, we provide pre-built binary packages of the ports required
by Emulab. 

(Do not let the names of the following tar files bother you, e.g.
"FreeBSD-4.10" or the apparent 20041102 date stamp.  These are indeed
the correct files to use with FreeBSD 4.11 and with the current (May 2005)
Emulab distribution.)

Download the packages tarball from:

(You can use the FreeBSD 'fetch' command to download the file.)

Now, untar this someplace (you need about 70MB of space, so don't use /tmp).
Let's say you untarred it into /usr/tmp. You would then run:

	env PKG_PATH=/usr/tmp/FreeBSD-4.10-20041102 pkg_add emulab-fs-1.4

Of course, if you untarred it somewhere else, put in the correct path.

Now you need to download a ports tree that corresponds to the above
packages.  We have run into many, many problems with versions of the
FreeBSD ports.  So, rather than using the /usr/ports tree that comes on the
FreeBSD installation media, we use one that we've tested against. You can
grab our 'approved' copy of the ports tree from:
Untar it, move it into place as /usr/ports (rename the old directory to
ports.old, or just remove it).  NOTE: You must download and copy the ports
tree into place, even if you do not intend to install any packages yourself.

##### Step 3 - Unpacking and running configure

At this point, you'll need to make a 'defs' file - You will use the same
defs file on boss, ops and fs. See defs-example in the top level directory
as a starting point. You want to make sure each of the variables has a
definition that makes sense in your environment. Feel free to ask Utah if
something is not clear.  Of particular note for the fs node is the setting
of FSDIR_* to match the filesystem layout from Step 1, and the setting of
FS_WITH_QUOTAS if you intend to use quotas on any filesystem.

Unpack the testbed source, and run it's configure script. A good place to
unpack the source code is /usr/testbed/src/testbed. You will use the
--with-TBDEFS option to configure to give it the path to your defs file:

	mkdir -p /usr/testbed/src/testbed
	mkdir -p /usr/testbed/obj/testbed
	cd /usr/testbed/src/testbed
	tar ....
	cd /usr/testbed/obj/testbed
	/usr/testbed/src/testbed/configure \

Typically, you would store your defs file in the source tree along with 
the other defs files that came in the tarball.

##### Step 4 - Running the fs installation script

In the object tree you've configured (say, /usr/testbed/obj/testbed), there's
an 'install' subdirectory, with a script called 'fs-install'. Just run this
script as root (note the same package directory argument as above).

	cd install
	env PKG_PATH=/usr/tmp/FreeBSD-4.10-20041102 perl fs-install

It will take care of installing any additional ports, and doing various
other configuration of FreeBSD required to make it into an fs node. The
script is designed so that you can run it as many times as you want, and
it'll just skip stuff it's already done. If it fails, send the output to
Utah so that we can fix it up. If it succeeds, follow any other
instructions it may have. The script will tell you to reboot the machine,
but you can wait until after you do the next step to do so, if you want.

(You may have to set the executable bit on this script, since configure won't;
we'd like to get this fixed at some point.)

You should be aware that, among other things, this script sets password-less
'sudo' for anyone in the 'wheel' group. If you don't want this for security
reasons, you can undo them after the installation script is finished.

##### Step 5 - Installing from source

To install the actual testbed software, simply do a 'gmake fs-install' in your
object directory.

(Note: If you're logged in as root, /usr/local/bin, where gmake lives on
FreeBSD, may not be in your path.)

##### Step 6 - Quotas (optional)

[ Note that this section is FreeBSD specific. ]

It you are planning to run quotas for Emulab users on the fs node, you
will need to establish a default quota value for all users.  Ideally,
the quota would be configurable per-user, but for now all quotas are
initialized from a "prototypical" user.  Note that you can change individual
user quotas later by running edquota(8) on the 'fs' node.  To establish
the default quota values, you will need a "prototype user" to which to
apply the quotas.  You will probably want to add a special user, say
'elabman', for this purpose.  The uid and gid for this user should be
the MIN_UNIX_UID and MIN_UNIX_GID values specified in your defs file
(10000 and 6000 by default).  Assuming those default values, you would do:

  pw useradd elabman -u 10000 -g 6000 -m -d /users/elabman -h - -s /bin/nologin

Now set the quota for that user on each quota-enabled filesystem, e.g.:

edquota -e /proj:2000000:2000000 -e /users:1000000:1000000 elabman

would set a 1GB quota on /proj and 512MB on /users for the prototype user.
Once the prototype user quotas are established, you can do:

edquota -p elabman 10000-15000

which would automatically apply the elabman quotas to any user created with
uids between 10000 and 15000, assuming that you wanted to allow up to 5000

##### Step 7 - Other miscellaneous things to set up

  [Nothing at this time]

Once you're done with all of this, reboot fs.