Commit c53d5827 authored by Mike Hibler's avatar Mike Hibler

Changes related to allowing seperate 'fs' (file server) node.

Entailed new instructions for manual setup as well as integration into
elabinelab framework.  First, the manual path:

setup.txt, setup-boss.txt, setup-ops.txt and new setup-fs.txt:
    Updated to reflect potential for separate fs node.  The org here
    is a little dicey and could be confusing with ops+fs vs. ops and fs.
    Has not been field tested yet.

*/GNUmakefile.in: new fs-install target.

configure, configure.in, defs-*:
    Somewhat unrelated, make min uid/gid to use be a defs setting.
    Also add config of fs-install.in script.

boss-install.in, ops-install.in and new fs-install.in:
    Handle distinct fs node.  If you have one, fs-install is run before
    ops-install.  All scripts rely on the defs file settings of FSNODE
    and USERNODE to determine if the fs node is seperate.

utils/checkquota.in:
    Just return "ok" if quotas are not used (i.e., if defs file FS_WITH_QUOTA
    string is null.

install/ports/emulab-fs:
    Meta port for fs node specific stuff.  Also a patch for the samba port
    Makefile so it doesn't drag in CUPs, etc.  Note that the current samba
    port Makefile has this change, I am just backporting to our version.

Elabinelab specific changes:

elabinelab-withfs.ns:
    NS fragment used in conjunction with
	tb-elab-in-elab-topology "withfs"
    to setup inner-elab with fs node.

elabinelab.ns:
    The hard work on the boss side.  Recognize seperate-fs config and handle
    running of rc.mkelab on that node.  fs setup happens before ops setup.

rc.mkelab:
    The hard work on the client side.  Recognize FsNode setup as well as
    differentiate ops+fs from ops setup.

Related stuff either not part of the repo or checked in previously:
    emulab-fs package
parent 317c82a5
......@@ -31,7 +31,8 @@ SUBDIRS = lib db assign www @optional_subdirs@ ipod security sensors \
all: all-subdirs
install:
@echo "Choose either boss-install (paper) or ops-install (plastic)"
@echo "Choose either boss-install, ops-install or fs-install"
@echo "Choose opsfs-install instead of ops-install for a combined ops/fs machine"
@echo "Choose tipserv-install for dedicated tip server machines."
#
......@@ -89,6 +90,13 @@ endif
@$(MAKE) -C tmcd control-install
@$(MAKE) -C account control-install
fs-install:
@$(MAKE) -C tbsetup fs-install
@$(MAKE) -C account fs-install
opsfs-install: ops-install fs-install
@echo "Combined ops/fs install done."
install-mkdirs:
-mkdir -p $(INSTALL_TOPDIR)/opsdir
-mkdir -p $(INSTALL_TOPDIR)/locks
......
......@@ -14,7 +14,7 @@ include $(OBJDIR)/Makeconf
SBIN_STUFF = tbacct addsfskey addpubkey mkusercert
LIBEXEC_STUFF = webtbacct webaddsfskey webaddpubkey webmkusercert
FS_SBIN_STUFF = quotamail
FSBIN_STUFF = quotamail
#
# Force dependencies on the scripts so that they will be rerun through
......@@ -42,10 +42,9 @@ post-install:
chown root $(INSTALL_SBINDIR)/mkusercert
chmod u+s $(INSTALL_SBINDIR)/mkusercert
# XXX ops == fs right now
control-install: fs-install
control-install:
fs-script-install: $(addprefix $(INSTALL_SBINDIR)/, $(FS_SBIN_STUFF))
fs-script-install: $(addprefix $(INSTALL_SBINDIR)/, $(FSBIN_STUFF))
fs-install: fs-script-install
......
......@@ -1365,6 +1365,8 @@ done
......@@ -1425,6 +1427,8 @@ BOSSEVENTPORT=2927
UNIFIED_BOSS_AND_OPS=0
FRISEBEEMCASTADDR="234.5.6"
FRISEBEEMCASTPORT=3564
MIN_UNIX_UID=10000
MIN_UNIX_GID=6000
#
# Okay, I know this is improper usage of --with. Too bad.
......@@ -1925,17 +1929,17 @@ for ac_hdr in ulxmlrpcpp/ulxr_config.h
do
ac_safe=`echo "$ac_hdr" | sed 'y%./+-%__p_%'`
echo $ac_n "checking for $ac_hdr""... $ac_c" 1>&6
echo "configure:1929: checking for $ac_hdr" >&5
echo "configure:1933: checking for $ac_hdr" >&5
if eval "test \"`echo '$''{'ac_cv_header_$ac_safe'+set}'`\" = set"; then
echo $ac_n "(cached) $ac_c" 1>&6
else
cat > conftest.$ac_ext <<EOF
#line 1934 "configure"
#line 1938 "configure"
#include "confdefs.h"
#include <$ac_hdr>
EOF
ac_try="$ac_cpp conftest.$ac_ext >/dev/null 2>conftest.out"
{ (eval echo configure:1939: \"$ac_try\") 1>&5; (eval $ac_try) 2>&5; }
{ (eval echo configure:1943: \"$ac_try\") 1>&5; (eval $ac_try) 2>&5; }
ac_err=`grep -v '^ *+' conftest.out | grep -v "^conftest.${ac_ext}\$"`
if test -z "$ac_err"; then
rm -rf conftest*
......@@ -1974,17 +1978,17 @@ for ac_hdr in linux/videodev.h
do
ac_safe=`echo "$ac_hdr" | sed 'y%./+-%__p_%'`
echo $ac_n "checking for $ac_hdr""... $ac_c" 1>&6
echo "configure:1978: checking for $ac_hdr" >&5
echo "configure:1982: checking for $ac_hdr" >&5
if eval "test \"`echo '$''{'ac_cv_header_$ac_safe'+set}'`\" = set"; then
echo $ac_n "(cached) $ac_c" 1>&6
else
cat > conftest.$ac_ext <<EOF
#line 1983 "configure"
#line 1987 "configure"
#include "confdefs.h"
#include <$ac_hdr>
EOF
ac_try="$ac_cpp conftest.$ac_ext >/dev/null 2>conftest.out"
{ (eval echo configure:1988: \"$ac_try\") 1>&5; (eval $ac_try) 2>&5; }
{ (eval echo configure:1992: \"$ac_try\") 1>&5; (eval $ac_try) 2>&5; }
ac_err=`grep -v '^ *+' conftest.out | grep -v "^conftest.${ac_ext}\$"`
if test -z "$ac_err"; then
rm -rf conftest*
......@@ -2017,7 +2021,7 @@ done
# Extract the first word of "gtk-config", so it can be a program name with args.
set dummy gtk-config; ac_word=$2
echo $ac_n "checking for $ac_word""... $ac_c" 1>&6
echo "configure:2021: checking for $ac_word" >&5
echo "configure:2025: checking for $ac_word" >&5
if eval "test \"`echo '$''{'ac_cv_prog_GTK_CONFIG'+set}'`\" = set"; then
echo $ac_n "(cached) $ac_c" 1>&6
else
......@@ -2096,7 +2100,7 @@ fi
# SVR4 /usr/ucb/install, which tries to use the nonexistent group "staff"
# ./install, which can be erroneously created by make from ./install.sh.
echo $ac_n "checking for a BSD compatible install""... $ac_c" 1>&6
echo "configure:2100: checking for a BSD compatible install" >&5
echo "configure:2104: checking for a BSD compatible install" >&5
if test -z "$INSTALL"; then
if eval "test \"`echo '$''{'ac_cv_path_install'+set}'`\" = set"; then
echo $ac_n "(cached) $ac_c" 1>&6
......@@ -2157,7 +2161,7 @@ esac
# Extract the first word of "rsync", so it can be a program name with args.
set dummy rsync; ac_word=$2
echo $ac_n "checking for $ac_word""... $ac_c" 1>&6
echo "configure:2161: checking for $ac_word" >&5
echo "configure:2165: checking for $ac_word" >&5
if eval "test \"`echo '$''{'ac_cv_path_RSYNC'+set}'`\" = set"; then
echo $ac_n "(cached) $ac_c" 1>&6
else
......@@ -2328,7 +2332,8 @@ outfiles="$outfiles Makeconf GNUmakefile \
cdrom/GNUmakefile cdrom/tbbootconfig/GNUmakefile \
cdrom/groklilo/GNUmakefile \
dhcpd/dhcpd.conf.template dhcpd/GNUmakefile \
install/GNUmakefile install/ops-install install/boss-install \
install/GNUmakefile \
install/ops-install install/boss-install install/fs-install \
install/newnode_sshkeys/GNUmakefile install/smb.conf.head \
mote/GNUmakefile mote/tbuisp mote/tbsgmotepower mote/newmote \
mote/sgtools/GNUmakefile \
......@@ -2605,6 +2610,8 @@ s%@FRISEBEEMCASTADDR@%$FRISEBEEMCASTADDR%g
s%@FRISEBEEMCASTPORT@%$FRISEBEEMCASTPORT%g
s%@WINSUPPORT@%$WINSUPPORT%g
s%@CVSSUPPORT@%$CVSSUPPORT%g
s%@MIN_UNIX_UID@%$MIN_UNIX_UID%g
s%@MIN_UNIX_GID@%$MIN_UNIX_GID%g
s%@TBOPSEMAIL@%$TBOPSEMAIL%g
s%@TBOPSEMAIL_NOSLASH@%$TBOPSEMAIL_NOSLASH%g
s%@TBLOGSEMAIL@%$TBLOGSEMAIL%g
......
......@@ -148,6 +148,8 @@ AC_SUBST(FRISEBEEMCASTADDR)
AC_SUBST(FRISEBEEMCASTPORT)
AC_SUBST(WINSUPPORT)
AC_SUBST(CVSSUPPORT)
AC_SUBST(MIN_UNIX_UID)
AC_SUBST(MIN_UNIX_GID)
#
# Offer both versions of the email addresses that have the @ escaped
......@@ -206,6 +208,8 @@ BOSSEVENTPORT=2927
UNIFIED_BOSS_AND_OPS=0
FRISEBEEMCASTADDR="234.5.6"
FRISEBEEMCASTPORT=3564
MIN_UNIX_UID=10000
MIN_UNIX_GID=6000
#
# Okay, I know this is improper usage of --with. Too bad.
......@@ -766,7 +770,8 @@ outfiles="$outfiles Makeconf GNUmakefile \
cdrom/GNUmakefile cdrom/tbbootconfig/GNUmakefile \
cdrom/groklilo/GNUmakefile \
dhcpd/dhcpd.conf.template dhcpd/GNUmakefile \
install/GNUmakefile install/ops-install install/boss-install \
install/GNUmakefile \
install/ops-install install/boss-install install/fs-install \
install/newnode_sshkeys/GNUmakefile install/smb.conf.head \
mote/GNUmakefile mote/tbuisp mote/tbsgmotepower mote/newmote \
mote/sgtools/GNUmakefile \
......
......@@ -40,6 +40,8 @@ MAILMANSUPPORT=1
WINSUPPORT=1
SFSSUPPORT=0
CVSSUPPORT=1
MIN_UNIX_UID=10000
MIN_UNIX_GID=6000
#
# SSL Certificate stuff. Used to customize config files in ssl directory.
# Note that OrganizationalUnit is set in the cnf file.
......
......@@ -31,11 +31,15 @@ WWWHOST=changeme
TBMAINSITE=0
THISHOMEBASE=changeme
PLABSUPPORT=0
WIKISUPPORT=0
BUGDBSUPPORT=0
MAILMANSUPPORT=0
WINSUPPORT=0
SFSSUPPORT=0
CVSSUPPORT=0
DISABLE_NSE=1
# This means it is an inner elab!
ELABINELAB=1
WINSUPPORT=0
# The name of the outer boss for inner boss to request services from.
OUTERBOSS_NODENAME=changeme
#
......@@ -54,6 +58,7 @@ TESTBED_NETWORK=changeme
TESTBED_NETMASK=changeme
BOSSNODE_IP=changeme
USERNODE_IP=changeme
FSNODE_IP=changeme
CONTROL_ROUTER_IP=changeme
CONTROL_NETWORK=changeme
CONTROL_NETMASK=changeme
......@@ -69,3 +74,6 @@ NAMED_FORWARDERS=changeme
# Must localize to avoid conflict with outer emulab frisbeed.
#
FRISEBEEMCASTADDR=changeme
# testing
MIN_UNIX_UID=500
MIN_UNIX_GID=500
......@@ -25,6 +25,17 @@ USERNODE=ops.example.emulab.net
# as the ops node)
FSNODE=fs.example.emulab.net
#
# Minimum Unix uid and gid values for Emulab users.
# Emulab will create the initial user with these values and all additional
# users will have values greater than these. It would be unwise to make
# these less than 1000 to avoid conflicts with "standard" BSD and Linux
# users. These can also be tweaked to avoid pre-existing accounts on the
# fileserver machine.
#
MIN_UNIX_UID=10000
MIN_UNIX_GID=6000
#
# Addresses to which email will be sent - These are expected to go to mailing
# lists. You can either host the lists on your ops node, send them off to
......@@ -75,6 +86,14 @@ FSDIR_GROUPS=/groups
FSDIR_PROJ=/q/proj
FSDIR_USERS=/users
FSDIR_SHARE=/share
#
# Filesystems on which quotas should be enforced.
# Note that if multiple of the FSDIR_* vars above are on the same filesystem
# (e.g., /q/proj and /q/users) then you should only specify the base of the
# common filesystem on which they all reside here (e.g., /q).
# Set to the empty string to turn off quota checking.
#
FS_WITH_QUOTAS="/q /groups /users"
#
......
#
# EMULAB-COPYRIGHT
# Copyright (c) 2002-2005 University of Utah and the Flux Group.
# All rights reserved.
#
#####
##### Setting up the Utah Network Testbed software
##### Most recently tested on FreeBSD 4.11.
#####
##### Step 0 - The Goal
In the past we have combined the filesystem function with the ops/users
functions. However, we recognize that people may have pre-existing, dedicated
(and usually very expensive :-) file server machine that they would like to
take advantage of. Or for performance reasons, you may want to separate
the filesystem service from user activities on the ops node.
So we have made a start toward separating the two. The general guiding
principle is to keep modifications to the fileserver box to a minimum,
since the ability to control the box may be greatly reduced; for example,
the fileserver might be a NetApp box or running MacOS X or Windows.
At a bare minimum however, we still require that you be able to manipulate
the access control lists (e.g., the BSD "exports" file) from the boss node.
Optionally, you may need to be able to monitor and manipulate disk quotas.
But that is all theory right now. In the current state of affairs, the
fileserver box still has to run FreeBSD and is assumed to be dedicated to
Emulab use (e.g., the Emulab boss node assigns uid/gids itself, with no
attempt to synchronize with pre-existing uid/gids on the fileserver).
With this caveat in mind, we can begin.
##### Step 1 - OS installation and filesystem setup
Install FreeBSD on the machine you'll be using for your fs node, using the
standard FreeBSD installation process. When asked by the installer, it's best
to choose the 'Developer' distribution set - this gets you full sources. When
it asks if you want to install the ports collection, answer *no*. Do not
install any packages at this time - you'll get a chance to later. You'll
need to partition your filesystems so that you have the proper amount of
space for certain directories - see below for details.
Make sure that you have the network correctly configured.
The following directories will need to exist on partitions that have enough
space to hold them:
/usr - Needs space for the ports tree and a system object tree.
At least 10GB. Be sure to build with plenty of
inodes (the ports tree itself uses about 200000, so
be safe and build with at least a million).
/usr/testbed/ - Needs space for testbed software and logs. For a dedicated
fileserver machine, this won't be very much, 100MB
should be plenty.
/users/ - Needs space for user home directories. Amount of space required
depends on how many users you expect to have.
Generally, though, we suggest that users store large
files related to their projects in the /proj directory.
/proj/ - Needs space for project files. We recommend that this be larger
than /users, to encourage people to store files here,
which aids per-project accountability.
/groups/ - Needs enough space for files shared by the sub-groups of
projects. These are primarily used by classes, if any.
/share/ - Exported read-only to all nodes, we use it for providing to
experimenters the source for the FreeBSD and Linux
versions we run as well as common packages and RPMs.
This could require anything from 1GB to 20GB+ depending
on what you want to make available.
You may want to enforce quotas on the user-writable filesystems. This is the
main reason you'd want to keep them in separate filesystems (i.e., so people
can have different /users/ and /proj/ quotas.) If you do not think you will
ever use quotas, then you could make /users and /proj part of the same
filesystem.
Note also since /share is exported read-only, FreeBSD requires that it be on
a separate filesystem from anything that is exported read-write. So while
/users, /proj and /groups can be on the same filesystem, /share cannot.
Make sure that, no matter how you decide to partition things up, you make
symlinks to the appropriate places. ie., if you make one big filesystem called
/z that has /users, /proj, and /groups in it, make sure you:
ln -s /z/users /users
ln -s /z/proj /proj
... etc.
In other words, we assume the existence of /users, /proj, /group and /share.
Do *not* create any user accounts, Emulab does not require that its users
have login accounts on the fileserver. For the purposes of this setup, just
log in as root. You can manually add login accounts for Emulab admins later
if you desire.
This would be a good time to reiterate that Emulab currently assumes that it
has complete control of the uid/gid namespace. However, it starts assigning
uids at 10000 and gids at 6000, so pre-existing accounts with values lower
than those should be ok. Other accounts will have to be removed or manually
synchronized with Emulab later.
##### Step 2 - Installing packages
To make sure that you're running with versions of software from the ports
collection that are known to work with our software, and to save you hours
of compile time, we provide pre-built binary packages of the ports required
by Emulab.
(Do not let the names of the following tar files bother you, e.g.
"FreeBSD-4.10" or the apparent 20041102 date stamp. These are indeed
the correct files to use with FreeBSD 4.11 and with the current (May 2005)
Emulab distribution.)
Download the packages tarball from:
http://www.emulab.net/downloads/FreeBSD-4.10-20041102.tar.gz
(You can use the FreeBSD 'fetch' command to download the file.)
Now, untar this someplace (you need about 70MB of space, so don't use /tmp).
Let's say you untarred it into /usr/tmp. You would then run:
env PKG_PATH=/usr/tmp/FreeBSD-4.10-20041102 pkg_add emulab-fs-1.4
Of course, if you untarred it somewhere else, put in the correct path.
Now you need to download a ports tree that corresponds to the above
packages. We have run into many, many problems with versions of the
FreeBSD ports. So, rather than using the /usr/ports tree that comes on the
FreeBSD installation media, we use one that we've tested against. You can
grab our 'approved' copy of the ports tree from:
http://www.emulab.net/downloads/ports-20041102.tar.gz
Untar it, move it into place as /usr/ports (rename the old directory to
ports.old, or just remove it). NOTE: You must download and copy the ports
tree into place, even if you do not intend to install any packages yourself.
##### Step 3 - Unpacking and running configure
At this point, you'll need to make a 'defs' file - You will use the same
defs file on boss, ops and fs. See defs-example in the top level directory
as a starting point. You want to make sure each of the variables has a
definition that makes sense in your environment. Feel free to ask Utah if
something is not clear. Of particular note for the fs node is the setting
of FSDIR_* to match the filesystem layout from Step 1, and the setting of
FS_WITH_QUOTAS if you intend to use quotas on any filesystem.
Unpack the testbed source, and run it's configure script. A good place to
unpack the source code is /usr/testbed/src/testbed. You will use the
--with-TBDEFS option to configure to give it the path to your defs file:
mkdir -p /usr/testbed/src/testbed
mkdir -p /usr/testbed/obj/testbed
cd /usr/testbed/src/testbed
tar ....
cd /usr/testbed/obj/testbed
/usr/testbed/src/testbed/configure \
--with-TBDEFS=/path/to/your/defs-file
Typically, you would store your defs file in the source tree along with
the other defs files that came in the tarball.
##### Step 4 - Running the fs installation script
In the object tree you've configured (say, /usr/testbed/obj/testbed), there's
an 'install' subdirectory, with a script called 'fs-install'. Just run this
script as root (note the same package directory argument as above).
cd install
env PKG_PATH=/usr/tmp/FreeBSD-4.10-20041102 perl fs-install
It will take care of installing any additional ports, and doing various
other configuration of FreeBSD required to make it into an fs node. The
script is designed so that you can run it as many times as you want, and
it'll just skip stuff it's already done. If it fails, send the output to
Utah so that we can fix it up. If it succeeds, follow any other
instructions it may have. The script will tell you to reboot the machine,
but you can wait until after you do the next step to do so, if you want.
(You may have to set the executable bit on this script, since configure won't;
we'd like to get this fixed at some point.)
You should be aware that, among other things, this script sets password-less
'sudo' for anyone in the 'wheel' group. If you don't want this for security
reasons, you can undo them after the installation script is finished.
##### Step 5 - Installing from source
To install the actual testbed software, simply do a 'gmake fs-install' in your
object directory.
(Note: If you're logged in as root, /usr/local/bin, where gmake lives on
FreeBSD, may not be in your path.)
##### Step 6 - Quotas (optional)
[ Note that this section is FreeBSD specific. ]
It you are planning to run quotas for Emulab users on the fs node, you
will need to establish a default quota value for all users. Ideally,
the quota would be configurable per-user, but for now all quotas are
initialized from a "prototypical" user. Note that you can change individual
user quotas later by running edquota(8) on the 'fs' node. To establish
the default quota values, you will need a "prototype user" to which to
apply the quotas. You will probably want to add a special user, say
'elabman', for this purpose. The uid and gid for this user should be
the MIN_UNIX_UID and MIN_UNIX_GID values specified in your defs file
(10000 and 6000 by default). Assuming those default values, you would do:
pw useradd elabman -u 10000 -g 6000 -m -d /users/elabman -h - -s /bin/nologin
Now set the quota for that user on each quota-enabled filesystem, e.g.:
edquota -e /proj:2000000:2000000 -e /users:1000000:1000000 elabman
would set a 1GB quota on /proj and 512MB on /users for the prototype user.
Once the prototype user quotas are established, you can do:
edquota -p elabman 10000-15000
which would automatically apply the elabman quotas to any user created with
uids between 10000 and 15000, assuming that you wanted to allow up to 5000
users.
##### Step 7 - Other miscellaneous things to set up
[Nothing at this time]
Once you're done with all of this, reboot fs.
......@@ -9,7 +9,7 @@
##### Most recently tested on FreeBSD 4.11.
#####
##### Step 0 - OS installation and setup
##### Step 1 - OS installation and setup
Install FreeBSD on the machine you'll be using for your ops node, using the
standard FreeBSD installation process. When asked by the installer, it's best
......@@ -30,6 +30,19 @@ space to hold them:
be safe and build with at least a million).
/usr/testbed/ - Needs space for testbed software and logs. Several (3-4) GB
should be enough.
Do *not* create any user accounts yet, and just log in as root for the time being.
Our software will create user accounts later, once you get boss set up. If you
already created any users, then delete them with the "pw" command and make sure
the home directories for them are removed as well.
##### Step 1a - Shared filesystem setup for combined ops/fs machine.
If you have a separate 'fs' node, you should have already setup your shared
filesystems and can skip to Step 2. If this node is also to be your file
server, you will also need:
/users/ - Needs space for user home directories. Amount of space required
depends on how many users you expect to have.
Generally, though, we suggest that users store large
......@@ -46,8 +59,10 @@ space to hold them:
on what you want to make available.
You may want to enforce quotas on the user-writable filesystems. This is the
main reason you'd want to keep them in separate filesystems (i.e., so people can
have different /users/ and /proj/ quotas.)
main reason you'd want to keep them in separate filesystems (i.e., so people
can have different /users/ and /proj/ quotas.) If you do not think you will
ever use quotas, then you could make /users and /proj part of the same
filesystem.
Note also since /share is exported read-only, FreeBSD requires that it be on
a separate filesystem from anything that is exported read-write. So while
......@@ -61,12 +76,10 @@ symlinks to the appropriate places. ie., if you make one big filesystem called
ln -s /z/proj /proj
... etc.
Do *not* create any user accounts yet, and just log in as root for the time being.
Our software will create user accounts later, once you get boss set up. If you
already created any users, then delete them with the "pw" command and make sure
the home directories for them are removed as well.
In other words, we assume the existence of /users, /proj, /group and /share.
##### Step 1 - Installing packages
##### Step 2 - Installing packages
To make sure that you're running with versions of software from the ports
collection that are known to work with our software, and to save you hours
......@@ -105,10 +118,10 @@ ops feel like 'home' (like emacs, jove, or whatever). NOTE: You must
download and copy the ports tree into place, even if you do not intend to
install any packages yourself.
##### Step 2 - Unpacking and running configure
##### Step 3 - Unpacking and running configure
At this point, you'll need to make a 'defs' file - You will use the same
defs file on boss and ops. See defs-example in the top level directory
defs file on boss, ops and fs. See defs-example in the top level directory
as a starting point. You want to make sure each of the variables has a
definition that makes sense in your environment. Feel free to ask Utah if
something is not clear.
......@@ -128,7 +141,7 @@ unpack the source code is /usr/testbed/src/testbed. You will use the
Typically, you would store your defs file in the source tree along with
the other defs files that came in the tarball.
##### Step 3 - Running the ops installation script
##### Step 4 - Running the ops installation script
In the object tree you've configured (say, /usr/testbed/obj/testbed), there's
an 'install' subdirectory, with a script called 'ops-install'. Just run this
......@@ -153,7 +166,7 @@ sets up password-less 'sudo' for anyone in the 'wheel' group. If you don't want
these for security reasons, you can undo them after the installation script is
finished.
##### Step 4 - Installing from source
##### Step 5 - Installing from source
To install the actual testbed software, simply do a 'gmake ops-install' in your
object directory.
......@@ -167,17 +180,17 @@ addition, you'll need to build a custom kernel with the appropriate driver (you
may even need to get patches from Utah for some boards). You'll also need to
set up a few things (not yet automated), described in doc/adding_nodes.txt .
##### Step 5 - Setting up mailing lists
##### Step 6 - Setting up mailing lists
Optional. The ops-install script set up some mailing lists for the email
addresses you set up in the defs file, in /etc/mail/lists . If you want to run
these mailing lists off your ops node, you can put some people in them now.
These are just standard sendmail list files: a list of addresses and/or
archive files, one per line. If you chose to use archive files, give a path
to the file you'd like sendmail to stick all correspondance in, and create
to the file you'd like sendmail to stick all correspondence in, and create
this file.
##### Step 6 - Other miscellaneous things to set up
##### Step 7 - Other miscellaneous things to set up
[Nothing at this time]
......
......@@ -5,7 +5,7 @@
#
#####
##### Setting up the Utah Network Testbed software on a boss node
##### Setting up the Utah Network Testbed software.
##### Tested on FreeBSD 4.11
#####
......@@ -30,11 +30,11 @@ financial, security, and support tradeoffs.
3. To be able to help you debug any problems you run into or answer
certain questions, we'll need to have accounts, preferably with root
access if allowed by your institution's AUP, on your Emulab's two
servers ('boss' and 'ops'), and will need to be able to access the
webserver on boss. This is crucial during testbed installation and
bringup; after that it's not so important, except when you are
upgrading to a new version of our software.
access if allowed by your institution's AUP, on your Emulab's servers
('boss', 'ops', and possibly 'fs' if you separate the ops/fs functions),
and will need to be able to access the webserver on boss. This is crucial
during testbed installation and bringup; after that it's not so important,
except when you are upgrading to a new version of our software.
4. Serial line consoles and remote power controllers must be connected
to at least two experimental nodes, so we can help debug.
......@@ -47,14 +47,20 @@ the resources to adapt it to every possible environment. So, you will need to
either work out a way to match the environment outlined below, or be willing to
invest some work in adapting the software.
(1) You will need at least two network interfaces on each node: one for the
control network and one for the experimental network. The experimental network
needs to be one on which we can make VLANs with SNMP. Currently, Emulab supports
(0) We assume a minimum of two dedicated server machines, known hereafter
as 'boss' and 'ops', for Emulab setup. While the 'ops' server normally
hosts the Emulab shared filesystems, we also support the use of a separate,
dedicated filesystem server, hereafter referred to as 'fs'.
(1) You will need at least two network interfaces on each experimental node:
one for the control network and one for the experimental network. The
experimental network needs to be one on which we can make VLANs with SNMP.
Currently, Emulab supports
* Cisco 6500 and 4000 series switches (though not all switches in these lines
have been tested - the 6513, 6509, 6506, 4006, and 4506 are known to work,
running both CatOS and IOS). Cisco 3750's probably work but have not been
tested. The 2950 and 2980 switches are known to work, though they are limited
to a small numer (64) of VLANs.
to a small number (64) of VLANs.
* many Nortel switches (those with RAPID-CITY Nortel mibs, ie, 5510-24T 5510-48T
5520-24T 5520-48T, and reasonably recent Accellars).
* some Foundry switches
......@@ -66,17 +72,17 @@ Emulab's Cisco support is the best, because those are the switches we have.
The control net must have full multicast support, including IGMP
snooping. Nodes' control network interfaces must support PXE.
(2) We highly, highly recommend that boss, ops, and all the nodes be in
publicly routed IP space. If this is not possible, then boss and ops should be
given two interfaces: One in the nodes' control network, and one in public IP
space. If you must use private IP space for the nodes' control net, we suggest
using the 192.168/16 subnet, which leaves the larger 10/8 subnet available for
the experimental network.
(2) We highly, highly recommend that boss, ops, fs, and all the nodes be in
publicly routed IP space. If this is not possible, then boss, ops and fs
should be given two interfaces: One in the nodes' control network, and one in
public IP space. If you must use private IP space for the nodes' control net,
we suggest using the 192.168/16 subnet, which leaves the larger 10/8 subnet
available for the experimental network.
(3) If you have a firewall, you will need to be able to get certain standard
ports through to boss and ops, such as the ports for http, https, ssh, named
(domain), and smtp. Any other strange network setup (such as NAT) between the
boss/ops and the outside world will cause really big headaches.
boss/ops/fs and the outside world will cause really big headaches.
(4) The whole testbed should be in a domain or subdomain for which boss can be
the name server.
......@@ -88,21 +94,26 @@ DHCP/BOOTP packets. Since the nodes will DHCP from boss, it is important that
there not be another DHCP server (ie. one for another part of your lab)
answering their requests.
(6) Boss and ops should have their own local disk space - in particular, the
/usr/testbed directory cannot be shared between them. It may be possible to
use an external machines (other than ops) as a fileserver - talk to Utah abut
this if you'd like to try.
(6) The boss node should have its own local disk space, for various reasons:
* For logistical reasons, /usr/testbed cannot be shared between boss and
ops, or between boss and fs, or between ops and fs. All three install
different (and alas, sometimes conflicting) subsets of the Emulab software.
* For security reasons, /usr/testbed/images, which is the home of the
"trusted" default disk images, should not be hosted on ops/fs since they
are potentially more vulnerable.
* Similarly, home directories for "real" (admin) users on boss should not be
shared with, or hosted from, ops and fs. See shellonboss.txt for details.
##### Other docs:
Useful summary info and diagrams can be found in "build-operate.ppt" and
"security-all.ppt" in http://www.cs.utah.edu/flux/testbed-docs/internals/
##### Step -1 - Set up "ops"
##### Step 1 - Set up "fs" and "ops"
Follow the instructions in the setup-ops.txt file before the ones in this file!
##### Step 0 - OS installation and setup
##### Step 2 - OS installation and setup
Install FreeBSD on the machine you'll be using for your boss node, using the
standard FreeBSD installation process. When asked by the installer, it's best
......@@ -129,7 +140,7 @@ set the root password use "passwd root".
If you already created users, then delete them with the "pw" command
and make sure the home directories for them are removed as well!
##### Step 1 - Installing packages
##### Step 3 - Installing packages
Again, almost the same as on ops. Download the same tarball, and follow
the same pkg_add procedure, except this time, you're going to install
......@@ -137,10 +148,10 @@ the emulab-boss-1.8 package instead of emulab-ops:
env PKG_PATH=/usr/tmp/FreeBSD-4.10-20041102 pkg_add emulab-boss-1.8
Also grab a copy og our approved ports tree and install it, the same as
Also grab a copy of our approved ports tree and install it, the same as
described in setup-ops.txt.
##### Step 2 - Unpacking source and creating a defs file
##### Step 4 - Unpacking source and creating a defs file
Unpack the source tarball:
......@@ -150,13 +161,13 @@ Unpack the source tarball:
tar ....
Now, you'll need to create a 'defs file', which is used by the configure
script to describe your enviroment, such as the hostnames of your boss and ops
script to describe your environment, such as the hostnames of your boss and ops
nodes, and email addresses that certain types of mail will be sent to.
Use the 'defs-example' file in the root of our source distribution as a
template. It contains comments explaining the important variables to set.
##### Step 3 - Configuring an object tree
##### Step 5 - Configuring an object tree
This works the same as it did on ops. Remember, you have to use the same
defs file on boss that you used on ops!
......@@ -166,7 +177,7 @@ defs file on boss that you used on ops!
/usr/testbed/src/testbed/configure \
--with-TBDEFS=/path/to/your/defs-file
##### Step 4 - Running the boss installation script
##### Step 6 - Running the boss installation script
Again, this works the same as it did on ops, except that you run
boss-install in the object tree, instead of ops-install. Just run this
......@@ -178,13 +189,13 @@ script as root (note the same package directory argument as above).
Like the ops-install script, boss-install sets up passwordless sudo for
anyone in the wheel group.
##### Step 5 - Installing from source.
##### Step 7 - Installing from source.
In your object directory, do a 'gmake && gmake boss-install'. Then, as root, do
a 'gmake post-install'. The post-install target needs to run as root, so that
it can make certain scripts setuid, etc.
##### Step 6 - Setting up root ssh from boss to ops
##### Step 8 - Setting up root ssh from boss to ops
This step is now done as part of boss-install/ops-install. To confirm
this, make sure this works:
......@@ -195,7 +206,7 @@ If this *FAILS*, you will need to do this by hand, typing a password:
scp /root/.ssh/identity.pub ops:/root/.ssh/authorized_keys
##### Step 7 - Setting up named
##### Step 9 - Setting up named
The testbed software manipulates DNS zone files for two reasons. First, it
adds your nodes to them so that you don't have to. Second, it creates CNAMEs
......@@ -257,7 +268,7 @@ basically, you'll want to put the public addresses for boss, ops, etc. into
OURDOMAIN.db.head, and their private addresses into
OURDOMAIN.internal.db.head .
##### Step 8 - Other miscellaneous things to set up
##### Step 10 - Other miscellaneous things to set up
There are a few things we haven't been able to completely automate just yet,
though we hope to soon.
......@@ -277,7 +288,7 @@ in:
(where <sitename> is OURDOMAIN from the configure defs file, which defaults
to boss's domain name).
boss-install already generated a temporary no-passhrase certificate for you
boss-install already generated a temporary no-passphrase certificate for you
and placed them in the locations specified above. However, we recommend
that you get a "real" certificate from Verisign (or one of the others).