Commit c53d5827 authored by Mike Hibler's avatar Mike Hibler

Changes related to allowing seperate 'fs' (file server) node.

Entailed new instructions for manual setup as well as integration into
elabinelab framework.  First, the manual path:

setup.txt, setup-boss.txt, setup-ops.txt and new setup-fs.txt:
    Updated to reflect potential for separate fs node.  The org here
    is a little dicey and could be confusing with ops+fs vs. ops and fs.
    Has not been field tested yet.

*/GNUmakefile.in: new fs-install target.

configure, configure.in, defs-*:
    Somewhat unrelated, make min uid/gid to use be a defs setting.
    Also add config of fs-install.in script.

boss-install.in, ops-install.in and new fs-install.in:
    Handle distinct fs node.  If you have one, fs-install is run before
    ops-install.  All scripts rely on the defs file settings of FSNODE
    and USERNODE to determine if the fs node is seperate.

utils/checkquota.in:
    Just return "ok" if quotas are not used (i.e., if defs file FS_WITH_QUOTA
    string is null.

install/ports/emulab-fs:
    Meta port for fs node specific stuff.  Also a patch for the samba port
    Makefile so it doesn't drag in CUPs, etc.  Note that the current samba
    port Makefile has this change, I am just backporting to our version.

Elabinelab specific changes:

elabinelab-withfs.ns:
    NS fragment used in conjunction with
	tb-elab-in-elab-topology "withfs"
    to setup inner-elab with fs node.

elabinelab.ns:
    The hard work on the boss side.  Recognize seperate-fs config and handle
    running of rc.mkelab on that node.  fs setup happens before ops setup.

rc.mkelab:
    The hard work on the client side.  Recognize FsNode setup as well as
    differentiate ops+fs from ops setup.

Related stuff either not part of the repo or checked in previously:
    emulab-fs package
parent 317c82a5
...@@ -31,7 +31,8 @@ SUBDIRS = lib db assign www @optional_subdirs@ ipod security sensors \ ...@@ -31,7 +31,8 @@ SUBDIRS = lib db assign www @optional_subdirs@ ipod security sensors \
all: all-subdirs all: all-subdirs
install: install:
@echo "Choose either boss-install (paper) or ops-install (plastic)" @echo "Choose either boss-install, ops-install or fs-install"
@echo "Choose opsfs-install instead of ops-install for a combined ops/fs machine"
@echo "Choose tipserv-install for dedicated tip server machines." @echo "Choose tipserv-install for dedicated tip server machines."
# #
...@@ -89,6 +90,13 @@ endif ...@@ -89,6 +90,13 @@ endif
@$(MAKE) -C tmcd control-install @$(MAKE) -C tmcd control-install
@$(MAKE) -C account control-install @$(MAKE) -C account control-install
fs-install:
@$(MAKE) -C tbsetup fs-install
@$(MAKE) -C account fs-install
opsfs-install: ops-install fs-install
@echo "Combined ops/fs install done."
install-mkdirs: install-mkdirs:
-mkdir -p $(INSTALL_TOPDIR)/opsdir -mkdir -p $(INSTALL_TOPDIR)/opsdir
-mkdir -p $(INSTALL_TOPDIR)/locks -mkdir -p $(INSTALL_TOPDIR)/locks
......
...@@ -14,7 +14,7 @@ include $(OBJDIR)/Makeconf ...@@ -14,7 +14,7 @@ include $(OBJDIR)/Makeconf
SBIN_STUFF = tbacct addsfskey addpubkey mkusercert SBIN_STUFF = tbacct addsfskey addpubkey mkusercert
LIBEXEC_STUFF = webtbacct webaddsfskey webaddpubkey webmkusercert LIBEXEC_STUFF = webtbacct webaddsfskey webaddpubkey webmkusercert
FS_SBIN_STUFF = quotamail FSBIN_STUFF = quotamail
# #
# Force dependencies on the scripts so that they will be rerun through # Force dependencies on the scripts so that they will be rerun through
...@@ -42,10 +42,9 @@ post-install: ...@@ -42,10 +42,9 @@ post-install:
chown root $(INSTALL_SBINDIR)/mkusercert chown root $(INSTALL_SBINDIR)/mkusercert
chmod u+s $(INSTALL_SBINDIR)/mkusercert chmod u+s $(INSTALL_SBINDIR)/mkusercert
# XXX ops == fs right now control-install:
control-install: fs-install
fs-script-install: $(addprefix $(INSTALL_SBINDIR)/, $(FS_SBIN_STUFF)) fs-script-install: $(addprefix $(INSTALL_SBINDIR)/, $(FSBIN_STUFF))
fs-install: fs-script-install fs-install: fs-script-install
......
...@@ -1365,6 +1365,8 @@ done ...@@ -1365,6 +1365,8 @@ done
...@@ -1425,6 +1427,8 @@ BOSSEVENTPORT=2927 ...@@ -1425,6 +1427,8 @@ BOSSEVENTPORT=2927
UNIFIED_BOSS_AND_OPS=0 UNIFIED_BOSS_AND_OPS=0
FRISEBEEMCASTADDR="234.5.6" FRISEBEEMCASTADDR="234.5.6"
FRISEBEEMCASTPORT=3564 FRISEBEEMCASTPORT=3564
MIN_UNIX_UID=10000
MIN_UNIX_GID=6000
# #
# Okay, I know this is improper usage of --with. Too bad. # Okay, I know this is improper usage of --with. Too bad.
...@@ -1925,17 +1929,17 @@ for ac_hdr in ulxmlrpcpp/ulxr_config.h ...@@ -1925,17 +1929,17 @@ for ac_hdr in ulxmlrpcpp/ulxr_config.h
do do
ac_safe=`echo "$ac_hdr" | sed 'y%./+-%__p_%'` ac_safe=`echo "$ac_hdr" | sed 'y%./+-%__p_%'`
echo $ac_n "checking for $ac_hdr""... $ac_c" 1>&6 echo $ac_n "checking for $ac_hdr""... $ac_c" 1>&6
echo "configure:1929: checking for $ac_hdr" >&5 echo "configure:1933: checking for $ac_hdr" >&5
if eval "test \"`echo '$''{'ac_cv_header_$ac_safe'+set}'`\" = set"; then if eval "test \"`echo '$''{'ac_cv_header_$ac_safe'+set}'`\" = set"; then
echo $ac_n "(cached) $ac_c" 1>&6 echo $ac_n "(cached) $ac_c" 1>&6
else else
cat > conftest.$ac_ext <<EOF cat > conftest.$ac_ext <<EOF
#line 1934 "configure" #line 1938 "configure"
#include "confdefs.h" #include "confdefs.h"
#include <$ac_hdr> #include <$ac_hdr>
EOF EOF
ac_try="$ac_cpp conftest.$ac_ext >/dev/null 2>conftest.out" ac_try="$ac_cpp conftest.$ac_ext >/dev/null 2>conftest.out"
{ (eval echo configure:1939: \"$ac_try\") 1>&5; (eval $ac_try) 2>&5; } { (eval echo configure:1943: \"$ac_try\") 1>&5; (eval $ac_try) 2>&5; }
ac_err=`grep -v '^ *+' conftest.out | grep -v "^conftest.${ac_ext}\$"` ac_err=`grep -v '^ *+' conftest.out | grep -v "^conftest.${ac_ext}\$"`
if test -z "$ac_err"; then if test -z "$ac_err"; then
rm -rf conftest* rm -rf conftest*
...@@ -1974,17 +1978,17 @@ for ac_hdr in linux/videodev.h ...@@ -1974,17 +1978,17 @@ for ac_hdr in linux/videodev.h
do do
ac_safe=`echo "$ac_hdr" | sed 'y%./+-%__p_%'` ac_safe=`echo "$ac_hdr" | sed 'y%./+-%__p_%'`
echo $ac_n "checking for $ac_hdr""... $ac_c" 1>&6 echo $ac_n "checking for $ac_hdr""... $ac_c" 1>&6
echo "configure:1978: checking for $ac_hdr" >&5 echo "configure:1982: checking for $ac_hdr" >&5
if eval "test \"`echo '$''{'ac_cv_header_$ac_safe'+set}'`\" = set"; then if eval "test \"`echo '$''{'ac_cv_header_$ac_safe'+set}'`\" = set"; then
echo $ac_n "(cached) $ac_c" 1>&6 echo $ac_n "(cached) $ac_c" 1>&6
else else
cat > conftest.$ac_ext <<EOF cat > conftest.$ac_ext <<EOF
#line 1983 "configure" #line 1987 "configure"
#include "confdefs.h" #include "confdefs.h"
#include <$ac_hdr> #include <$ac_hdr>
EOF EOF
ac_try="$ac_cpp conftest.$ac_ext >/dev/null 2>conftest.out" ac_try="$ac_cpp conftest.$ac_ext >/dev/null 2>conftest.out"
{ (eval echo configure:1988: \"$ac_try\") 1>&5; (eval $ac_try) 2>&5; } { (eval echo configure:1992: \"$ac_try\") 1>&5; (eval $ac_try) 2>&5; }
ac_err=`grep -v '^ *+' conftest.out | grep -v "^conftest.${ac_ext}\$"` ac_err=`grep -v '^ *+' conftest.out | grep -v "^conftest.${ac_ext}\$"`
if test -z "$ac_err"; then if test -z "$ac_err"; then
rm -rf conftest* rm -rf conftest*
...@@ -2017,7 +2021,7 @@ done ...@@ -2017,7 +2021,7 @@ done
# Extract the first word of "gtk-config", so it can be a program name with args. # Extract the first word of "gtk-config", so it can be a program name with args.
set dummy gtk-config; ac_word=$2 set dummy gtk-config; ac_word=$2
echo $ac_n "checking for $ac_word""... $ac_c" 1>&6 echo $ac_n "checking for $ac_word""... $ac_c" 1>&6
echo "configure:2021: checking for $ac_word" >&5 echo "configure:2025: checking for $ac_word" >&5
if eval "test \"`echo '$''{'ac_cv_prog_GTK_CONFIG'+set}'`\" = set"; then if eval "test \"`echo '$''{'ac_cv_prog_GTK_CONFIG'+set}'`\" = set"; then
echo $ac_n "(cached) $ac_c" 1>&6 echo $ac_n "(cached) $ac_c" 1>&6
else else
...@@ -2096,7 +2100,7 @@ fi ...@@ -2096,7 +2100,7 @@ fi
# SVR4 /usr/ucb/install, which tries to use the nonexistent group "staff" # SVR4 /usr/ucb/install, which tries to use the nonexistent group "staff"
# ./install, which can be erroneously created by make from ./install.sh. # ./install, which can be erroneously created by make from ./install.sh.
echo $ac_n "checking for a BSD compatible install""... $ac_c" 1>&6 echo $ac_n "checking for a BSD compatible install""... $ac_c" 1>&6
echo "configure:2100: checking for a BSD compatible install" >&5 echo "configure:2104: checking for a BSD compatible install" >&5
if test -z "$INSTALL"; then if test -z "$INSTALL"; then
if eval "test \"`echo '$''{'ac_cv_path_install'+set}'`\" = set"; then if eval "test \"`echo '$''{'ac_cv_path_install'+set}'`\" = set"; then
echo $ac_n "(cached) $ac_c" 1>&6 echo $ac_n "(cached) $ac_c" 1>&6
...@@ -2157,7 +2161,7 @@ esac ...@@ -2157,7 +2161,7 @@ esac
# Extract the first word of "rsync", so it can be a program name with args. # Extract the first word of "rsync", so it can be a program name with args.
set dummy rsync; ac_word=$2 set dummy rsync; ac_word=$2
echo $ac_n "checking for $ac_word""... $ac_c" 1>&6 echo $ac_n "checking for $ac_word""... $ac_c" 1>&6
echo "configure:2161: checking for $ac_word" >&5 echo "configure:2165: checking for $ac_word" >&5
if eval "test \"`echo '$''{'ac_cv_path_RSYNC'+set}'`\" = set"; then if eval "test \"`echo '$''{'ac_cv_path_RSYNC'+set}'`\" = set"; then
echo $ac_n "(cached) $ac_c" 1>&6 echo $ac_n "(cached) $ac_c" 1>&6
else else
...@@ -2328,7 +2332,8 @@ outfiles="$outfiles Makeconf GNUmakefile \ ...@@ -2328,7 +2332,8 @@ outfiles="$outfiles Makeconf GNUmakefile \
cdrom/GNUmakefile cdrom/tbbootconfig/GNUmakefile \ cdrom/GNUmakefile cdrom/tbbootconfig/GNUmakefile \
cdrom/groklilo/GNUmakefile \ cdrom/groklilo/GNUmakefile \
dhcpd/dhcpd.conf.template dhcpd/GNUmakefile \ dhcpd/dhcpd.conf.template dhcpd/GNUmakefile \
install/GNUmakefile install/ops-install install/boss-install \ install/GNUmakefile \
install/ops-install install/boss-install install/fs-install \
install/newnode_sshkeys/GNUmakefile install/smb.conf.head \ install/newnode_sshkeys/GNUmakefile install/smb.conf.head \
mote/GNUmakefile mote/tbuisp mote/tbsgmotepower mote/newmote \ mote/GNUmakefile mote/tbuisp mote/tbsgmotepower mote/newmote \
mote/sgtools/GNUmakefile \ mote/sgtools/GNUmakefile \
...@@ -2605,6 +2610,8 @@ s%@FRISEBEEMCASTADDR@%$FRISEBEEMCASTADDR%g ...@@ -2605,6 +2610,8 @@ s%@FRISEBEEMCASTADDR@%$FRISEBEEMCASTADDR%g
s%@FRISEBEEMCASTPORT@%$FRISEBEEMCASTPORT%g s%@FRISEBEEMCASTPORT@%$FRISEBEEMCASTPORT%g
s%@WINSUPPORT@%$WINSUPPORT%g s%@WINSUPPORT@%$WINSUPPORT%g
s%@CVSSUPPORT@%$CVSSUPPORT%g s%@CVSSUPPORT@%$CVSSUPPORT%g
s%@MIN_UNIX_UID@%$MIN_UNIX_UID%g
s%@MIN_UNIX_GID@%$MIN_UNIX_GID%g
s%@TBOPSEMAIL@%$TBOPSEMAIL%g s%@TBOPSEMAIL@%$TBOPSEMAIL%g
s%@TBOPSEMAIL_NOSLASH@%$TBOPSEMAIL_NOSLASH%g s%@TBOPSEMAIL_NOSLASH@%$TBOPSEMAIL_NOSLASH%g
s%@TBLOGSEMAIL@%$TBLOGSEMAIL%g s%@TBLOGSEMAIL@%$TBLOGSEMAIL%g
......
...@@ -148,6 +148,8 @@ AC_SUBST(FRISEBEEMCASTADDR) ...@@ -148,6 +148,8 @@ AC_SUBST(FRISEBEEMCASTADDR)
AC_SUBST(FRISEBEEMCASTPORT) AC_SUBST(FRISEBEEMCASTPORT)
AC_SUBST(WINSUPPORT) AC_SUBST(WINSUPPORT)
AC_SUBST(CVSSUPPORT) AC_SUBST(CVSSUPPORT)
AC_SUBST(MIN_UNIX_UID)
AC_SUBST(MIN_UNIX_GID)
# #
# Offer both versions of the email addresses that have the @ escaped # Offer both versions of the email addresses that have the @ escaped
...@@ -206,6 +208,8 @@ BOSSEVENTPORT=2927 ...@@ -206,6 +208,8 @@ BOSSEVENTPORT=2927
UNIFIED_BOSS_AND_OPS=0 UNIFIED_BOSS_AND_OPS=0
FRISEBEEMCASTADDR="234.5.6" FRISEBEEMCASTADDR="234.5.6"
FRISEBEEMCASTPORT=3564 FRISEBEEMCASTPORT=3564
MIN_UNIX_UID=10000
MIN_UNIX_GID=6000
# #
# Okay, I know this is improper usage of --with. Too bad. # Okay, I know this is improper usage of --with. Too bad.
...@@ -766,7 +770,8 @@ outfiles="$outfiles Makeconf GNUmakefile \ ...@@ -766,7 +770,8 @@ outfiles="$outfiles Makeconf GNUmakefile \
cdrom/GNUmakefile cdrom/tbbootconfig/GNUmakefile \ cdrom/GNUmakefile cdrom/tbbootconfig/GNUmakefile \
cdrom/groklilo/GNUmakefile \ cdrom/groklilo/GNUmakefile \
dhcpd/dhcpd.conf.template dhcpd/GNUmakefile \ dhcpd/dhcpd.conf.template dhcpd/GNUmakefile \
install/GNUmakefile install/ops-install install/boss-install \ install/GNUmakefile \
install/ops-install install/boss-install install/fs-install \
install/newnode_sshkeys/GNUmakefile install/smb.conf.head \ install/newnode_sshkeys/GNUmakefile install/smb.conf.head \
mote/GNUmakefile mote/tbuisp mote/tbsgmotepower mote/newmote \ mote/GNUmakefile mote/tbuisp mote/tbsgmotepower mote/newmote \
mote/sgtools/GNUmakefile \ mote/sgtools/GNUmakefile \
......
...@@ -40,6 +40,8 @@ MAILMANSUPPORT=1 ...@@ -40,6 +40,8 @@ MAILMANSUPPORT=1
WINSUPPORT=1 WINSUPPORT=1
SFSSUPPORT=0 SFSSUPPORT=0
CVSSUPPORT=1 CVSSUPPORT=1
MIN_UNIX_UID=10000
MIN_UNIX_GID=6000
# #
# SSL Certificate stuff. Used to customize config files in ssl directory. # SSL Certificate stuff. Used to customize config files in ssl directory.
# Note that OrganizationalUnit is set in the cnf file. # Note that OrganizationalUnit is set in the cnf file.
......
...@@ -31,11 +31,15 @@ WWWHOST=changeme ...@@ -31,11 +31,15 @@ WWWHOST=changeme
TBMAINSITE=0 TBMAINSITE=0
THISHOMEBASE=changeme THISHOMEBASE=changeme
PLABSUPPORT=0 PLABSUPPORT=0
WIKISUPPORT=0
BUGDBSUPPORT=0
MAILMANSUPPORT=0
WINSUPPORT=0
SFSSUPPORT=0 SFSSUPPORT=0
CVSSUPPORT=0
DISABLE_NSE=1 DISABLE_NSE=1
# This means it is an inner elab! # This means it is an inner elab!
ELABINELAB=1 ELABINELAB=1
WINSUPPORT=0
# The name of the outer boss for inner boss to request services from. # The name of the outer boss for inner boss to request services from.
OUTERBOSS_NODENAME=changeme OUTERBOSS_NODENAME=changeme
# #
...@@ -54,6 +58,7 @@ TESTBED_NETWORK=changeme ...@@ -54,6 +58,7 @@ TESTBED_NETWORK=changeme
TESTBED_NETMASK=changeme TESTBED_NETMASK=changeme
BOSSNODE_IP=changeme BOSSNODE_IP=changeme
USERNODE_IP=changeme USERNODE_IP=changeme
FSNODE_IP=changeme
CONTROL_ROUTER_IP=changeme CONTROL_ROUTER_IP=changeme
CONTROL_NETWORK=changeme CONTROL_NETWORK=changeme
CONTROL_NETMASK=changeme CONTROL_NETMASK=changeme
...@@ -69,3 +74,6 @@ NAMED_FORWARDERS=changeme ...@@ -69,3 +74,6 @@ NAMED_FORWARDERS=changeme
# Must localize to avoid conflict with outer emulab frisbeed. # Must localize to avoid conflict with outer emulab frisbeed.
# #
FRISEBEEMCASTADDR=changeme FRISEBEEMCASTADDR=changeme
# testing
MIN_UNIX_UID=500
MIN_UNIX_GID=500
...@@ -25,6 +25,17 @@ USERNODE=ops.example.emulab.net ...@@ -25,6 +25,17 @@ USERNODE=ops.example.emulab.net
# as the ops node) # as the ops node)
FSNODE=fs.example.emulab.net FSNODE=fs.example.emulab.net
#
# Minimum Unix uid and gid values for Emulab users.
# Emulab will create the initial user with these values and all additional
# users will have values greater than these. It would be unwise to make
# these less than 1000 to avoid conflicts with "standard" BSD and Linux
# users. These can also be tweaked to avoid pre-existing accounts on the
# fileserver machine.
#
MIN_UNIX_UID=10000
MIN_UNIX_GID=6000
# #
# Addresses to which email will be sent - These are expected to go to mailing # Addresses to which email will be sent - These are expected to go to mailing
# lists. You can either host the lists on your ops node, send them off to # lists. You can either host the lists on your ops node, send them off to
...@@ -75,6 +86,14 @@ FSDIR_GROUPS=/groups ...@@ -75,6 +86,14 @@ FSDIR_GROUPS=/groups
FSDIR_PROJ=/q/proj FSDIR_PROJ=/q/proj
FSDIR_USERS=/users FSDIR_USERS=/users
FSDIR_SHARE=/share FSDIR_SHARE=/share
#
# Filesystems on which quotas should be enforced.
# Note that if multiple of the FSDIR_* vars above are on the same filesystem
# (e.g., /q/proj and /q/users) then you should only specify the base of the
# common filesystem on which they all reside here (e.g., /q).
# Set to the empty string to turn off quota checking.
#
FS_WITH_QUOTAS="/q /groups /users" FS_WITH_QUOTAS="/q /groups /users"
# #
......
#
# EMULAB-COPYRIGHT
# Copyright (c) 2002-2005 University of Utah and the Flux Group.
# All rights reserved.
#
#####
##### Setting up the Utah Network Testbed software
##### Most recently tested on FreeBSD 4.11.
#####
##### Step 0 - The Goal
In the past we have combined the filesystem function with the ops/users
functions. However, we recognize that people may have pre-existing, dedicated
(and usually very expensive :-) file server machine that they would like to
take advantage of. Or for performance reasons, you may want to separate
the filesystem service from user activities on the ops node.
So we have made a start toward separating the two. The general guiding
principle is to keep modifications to the fileserver box to a minimum,
since the ability to control the box may be greatly reduced; for example,
the fileserver might be a NetApp box or running MacOS X or Windows.
At a bare minimum however, we still require that you be able to manipulate
the access control lists (e.g., the BSD "exports" file) from the boss node.
Optionally, you may need to be able to monitor and manipulate disk quotas.
But that is all theory right now. In the current state of affairs, the
fileserver box still has to run FreeBSD and is assumed to be dedicated to
Emulab use (e.g., the Emulab boss node assigns uid/gids itself, with no
attempt to synchronize with pre-existing uid/gids on the fileserver).
With this caveat in mind, we can begin.
##### Step 1 - OS installation and filesystem setup
Install FreeBSD on the machine you'll be using for your fs node, using the
standard FreeBSD installation process. When asked by the installer, it's best
to choose the 'Developer' distribution set - this gets you full sources. When
it asks if you want to install the ports collection, answer *no*. Do not
install any packages at this time - you'll get a chance to later. You'll
need to partition your filesystems so that you have the proper amount of
space for certain directories - see below for details.
Make sure that you have the network correctly configured.
The following directories will need to exist on partitions that have enough
space to hold them:
/usr - Needs space for the ports tree and a system object tree.
At least 10GB. Be sure to build with plenty of
inodes (the ports tree itself uses about 200000, so
be safe and build with at least a million).
/usr/testbed/ - Needs space for testbed software and logs. For a dedicated
fileserver machine, this won't be very much, 100MB
should be plenty.
/users/ - Needs space for user home directories. Amount of space required
depends on how many users you expect to have.
Generally, though, we suggest that users store large
files related to their projects in the /proj directory.
/proj/ - Needs space for project files. We recommend that this be larger
than /users, to encourage people to store files here,
which aids per-project accountability.
/groups/ - Needs enough space for files shared by the sub-groups of
projects. These are primarily used by classes, if any.
/share/ - Exported read-only to all nodes, we use it for providing to
experimenters the source for the FreeBSD and Linux
versions we run as well as common packages and RPMs.
This could require anything from 1GB to 20GB+ depending
on what you want to make available.
You may want to enforce quotas on the user-writable filesystems. This is the
main reason you'd want to keep them in separate filesystems (i.e., so people
can have different /users/ and /proj/ quotas.) If you do not think you will
ever use quotas, then you could make /users and /proj part of the same
filesystem.
Note also since /share is exported read-only, FreeBSD requires that it be on
a separate filesystem from anything that is exported read-write. So while
/users, /proj and /groups can be on the same filesystem, /share cannot.
Make sure that, no matter how you decide to partition things up, you make
symlinks to the appropriate places. ie., if you make one big filesystem called
/z that has /users, /proj, and /groups in it, make sure you:
ln -s /z/users /users
ln -s /z/proj /proj
... etc.
In other words, we assume the existence of /users, /proj, /group and /share.
Do *not* create any user accounts, Emulab does not require that its users
have login accounts on the fileserver. For the purposes of this setup, just
log in as root. You can manually add login accounts for Emulab admins later
if you desire.
This would be a good time to reiterate that Emulab currently assumes that it
has complete control of the uid/gid namespace. However, it starts assigning
uids at 10000 and gids at 6000, so pre-existing accounts with values lower
than those should be ok. Other accounts will have to be removed or manually
synchronized with Emulab later.
##### Step 2 - Installing packages
To make sure that you're running with versions of software from the ports
collection that are known to work with our software, and to save you hours
of compile time, we provide pre-built binary packages of the ports required
by Emulab.
(Do not let the names of the following tar files bother you, e.g.
"FreeBSD-4.10" or the apparent 20041102 date stamp. These are indeed
the correct files to use with FreeBSD 4.11 and with the current (May 2005)
Emulab distribution.)
Download the packages tarball from:
http://www.emulab.net/downloads/FreeBSD-4.10-20041102.tar.gz
(You can use the FreeBSD 'fetch' command to download the file.)
Now, untar this someplace (you need about 70MB of space, so don't use /tmp).
Let's say you untarred it into /usr/tmp. You would then run:
env PKG_PATH=/usr/tmp/FreeBSD-4.10-20041102 pkg_add emulab-fs-1.4
Of course, if you untarred it somewhere else, put in the correct path.
Now you need to download a ports tree that corresponds to the above
packages. We have run into many, many problems with versions of the
FreeBSD ports. So, rather than using the /usr/ports tree that comes on the
FreeBSD installation media, we use one that we've tested against. You can
grab our 'approved' copy of the ports tree from:
http://www.emulab.net/downloads/ports-20041102.tar.gz
Untar it, move it into place as /usr/ports (rename the old directory to
ports.old, or just remove it). NOTE: You must download and copy the ports
tree into place, even if you do not intend to install any packages yourself.
##### Step 3 - Unpacking and running configure
At this point, you'll need to make a 'defs' file - You will use the same
defs file on boss, ops and fs. See defs-example in the top level directory
as a starting point. You want to make sure each of the variables has a
definition that makes sense in your environment. Feel free to ask Utah if
something is not clear. Of particular note for the fs node is the setting
of FSDIR_* to match the filesystem layout from Step 1, and the setting of
FS_WITH_QUOTAS if you intend to use quotas on any filesystem.
Unpack the testbed source, and run it's configure script. A good place to
unpack the source code is /usr/testbed/src/testbed. You will use the
--with-TBDEFS option to configure to give it the path to your defs file:
mkdir -p /usr/testbed/src/testbed
mkdir -p /usr/testbed/obj/testbed
cd /usr/testbed/src/testbed
tar ....
cd /usr/testbed/obj/testbed
/usr/testbed/src/testbed/configure \
--with-TBDEFS=/path/to/your/defs-file
Typically, you would store your defs file in the source tree along with
the other defs files that came in the tarball.
##### Step 4 - Running the fs installation script
In the object tree you've configured (say, /usr/testbed/obj/testbed), there's
an 'install' subdirectory, with a script called 'fs-install'. Just run this
script as root (note the same package directory argument as above).
cd install
env PKG_PATH=/usr/tmp/FreeBSD-4.10-20041102 perl fs-install
It will take care of installing any additional ports, and doing various
other configuration of FreeBSD required to make it into an fs node. The
script is designed so that you can run it as many times as you want, and
it'll just skip stuff it's already done. If it fails, send the output to
Utah so that we can fix it up. If it succeeds, follow any other
instructions it may have. The script will tell you to reboot the machine,
but you can wait until after you do the next step to do so, if you want.
(You may have to set the executable bit on this script, since configure won't;
we'd like to get this fixed at some point.)
You should be aware that, among other things, this script sets password-less
'sudo' for anyone in the 'wheel' group. If you don't want this for security
reasons, you can undo them after the installation script is finished.
##### Step 5 - Installing from source
To install the actual testbed software, simply do a 'gmake fs-install' in your
object directory.
(Note: If you're logged in as root, /usr/local/bin, where gmake lives on
FreeBSD, may not be in your path.)
##### Step 6 - Quotas (optional)
[ Note that this section is FreeBSD specific. ]
It you are planning to run quotas for Emulab users on the fs node, you
will need to establish a default quota value for all users. Ideally,
the quota would be configurable per-user, but for now all quotas are
initialized from a "prototypical" user. Note that you can change individual
user quotas later by running edquota(8) on the 'fs' node. To establish
the default quota values, you will need a "prototype user" to which to
apply the quotas. You will probably want to add a special user, say
'elabman', for this purpose. The uid and gid for this user should be
the MIN_UNIX_UID and MIN_UNIX_GID values specified in your defs file
(10000 and 6000 by default). Assuming those default values, you would do:
pw useradd elabman -u 10000 -g 6000 -m -d /users/elabman -h - -s /bin/nologin
Now set the quota for that user on each quota-enabled filesystem, e.g.:
edquota -e /proj:2000000:2000000 -e /users:1000000:1000000 elabman
would set a 1GB quota on /proj and 512MB on /users for the prototype user.
Once the prototype user quotas are established, you can do:
edquota -p elabman 10000-15000
which would automatically apply the elabman quotas to any user created with
uids between 10000 and 15000, assuming that you wanted to allow up to 5000
users.
##### Step 7 - Other miscellaneous things to set up
[Nothing at this time]
Once you're done with all of this, reboot fs.
...@@ -9,7 +9,7 @@ ...@@ -9,7 +9,7 @@
##### Most recently tested on FreeBSD 4.11. ##### Most recently tested on FreeBSD 4.11.
##### #####
##### Step 0 - OS installation and setup ##### Step 1 - OS installation and setup
Install FreeBSD on the machine you'll be using for your ops node, using the Install FreeBSD on the machine you'll be using for your ops node, using the
standard FreeBSD installation process. When asked by the installer, it's best standard FreeBSD installation process. When asked by the installer, it's best
...@@ -30,6 +30,19 @@ space to hold them: ...@@ -30,6 +30,19 @@ space to hold them:
be safe and build with at least a million). be safe and build with at least a million).
/usr/testbed/ - Needs space for testbed software and logs. Several (3-4) GB /usr/testbed/ - Needs space for testbed software and logs. Several (3-4) GB
should be enough. should be enough.
Do *not* create any user accounts yet, and just log in as root for the time being.
Our software will create user accounts later, once you get boss set up. If you
already created any users, then delete them with the "pw" command and make sure
the home directories for them are removed as well.
##### Step 1a - Shared filesystem setup for combined ops/fs machine.
If you have a separate 'fs' node, you should have already setup your shared
filesystems and can skip to Step 2. If this node is also to be your file
server, you will also need:
/users/ - Needs space for user home directories. Amount of space required /users/ - Needs space for user home directories. Amount of space required
depends on how many users you expect to have. depends on how many users you expect to have.
Generally, though, we suggest that users store large Generally, though, we suggest that users store large
...@@ -46,8 +59,10 @@ space to hold them: ...@@ -46,8 +59,10 @@ space to hold them:
on what you want to make available. on what you want to make available.
You may want to enforce quotas on the user-writable filesystems. This is the You may want to enforce quotas on the user-writable filesystems. This is the
main reason you'd want to keep them in separate filesystems (i.e., so people can main reason you'd want to keep them in separate filesystems (i.e., so people
have different /users/ and /proj/ quotas.) can have different /users/ and /proj/ quotas.) If you do not think you will
ever use quotas, then you could make /users and /proj part of the same
filesystem.
Note also since /share is exported read-only, FreeBSD requires that it be on Note also since /share is exported read-only, FreeBSD requires that it be on
a separate filesystem from anything that is exported read-write. So while a separate filesystem from anything that is exported read-write. So while
...@@ -61,12 +76,10 @@ symlinks to the appropriate places. ie., if you make one big filesystem called ...@@ -61,12 +76,10 @@ symlinks to the appropriate places. ie., if you make one big filesystem called
ln -s /z/proj /proj ln -s /z/proj /proj
... etc. ... etc.
Do *not* create any user accounts yet, and just log in as root for the time being. In other words, we assume the existence of /users, /proj, /group and /share.
Our software will create user accounts later, once you get boss set up. If you
already created any users, then delete them with the "pw" command and make sure
the home directories for them are removed as well.
##### Step 1 - Installing packages ##### Step 2 - Installing packages
To make sure that you're running with versions of software from the ports To make sure that you're running with versions of software from the ports
collection that are known to work with our software, and to save you hours collection that are known to work with our software, and to save you hours
...@@ -105,10 +118,10 @@ ops feel like 'home' (like emacs, jove, or whatever). NOTE: You must ...@@ -105,10 +118,10 @@ ops feel like 'home' (like emacs, jove, or whatever). NOTE: You must