In particular, make it work with d430s as inner nodes by supporting
an "sio2" pxeboot, the 64-bit MFS, and a more modern kernel in the MFS.

Support FreeBSD 10.3. We will need to be moving to this before long
as 10.2 EOLs in two days.

Support setup of "Emulab-aware" ZFS use in install scripts. Note that
the core support code was already done (WITHZFS, WITHAMD). Mostly this
involves changes to setup either amd (WITHAMD==1) or autofs (WITHAMD==0)
on the boss node and to NOT add mounts of /{users,groups,proj} to
/etc/fstab. We still need to add a section to the install documentation
about setting up a zpool for Emulab to use. There was also a fix to the
firstuser script which did not do the account setup correctly.

Support setup of ZFS in elabinelab. The elabinelab attributes CONFIG_ZFS
and CONFIG_AUTOFS are used to convey intent here. Currently they can only
be used in an "ops+fs" config (e.g., the standard boss and ops config,
NOT the seperate fs node config). It should work with either the physical
or virtual node setups:

* For the physical node setup, we actually use local blockstores in the
  ops node config: a SYSVOL blockstore for /usr/testbed and a tiny 1Mib
  NONSYSVOL blockstore. The latter blockstore is not actually used, we
  just make it to force setup of a ZFS zpool which we then use for the
  inner elab.

* For the virtual node setup, we just identify the virtual EXTRADISK
  intended for "/q" and create a zpool on that device.

I would like to change all physical elabinelab setups to use blockstors
rather than the current hacky mkextrafs usage. But that is a task for
another day.

Finally, a couple of random changes in elabinelab code: change the
CentOS image downloaded to CENTOS7-64-STD, increased the default sizes
of the EXTRADISKS used in the VM config.

stopped setting this, but does not really matter since we do not use it
anymore.

your defs file to avoid a warning.

As of FreeBSD 9, the race condition in mountd has been addressed.
We automatically (based on the OS running) add the option to mountd (-S)
as part of ops-install. There is also a configuration variable, NFSRACY,
that we auto-detect and set in elabinelab setup. For regular installs,
you need to set this manually in the defs-* file. Note that the setting
in the defs file does not affect very much. It is passed to the client-side
which uses it in one copy function (used as a fall back for getting the
topology file).

There are still a couple of places in the Emulab code where handling
the race condition is hardwired into the code (most notably, imagezip).
These remain.

Add "-Q <interval>" option to the master server to allow it to act as an
IGMP V2 querier in environment where there is otherwise not one. It does
essentially what the perl-based querier (code.google.com/p/perl-igmp-querier/)
does, sending out a v2 membership query at the specified interval.

This eliminates the need to run mrouted in some environments (e.g., elabinelab)
just to issue IGMP queries. As a result, all the boss-install and elabinelab
setup related to using mrouted to perform this function has been removed.
The elabinelab CONFIG_MROUTED option has been changed to CONFIG_QUERIER
(the former is still recognized and mapped to the latter). The undocumented
defs-* variable NEEDMROUTED has been changed to NEEDMCQUERIER (the former
still exists in install/installvars.pm.in but is always set to 0) to more
accurately reflect the variable's purpose. If NEEDMCQUERIER is set, then
the mfrisbeed startup script is modified to add the "-Q 30" option.

The implementation of the client and server "-K <interval>" keep-alive option
has been changed to directly send IGMP v2 membership reports containing the
associated MC address.

Note that the -K options have always been a hack to work-around assorted
IGMP-related misconfigurations and incompatibilities, and really should
only be used as a last resort. As implemented, they could cause the host
machine to be pruned out of other MC groups at the nearest switch since
they only report membership in the frisbee MC group. With the master server
acting as an IGMP querier, instances of the frisbee server on that host
should no longer need to do keep alives. We still have one case where it
is needed on the client-side: a FreeBSD 8.x or later host connected to an
IGMPv2-only switch. It appears that the IGMPv3 implementation added in
FreeBSD 8.x always sends v3 reports, even when the default is configured
(via sysctl or even recompiling the kernel) as v2.

Bogus param list for AddPerExperimentSwitchStack; can't have scalar args
after a list arg.

Missing mailing list TBERRORSEMAIL in defs-elabinelab. Caused redundant
"testbed-ops" addresses in /etc/aliases.

Weed out said redundant mailing list names when creating aliases during
ops-install.

named.reload script has gone away. Use rndc instead.

OPSVM_ENABLE=changeme
OPSVM_MOUNTPOINT=changeme

This is specifically intended for elabinelab right now where configurations
with an "inner control network" need a multicast router for inner frisbee
to work. There is (an undocumented) NEEDMROUTED defs variable that controls
whether the machinery is triggered or not. It is not documented because, though
in theory this would work for a real boss setup, it has not been tested and
each install would probably need a custom mrouted.conf file.

For an inner elab, this variable is automatically set in defs-elabinelab
if there is an inner control network. The result is that the mrouted port
will be installed, it will be enabled in rc.conf, and a stub mrouted.conf
file is created (to force mrouted to ignore the real control network).

current and maintained DBI::mysql module. A couple of things make this
a little more work then you might think.

Mysql exports a slightly different API then DBI, both at the DB *and*
the statement level. The former required some restructuring of
emdbi.pm, partly cause we want external sites to continue using Mysql
for a while longer. So, emdbi suppports both interfaces, via the
configure variable TBUSEDBI.

I also took the opportunity to also scrap the existing fork()
detection code and redo it in an easier to understand manner.
Actually, I had no idea what the previous code was trying to do, so it
was easier to just get rid of it, rather then try to make it work for
the DBI API.

There are also API differences in the "statement" class, but
fortunately this can be hidden by wrapping the statement class with a
wrapper that adds the routines we need to avoid making silly changes
to 1000s of queries. They are all trivial little things since mostly
its a matter of naming (numrows --> rows).

I also changed the library we use on ops (db/libtbdb.pm.in) to use
DBI, but in this case I just switched it over. Seemed like overkill to
worry about supporting both APIs on ops. If it works it works, and so
far it does. 

Lastly, the following modules still use Mysql directly. They all need
to be changed, but none of these are on the critical path to swapin
and swapout, so they can change later.

db/dumperrorlog.proxy.in
db/showgraph.in
db/sitevarscheck.in
bgmon/find-asymmetric
db/pelab_opspush.proxy.in
slothd/sdisrunning.in
utils/export_tables.in
utils/setbuildinfo.in
pelab/bgmon/libpelabdb.pm
pelab/dbmonitor/libtbdb.pm

(eventual) support for NFS servers without race conditions!

This means no NFS between nodes and ops/fs. There are still NFS mounts of
ops on boss however.

Added new defs-* variable NOSHAREDFS, which when set non-zero will disable
the export of NFS filesystems to nodes.  Involved lots of little changes:

 * /users, /proj, and /share filesystems are not exported to nodes.

 * Returned mount info now includes an FSTYPE key which will be set to "LOCAL"
   if NOSHAREDFS is in effect (by default it is set to "NFS-RACY"; more on
   this later).  In the case where it is set to LOCAL, the other mount lines
   no longer contain REMOTE=foo settings.  Because of this change,
   THE TMCD VERSION NUMBER HAS BEEN BUMPED TO 32.

 * The client rc.mounts script will now create local versions of /users/*,
   /proj/<pid>, and /share when FSTYPE=LOCAL.  It first runs mkextrafs to
   create a large partition for these, since someday we will likely want
   to pre-populate these with a non-trivial amount of data.  Right now,
   the only thing that is put in the user's homedir is the standard dotfiles
   for the OS and the Emulab authorized_keys file (so you can login).

 * Linktest had to be modified to fetch the various results files (via
   loghole) rather than just assuming they were in /proj.  And also changed
   to invoke tevc with the local copy of the event key so it won't try to
   read it over NFS.

 * create_image was modified to ssh to the node and run the imagezip
   command, capturing the output of ssh.  This is controlled via the "-s"
   option which defaults to on for a NOSHAREDFS system, but can also be
   used on a normal system.

 * elabinelab's can be configured with/without a shared FS via the
   CONFIG_SHAREDFS attribute (note polarity change) which defaults to 1.

Another new defs-* variable, NFSRACY, will some day allow you to specify
(by setting to 0) that your NFS server does NOT have the nefarious mountd
race condition when changing /etc/exports.  Currently, this defaults to 1
since all versions of FreeBSD supported as an "fs" node have this "feature."
Rumor has it that FreeBSD 8 does not have this problem nor, presumably,
would a Linux NFS server.

The only use of this variable right now is to set the FSTYPE returned by the
tmcd "mounts" call, which in turn is used by one client script, rc.topomap
(via a libsetup function) to determine whether it should try copying
the topo file multiple times.

Random: add python2.6 to list of python's checked for in configure.
Random: resync defs-example-privatecnet with defs-example.
Random: did a little code-pissin here and there.

Need to enable/disable the SelfLoader based on the version of perl installed.
Other minor nits.

But opsrecv and friends were never officially installed, so took care of
that too (conditional on PELABSUPPORT in the defs file).

Two-day boondoggle to support "/scratch", an optional large, shared filesystem
for users.  To do this, I needed to find all the instances where /proj is used
and behave accordingly.  The boondoggle part was the decision to gather up all
the hardwired instances of shared directory names ("/proj", "/users", etc.)
so that they are set in a common place (via unexposed configure variables).
This is a boondoggle because:

1. I didn't change the client-side scripts.  They need a different mechanism
   (e.g., tmcd) to get the info, configure is the wrong way.

2. Even if I had done #1 it is likely--no, certain--that something would
   fail if you tried to rename "/proj" to be "/mike".  These names are just
   too ingrained.

3. We may not even use "/scratch" as it turns out.

Note, I also didn't fix any of the .html documentation.  Anyway, it is done.
To maintain my illusion in the future you should:

1. Have perl scripts include "use libtestbed" and use the defined PROJROOT(),
   et.al. functions where possible.  If not possible, make sure they run
   through configure and use @PROJROOT_DIR@, etc.

2. Use the configure method for python, C, php and other languages.

3. There are perl (TBValidUserDir) and php (VALIDUSERPATH) functions which
   you should call to determine if an NS, template parameter, tarball or
   other file are in "an acceptable location."  Use these functions where
   possible.  They know about the optional "scratch" filesystem.  Note that
   the perl function is over-engineered to handles cases that don't occur
   in nature.

browser confusion that results when the same cookie name is used on
multiple elabinelab experiments.

version of TWiki, which will not run on ops until it is updated.

public stuff needed on ops (cvsd).

* Add mailman to the install process. This includes the package
  install, all the setup needed in ops-install, and the little patches
  we apply locally.

  Note: Only under 6.0: no point in going backwards since its a pain
  in the ass to build these packages for the 4.10 image.

* Push more of the package install back down into the ops/boss install
  scripts. This improves testing within those scripts, and avoids
  duplication. There are some changes in rc.mkelab that will temp
  mount the package dir on /packages for the benefit of the scripts.

Entailed new instructions for manual setup as well as integration into
elabinelab framework.  First, the manual path:

setup.txt, setup-boss.txt, setup-ops.txt and new setup-fs.txt:
    Updated to reflect potential for separate fs node.  The org here
    is a little dicey and could be confusing with ops+fs vs. ops and fs.
    Has not been field tested yet.

*/GNUmakefile.in: new fs-install target.

configure, configure.in, defs-*:
    Somewhat unrelated, make min uid/gid to use be a defs setting.
    Also add config of fs-install.in script.

boss-install.in, ops-install.in and new fs-install.in:
    Handle distinct fs node.  If you have one, fs-install is run before
    ops-install.  All scripts rely on the defs file settings of FSNODE
    and USERNODE to determine if the fs node is seperate.

utils/checkquota.in:
    Just return "ok" if quotas are not used (i.e., if defs file FS_WITH_QUOTA
    string is null.

install/ports/emulab-fs:
    Meta port for fs node specific stuff.  Also a patch for the samba port
    Makefile so it doesn't drag in CUPs, etc.  Note that the current samba
    port Makefile has this change, I am just backporting to our version.

Elabinelab specific changes:

elabinelab-withfs.ns:
    NS fragment used in conjunction with
	tb-elab-in-elab-topology "withfs"
    to setup inner-elab with fs node.

elabinelab.ns:
    The hard work on the boss side.  Recognize seperate-fs config and handle
    running of rc.mkelab on that node.  fs setup happens before ops setup.

rc.mkelab:
    The hard work on the client side.  Recognize FsNode setup as well as
    differentiate ops+fs from ops setup.

Related stuff either not part of the repo or checked in previously:
    emulab-fs package

variable in the defs file. Nice for elabinelab to avoid inner named
from having to do too much work.

configure so that we can localize it for inner emulab. Not good to
overlap the address space between inner and outer emulab if inner
boss is going to use frisbee client to suck images from outer frisbeed.

port, and the location of the SSL certificate used to talk to the server