No more hardwiring in bsdefs.h (which has been moved out
of the clientside dir since it is only used in tmcd.c).

Clientside builds on HP Moonshot, haven't tried running anything.

The most significant change here is moving to autoconf 2.69, which
is the first that supports "aarch64". This required replacing the
scripts in autoconf/ and significant textual changes to configure.in
files.

Other than that, it was all lint.

shared hosts. This is required for OpenVZ but not for XEN, so this does
mean that when using ZFS, shared OpenVZ hosts are not supported. Not likely
to be a problem.

For every tarball and rpm, tmcd will now pass a SERVER= string as well
telling the client where the file should be downloaded from (if using
the web rather than NFS). Right now this value is the same for all
tarballs and rpms, and is hardwired in tmcd as either "www" (if
SPEWFROMOPS=0) or "users" (if 1). Note: BUMPED THE TMCC VERSION NUMBER
for this.

Made a common routine for doing an error-check-and-retry copy of a file
across "racy" NFS. This is used by install-{tarfile,rpm} and rc.topomap.

    
    OPSVM_ENABLE=0
    OPSVM_MOUNTPOINT=""

Primarily, so we can let frisbee upload to boss work through our firewall
without opening a gaping hole.

YOU MUST RECONFIGURE YOUR TREE before installing.

There are a couple of new packet types in the frisbee protocol which are
exchanged via TCP with the master server: GETREQUEST and GETREPLY.  The
client passes to the master server an opaque imageid and a couple of options
and gets back the addr/port to use to actually download the image.  The
implementation of the master server is fragile and is more of a test
framework, Grant is working on a more robust master server.  I am mostly
doing a backend that communicates with the Emulab DB to do its authentication
and making the client changes.

The client now uses the -S option to specify the IP address of the master
server and the -F option to specify an imageid.  If no error is returned,
the image is downloaded using the returned addr/port.  If -Q is used in place
of -F, then the client makes a "status only" call getting back info about
whether the named image is accessible to the client and whether a server is
currently running.

On the server side, the new master server (mserver.c) has an Emulab
configuration "backend" that supports host-based authentication.
The IP address of the caller is mapped to a node_id/pid/gid/eid combo
that is used to determine access.  On a request, the specified imageid is
treated either as a pathname (if it starts with '/') or an image identifier
of the form "<pid>/<imagename>".  If it is a pathname, we check to make
sure that pathname (after running through "realpath") is contained in one
of the directories accessible to that node in its current experiment context;
i.e., /share, /proj/<pid>, /groups/<pid>/<gid>, or /users/<swapper-uid>.
If it is an image identifier, the DB is queried to ensure that access is
allowed to that image; i.e., it must be "global" or in the appropriate
project/group.

The master server forks a frisbeed for each valid request, if one is not
already running.  The multicast address selection is still based on the
emulab_indicies.frisbee_index field, but the address/port/server info is no
longer stored in the frisbee_blobs table (frisbee_pid, load_address,
load_busy are not set).

Note that this is not yet integrated in the os_load path.  Further work is
required to replace frisbeelauncher.

(eventual) support for NFS servers without race conditions!

This means no NFS between nodes and ops/fs. There are still NFS mounts of
ops on boss however.

Added new defs-* variable NOSHAREDFS, which when set non-zero will disable
the export of NFS filesystems to nodes.  Involved lots of little changes:

 * /users, /proj, and /share filesystems are not exported to nodes.

 * Returned mount info now includes an FSTYPE key which will be set to "LOCAL"
   if NOSHAREDFS is in effect (by default it is set to "NFS-RACY"; more on
   this later).  In the case where it is set to LOCAL, the other mount lines
   no longer contain REMOTE=foo settings.  Because of this change,
   THE TMCD VERSION NUMBER HAS BEEN BUMPED TO 32.

 * The client rc.mounts script will now create local versions of /users/*,
   /proj/<pid>, and /share when FSTYPE=LOCAL.  It first runs mkextrafs to
   create a large partition for these, since someday we will likely want
   to pre-populate these with a non-trivial amount of data.  Right now,
   the only thing that is put in the user's homedir is the standard dotfiles
   for the OS and the Emulab authorized_keys file (so you can login).

 * Linktest had to be modified to fetch the various results files (via
   loghole) rather than just assuming they were in /proj.  And also changed
   to invoke tevc with the local copy of the event key so it won't try to
   read it over NFS.

 * create_image was modified to ssh to the node and run the imagezip
   command, capturing the output of ssh.  This is controlled via the "-s"
   option which defaults to on for a NOSHAREDFS system, but can also be
   used on a normal system.

 * elabinelab's can be configured with/without a shared FS via the
   CONFIG_SHAREDFS attribute (note polarity change) which defaults to 1.

Another new defs-* variable, NFSRACY, will some day allow you to specify
(by setting to 0) that your NFS server does NOT have the nefarious mountd
race condition when changing /etc/exports.  Currently, this defaults to 1
since all versions of FreeBSD supported as an "fs" node have this "feature."
Rumor has it that FreeBSD 8 does not have this problem nor, presumably,
would a Linux NFS server.

The only use of this variable right now is to set the FSTYPE returned by the
tmcd "mounts" call, which in turn is used by one client script, rc.topomap
(via a libsetup function) to determine whether it should try copying
the topo file multiple times.

Random: add python2.6 to list of python's checked for in configure.
Random: resync defs-example-privatecnet with defs-example.
Random: did a little code-pissin here and there.

(including an update to make schemacheck happy), and client-side script
modifications.

Tested by me on a few different machines. Ok by Mike.

config.h for tmcd.c

widearea reloading support.

* New slot in the images table to store an access key which remote
  sites must provide in order to download an image (via https).

* tmcd returns a different kind of ADDRESS field from doloadinfo.
  Instead of the multicast stuff, return a URL that points to boss'
  web server. The URL is of the form:

   https://www.myemulab.net/spewimage.php?imageid=10013&access_key=abcdef

  which as you can see is fully specified; the client does not need
  to know anything else.

* New webpage and backend scripts appropriately called "spewimage"
  which also includes support for the http HEAD request (from wget) to
  avoid downloading images that are already on the node. I just
  learned about this HEAD request stuff today ... but otherwise these
  operate as expected, spewing the image if the access key is provided.

* Changes to rc.frisbee to deal with remote loading. In addition to
  URL support, I also added support for simple paths, the intent being
  that we will probably distribute images offline (say, at night) so
  that when a node reboots it doesn't actually have to wait 60 minutes
  for an image to download. I have not added any server side support
  for this yet though. Maybe later this week.

* Other bits and pieces and fixes to make this work.

file: we return info for boss/ops in the "hostnames" command, but only if
the command would have otherwise returned something.

This is because the default hosts file we distribute in the rootball already
has boss/ops in it.  But if, during bootup, tmcd returns hostname info, that
hosts file gets overwritten and boss/ops info gets lost.

I could just grep out the info from the original hosts file and transfer it
to the one we are building, but what do I grep for: "boss", "ops",
"emulab.net"?  Don't really want to hardwire any of those in the client-side
script.

By returning this info via tmcd, I also don't have to modify the client-side
and thus don't need to build a new rootball!

Two-day boondoggle to support "/scratch", an optional large, shared filesystem
for users.  To do this, I needed to find all the instances where /proj is used
and behave accordingly.  The boondoggle part was the decision to gather up all
the hardwired instances of shared directory names ("/proj", "/users", etc.)
so that they are set in a common place (via unexposed configure variables).
This is a boondoggle because:

1. I didn't change the client-side scripts.  They need a different mechanism
   (e.g., tmcd) to get the info, configure is the wrong way.

2. Even if I had done #1 it is likely--no, certain--that something would
   fail if you tried to rename "/proj" to be "/mike".  These names are just
   too ingrained.

3. We may not even use "/scratch" as it turns out.

Note, I also didn't fix any of the .html documentation.  Anyway, it is done.
To maintain my illusion in the future you should:

1. Have perl scripts include "use libtestbed" and use the defined PROJROOT(),
   et.al. functions where possible.  If not possible, make sure they run
   through configure and use @PROJROOT_DIR@, etc.

2. Use the configure method for python, C, php and other languages.

3. There are perl (TBValidUserDir) and php (VALIDUSERPATH) functions which
   you should call to determine if an NS, template parameter, tarball or
   other file are in "an acceptable location."  Use these functions where
   possible.  They know about the optional "scratch" filesystem.  Note that
   the perl function is over-engineered to handles cases that don't occur
   in nature.

Mote and robot related stuff.  The main thing is the addition of relay
capabilities to capture and related things.

	* GNUmakefile.in: Add the capture and tip subdirectories to the
	client and client-install targets.

	* configure, configure.in, config.h.in: Detect srandomdev() for
	capture and add "mote/newmote" script.

	* capture/GNUmakefile.in, capture/capture.c: Add "relay"
	capabilities to capture.

	* capture/capquery.c: Query the capserver for the relay receiver's
	port number.

	* capture/capserver.c: Small hack to return the port number
        for a node.

	* db/libdb.pm.in, db/xmlconvert.in: Add virt_tiptunnels table.

	* event/program-agent/program-agent.c: Change log file names to
	something a little more user-friendly.  Add a "MODIFY" event
	handler that lets the user set agent attributes (command, tag,
	timeout) without having to run a program.

	* event/sched/GNUmakefile.in, event/sched/console-agent.cc,
	event/sched/console-agent.h, event/sched/event-sched.c: Add
	console agents that can be used to snapshot a section of the
	capture log file.

	* event/sched/node-agent.cc: Some minor cleanup.

	* event/sched/simulator-agent.cc, event/sched/simulator-agent.h:
	Add the config data to the report mail.  Add a "RESET" event
	handler that runs "loghole clean".  Save the report mail in a file
	so it gets archived with the rest of the logs.

	* lib/libtb/tbdefs.h: Add CONSOLE object type.

	* mote/GNUmakefile.in, mote/newmote: Add newmote script, just a
	quick hack to add motes to the DB.

	* mote/tbuisp.in: Add another backend for loading motes through
	their relay capture server.

	* robots/mtp/mtp_dump.c: Dump the min/max values for x and y,
	handy for figuring out the bounds of the camera.

	* sql/database-fill.sql: Change the RELOAD-MOTE/SHUTDOWN ->
	ALWAYSUP/SHUTDOWN mode transition to ALWAYSUP/ISUP since stated
	doesn't seem to run triggers after a state change by a mode
	transition.

	* tbsetup/tbreport.in: Change the ordering of the eventlist so it
	displays event-sequences appropriately.

	* tbsetup/ns2ir/GNUmakefile.in, tbsetup/ns2ir/console.tcl,
	tbsetup/ns2ir/node.tcl, tbsetup/ns2ir/parse.tcl.in,
	tbsetup/ns2ir/sim.tcl.in: Add a "console" agent that represents
	the serial console for a node.

	* tbsetup/ns2ir/sequence.tcl: Add an "append" method so it is
	easier to build sequences dynamically.

	* tbsetup/ns2ir/topography.tcl: Make checkdest available to
	regular users.

	* tip/GNUmakefile.in, tip/tiptunnel.c: Add support for uploading a
	file to a relay version of capture and exporting the end
	connection as a pty.

	* tmcd/decls.h, tmcd/common/libsetup.pm: Bump version number since
	the dosubnodelist change is not backwards compatible.

	* tmcd/tmcd.c: Make dosubnodelist and dosubconfig callable even
	when a node isn't allocated.  Add dotiptunnels command that
	returns which serial consoles are to be mounted on a node.  Add
	mote version of subconfig that returns information needed to
	startup the relay version of capture.

	* tmcd/common/bootsubnodes: For motes, startup the relay version
	of capture (XXX stargate specific).

	* tmcd/common/libsetup.pm, tmcd/common/libtmcc.pm,
	tmcd/common/config/rc.config, tmcd/common/config/rc.tiptunnels:
	Client side changes for mounting another nodes serial line.

	* tmcd/common/rc.bootsetup: Always boot the subnodes, even when
	free.  This is used for motes since their capture needs to be up
	for reloading at the time.

	* tmcd/linux/ixpboot: Shuffle some code around so the script
	doesn't fail if the ixp isn't allocated.

	* utils/loghole.in: Add "digest.out" and "report.mail" as global
	logs to be saved in archives and display the "report.mail" file
	when showing a loghole archive.

	* xmlrpc/emulabserver.py.in: Scrub more of the return values to
	get rid of "None"s.

	FSNODE_IP
	EXTERNAL_FSNODE_IP

which default to ops' IP address if not set.  This allows for separate
ops/fs nodes, a config Aero has.

Also push some IP addresses (boss, ops, fs) into config.h so they can
be used in C programs.  Specifically tmcd for returning values of
"firewall variables" that can be used in default firewall rules.
But I will likely wind up putting the var/value pairs in the DB rather
than compiling them into tmcd.

Some cleanup of the robot vision system:

	* Makeconf.in: Add INSTALL_INCDIR for includes, EVENTSYS for
	whether or not the event system is available, HAVE_MEZZANINE for
	whether or not we'll be building mezzanine, and GTK_CONFIG which
	refers to the gtk-config binary, if there is one.

	* config.h.in: Add HAVE_LINUX_VIDEODEV_H and HAVE_MEZZANINE
	defines.

	* configure, configure.in: Check for the robot vision system
	dependencies.  Add mezzanine template files.

	* robots/GNUmakefile.in: Add some conditionals for directories
	that depend on the event-system and mezzanine.

	* robots/mezzanine/GNUmakefile.in,
	robots/mezzanine/libfg/GNUmakefile.in,
	robots/mezzanine/libmezz/GNUmakefile.in,
	robots/mezzanine/mezzanine/GNUmakefile.in,
	robots/mezzanine/mezzcal/GNUmakefile.in,
	robots/mezzanine/rtk2/GNUmakefile.in: Fold mezzanine into the
	testbed's build system.

	* robots/vmcd/GNUmakefile.in: When building the vmc-client, use
	the mezzanine that we build locally instead of an installed
	version.

	* robots/vmcd/test_vmc-client.sh.in, robots/vmcd/test_vmcd.sh.in,
	robots/vmcd/test_vmcd2.sh, robots/vmcd/test_vmcd3.sh,
	robots/vmcd/test_vmcd4.sh: Bring the test cases up-to-date with
	respect to the actual code.

	* robots/vmcd/vmc-client.c: A bunch of cleanups and bug fixes: add
	comments, set TCP_NODELAY on the client sockets (doh), etc...

its an inner elab. Define ELABINELAB in the inner defs file. Actual hack
is coming in a bit.

as defined in the defs-* file (e.g. "TBLOGFACIL=local2").  The default is
"local5" which is what we are setup to use so you shouldn't need to mess
with your defs- file!

perl scripts just get this value configured in when configure is run.
C programs get the value in two ways.  For programs that are intimate with
the testbed infrastructure, and include "config.h", they just get it from
that file.  For programs that we sometimes use outside the Emulab build
environment (e.g., frisbee, capture) and that don't include config.h,
the value is set via a "-DLOG_TESTBED=..." in the GNUmakefile build line.
If the value isn't set, it defaults to what it used to be (usually LOG_USER).

Still to do: healthd, hmcd (whose build doesn't seem to be completely
integrated) and plabdaemon.in (since its icky python :-)

and passed back from tmcd mounts command to the nodes. If not defined
in the defs file, it will not be compiled into tmcd.

is installed on ops. Makes the use of SFS more explicit, which is
good.

fact is defaulted in configure.in but can be overridden in the defs
file. Changed the one perl script that had a hardwired flux group. The
other dozen or so uses are in the web page code. I'll do those next.

it is converted into a proper config.h file in the object directory.
Look for AC_DEFINE_UNQUOTED in configure.in. This removes some of the
headache of defining variables for C code in the makefiles. Be sure
sure to add -I.. to CFLAGS and include "config.h" in your source code.