stale.

Remove a "use Fcntl" that Gary says isn't needed any longer.
This "use" broke the script under old versions of perl.

directory.

We need blobs potentially before rc.keys runs to grab the keyhash, so we
have to grab it right away if the file doesn't exist.

"tb-set-tarfiles" is like "tb-set-node-tarfiles" except that it
distributes the tarfile to all nodes rather than just one and that it
uses frisbee to distribute the file.

These changes involved 1) refactoring frisbee info from images table
into a new table, frisbee_blobs, 2) a new experiment_blobs table, and
3) a new tmcd command so the node knows how to get the files from the
server.

The changes where designed to be general purpose enough to eventually
support:
  1) Distributing arbitrary files (not just tarfiles) to nodes
  2) Perform arbitrary actions on those files
  3) Use arbitrary methods to get the files

As such the tmcd line is as follows:
  URL=* ACTION=*

where URL is currently:
  frisbee.mcast://<ADDR>/<FILE>
for example
  frisbee.mcast://234.16.184.192:18092/users/kevina/home-dir.tar.gz
and when we get around to using a master Frisbee server it could be
  frisbee://*
or it could be a file://, http://, etc.

and ACTION is currently:
  unpack:<LOCATION>
for example
  unpackt:/users
with future syntax to be determined.

- Add support for new tmcd dhcpdconf command to watchdog
- Fix dhcpd.conf template so that ddns-update-style is set to none
- Pull some utility functions from boss's libtestbed.pm into the client
  libtestbed.pm for use by subboss_dhcpd_makeconf and daemon_wrapper.
- Add stuff to simplify getting control interface IP address.

(including an update to make schemacheck happy), and client-side script
modifications.

Tested by me on a few different machines. Ok by Mike.

proxy to service an inet addr/port rather than a Unix domain socket.
Verified that this version for Xen, OpenVZ, and jails.

the tb-set-node-os command with a second optional argument; if that is
present, the first arg is the child OS and the second is the parent OS.
We add some new features in ptopgen (OS-parentOSname-childOSname) based
off a new table that maps which child OSes can run on which parents, and
the right desires get added to match.  We setup the reloads in os_setup
along with the parents.  Also needed a new opmode, RELOAD-PCVM, to handle
all this.

For now, users only have to specify that their images can run on pcvms, a
special hack for which type the images can run on.  This makes sense in
general since there is no point conditionalizing childOS loading on
hardware type at the moment, but rather on parentOS.  Hopefully this stuff
wiill mostly work on shared nodes too, although we'll have to be more
aggressive on the client side garbage collecting old frisbee'd images for
long-lived shared hosts.

I only made these changes in libvtop, so assign_wrapper folks are left in
the dark.

Currently, the client side supports frisbee.  Only in openvz for now, and
this probably breaks libvnode_xen.pm.  Also in here are some openvz
improvements, like ability to sniff out which network is the public
control net, and which is the fake virtual control net.

raw geni node.

(and vnodes). Each time a node is allocated to an experiment it gets a
new root password (using the node_attributes table). The watchdog has
a new section that resets the root password (defaults to hourly).  We
still using a common password in the image to avoid totally bricking
ourselves, but once a node boots into an experiment it gets a new root
password.

This prevents hundreds of nodes with the same password, and all of the
problems associated with that.

integrates some of the early RON support, adding a fake jail setup that
does a lot of what happens inside a jail, but without the actual jail.
Remains to be seen how well this is going to work out.

that clients and servers can avoid using hardwired ports on those
experimental nodes. I have added the following tmcd operation:

	tmcc portregister <service> [<port>]

where we assume its the control network IP (from the DB), and the pid/eid
of the node the experiment belongs to. The given port is entered into
the port_registration table for the experiment, using the service as the
tag. Supplying port=0 clears the registration from the table.

When called like:

	tmcc portregister <service>

we return the registered port, or nothing.

I hacked up a little C library module in libtb so that there is something
that looks like a C interface to this:

 	int
 	PortRegister(char *service, int port);

 	int
 	PortLookup(char *service, char *hostname, int namelen, int *port);

The above routines call out to tmcc of course.

Lastly, I changed the sync server and client to use the new port
registration, via the library calls above.

There are other emulab services that need to be changed as well, but
they can be done on an as needed basis.

Add "plabeventkeys" tmcc command to libtmcc.pm

Changes to support reporting back the port elvind bound to in the
management slice, and also sending this back to non-MS slivers.

It breaks the windows port and also doesn't accomplish anything.
(a non-root user can always just make the tmcc call to boss to
get the info).

 * ipodinfo and watchdoginfo are now fullconfig commands
 * ipod setup moved later in startup to take advantage of fullconfig
 * watchdog changed to use cached info on the first call
 * move firewall setup later to take advantage of fullconfig
Other stuff:
 * accounts/ipodinfo cache files protected 600

Along with Tim's changes, this should eliminate 4 more boot time calls!

some details can be found in the advanced tutorial that I wrote up.
See this link:

http://www.emulab.net/tutorial/docwrapper.php3?docname=advanced.html#Tracing

The basic idea is that each virt_lan entry gets a couple of new slots
describing the type of tracing that is desired.

  traced tinyint(1) default '0',
  trace_type enum('header','packet','monitor') NOT NULL default 'header',
  trace_expr tinytext,
  trace_snaplen int(11) NOT NULL default '0',
  trace_endnode tinyint(1) NOT NULL default '0',

There is a new physical table called "traces" that is a little bit
like the current delays table. A new tmcd command returns the trace
configuration to the client nodes (tmcd/common/config/rc.trace).

The delays table got a new boolean called "noshaping" that tells the
delay node to bridge, but not set up any pipes. This allows us to
capture traffic at the delay node, but without much less overhead on
the packets.

The pcapper got bloated up to do packet capture and more event stuff.
I also had to add some mutex locking around calls into the pcap
library and around malloc, since the current setup used linuxthreads,
which is not compatable with the standard libc_r library. I was
getting all kinds of memory corruption, and I am sure that if someone
breathes on the pcapper again, it will break in some new way.

Mote and robot related stuff.  The main thing is the addition of relay
capabilities to capture and related things.

	* GNUmakefile.in: Add the capture and tip subdirectories to the
	client and client-install targets.

	* configure, configure.in, config.h.in: Detect srandomdev() for
	capture and add "mote/newmote" script.

	* capture/GNUmakefile.in, capture/capture.c: Add "relay"
	capabilities to capture.

	* capture/capquery.c: Query the capserver for the relay receiver's
	port number.

	* capture/capserver.c: Small hack to return the port number
        for a node.

	* db/libdb.pm.in, db/xmlconvert.in: Add virt_tiptunnels table.

	* event/program-agent/program-agent.c: Change log file names to
	something a little more user-friendly.  Add a "MODIFY" event
	handler that lets the user set agent attributes (command, tag,
	timeout) without having to run a program.

	* event/sched/GNUmakefile.in, event/sched/console-agent.cc,
	event/sched/console-agent.h, event/sched/event-sched.c: Add
	console agents that can be used to snapshot a section of the
	capture log file.

	* event/sched/node-agent.cc: Some minor cleanup.

	* event/sched/simulator-agent.cc, event/sched/simulator-agent.h:
	Add the config data to the report mail.  Add a "RESET" event
	handler that runs "loghole clean".  Save the report mail in a file
	so it gets archived with the rest of the logs.

	* lib/libtb/tbdefs.h: Add CONSOLE object type.

	* mote/GNUmakefile.in, mote/newmote: Add newmote script, just a
	quick hack to add motes to the DB.

	* mote/tbuisp.in: Add another backend for loading motes through
	their relay capture server.

	* robots/mtp/mtp_dump.c: Dump the min/max values for x and y,
	handy for figuring out the bounds of the camera.

	* sql/database-fill.sql: Change the RELOAD-MOTE/SHUTDOWN ->
	ALWAYSUP/SHUTDOWN mode transition to ALWAYSUP/ISUP since stated
	doesn't seem to run triggers after a state change by a mode
	transition.

	* tbsetup/tbreport.in: Change the ordering of the eventlist so it
	displays event-sequences appropriately.

	* tbsetup/ns2ir/GNUmakefile.in, tbsetup/ns2ir/console.tcl,
	tbsetup/ns2ir/node.tcl, tbsetup/ns2ir/parse.tcl.in,
	tbsetup/ns2ir/sim.tcl.in: Add a "console" agent that represents
	the serial console for a node.

	* tbsetup/ns2ir/sequence.tcl: Add an "append" method so it is
	easier to build sequences dynamically.

	* tbsetup/ns2ir/topography.tcl: Make checkdest available to
	regular users.

	* tip/GNUmakefile.in, tip/tiptunnel.c: Add support for uploading a
	file to a relay version of capture and exporting the end
	connection as a pty.

	* tmcd/decls.h, tmcd/common/libsetup.pm: Bump version number since
	the dosubnodelist change is not backwards compatible.

	* tmcd/tmcd.c: Make dosubnodelist and dosubconfig callable even
	when a node isn't allocated.  Add dotiptunnels command that
	returns which serial consoles are to be mounted on a node.  Add
	mote version of subconfig that returns information needed to
	startup the relay version of capture.

	* tmcd/common/bootsubnodes: For motes, startup the relay version
	of capture (XXX stargate specific).

	* tmcd/common/libsetup.pm, tmcd/common/libtmcc.pm,
	tmcd/common/config/rc.config, tmcd/common/config/rc.tiptunnels:
	Client side changes for mounting another nodes serial line.

	* tmcd/common/rc.bootsetup: Always boot the subnodes, even when
	free.  This is used for motes since their capture needs to be up
	for reloading at the time.

	* tmcd/linux/ixpboot: Shuffle some code around so the script
	doesn't fail if the ixp isn't allocated.

	* utils/loghole.in: Add "digest.out" and "report.mail" as global
	logs to be saved in archives and display the "report.mail" file
	when showing a loghole archive.

	* xmlrpc/emulabserver.py.in: Scrub more of the return values to
	get rid of "None"s.

Checkin some changes related to experiment automation and vnode feedback:

	* configure, configure.in: Add sensors/canaryd/feedbacklogs
	template.

	* db/libdb.pm.in, db/xmlconvert.in: Add "virt_user_environment"
	table that holds environment variable names and values.

	* event/lib/event.c: Allocate memory of the right size for
	event_notifications.

	* event/program-agent/GNUmakefile.in: Add version.c file and
	add install targets for the man page.

	* event/program-agent/program-agent.8: Man page describing the
	program-agent daemon.

	* event/program-agent/program-agent.c: Add a bunch of convenience
	features: let the user specify the working directory for commands;
	save output to separate files on every invocation of an agent; let
	the user specify a timeout for a command; make the set of
	environment variables sane and add vars given in the NS file in
	the opt array; a "status" file containing process information is
	written out when children are collected.  Internal changes: child
	processes are collected immediately, instead of waiting for the
	next START event, so we can send back COMPLETE events; the daemon
	now runs with a real-time priority, to increase the chances of
	receiving events.

	* event/proxy/evproxy.c: Made it bidirectional so the
	program-agent's COMPLETE events make it back to the scheduler.

	* event/sched/error-record.c: Change the default log directory.

	* event/sched/event-sched.h, event/sched/event-sched.c: Setup an
	environment similar to a program-agent to run the user's log
	digester.

	* event/sched/node-agent.cc: Add a handler for the SNAPSHOT event
	that runs create_image for the node.

	* event/sched/simulator-agent.h, event/sched/simulator-agent.cc:
	Let the user specify a "DIGESTER" script that digests the log
	files into a summary of the results.  Add event handler for
	remapping a vnode experiment.

	* event/sched/timeline-agent.c: Accept the RUN event as well as
	the START event.

	* os/GNUmakefile.in: Install the install-tarfile.1 man page.

	* os/install-tarfile: Automatically chown/chgrp any files that do
	not have valid user or group IDs, the new owner will be the user
	that swapped in the experiment.  Include the install directory in
	the DB file.  Add a "list" mode that just dumps what files have
	been installed and where.  Add a "force" option so the user can
	forcefully install the file, even though the DB says its already
	there.

	* os/install-tarfile.1: Man page describing the install-tarfile
	tool.

	* os/syncd/GNUmakefile.in: Install man pages on ops.

	* sensors/canaryd/GNUmakefile.in: Link canaryd statically and
	install "feedbacklogs" tool.

	* sensors/canaryd/canaryd.c: Dump dummynet pipe data.

	* sensors/canaryd/canarydEvents.c: Log errors.

	* sensors/canaryd/feedbacklogs.in: Tool used to generate feedback
	data from canaryd log files.

	* sensors/slothd/GNUmakefile.in: Install digest-slothd on ops.

	* sensors/slothd/digest-slothd: Fix some bugs and write out an
	"alert" file with all the nodes/links that were overloaded.

	* tbsetup/os_load.in, tbsetup/libosload.pm.in: Add "waitmode"
	argument that lets you specify that you want to wait for the disk
	to finish loading and/or wait for the node to come back up in the
	new OS.

	* tbsetup/power.in: Remove debugging printf.

	* tbsetup/ns2ir/node.tcl, tbsetup/ns2ir/program.tcl,
	tbsetup/ns2ir/sequence.tcl, tbsetup/ns2ir/sim.tcl.in: Fix some
	quoting problems with event-sequences.  Add -expected-exit-code
	and -tag options to the "$program run" event.  Add -digester to
	the "$ns report" event that lets the user specify a program to run
	to digest the log files.

	* tbsetup/ns2ir/tb_compat.tcl.in: Change the initial scaling
	factor for feedback nodes to 1%, instead of 100%.

	* tmcd/tmcd.c, tmcd/common/libtmcc.pm: Add "userenv" command that
	returns the values in "virt_user_environment".  Return new program
	agent fields: dir, timeout, and expected_exit_code.

	* tmcd/common/GNUmakefile.in: Install rc.canaryd.

	* tmcd/common/bootvnodes: Add hack to boost the program-agents to
	a real-time priority, they can't do it from inside the jail.

	* tmcd/common/rc.canaryd: Rc script for canaryd.

	* tmcd/common/watchdog: Don't fail outright if there is a bad line
	in the battery.log

	* tmcd/common/rc.progagent: Append "userenv" data to the
	program-agent config file.

	* utils/GNUmakefile.in: Install loghole and its man page on ops.

	* utils/loghole.1: Document "clean" command and the change in
	loghole directories.

	* utils/loghole.in: Add "clean" command and parallelization.

	* xmlrpc/emulabserver.py.in: Add "virt_user_environment" table.
	Order the eventlist by "idx" and time, needed for sequences.  And
	removed unnecessary nologin checks.

Report battery stats using the watchdog:

	* robots/emc/emcd.c: Don't send events with battery levels
	anymore.

	* tmcd/tmcd.c: Add 'battery' command that updates the
	battery_percentage, battery_voltage, and battery_timestamp values
	in the nodes table.  It's currently only called by the watchdog
	running on the garcias.

	* tmcd/common/libtmcc.pm: Add TMCCCMD_BATTERY function.

	* tmcd/common/watchdog: Updated to send back battery information
	obtained from garcia-pilot.

* Add boot_errno to the nodes table so that nodes can report in a
  subcode to indicate what went wrong. At present, we do not report any
  real error codes; that is going to take some time to work out since it
  will reqiure a bunch of changes to the boot scripts.

* Add new table node_bootlogs to store logs provided by the nodes. Not
  a full console log, but a log of the tmcd client side part. We can
  make it a full log if we want though; just means mucking about with
  the boot phase a bit.

* Add new state transition to NORMALv2 and PCVM state machines. "TBFAILED"
  is a new state that is sent (after TBSETUP) if a node fails somewhere in
  the tmcd client side.

* Change TBNodeStateWait() to take a list of states (instead of single
  state) and an optional pass by reference parameter to return the actual
  state that the node landed in. Change all calls to TBNodeStateWait() of
  course.

* Change os_setup (and libreboot in wait mode) to look for both TBFAILED
  and ISUP. If a TBFAILED event is seen, we can terminate the wait early
  and not retry os_setup on physical nodes (although still retry virtual
  nodes). The nice thing about this is that the wait should terminate much
  earlier (rather then waiting for timeout), especially for virtual nodes
  which can take a really long time when there are a couple of hundred.

* Add new routines dobooterrno() and dobootlog() to tmcd. Bump version
  number and increase the buffer size to allow for the larger packets that
  a console log wikk generate (added MAXTMCDPACKET variable, set to 0x4000).

* Add new -f option to tmcc to specify a datafile to send along as the last
  argument to tmcd. This is more pleasing then trying to send a console log
  in on the command line. For example: "tmcc -f /tmp/log BOOTLOG" will send
  a BOOTLOG command along with the contents of /tmp/log.

  Also close the write side of the pipe so that server sees EOF on
  read. See aside comment below.

* Changes to rc.bootsetup:
     1. Use perl tricks to capture all output, duping to the console and to
        a log file in /var/emulab/logs.
     2. On any error, send a status code (boot_errno) and the bootlog to
        tmcd.
     3. Generate a TBFAILED state transition.

* Changes to rc.injail:
     1. Same as rc.bootsetup, but do not send log files; that would pummel
        boss. Leave them on the physical node.

* Change vnodesetup (which calls mkjail) to watch for any error and send a
  TBFAILED state transition. This should catch almost all errors, and
  dramatically reduce waiting when something fails.

* Changes to rc.cdboot are essentially the same as rc.bootsetup, although a
  bootlog is sent all the time (success or failure), and I do not generate
  a boot_errno yet. Also, instead of TBFAILED, generate a PXEFAILED state
  since the CDROM is actually operating within the PXEFBSD opmode. I have
  yet to work this into the rest of the system though; waiting to get a new
  CD built and actually experiment with it.

* Add new menu option and web page to display the node bootlog. We store
  only the lastest bootlog, but maybe someday store more then one. Display
  boot_errno on node page.

Aside: I made a big mistake in the tmcd protocol; I did not envision
passing more then a small amount of data (one fragment) and so I do not
include a record terminator (ie: close of the write side on the client
sends EOF) or a size field at the beginning. No big deal since small
requests are sent in one fragment and the server sees the entire
thing. Well, with a large console log, that will end up as multiple
fragments, and the server will often not get the entire thing on the first
read, and there are no subsequent reads (with no EOF or known size, it
would block forever). Well, fixing this in a backwards compatable manner
(for old images) was way too much pain. Instead, tmcc now closes the write
side, and the server does subsequent reads *only* in the new dobbootlog()
routine. Note that it *is* possible to fix this in a backwards compatable
manner, but I did not want to go down that path just yet.

installs a new root pubkey into /root/.ssh, which comes from a
sitevar.

CDVERSION= for CD-booted machines.

Add "emulabconfig" goo to libtmcc.

handy by itself; add -i option to tmcc (C and perl version) to turn
off SSL mode. So instead of a separate nossl binary, the SSL binary
can now operate in nossl mode. Like the previous revision, this is
also controlled by an environment variable so that rc.inelab can make
sure that all children use the nossl mode. Why? Cause the inner elab
will have its own set of certificates, and the outer elab tmcd will
reject the connections. Simpler to just speak nossl to the outer elab,
rather than try to mess with two sets of certs.

Overview of simply firewall setup.

Experimentor specifies in their ns file:

     set fw [new Firewall $ns]
     $fw style <open|closed|basic>

to set up an "open" ("allow any"), "closed" ("deny any"), or "basic"
(allow ICMP and ssh) firewall.  "basic is the default.  Additional rules
can be added with:

     $fw add-rule <IPFW format rule>
     $fw add-numbered-rule <1-50000> <IPFW format rule>

where the former implicitly numbers rules such that the firewall processes
them in the order given in the NS file.  The latter allows explicit
specification of the numbering.  Currently the rules are fixed strings,
there is no variable substitution.  There is also no syntax checking done
on the rules at parse time.

We allocate an extra node to the experiment to serve as a firewall.
Currently that node runs FreeBSD and uses IPFW.  In the initial configuration,
all other nodes in the experiment will just be setup with a default route
that points to the firewall node.  So all outbound traffic will pass through
it.  Inbound traffic will still travel straight to the node.  This should
prevent nodes from accidentally initiating attacks on the outside world.
Long term we will of course enforce the firewall on all traffic, that should
not have any effect on the NS syntax above.

When a node boots, there will be an rc.firewall script that checks to see
if there is a firewall for the experiment and if so, which node it is.
This is done with the TMCD "firewallinfo" command which returns:

      TYPE=none

      TYPE=remote FWIP=N.N.N.N

      TYPE=<fwtype> STYLE=<fwstyle> IN_IF=<macaddr> OUT_IF=<macaddr>
      RULENO=<num> RULE="<ipfw command string>"
      RULENO=...
      ...

In the case of no firewall we get back TYPE=none, and we continue as normal.
Otherwise, there are two types of replies, one for a node that is being
firewalled (TYPE=remote) and one for a node that is a firewall
(TYPE=<fwtype> + RULES).

In the TYPE=remote case, the firewall node indicated by FWIP.  This is
the address we use for the default route.

For TYPE=<fwtype>, we are the firewall, and we get STYLE and IN_IF/OUT_IF
info.  Here TYPE indicates whether we should use ipfw or whatever.
For now it is always ipfw.  IN_IF and OUT_IF may someday indicate the
interfaces to use for the internal and external connections, right now
both will indicate the control net interface.  So, after ensuring that
the ipfw modules is loaded, we grab the provided RULE info, which includes
both per-experiment and default rules, and setup ipfw.

Issues to resolve:
       - synchronization: how to ensure firewall comes up first
       - how to better implement the firewalling
         (i.e., without the cooperation of the nodes)
       - support the equiv of linkdelays (on-node firewalling)?
       - allow firewalls within experiments?
         (ie., on experimental interfaces)
       - dynamic changing of firewall rules via events?
       - how to show firewall state in various web pages