doing a Start Run. On the web page, there is a new checkbox, and
on ops, template_startrun takes a new -m option.

Caveat: You cannot specify a new NS file, yet. The original file is
reparsed, and the idea is that a change in the template parameters
will result in a change to the topology. I will add the ability to
specify a new NS file in the next revision of this change.

If you really really want to change the NS file, go to
/proj/$pid/exp/$eid/archive/nsdata and edit nsfile.ns ...

In addtion, DATASTORE is now defined while parsing the NS file. This
turned to be quite the headache!

to make stoprun waiting work correctly.

When tevc is invoked with the -w (wait for completion) option, tevc
generates a token to put into the notification. The event scheduler will
not generate a new token if there is already on in the notification, but
instead pass it on.

For the specific case of stoprun, the simulator agent has to pass that
token along to boss and template_exprun, which generates the completion
event (for reasons discussed in prior commit message).

it got more complicated as it progressed.

The bulk of the change was changing template_exprun so that it can take a
pid/eid as an alternative to eid/guid. This is a big convenience since its
easy to find the template from a running experiment, and it makes it
possible to invoke from the event scheduler, which has never heard of a
template before (and its not something I wanted to teach it about).  Its
also easier on users.

Anyway, back to the stoprun event. You can now do this:

	$ns at 100 "$ns stoprun"
or
	tevc -e pid/eid now ns stoprun

You can add the -w option to wait for the completion event that is sent,
but this brings me to the glaring problems with this whole thing.

* First, the scheduler has to fire off the stoprun in the background,
  since if it waits, we get deadlock. Why? Cause the implementation of
  stoprun uses the event system (SNAPSHOT event, other things), and if
  the scheduler is sitting and waiting, nothing happens.

  Okay, the solution to this was to generate a COMPLETION event from
  template_exprun once the stop operation is complete. This brings me
  to the second problem ...

* Worse, is that the "ns" events that are sent to implement stoprun (like
  snapshot) send their own completion events, and that confuses anyone
  waiting on the original stoprun event (it returns early).

  So what to do about this? There is a "token" field in the completion
  event structure, which I presume is to allow you to match things up.  But
  there is no way to set this token using tevc (and then wait for it), and
  besides, the event scheduler makes them up anyway and sticks them into
  the event. So, the seed of a fix are already germinating in my mind, but
  I wanted to get this commit in so that Mike would have fun reading this
  commit log.

so that users can schedule program events to run there. For example:

	set myprog [new Program $ns]
	$myprog set node "ops"
	$myprog set command "/usr/bin/env >& /tmp/foo"

	$ns at 10 "$myprog start"
or
	tevc -e pid/eid now myprog start

Since the program agent cannot talk to tmcd from ops, there are new
routines to create the config files that the program agent uses, in
the expertment tbdata directory.

I also rewrote the eventsys.proxy script that starts the event
scheduler on ops; I rolled the startup of the program agent into this
script, via new -a option which is passed over from boss when an ops
program agent is detected in the virt topology. This keep the number
of new processes on ops to a small number.

Also part of the above rewrite is that we now catch when event
scheduler (or the program agent) exits abnormally, sending email to
tbops and the swapper of the experiment. We have been seeing abnormal
exits of the scheduler and it would good to detect and see if we can
figure out what is going wrong.

Other small bug fixes in experiment run.

* Add XMLRPC interface for template swapin,stoprun,startrun,swapout and
  add the appropriate wrappers to the script_wrapper on ops.

* Allow parameter descriptions in NS files. This is probably not in its
  final form since its a bit confusing as to what has priority; something
  in the NS file or a metadata item. Anyway, you can do this in your NS
  file:

	$ns define-template-parameter GUID "0/0" "The GUID to be analyzed"

  The rules are currently that the NS file description has priority and
  is copied to child templates, unless the user has modified a description
  via the web interface, in which case the NS file description is ignored.
  I know, sounds awful, but for the most part people are going to use the
  NS file anyway.

* Add "clear" option when starting a new experiment run; the per
  experiment DB at the logholes are cleared. Note that this is *not* the
  default behaviour; you have to either check the checkbox on the web form
  or use the -c option to the script wrapper, or clear=yes if talking
  directly to the XMLRPC server.

* Fix up how email is generated for template_swapin and template_create,
  so that Kevin can debug tblog/tbreport stuff, but also so that we maintain
  mail logs as before. I have made some improvements to libaudit so as to
  centralize the mail goo, and avoid duplicating all that stuff.

* Minor fixes to the program agent so that the new environment strings are
  sent before the program agent exits and reloads them!

* Other minor little things.

* Export the above via the XMLRPC interface and add a wrapper function
  to the script_wrapper. This allows you do to this on ops:

	cd /proj/testbed/templates/10023/1
        Edit some files
        template_commit

  Which creates a new template, using the current directory to infer
  the template. Otherwise, provide the template GUID on the command line.
  Hmm, maybe this should be called template_modify? Either way, the
  name does not quite match

* Export template_export via the XMLRPC wrapper. This allows you to
  export a template (instance) record from the command line on ops.


	cd /proj/testbed/templates/10023/1
        template_export -i 12
        Exported to /proj/testbed/export/10000/3/12

  Which exports the template record for instance number 12. Again, the
  GUID is infered, but you can specify one on the command line. The export
  directory is printed so you know where it went. Note that export does
  *not* populate a DB on ops with the old DB data.

draft is that the user will at the end of an experiment run, log into one
of his nodes and perform some analysis which is intended to be repeated at
the end of the next run, and in future instantiations of the template.

A new table called experiment_template_events holds the dynamic events for
the template. Right now I am supporting just program events, but it will be
easy to support arbitrary events later. As an absurd example:

	node6> /usr/local/bin/template_analyze ~/data_analyze arg arg ...

The user is currently responsible for making sure the output goes into a
file in the archive. I plan to make the template_analyze wrapper handle
that automatically later, but for now what you really want is to invoke a
script that encapsulates that, redirecting output to $ARCHIVE (this
variable is installed in the environment template_analyze.

The wrapper script will save the current time, and then run the program.
If the program terminat...

into per-experiment databases on ops. Additional support for reconsituting
those databases back into temporary databases on ops, for post processing.

* This revision relies on the "snort" port (/usr/ports/security/snort)
  to read the pcap files and load them into a database. The schema is
  probably not ideal, but its better then nothing. See the file
  ops:/usr/local/share/examples/snort/create_mysql for the schema.

* For simplicity, I have hooked into loghole, which already had all
  the code for downloading the trace data. I added some new methods to
  the XMLRPC server for loghole to use, to get the users DB password
  and the name of the per-experiment database. There is a new slot in
  the traces table that indicates that the trace should be snorted to
  its DB. In case you forgot, at the end of a run or when the instance
  is swapped out, loghole is run to download the trace data.

* For reconsituting, there are lots of additions to opsdb_control and
  opsdb_control.proxy to create "temporary" databases and load them
  from a dump file that is stored in the archive. I've added a button
  to the Template Record page, inappropriately called "Analyze" since
  right now all it does is reconsitute the trace data into a DB on
  ops.

  Currently, the only indication of what has been done (the name of
  the DBs created on ops) is the log email that the user gets. A
  future project is tell the user this info in the web interface.

* To turn on database capturing of trace data, do this in your NS
  file:

	set link0 ...
	$link0 trace
	$link0 trace_snaplen 128
	$link0 trace_db 1

   the increase in snaplen is optional, but a good idea if you want
   snort to undertand more then just ip headers.

* Also some changes to the parser to allow plain experiments to take
  advantage of all this stuff. To simple get yourself a per-experiment
  DB, put this in your NS file:

	tb-set-dpdb 1

  however, anytime you turn trace_db on for a link or lan, you
  automatically get a per-experiment DB.

* To capture the trace data to the DB, you can run loghole by hand:

	loghole sync -s

  the -s option turns on the "post-process" phase of loghole.

Actually, most of the changes here were just to generalize the "virtual
interface" state in the DB.  Other than the client-side scripts, there
is very little specific here specific to tagged VLANs.

In fact, you cannot specify "vlan" as a type yet as we haven't done the
snmpit support for setting up the switches.

For more info see bas:~mike/flux/doc/testbed-virtinterfaces.txt, which I
will integrate into the knowledge base and the Emulab doc at some point.

wrong).

collab_password). I'm using this from the event scheduler so it can
access the per-experiment DB to store the event trace.

I suppose it would have been easier to stick them in a file?

does not handle. Just kill it from the response in osid.info; the caller is
not interested.

timeline to __ns_teardown so the evproxies unsubscribe from the
experiment.

Some pelab/plab event system hacks.

	* event/lib/event.h, event/lib/event.c: Add event_subscribe_auth
	which lets you specify whether any authentication should be done
	for events received through this subscription.

	* event/sched/event-sched.c: Handle EVPROXY objects.  Add a
	separate subscription for EVPROXY UPDATE events for each plab
	pnode (which might be too many...).  Also, need to update the EXPT
	field for events received through a noauth subscription so the
	proxies can figure out which experiments are active.

	* lib/libtb/tbdefs.h, lib/libtb/tbdefs.c: Add UPDATE event defs.

	* xmlrpc/emulabserver.py.in: Inject __plab_setup and
	__plab_teardown timelines into the eventlist when an experiment
	has plab nodes.  The __plab_setup timeline sends EVPROXY UPDATE
	events to each physical node while the __plab_teardown sends
	EVPROXY CLEAR events.  The __plab_setup timeline is run when the
	scheduler starts up, the __plab_teardown isn't run automatically
	yet.

an inner emulab.

scheduler can startup.

and a few other tweaks.  Also bumped parallelism from 10 to 25 rsyncs
at a time.

can run.

number.

Make the netlab client applet available to locals.

	* configure, configure.in: Add xmlrpcpipe.php3.

	* xmlrpc/emulabserver.py.in: Add missing virtual_tables.  Add
	getareas call to get the list of robot areas.  Add node.getlist
	and node.typeinfo methods for getting information about the nodes.
	Add a "nic" argument to node.available to get the count of
	wireless nodes.  Add "exclude" argument to
	experiment.virtual_topology so we don't have to download the
	massive route table, also delete the 'pid'/'eid' fields for the
	same reason.  Don't return string output of virtual_topology, it's
	huge.  Return some more info in experiment.getlist().

	* www/GNUmakefile.in: Add xmlrpcpipe.php3.

	* www/beginexp_form.php3, www/modifyexp.php3: Add links to the
	client gui.

	* www/netlab-client.jar: The client binary.

info to admin users.

tipacl data. Sends it to stdout.

experiment.nsfile methods.  Prune some more fields out of the metadata
return value.

experiment.

Add a minor feature to loghole to support network link trace files.

	* utils/loghole.in, utils/loghole.1: Create a directory for traced
	network links that contains symbolic links to the pcap files.

	* www/tutorial/advanced.html: Fix the link to the loghole doc.

	* xmlrpc/emulabserver.py.in: Add a 'traces' aspect to the
	experiment.info method (used by loghole).

Event system documentation and tweaks:

	* event/sched/event-sched.c: The snapshot event got left out of
	the list of events that send back a complete event.  If a top
	level sequence stops with an error, automatically send a report.

	* event/sched/group-agent.c, event/sched/timeline-agent.c: When
	handling a complete event, return true if the event was consumed.

	* event/sched/node-agent.cc: Clean the "/local/logs" loghole on
	the node before taking a snapshot of the disk image.

	* event/sched/simulator-agent.cc, event/sched/simulator-agent.h:
	Made send_report non-static so it can be called when a sequence
	errors out.  Actually check the clear flag when dumping report
	data.

	* lib/libtb/tbdefs.c: Add a comment about updating the event
	scheduler when a new object type is added.

	* tbsetup/ns2ir/sim.tcl.in: Check the OSIDs passed to the "reload"
	and "snapshot-to" events.

	* utils/loghole.1: Mention early and often the directories that
	gets sync'd.  Document the 'change' action and new options for
	'clean'.

	* utils/loghole.in: Use rsync's '--include-from' option so we
	don't have to run rsync separately for each directory.  Add a
	'change' action that lets people mess with the metadata of an
	existing archive.  Make the 'clean' action also remove log files
	on the nodes.  Only rsync node's whose OS's support ssh.  Grab
	logs from delay nodes.

	* www/tutorial/advanced.html: Update the program agent examples to
	use the new form.

	* www/tutorial/eventsystem.html: First cut at an event system
	reference manual that includes sequences, timelines, and other
	new features.

	* www/tutorial/loghole.html: Just a pointer to the man page on ops
	for now.

	* www/tutorial/nscommands.html: Add a section about capturing
	parameters from the 'opt' array.

	* xmlrpc/emulabserver.py.in: Add an osid.info() method that
	returns information about a single OSID.

experiments can be auto fired during swapin.

the latter gives us one big argument with spaces, etc. escaped.

work inside ElabInElab.

The crux of this is to emulate what switchmac does when incorporating nodes
into the testbed. Rather then using snmpit to ask the switches for MAC info,
we can just ask outer emulab via the proxy to get that same info from the
DB. Note, there are no changes to the newnode MFS; it will boot and happily
report it after launching teachswitch, but those packets are harmless to the situation.

Another wrinkle is that we need the outer emulab to tell us which
interfaces are control and which are experimental, since inside there
is no real way to determine that. I did this by overloading the final
field in the return from switchmac, and using that to override the
"role" setting that utils/newnode would normally choose on its own.

There is another new routine in the XMLRPC server that is intended to
be used for bypassing the newnode path. This is not hooked into
anything yet, but the intent is that rather then using the web
interface to "add" nodes, we just return everything needed to seed the
new_nodes and new_interfaces table, and then run newnodes directly.

Mote and robot related stuff.  The main thing is the addition of relay
capabilities to capture and related things.

	* GNUmakefile.in: Add the capture and tip subdirectories to the
	client and client-install targets.

	* configure, configure.in, config.h.in: Detect srandomdev() for
	capture and add "mote/newmote" script.

	* capture/GNUmakefile.in, capture/capture.c: Add "relay"
	capabilities to capture.

	* capture/capquery.c: Query the capserver for the relay receiver's
	port number.

	* capture/capserver.c: Small hack to return the port number
        for a node.

	* db/libdb.pm.in, db/xmlconvert.in: Add virt_tiptunnels table.

	* event/program-agent/program-agent.c: Change log file names to
	something a little more user-friendly.  Add a "MODIFY" event
	handler that lets the user set agent attributes (command, tag,
	timeout) without having to run a program.

	* event/sched/GNUmakefile.in, event/sched/console-agent.cc,
	event/sched/console-agent.h, event/sched/event-sched.c: Add
	console agents that can be used to snapshot a section of the
	capture log file.

	* event/sched/node-agent.cc: Some minor cleanup.

	* event/sched/simulator-agent.cc, event/sched/simulator-agent.h:
	Add the config data to the report mail.  Add a "RESET" event
	handler that runs "loghole clean".  Save the report mail in a file
	so it gets archived with the rest of the logs.

	* lib/libtb/tbdefs.h: Add CONSOLE object type.

	* mote/GNUmakefile.in, mote/newmote: Add newmote script, just a
	quick hack to add motes to the DB.

	* mote/tbuisp.in: Add another backend for loading motes through
	their relay capture server.

	* robots/mtp/mtp_dump.c: Dump the min/max values for x and y,
	handy for figuring out the bounds of the camera.

	* sql/database-fill.sql: Change the RELOAD-MOTE/SHUTDOWN ->
	ALWAYSUP/SHUTDOWN mode transition to ALWAYSUP/ISUP since stated
	doesn't seem to run triggers after a state change by a mode
	transition.

	* tbsetup/tbreport.in: Change the ordering of the eventlist so it
	displays event-sequences appropriately.

	* tbsetup/ns2ir/GNUmakefile.in, tbsetup/ns2ir/console.tcl,
	tbsetup/ns2ir/node.tcl, tbsetup/ns2ir/parse.tcl.in,
	tbsetup/ns2ir/sim.tcl.in: Add a "console" agent that represents
	the serial console for a node.

	* tbsetup/ns2ir/sequence.tcl: Add an "append" method so it is
	easier to build sequences dynamically.

	* tbsetup/ns2ir/topography.tcl: Make checkdest available to
	regular users.

	* tip/GNUmakefile.in, tip/tiptunnel.c: Add support for uploading a
	file to a relay version of capture and exporting the end
	connection as a pty.

	* tmcd/decls.h, tmcd/common/libsetup.pm: Bump version number since
	the dosubnodelist change is not backwards compatible.

	* tmcd/tmcd.c: Make dosubnodelist and dosubconfig callable even
	when a node isn't allocated.  Add dotiptunnels command that
	returns which serial consoles are to be mounted on a node.  Add
	mote version of subconfig that returns information needed to
	startup the relay version of capture.

	* tmcd/common/bootsubnodes: For motes, startup the relay version
	of capture (XXX stargate specific).

	* tmcd/common/libsetup.pm, tmcd/common/libtmcc.pm,
	tmcd/common/config/rc.config, tmcd/common/config/rc.tiptunnels:
	Client side changes for mounting another nodes serial line.

	* tmcd/common/rc.bootsetup: Always boot the subnodes, even when
	free.  This is used for motes since their capture needs to be up
	for reloading at the time.

	* tmcd/linux/ixpboot: Shuffle some code around so the script
	doesn't fail if the ixp isn't allocated.

	* utils/loghole.in: Add "digest.out" and "report.mail" as global
	logs to be saved in archives and display the "report.mail" file
	when showing a loghole archive.

	* xmlrpc/emulabserver.py.in: Scrub more of the return values to
	get rid of "None"s.

Fix a couple of things I missed in my last checkin.

	* event/program-agent/program-agent.c: Setup of the user
	environment variables wasn't ordered correctly.

	* event/sched/event-sched.c, event/sched/rpc.h,
	event/sched/rpc.cc: Setup the expected user environment variables,
	just like in program-agent.

	* event/sched/simulator-agent.cc: Rename the old feedback data so
	we don't lose it in case the vnode remapping modify fails.

	* xmlrpc/emulabserver.py.in: Add virt_user_environment table data
	to the result of the experiment.metadata method, so the
	event-scheduler can get it.

Checkin some changes related to experiment automation and vnode feedback:

	* configure, configure.in: Add sensors/canaryd/feedbacklogs
	template.

	* db/libdb.pm.in, db/xmlconvert.in: Add "virt_user_environment"
	table that holds environment variable names and values.

	* event/lib/event.c: Allocate memory of the right size for
	event_notifications.

	* event/program-agent/GNUmakefile.in: Add version.c file and
	add install targets for the man page.

	* event/program-agent/program-agent.8: Man page describing the
	program-agent daemon.

	* event/program-agent/program-agent.c: Add a bunch of convenience
	features: let the user specify the working directory for commands;
	save output to separate files on every invocation of an agent; let
	the user specify a timeout for a command; make the set of
	environment variables sane and add vars given in the NS file in
	the opt array; a "status" file containing process information is
	written out when children are collected.  Internal changes: child
	processes are collected immediately, instead of waiting for the
	next START event, so we can send back COMPLETE events; the daemon
	now runs with a real-time priority, to increase the chances of
	receiving events.

	* event/proxy/evproxy.c: Made it bidirectional so the
	program-agent's COMPLETE events make it back to the scheduler.

	* event/sched/error-record.c: Change the default log directory.

	* event/sched/event-sched.h, event/sched/event-sched.c: Setup an
	environment similar to a program-agent to run the user's log
	digester.

	* event/sched/node-agent.cc: Add a handler for the SNAPSHOT event
	that runs create_image for the node.

	* event/sched/simulator-agent.h, event/sched/simulator-agent.cc:
	Let the user specify a "DIGESTER" script that digests the log
	files into a summary of the results.  Add event handler for
	remapping a vnode experiment.

	* event/sched/timeline-agent.c: Accept the RUN event as well as
	the START event.

	* os/GNUmakefile.in: Install the install-tarfile.1 man page.

	* os/install-tarfile: Automatically chown/chgrp any files that do
	not have valid user or group IDs, the new owner will be the user
	that swapped in the experiment.  Include the install directory in
	the DB file.  Add a "list" mode that just dumps what files have
	been installed and where.  Add a "force" option so the user can
	forcefully install the file, even though the DB says its already
	there.

	* os/install-tarfile.1: Man page describing the install-tarfile
	tool.

	* os/syncd/GNUmakefile.in: Install man pages on ops.

	* sensors/canaryd/GNUmakefile.in: Link canaryd statically and
	install "feedbacklogs" tool.

	* sensors/canaryd/canaryd.c: Dump dummynet pipe data.

	* sensors/canaryd/canarydEvents.c: Log errors.

	* sensors/canaryd/feedbacklogs.in: Tool used to generate feedback
	data from canaryd log files.

	* sensors/slothd/GNUmakefile.in: Install digest-slothd on ops.

	* sensors/slothd/digest-slothd: Fix some bugs and write out an
	"alert" file with all the nodes/links that were overloaded.

	* tbsetup/os_load.in, tbsetup/libosload.pm.in: Add "waitmode"
	argument that lets you specify that you want to wait for the disk
	to finish loading and/or wait for the node to come back up in the
	new OS.

	* tbsetup/power.in: Remove debugging printf.

	* tbsetup/ns2ir/node.tcl, tbsetup/ns2ir/program.tcl,
	tbsetup/ns2ir/sequence.tcl, tbsetup/ns2ir/sim.tcl.in: Fix some
	quoting problems with event-sequences.  Add -expected-exit-code
	and -tag options to the "$program run" event.  Add -digester to
	the "$ns report" event that lets the user specify a program to run
	to digest the log files.

	* tbsetup/ns2ir/tb_compat.tcl.in: Change the initial scaling
	factor for feedback nodes to 1%, instead of 100%.

	* tmcd/tmcd.c, tmcd/common/libtmcc.pm: Add "userenv" command that
	returns the values in "virt_user_environment".  Return new program
	agent fields: dir, timeout, and expected_exit_code.

	* tmcd/common/GNUmakefile.in: Install rc.canaryd.

	* tmcd/common/bootvnodes: Add hack to boost the program-agents to
	a real-time priority, they can't do it from inside the jail.

	* tmcd/common/rc.canaryd: Rc script for canaryd.

	* tmcd/common/watchdog: Don't fail outright if there is a bad line
	in the battery.log

	* tmcd/common/rc.progagent: Append "userenv" data to the
	program-agent config file.

	* utils/GNUmakefile.in: Install loghole and its man page on ops.

	* utils/loghole.1: Document "clean" command and the change in
	loghole directories.

	* utils/loghole.in: Add "clean" command and parallelization.

	* xmlrpc/emulabserver.py.in: Add "virt_user_environment" table.
	Order the eventlist by "idx" and time, needed for sequences.  And
	removed unnecessary nologin checks.