that works for people who are just "average" ...

thankless job but someone has to do it. I'm expecting to finish by the
time Bush 43 leaves office.

checkboxes.

I have disabled the speculative run code, and restored the original
code.  It was a pretty minor change to bring it back, and we can
around circles as many times as we like.

no such luck. IE sucks.

annotation.

  register_globals=1 to turn POST/GET/COOKIES arguments in local variables.
  This is known to be a terrible security risk, and we keep saying we are
  going to fix it, and now I am. In order to accomplish this on a
  transitional basis (since I don't want the entire web interface to stop
  working while I debug it), and because the code just needs the cleanup, I
  am doing it like this: Each page will sport new declarations at the top:

	RequiredPageArguments("experiment", PAGEARG_EXPERIMENT,
                              "template",   PAGEARG_TEMPLATE,
                              "instance",   PAGEARG_INSTANCE,
                              "metadata",   PAGEARG_METADATA,
                              "osinfo",     PAGEARG_OSINFO,
                              "image",      PAGEARG_IMAGE,
                              "project",    PAGEARG_PROJECT,
                              "group",      PAGEARG_GROUP,
                              "user",       PAGEARG_USER,
			      "node",       PAGEARG_NODE,
			      "yesno",      PAGEARG_BOOLEAN,
			      "message",    PAGEARG_STRING,
			      "age",        PAGEARG_INTEGER,
                              "cost",       PAGEARG_NUMERIC,
                              "formfields", PAGEARG_ARRAY,
                              "unknown",    PAGEARG_ANYTHING);

	OptionalPageArguments("canceled", PAGEARG_BOOLEAN);

  The first token in each pair is the name of the global variable to
  set, and the second token is the type. So, for "experiment" we look at
  the URL for a pid/eid or exptidx, etc, sanity check them (safe for a
  DB query), and then try to find that experiment in the DB. If it maps
  to an experiment, set global variable $experiment to the object. Since
  its a required argument, produce an error if not supplied. Similar
  treatment for optional arguments, with the obvious difference.

  The goal is to have ALL argument processing in one place, consistent,
  and correct. I've found numerous places where we leak unchecked
  arguments into queries. It also cuts out a lot of duplicated code.

* To make the above easier to deal with, I've been replacing lots of
  hardcoded URLS in the code of the form:

	foo.php3?pid=$pid&eid=$eid ...

  with

        CreateURL("foo", $experiment)

  which creates and returns the neccessary url string, by looking at
  the type of its arguments (experiment, template, instance, etc.)

  Eventually plan to replace them all so that URL handling throughout
  the code is all defined in one place (all the new URL code is in
  url_defs.php).

* I have cranked up error reporting to tell me anytime a variable is
  used before it is initialized, plus a bunch of other stuff that PHP
  deems improper. Think of it like -Wall ... and boy we get a lot of
  warnings.  A very large percentage of the diffs are to fix all these
  warnings.

  The warnings are currently going to /usr/testbed/log/php-errors.log,
  and I'll be adding a script to capture them each night and mail them
  to tbops. This file also gets errors (this will be a change for
  developers; rather then seeing errors and warnings dumped in the
  middle of web pages, they will go to this file instead).

* Major refactoring of the code. More objects (nodes, images, osids).
  Moving tons of queries into the objects in the hopes of someday
  getting to a point where we can split the web interface onto a
  different server.  Lots of general cleanup.

converting to locally unique ids and later globally unique ids.

When a run is over changed:
  Template: 10424/23 (Current Run: bt-R54)
to
  Template: 10424/23 (Last Run: bt-R54)
Rather Than displaying
  Template: 10424/23 (Current Run:)

Also changed "Instance Bindings" to "Last Run Bindings" since that makes
much more sense, at least to me. (FS # 128:
https://users.emulab.net/flyspray/?do=details&id=128)
The function to show Instance Bindings still exists so it can easily be
changed back.

not swapped in (equiv of plain experiment preload), and then swapped
in later. This fulfills flyspray request FS#100.

* New version of template_record just for ops, since so much is
  different about ops, not bothering to maintain a single version.

* Various fixes to how the recorded events are stored and reconstituted.
  The big fix is to wrap them in a sequence to that they get fired
  properly (waiting for completion of previous event in recording).

* New buttons to Pause and Continue event time, which is used when
  adding recorded events. This allows users to pause time while they
  "think" so when an event is recorded, the thinking time is not actually
  in the timeline. Eventually hope to figure this out automatically, but
  that will take some real, uh, thinking.

* Add a new event editor (linked off the template page) that allows
  you to delete and change the recordings. Note that you can only edit
  the events at the template level; you cannot edit the events of an
  instance (swapped in experiment), and you can only edit the recorded
  events, not any other events. Not sure its useful to be able to do
  either of these yet, but probably not too hard to add at some point.

the Show Experiment menu to see if anyone uses it.

* Add XMLRPC interface for template swapin,stoprun,startrun,swapout and
  add the appropriate wrappers to the script_wrapper on ops.

* Allow parameter descriptions in NS files. This is probably not in its
  final form since its a bit confusing as to what has priority; something
  in the NS file or a metadata item. Anyway, you can do this in your NS
  file:

	$ns define-template-parameter GUID "0/0" "The GUID to be analyzed"

  The rules are currently that the NS file description has priority and
  is copied to child templates, unless the user has modified a description
  via the web interface, in which case the NS file description is ignored.
  I know, sounds awful, but for the most part people are going to use the
  NS file anyway.

* Add "clear" option when starting a new experiment run; the per
  experiment DB at the logholes are cleared. Note that this is *not* the
  default behaviour; you have to either check the checkbox on the web form
  or use the -c option to the script wrapper, or clear=yes if talking
  directly to the XMLRPC server.

* Fix up how email is generated for template_swapin and template_create,
  so that Kevin can debug tblog/tbreport stuff, but also so that we maintain
  mail logs as before. I have made some improvements to libaudit so as to
  centralize the mail goo, and avoid duplicating all that stuff.

* Minor fixes to the program agent so that the new environment strings are
  sent before the program agent exits and reloads them!

* Other minor little things.

draft is that the user will at the end of an experiment run, log into one
of his nodes and perform some analysis which is intended to be repeated at
the end of the next run, and in future instantiations of the template.

A new table called experiment_template_events holds the dynamic events for
the template. Right now I am supporting just program events, but it will be
easy to support arbitrary events later. As an absurd example:

	node6> /usr/local/bin/template_analyze ~/data_analyze arg arg ...

The user is currently responsible for making sure the output goes into a
file in the archive. I plan to make the template_analyze wrapper handle
that automatically later, but for now what you really want is to invoke a
script that encapsulates that, redirecting output to $ARCHIVE (this
variable is installed in the environment template_analyze.

The wrapper script will save the current time, and then run the program.
If the program terminates with a zero exit status, it will ssh over to ops
and invoke an xmlrpc routine to tell boss to add a program event to both
the eventlist for the current instance, and to the template_eventlist for
future instances. The time of the event is the relative start time that was
saved above (remember, each experiment run replays the event stream from
time zero).

For the future, we want to allow this to be done on ops as well, but
that will take more infrastructure, to run "program agents" on ops.

It would be nice to install the ssl xmlrpc client side on our images so
that we do not have to ssh to ops to invoke the client.

kinds of graphs for a template instantiation.

into per-experiment databases on ops. Additional support for reconsituting
those databases back into temporary databases on ops, for post processing.

* This revision relies on the "snort" port (/usr/ports/security/snort)
  to read the pcap files and load them into a database. The schema is
  probably not ideal, but its better then nothing. See the file
  ops:/usr/local/share/examples/snort/create_mysql for the schema.

* For simplicity, I have hooked into loghole, which already had all
  the code for downloading the trace data. I added some new methods to
  the XMLRPC server for loghole to use, to get the users DB password
  and the name of the per-experiment database. There is a new slot in
  the traces table that indicates that the trace should be snorted to
  its DB. In case you forgot, at the end of a run or when the instance
  is swapped out, loghole is run to download the trace data.

* For reconsituting, there are lots of additions to opsdb_control and
  opsdb_control.proxy to create "temporary" databases and load them
  from a dump file that is stored in the archive. I've added a button
  to the Template Record page, inappropriately called "Analyze" since
  right now all it does is reconsitute the trace data into a DB on
  ops.

  Currently, the only indication of what has been done (the name of
  the DBs created on ops) is the log email that the user gets. A
  future project is tell the user this info in the web interface.

* To turn on database capturing of trace data, do this in your NS
  file:

	set link0 ...
	$link0 trace
	$link0 trace_snaplen 128
	$link0 trace_db 1

   the increase in snaplen is optional, but a good idea if you want
   snort to undertand more then just ip headers.

* Also some changes to the parser to allow plain experiments to take
  advantage of all this stuff. To simple get yourself a per-experiment
  DB, put this in your NS file:

	tb-set-dpdb 1

  however, anytime you turn trace_db on for a link or lan, you
  automatically get a per-experiment DB.

* To capture the trace data to the DB, you can run loghole by hand:

	loghole sync -s

  the -s option turns on the "post-process" phase of loghole.

create a new template (well, really a modify) from the current
swapped in experiment. This allows you to create a template, swap in
an instance, modify the datastore in the instance (which is a copy
of the datastore in the template), and then create a new template
using the datastore and nsfile from the instance. This is a new menu
item on the showexp page for the instance.

Also in this commit are fixes and improvements to the new navagation
bar that I recently added.

* add an "active" flag to the template record, which will be used by
  the user to indicate what templates he wants listed (rather then the
  roots). Basically, the current working templates, rather then a big
  graph.

But I never actually finished that cause it sorta morphed into:

* Added a vis_graphs table to cache the last generated visualization
  rendering in the database so that we do not have to wait so long for:

* Add new buttons to showexp and template_show pages, to display in the
  same page either the settings (current view), the NS file, or the
  visualization (along with zoom in/out buttons).

And now I can go back to that "active" thing I mentioned up above ...

template show page ...

but that stuff is still not totally hooked in yet.

much "fun" as the Perl conversion, but nothing about PHP is as much
fun as it is in perl.

"object" and this was a good opportunity to see if they are useful and
easy enough to use. Yep they are; the code is much cleaner with many
fewer utility functions to get at stuff. I recommend this approach
from now on.

The problem is the php side, which ends up duplicating some stuff, but
in the old style. This is not so bad for the template code since I
have made it a point not to do anything but display functions in php;
all modifications are handled in the backend.

needed (reparsing of NS file with current bindings).

Mostly I decided that I made a huge mistake in using the exptidx as the
main cross index. I started to change that, which took a bunch of time.
Still more tables to fix up, but I think all of this code is gonna need
a major reorg once I am no longer under the gun to get it done.

now in place (create and display), and the backend part that deals with
setting up the DB records. Nothing actually happens on the nodes yet.

you can turn it on in your devel tree by setting the $EXPOSETEMPLATES
variable in www/defs.php3.in to 1. BE SURE NOT TO CHECK THAT CHANGE IN!

Changed format of email sent to user on errors.  The error will now
appear instead of the generic message when I am confident it is
accurate.  The subject line will also change to reflect the cause of
an error.

Avoid sending mail to testbed-ops during failed swap related evenets
in some cases.  It will instead be sent to a new mailing list
testbed-errors.

Added a new row in the experiment info table "Last Error:" which
states the cause of the error, and links to a new page displaying the
error.

Made some assign/assign_wrapper errors more informative.

The error (as determined by tblog) is now stored in the database in a
more structured fashion.  This inlcudes adding a column for the session
(in the log table) to testbed_stats to link eash swap event with the
logs and possible the error.

Other changes to the database, see sql/database-migrate.txt