associated with an experiment (or a project/group). This is to make it
easier to generate the proper permission lists for db access.

into per-experiment databases on ops. Additional support for reconsituting
those databases back into temporary databases on ops, for post processing.

* This revision relies on the "snort" port (/usr/ports/security/snort)
  to read the pcap files and load them into a database. The schema is
  probably not ideal, but its better then nothing. See the file
  ops:/usr/local/share/examples/snort/create_mysql for the schema.

* For simplicity, I have hooked into loghole, which already had all
  the code for downloading the trace data. I added some new methods to
  the XMLRPC server for loghole to use, to get the users DB password
  and the name of the per-experiment database. There is a new slot in
  the traces table that indicates that the trace should be snorted to
  its DB. In case you forgot, at the end of a run or when the instance
  is swapped out, loghole is run to download the trace data.

* For reconsituting, there are lots of additions to opsdb_control and
  opsdb_control.proxy to create "temporary" databases and load them
  from a dump file that is stored in the archive. I've added a button
  to the Template Record page, inappropriately called "Analyze" since
  right now all it does is reconsitute the trace data into a DB on
  ops.

  Currently, the only indication of what has been done (the name of
  the DBs created on ops) is the log email that the user gets. A
  future project is tell the user this info in the web interface.

* To turn on database capturing of trace data, do this in your NS
  file:

	set link0 ...
	$link0 trace
	$link0 trace_snaplen 128
	$link0 trace_db 1

   the increase in snaplen is optional, but a good idea if you want
   snort to undertand more then just ip headers.

* Also some changes to the parser to allow plain experiments to take
  advantage of all this stuff. To simple get yourself a per-experiment
  DB, put this in your NS file:

	tb-set-dpdb 1

  however, anytime you turn trace_db on for a link or lan, you
  automatically get a per-experiment DB.

* To capture the trace data to the DB, you can run loghole by hand:

	loghole sync -s

  the -s option turns on the "post-process" phase of loghole.

of the most recent child (modification). Better then passing this info
back in a print statement to be parsed.

the decimal point in decimal slots.

* add an "active" flag to the template record, which will be used by
  the user to indicate what templates he wants listed (rather then the
  roots). Basically, the current working templates, rather then a big
  graph.

But I never actually finished that cause it sorta morphed into:

* Added a vis_graphs table to cache the last generated visualization
  rendering in the database so that we do not have to wait so long for:

* Add new buttons to showexp and template_show pages, to display in the
  same page either the settings (current view), the NS file, or the
  visualization (along with zoom in/out buttons).

And now I can go back to that "active" thing I mentioned up above ...

most of what is in the node_types table (and node_types_auxtypes).
The goal is to make most of the type information free form key/value
pairs so that type information is more extensible.

This commit simply creates the table and populates it with initial
info using init_nodeattrs.pl. I expect this script to change, but it
can be rerun later so no big deal.

Many changes to tblog code.  Database update needed:

1) Added summary of failed nodes is os_setup.  The cause of the error is now
classified as "user" if it is only user images that failed and the user
image failed on every pc of a particular type.  Otherwise I leave the cause
as "unknown" since it is really hard to tell what the real cause is.

2) Raised the confidence threshold for most errors so that they will appear
on the top.

3) Added a special error when an experiment is canceled.  The cause is
"canceled" and testbed-ops won't see these errors.

4) Fixed a bug in assign_wrapper where it will incorrectly report "This
experiment cannot be instantiated on this testbed..." when really the user
canceled the swapin.

5) Fixed a bug where os_setup errors where being incorrectly reported as
assign errors.  This happens when os_setup fails for some reason and
tbswap tries again, but the second time around there are not enough nodes.
So the last error is coming from assign even though the true cause of the
error is due to failed nodes.  The fix for this involved added a new column
to the log table, "attempt", which will be 1 for the first attempt and then
incremented for each new attempt.  tblog_find_error will then simply ignore
any errors with "attempt > 1".

6) Also fixed a potential problem when there is an error during the cleanup
phase by adding another column "cleanup".  tblog_find_error will
also ignore any errors with the cleanup bit set.

revision 4.70.

Actually, most of the changes here were just to generalize the "virtual
interface" state in the DB.  Other than the client-side scripts, there
is very little specific here specific to tagged VLANs.

In fact, you cannot specify "vlan" as a type yet as we haven't done the
snmpit support for setting up the switches.

For more info see bas:~mike/flux/doc/testbed-virtinterfaces.txt, which I
will integrate into the knowledge base and the Emulab doc at some point.

Each template has a datastore, which is really just a subdirectory that can
be populated with files, and committed to the subversion archive.  Note,
the datastore os specific to the template itself. The Template Archive link
on the Show Template page takes you to the subdirectory, which by
convention I am calling "datastore".

The directory actually lives in /proj/pid/exp/eid/TGUID-VERS ... but that
path is printed out for you on the archive page.

Anyway, put stuff in the datastore directory, and then commit the template
archive so there is a tag associated with it.

When an instance is created, a checkout of the datastore is placed in the
experiment directory (/proj/pid/eid/exp/template_datastore). The current
tag (from above) is stored with the instance so that we can later recreate
the enviroment for the instance, say for rerun.

Tarfiles and rpms in the datastore can be referenced as xxx://foo.rpm (in
your NS file).  tarfiles_setup transforms those when the instance is
swapped in, sorta like it does other URLs, only it does not actually fetch
them, just need to rewrite the paths so they reference datastore.

The program agent gets another environment variable so you can refer to the
datastore without hardwiring paths ($DATASTORE). Eventually I want to move
the checkout someplace else, but it was easy to drop it into the experiment
directory for now.

The adminon/off bit is now kept in the login table instead of the
users table so that you can be reddot on a per-browser (login) basis.

entries, but sheesh, they were screwed up! Hopefully fixed now.

the user has requested a per-experiment DB on ops (or wherever the DP
is). As you see, its an optional thing that will probably only be used
for templates.

* Rework metadata to be somewhat more general so that they can be used for
  things like parameter descriptions.

* My own little part of javascript hell ... add a highlight around the
  "current" template in the template graph. Sheesh!

* Add more tooltips (using the tooltip library). When you mouseover the
  metadata or descriptions, the formatted text is shown in a tooltip,
  which is handy when the valus is truncated cause its too long for the
  tables.

* setup some fields that parallel the
  elab-in-elab ones, for modifying the dhcpd.conf file.

* tb-set-node-plab-role to set a node as either 'plc' or 'node'
  used to set the DB role field above

show template page, to some primitive zoom code in the backend. It will
do for now.

Refactor "log" table to move some stuff into a new table.

are now reserved words mysql 5.X. Skip to the next entry ...

heavyweight queries in idlemail

node_types pxe_boot_path is going to be needed after all, so it has
made a triumphant return.

needed (reparsing of NS file with current bindings).

Mostly I decided that I made a huge mistake in using the exptidx as the
main cross index. I started to change that, which took a bunch of time.
Still more tables to fix up, but I think all of this code is gonna need
a major reorg once I am no longer under the gun to get it done.

an alternate "filename" for the dhcpd.conf file.  "Inner plabs" are likely
to need this (to boot pxelinux for emulating the plab boot CD).

dhcpd_makeconf picks up the DB field and sets the filename field accordingly.

nfree will invoke dhcpd_makeconf if any node being freed has a non-NULL
pxe_boot_path value, in order to reset it.

you can turn it on in your devel tree by setting the $EXPOSETEMPLATES
variable in www/defs.php3.in to 1. BE SURE NOT TO CHECK THAT CHANGE IN!

Changed format of email sent to user on errors.  The error will now
appear instead of the generic message when I am confident it is
accurate.  The subject line will also change to reflect the cause of
an error.

Avoid sending mail to testbed-ops during failed swap related evenets
in some cases.  It will instead be sent to a new mailing list
testbed-errors.

Added a new row in the experiment info table "Last Error:" which
states the cause of the error, and links to a new page displaying the
error.

Made some assign/assign_wrapper errors more informative.

The error (as determined by tblog) is now stored in the database in a
more structured fashion.  This inlcudes adding a column for the session
(in the log table) to testbed_stats to link eash swap event with the
logs and possible the error.

Other changes to the database, see sql/database-migrate.txt