More tbreport changes from Mike Kasick <mkasick@andrew.cmu.edu>:

- Added tblog support to nscheck.

- Added ns_parse_failed error to nscheck.

- Added invocation column to report_errors to differentiate between assign
  runs in infeasible resource assignments.

Prep for Mike Kasick report code.  Updated database schema and
installed hooks for his code.

Cleaned up how errors were handled in tblog(...).

Allow SENDMAIL to be called before the path is untained in '-T' scripts.

Other small changes.

draft is that the user will at the end of an experiment run, log into one
of his nodes and perform some analysis which is intended to be repeated at
the end of the next run, and in future instantiations of the template.

A new table called experiment_template_events holds the dynamic events for
the template. Right now I am supporting just program events, but it will be
easy to support arbitrary events later. As an absurd example:

	node6> /usr/local/bin/template_analyze ~/data_analyze arg arg ...

The user is currently responsible for making sure the output goes into a
file in the archive. I plan to make the template_analyze wrapper handle
that automatically later, but for now what you really want is to invoke a
script that encapsulates that, redirecting output to $ARCHIVE (this
variable is installed in the environment template_analyze.

The wrapper script will save the current time, and then run the program.
If the program terminates with a zero exit status, it will ssh over to ops
and invoke an xmlrpc routine to tell boss to add a program event to both
the eventlist for the current instance, and to the template_eventlist for
future instances. The time of the event is the relative start time that was
saved above (remember, each experiment run replays the event stream from
time zero).

For the future, we want to allow this to be done on ops as well, but
that will take more infrastructure, to run "program agents" on ops.

It would be nice to install the ssl xmlrpc client side on our images so
that we do not have to ssh to ops to invoke the client.

associated with an experiment (or a project/group). This is to make it
easier to generate the proper permission lists for db access.

into per-experiment databases on ops. Additional support for reconsituting
those databases back into temporary databases on ops, for post processing.

* This revision relies on the "snort" port (/usr/ports/security/snort)
  to read the pcap files and load them into a database. The schema is
  probably not ideal, but its better then nothing. See the file
  ops:/usr/local/share/examples/snort/create_mysql for the schema.

* For simplicity, I have hooked into loghole, which already had all
  the code for downloading the trace data. I added some new methods to
  the XMLRPC server for loghole to use, to get the users DB password
  and the name of the per-experiment database. There is a new slot in
  the traces table that indicates that the trace should be snorted to
  its DB. In case you forgot, at the end of a run or when the instance
  is swapped out, loghole is run to download the trace data.

* For reconsituting, there are lots of additions to opsdb_control and
  opsdb_control.proxy to create "temporary" databases and load them
  from a dump file that is stored in the archive. I've added a button
  to the Template Record page, inappropriately called "Analyze" since
  right now all it does is reconsitute the trace data into a DB on
  ops.

  Currently, the only indication of what has been done (the name of
  the DBs created on ops) is the log email that the user gets. A
  future project is tell the user this info in the web interface.

* To turn on database capturing of trace data, do this in your NS
  file:

	set link0 ...
	$link0 trace
	$link0 trace_snaplen 128
	$link0 trace_db 1

   the increase in snaplen is optional, but a good idea if you want
   snort to undertand more then just ip headers.

* Also some changes to the parser to allow plain experiments to take
  advantage of all this stuff. To simple get yourself a per-experiment
  DB, put this in your NS file:

	tb-set-dpdb 1

  however, anytime you turn trace_db on for a link or lan, you
  automatically get a per-experiment DB.

* To capture the trace data to the DB, you can run loghole by hand:

	loghole sync -s

  the -s option turns on the "post-process" phase of loghole.

of the most recent child (modification). Better then passing this info
back in a print statement to be parsed.

the decimal point in decimal slots.

* add an "active" flag to the template record, which will be used by
  the user to indicate what templates he wants listed (rather then the
  roots). Basically, the current working templates, rather then a big
  graph.

But I never actually finished that cause it sorta morphed into:

* Added a vis_graphs table to cache the last generated visualization
  rendering in the database so that we do not have to wait so long for:

* Add new buttons to showexp and template_show pages, to display in the
  same page either the settings (current view), the NS file, or the
  visualization (along with zoom in/out buttons).

And now I can go back to that "active" thing I mentioned up above ...

most of what is in the node_types table (and node_types_auxtypes).
The goal is to make most of the type information free form key/value
pairs so that type information is more extensible.

This commit simply creates the table and populates it with initial
info using init_nodeattrs.pl. I expect this script to change, but it
can be rerun later so no big deal.

Many changes to tblog code.  Database update needed:

1) Added summary of failed nodes is os_setup.  The cause of the error is now
classified as "user" if it is only user images that failed and the user
image failed on every pc of a particular type.  Otherwise I leave the cause
as "unknown" since it is really hard to tell what the real cause is.

2) Raised the confidence threshold for most errors so that they will appear
on the top.

3) Added a special error when an experiment is canceled.  The cause is
"canceled" and testbed-ops won't see these errors.

4) Fixed a bug in assign_wrapper where it will incorrectly report "This
experiment cannot be instantiated on this testbed..." when really the user
canceled the swapin.

5) Fixed a bug where os_setup errors where being incorrectly reported as
assign errors.  This happens when os_setup fails for some reason and
tbswap tries again, but the second time around there are not enough nodes.
So the last error is coming from assign even though the true cause of the
error is due to failed nodes.  The fix for this involved added a new column
to the log table, "attempt", which will be 1 for the first attempt and then
incremented for each new attempt.  tblog_find_error will then simply ignore
any errors with "attempt > 1".

6) Also fixed a potential problem when there is an error during the cleanup
phase by adding another column "cleanup".  tblog_find_error will
also ignore any errors with the cleanup bit set.

revision 4.70.

Actually, most of the changes here were just to generalize the "virtual
interface" state in the DB.  Other than the client-side scripts, there
is very little specific here specific to tagged VLANs.

In fact, you cannot specify "vlan" as a type yet as we haven't done the
snmpit support for setting up the switches.

For more info see bas:~mike/flux/doc/testbed-virtinterfaces.txt, which I
will integrate into the knowledge base and the Emulab doc at some point.

Each template has a datastore, which is really just a subdirectory that can
be populated with files, and committed to the subversion archive.  Note,
the datastore os specific to the template itself. The Template Archive link
on the Show Template page takes you to the subdirectory, which by
convention I am calling "datastore".

The directory actually lives in /proj/pid/exp/eid/TGUID-VERS ... but that
path is printed out for you on the archive page.

Anyway, put stuff in the datastore directory, and then commit the template
archive so there is a tag associated with it.

When an instance is created, a checkout of the datastore is placed in the
experiment directory (/proj/pid/eid/exp/template_datastore). The current
tag (from above) is stored with the instance so that we can later recreate
the enviroment for the instance, say for rerun.

Tarfiles and rpms in the datastore can be referenced as xxx://foo.rpm (in
your NS file).  tarfiles_setup transforms those when the instance is
swapped in, sorta like it does other URLs, only it does not actually fetch
them, just need to rewrite the paths so they reference datastore.

The program agent gets another environment variable so you can refer to the
datastore without hardwiring paths ($DATASTORE). Eventually I want to move
the checkout someplace else, but it was easy to drop it into the experiment
directory for now.

The adminon/off bit is now kept in the login table instead of the
users table so that you can be reddot on a per-browser (login) basis.

entries, but sheesh, they were screwed up! Hopefully fixed now.

the user has requested a per-experiment DB on ops (or wherever the DP
is). As you see, its an optional thing that will probably only be used
for templates.

* Rework metadata to be somewhat more general so that they can be used for
  things like parameter descriptions.

* My own little part of javascript hell ... add a highlight around the
  "current" template in the template graph. Sheesh!

* Add more tooltips (using the tooltip library). When you mouseover the
  metadata or descriptions, the formatted text is shown in a tooltip,
  which is handy when the valus is truncated cause its too long for the
  tables.

* setup some fields that parallel the
  elab-in-elab ones, for modifying the dhcpd.conf file.

* tb-set-node-plab-role to set a node as either 'plc' or 'node'
  used to set the DB role field above

show template page, to some primitive zoom code in the backend. It will
do for now.

Refactor "log" table to move some stuff into a new table.

are now reserved words mysql 5.X. Skip to the next entry ...

heavyweight queries in idlemail

node_types pxe_boot_path is going to be needed after all, so it has
made a triumphant return.

needed (reparsing of NS file with current bindings).

Mostly I decided that I made a huge mistake in using the exptidx as the
main cross index. I started to change that, which took a bunch of time.
Still more tables to fix up, but I think all of this code is gonna need
a major reorg once I am no longer under the gun to get it done.