This change attempts to make the stats gathering code more reliable by
not relying on the testbed_stats records to reconstruct usage
statistics.  The main source of errors and total confusion in the
current stats code is that testbed_stats includes all the errors and
transitions, from which I have to reconstruct what happened in order
to determine usage by a project or user.

The new stats code still generates the testbed_stats code, but actual
usage is recorded as it happens, in the experiment_resources table, as
swapins, swapouts, and swapmods occur. Its also much faster to compute
the data for the tables in the web interface, not having to scan a
zillion testbed_stats records in php.

There is a time consuming update to the records that takes place with
a lot of tables locked.

cache latest statistics about planetlab nodes (a reduction of the
data in plab_nodehist).

deal with the issue of what a hidden run means when doing a replay, cause
more generally, replay needs a lot of work.

Other stuff leaked in too ...

I did separate out a lot of tbsetup/libArchive into db/Archive, and
whats left in libArchive.pm will eventually move over into the
Template library.

Note that I have dropped archiving of plain experiments; this is not
really owrth it outside the workbench context, and it just wastes
space and makes a lot if stuff painful in the web interface.

this as a temporary measure.

Also, renumber the last few database-migrate entries, there was a missing
number and things had gotten out of sync with database-create.

everybody's stuff up.

working on this a few weeks ago when I got sidetracked by the pubsub
project.

3.x to 4.x continues).

connected.

table. Note that workbench permission implies archive permission too.
Added the toggle to the project page.

Added a "stud" toggle to the user page while I was there.

(also knows an integers).

* install/dump-descriptors <filename> will write out a set of insert
  statements for the images and os_info table, slightly munged. In
  fact, what I do is create temporary tables called temp_images and
  temp_os_info, clean them a bit, and then write out the insert
  statements to load them into new tables of the same name.

  There are some arrays at the top of this script that says what images and
  osids to write out.

* install/load-descriptors <filename> takes the output of
  dump-descriptors, creates the two temporary tables and loads the
  data into them. Then it (optionally) updates those tables with the
  local indicies of elabman and the emulab-ops project and group.

  Then it computes an osidtoimageid table for all class='pc' types. On a
  new testbed this is a reasonable approach, in my opinion.

  Next it takes the contents of the two temp tables and moves them across
  to the actual tables.

* install/descriptors.sql is the current data set which has everything
  contained in sql/database-fill-supplemental.sql and install/images/*

indexed by exptidx. I also got the last of the pid and pid,gid tables.

interface to the backend. There are new scripts that can be called
from the command line:

	newuser xmlfile
	newproj xmlfile

They both run from small xmlfiles that are generated by the web
interface from the form data. I also moved user verification to the
backend so that we do not have duplicated email functions, but that
was a small change.

Upon error, the xmlfile is saved and sent to tbops so that we can
rerun the command by hand, rather then force user to fill out form
again. I also do a better job of putting the form back up intact when
there are internal errors.

If the user provides an initial public key, that is put into the xml
file as well and addpubkey is called from newuser instead of the web
interface. A more general change to addpukey is that it is now
*always* called as "nobody". This script was a morass of confusion
cause of having to call it as nobody before the user actually
exists. In fact, another of my ongoing projects is to reduce the
number of scripts called as a particular user, but thats a story for
another day. Anyway, the script is always called as nobody, but we
pass along the implied user in the environment so that it can do
permission checks.

most of the rest of the tables in the system (still a few exceptions).
Bound to be some bugs ...

uid_idx to the uid for compatability with unchanged code.

Some of these are on rinky-dink scripts, but it just makes life easier to
be consistent.

declared NOT NULL (not necessary in 5.0).

ElabInElab experiments, so as not to consume an interface.

I actually modified Keith's changes so that we can dynmaically choose
the single or dual model in the NS file (Keith's changes hardwired the
system to only single control network) since the single network model
has the disadvantage of not allowing new nodes to be added to an inner
elab, which would make it impossible to test some things (like the
newnode path!).

To choose the single control network model, place this in your NS file:

	tb-elabinelab-singlenet

other it defaults to the older dual network model.