option to also unlock tables at the same time, but before the callout
to the event system. This should hopefully reduce the incidence of
complete system lockup cause a process is holding the experiments
table locked while the event system is wedged up.

thankless job but someone has to do it. I'm expecting to finish by the
time Bush 43 leaves office.

  DB alongside the resource records. Previously, we stored only the
  nsfiles for current experiments, and purged them when the experiment
  was terminated. The new approach saves them forever using the resource
  record ID. Note that we do not store copies of NS files, but reference
  them indirectly instead so that we can MD5 them and avoid the dups.

  I put a "compressed" bit into the table cause at some point we will
  start compressing the data before storing them into the DB. Or maybe
  we bag this and start using GFS!

  Also note that this addresses the problem of losing the NS file
  history when using swapmod, since the NS file is overwritten.

* Add a pmapping table stores the nodes (and their types) used by an
  experiment. This data is also saved forever, alongside the resource
  records, so that we can more accurately replay an experiment. As Rob
  points out, the node names can also be used in conjunction with the
  ptop files that are saved, to get a 100% accurate remap of resources.

experiment underlying a template. Rather then using the experiment directory
we use the template directory in /proj/$pid/templates ...

pointer. This indicates a previous failure that really got hosed and
left the reources table and/or stats table in a bad state which must
be fixed by hand (for now).

This change attempts to make the stats gathering code more reliable by
not relying on the testbed_stats records to reconstruct usage
statistics.  The main source of errors and total confusion in the
current stats code is that testbed_stats includes all the errors and
transitions, from which I have to reconstruct what happened in order
to determine usage by a project or user.

The new stats code still generates the testbed_stats code, but actual
usage is recorded as it happens, in the experiment_resources table, as
swapins, swapouts, and swapmods occur. Its also much faster to compute
the data for the tables in the web interface, not having to scan a
zillion testbed_stats records in php.

There is a time consuming update to the records that takes place with
a lot of tables locked.

  "$DBQUERY_MAXTRIES" is not exported by the libdb module
All I did was exactly what Template.pm does, but obviously I am a bozo
for not understanding something!

* Records are now "help open" when a run is stopped. When the next run
  is started, a check is made to see if the files
  (/project/$pid/exp/$eid) have changed, and if so a new version of the
  archive is committed before the next run is started.

* Change the way swapmod is handled within an instance. A new option
  on the ShowExp page called Modify Resources. The intent is to allow
  an instance to be modified without having to start and stop runs,
  which tends to clutter things up, according to our user base. So, if
  you are within a run, that run is reset (reused) after the swapmod is
  finished. You can do this as many times as you like. If you are
  between runs (last operation was a stoprun), do the swapmod and then
  "speculatively" start a new run. Subsequent modifies reuse the that
  run again, as above.

  I think this is what Kevin was after ... there are some UI issues
  that may need to be resolved, will wait to hear what people have to
  say.

* Revising a record is now supported. Export, change in place, and
  then use the Revise link on the ShowRun page. Currently this has to
  happen from the export directory on ops, but eventually allow an
  upload (to correspond to downloaded exports)

* Check to see if export already exists, and give warning. Added a
  checkbox that allows user to overwrite the export.

* A bunch of minor UI changes to the various template pages.

Other stuff leaked in too ...

I did separate out a lot of tbsetup/libArchive into db/Archive, and
whats left in libArchive.pm will eventually move over into the
Template library.

Note that I have dropped archiving of plain experiments; this is not
really owrth it outside the workbench context, and it just wastes
space and makes a lot if stuff painful in the web interface.

stuff into the library since we do that in many places.

working on this a few weeks ago when I got sidetracked by the pubsub
project.

indexed by exptidx. I also got the last of the pid and pid,gid tables.

user directory.

run display pages.

* Add new code to archive_control and libarchive to create tags (in
  optional subdirs) of the tags directory. These tags are simply tags
  on the current trunk at the time.

* Change to template instantiate and start/top run code to create a
  "runs" directory. Within the runs directory are subdirs named for
  each runid, and within this directory are start and stop run tags.
  Eventually add some other tags when doing swapmods, but do not have
  that figured out yet.

* Change the viewcvs code to add an "embedded" view of an archive.
  This is just an alternate view that has no header/footer goo, but
  only the actual svn contents. The main reason for this change is
  that the header and footers take up a huge amount of space and
  contribute nothing. Smaller means its easier to add to the instance
  and run show pages.

* Add the above mentioned embedded archive view to the instance and
  run show pages. On the instance page you see the directory of all
  runids, and on the show run page you see the subdir for that run.

most of the rest of the tables in the system (still a few exceptions).
Bound to be some bugs ...

kludgy code in template_exprun that tries to ignore dead nodes during
start and stop run.

Now named by "E${exptidx" which sorta sucks but not much to do.

inserted for regular experiments as well as template instances.

* Create a per-experiment DB user for the per-experiment DB; the user
  name is equal to the DB name.

* Add a dpdbpassword field to the experiments table; this is the
  randomly generated password for the DB user mentioned above.

* For Templates, use the above user/password in the environment,
  instead of the swapper uid/password.

* Add experiment dbname/dbpassword to the Show Experiment page.

template record to replay an instance, but all it really does is
replay the first run. There is also support to replay a specific run
in an instance, but that is not hooked up to the web page yet.

Note that the bulk of the replay code went in by mistake a couple of
days ago.

globbing since that was not working properly (not sure why, but this
is more efficient anyway.

Add backup routine to back up failed template swapins to .$eid-failed.

doing a Start Run. On the web page, there is a new checkbox, and
on ops, template_startrun takes a new -m option.

Caveat: You cannot specify a new NS file, yet. The original file is
reparsed, and the idea is that a change in the template parameters
will result in a change to the topology. I will add the ability to
specify a new NS file in the next revision of this change.

If you really really want to change the NS file, go to
/proj/$pid/exp/$eid/archive/nsdata and edit nsfile.ns ...

In addtion, DATASTORE is now defined while parsing the NS file. This
turned to be quite the headache!

reload and halt events send proper completion events. This is required
for stoprun and startrun to work correctly. On stoprun the logs are
not collected until the programs have stopped, and on startrun we do
not want to proceed until all the agents have reloaded their
environments.

so that users can schedule program events to run there. For example:

	set myprog [new Program $ns]
	$myprog set node "ops"
	$myprog set command "/usr/bin/env >& /tmp/foo"

	$ns at 10 "$myprog start"
or
	tevc -e pid/eid now myprog start

Since the program agent cannot talk to tmcd from ops, there are new
routines to create the config files that the program agent uses, in
the expertment tbdata directory.

I also rewrote the eventsys.proxy script that starts the event
scheduler on ops; I rolled the startup of the program agent into this
script, via new -a option which is passed over from boss when an ops
program agent is detected in the virt topology. This keep the number
of new processes on ops to a small number.

Also part of the above rewrite is that we now catch when event
scheduler (or the program agent) exits abnormally, sending email to
tbops and the swapper of the experiment. We have been seeing abnormal
exits of the scheduler and it would good to detect and see if we can
figure out what is going wrong.

Other small bug fixes in experiment run.

do!

The original operation was to save up every log file forever in the
work directory, and copy that out to both the user directory and the
info directory (long term archive). When I cleaned /proj on ops
yesterday of all this old cruft, I recoved 17GB of disk space. Yow!

So, the new operation is:

* Only files that end in .log are copied to the user directory. No
  longer copying out .top, .ptop, and a couple of other logs; 99% of
  users never look at these things. We still have them available to us
  though, on boss.

* At the beginning of each swap operation, clean out the work
  directory of all the old log files. These are named a variety of
  ways, so I use some pattern patches to do this.

* Jigger the names a little so that we do not name things in the form
  "$$.log", to avoid copying out different named files to the user
  directory each time; instead link the .log file to the real output
  file so that it gets overwritten each time, while still getting the
  per-swap files for long term storage.

that clients and servers can avoid using hardwired ports on those
experimental nodes. I have added the following tmcd operation:

	tmcc portregister <service> [<port>]

where we assume its the control network IP (from the DB), and the pid/eid
of the node the experiment belongs to. The given port is entered into
the port_registration table for the experiment, using the service as the
tag. Supplying port=0 clears the registration from the table.

When called like:

	tmcc portregister <service>

we return the registered port, or nothing.

I hacked up a little C library module in libtb so that there is something
that looks like a C interface to this:

 	int
 	PortRegister(char *service, int port);

 	int
 	PortLookup(char *service, char *hostname, int namelen, int *port);

The above routines call out to tmcc of course.

Lastly, I changed the sync server and client to use the new port
registration, via the library calls above.

There are other emulab services that need to be changed as well, but
they can be done on an as needed basis.