Other stuff leaked in too ...

I did separate out a lot of tbsetup/libArchive into db/Archive, and
whats left in libArchive.pm will eventually move over into the
Template library.

Note that I have dropped archiving of plain experiments; this is not
really owrth it outside the workbench context, and it just wastes
space and makes a lot if stuff painful in the web interface.

stuff into the library since we do that in many places.

working on this a few weeks ago when I got sidetracked by the pubsub
project.

indexed by exptidx. I also got the last of the pid and pid,gid tables.

user directory.

run display pages.

* Add new code to archive_control and libarchive to create tags (in
  optional subdirs) of the tags directory. These tags are simply tags
  on the current trunk at the time.

* Change to template instantiate and start/top run code to create a
  "runs" directory. Within the runs directory are subdirs named for
  each runid, and within this directory are start and stop run tags.
  Eventually add some other tags when doing swapmods, but do not have
  that figured out yet.

* Change the viewcvs code to add an "embedded" view of an archive.
  This is just an alternate view that has no header/footer goo, but
  only the actual svn contents. The main reason for this change is
  that the header and footers take up a huge amount of space and
  contribute nothing. Smaller means its easier to add to the instance
  and run show pages.

* Add the above mentioned embedded archive view to the instance and
  run show pages. On the instance page you see the directory of all
  runids, and on the show run page you see the subdir for that run.

most of the rest of the tables in the system (still a few exceptions).
Bound to be some bugs ...

kludgy code in template_exprun that tries to ignore dead nodes during
start and stop run.

Now named by "E${exptidx" which sorta sucks but not much to do.

inserted for regular experiments as well as template instances.

* Create a per-experiment DB user for the per-experiment DB; the user
  name is equal to the DB name.

* Add a dpdbpassword field to the experiments table; this is the
  randomly generated password for the DB user mentioned above.

* For Templates, use the above user/password in the environment,
  instead of the swapper uid/password.

* Add experiment dbname/dbpassword to the Show Experiment page.

template record to replay an instance, but all it really does is
replay the first run. There is also support to replay a specific run
in an instance, but that is not hooked up to the web page yet.

Note that the bulk of the replay code went in by mistake a couple of
days ago.

globbing since that was not working properly (not sure why, but this
is more efficient anyway.

Add backup routine to back up failed template swapins to .$eid-failed.

doing a Start Run. On the web page, there is a new checkbox, and
on ops, template_startrun takes a new -m option.

Caveat: You cannot specify a new NS file, yet. The original file is
reparsed, and the idea is that a change in the template parameters
will result in a change to the topology. I will add the ability to
specify a new NS file in the next revision of this change.

If you really really want to change the NS file, go to
/proj/$pid/exp/$eid/archive/nsdata and edit nsfile.ns ...

In addtion, DATASTORE is now defined while parsing the NS file. This
turned to be quite the headache!

reload and halt events send proper completion events. This is required
for stoprun and startrun to work correctly. On stoprun the logs are
not collected until the programs have stopped, and on startrun we do
not want to proceed until all the agents have reloaded their
environments.

so that users can schedule program events to run there. For example:

	set myprog [new Program $ns]
	$myprog set node "ops"
	$myprog set command "/usr/bin/env >& /tmp/foo"

	$ns at 10 "$myprog start"
or
	tevc -e pid/eid now myprog start

Since the program agent cannot talk to tmcd from ops, there are new
routines to create the config files that the program agent uses, in
the expertment tbdata directory.

I also rewrote the eventsys.proxy script that starts the event
scheduler on ops; I rolled the startup of the program agent into this
script, via new -a option which is passed over from boss when an ops
program agent is detected in the virt topology. This keep the number
of new processes on ops to a small number.

Also part of the above rewrite is that we now catch when event
scheduler (or the program agent) exits abnormally, sending email to
tbops and the swapper of the experiment. We have been seeing abnormal
exits of the scheduler and it would good to detect and see if we can
figure out what is going wrong.

Other small bug fixes in experiment run.

do!

The original operation was to save up every log file forever in the
work directory, and copy that out to both the user directory and the
info directory (long term archive). When I cleaned /proj on ops
yesterday of all this old cruft, I recoved 17GB of disk space. Yow!

So, the new operation is:

* Only files that end in .log are copied to the user directory. No
  longer copying out .top, .ptop, and a couple of other logs; 99% of
  users never look at these things. We still have them available to us
  though, on boss.

* At the beginning of each swap operation, clean out the work
  directory of all the old log files. These are named a variety of
  ways, so I use some pattern patches to do this.

* Jigger the names a little so that we do not name things in the form
  "$$.log", to avoid copying out different named files to the user
  directory each time; instead link the .log file to the real output
  file so that it gets overwritten each time, while still getting the
  per-swap files for long term storage.

that clients and servers can avoid using hardwired ports on those
experimental nodes. I have added the following tmcd operation:

	tmcc portregister <service> [<port>]

where we assume its the control network IP (from the DB), and the pid/eid
of the node the experiment belongs to. The given port is entered into
the port_registration table for the experiment, using the service as the
tag. Supplying port=0 clears the registration from the table.

When called like:

	tmcc portregister <service>

we return the registered port, or nothing.

I hacked up a little C library module in libtb so that there is something
that looks like a C interface to this:

 	int
 	PortRegister(char *service, int port);

 	int
 	PortLookup(char *service, char *hostname, int namelen, int *port);

The above routines call out to tmcc of course.

Lastly, I changed the sync server and client to use the new port
registration, via the library calls above.

There are other emulab services that need to be changed as well, but
they can be done on an as needed basis.

into per-experiment databases on ops. Additional support for reconsituting
those databases back into temporary databases on ops, for post processing.

* This revision relies on the "snort" port (/usr/ports/security/snort)
  to read the pcap files and load them into a database. The schema is
  probably not ideal, but its better then nothing. See the file
  ops:/usr/local/share/examples/snort/create_mysql for the schema.

* For simplicity, I have hooked into loghole, which already had all
  the code for downloading the trace data. I added some new methods to
  the XMLRPC server for loghole to use, to get the users DB password
  and the name of the per-experiment database. There is a new slot in
  the traces table that indicates that the trace should be snorted to
  its DB. In case you forgot, at the end of a run or when the instance
  is swapped out, loghole is run to download the trace data.

* For reconsituting, there are lots of additions to opsdb_control and
  opsdb_control.proxy to create "temporary" databases and load them
  from a dump file that is stored in the archive. I've added a button
  to the Template Record page, inappropriately called "Analyze" since
  right now all it does is reconsitute the trace data into a DB on
  ops.

  Currently, the only indication of what has been done (the name of
  the DBs created on ops) is the log email that the user gets. A
  future project is tell the user this info in the web interface.

* To turn on database capturing of trace data, do this in your NS
  file:

	set link0 ...
	$link0 trace
	$link0 trace_snaplen 128
	$link0 trace_db 1

   the increase in snaplen is optional, but a good idea if you want
   snort to undertand more then just ip headers.

* Also some changes to the parser to allow plain experiments to take
  advantage of all this stuff. To simple get yourself a per-experiment
  DB, put this in your NS file:

	tb-set-dpdb 1

  however, anytime you turn trace_db on for a link or lan, you
  automatically get a per-experiment DB.

* To capture the trace data to the DB, you can run loghole by hand:

	loghole sync -s

  the -s option turns on the "post-process" phase of loghole.

Also add an Update method.

table, into a new table called node_type_attributes, which is intended
to be a more extensible way of describing nodes.

The only things left in the node_types table will be type,class and the
various isXXX boolean flags, since we use those in numerous joins all over
the system (ie: when discriminating amongst nodes).

For the most part, all of that other stuff is rarely used, or used in
contexts where the information is needed, but not for type descrimination.
Still, it made for a lot of queries to change!

Along the way I added a NodeType library module that represents the type
info as a perl object. I also beefed up the existing Node module, and
started using it in more places. I also added an Interfaces module, but I
have not done much with that yet.

I have not yet removed all the slots from the node_types table; I plan to
run the new code for a few days and then remove the slots.

Example using the new NodeType object:

	use NodeType;

	my $typeinfo = NodeType->Lookup($type);

        if ($typeinfo->control_interface(\$control_iface) ||
            !$control_iface) {
  	    warn "No control interface for $type is defined in the DB!\n";
        }

or using the Node:

	use Node;

        my $nodeobject = Node->Lookup($node_id);
        my $imageable  = $nodeobject->NodeTypeInfo()->imageable();
or
        my $rebootable = $nodeobject->isrebootable();
or
        $nodeobject->NodeTypeAttribute("control_interface", \$control_iface);

Lots of way to accomplish the same thing, but the main point is that the
Node is able to override the NodeType (if it wants to), which I think is
necessary for flexibly describing one/two of a kind things like switches, etc.

to do everything!

It mostly incorporates (part of) what is currently spread all over
libdb.pm, and so far I have converted just batchexp to use it.
I will be converting scripts over time ... but you might want to look
at this module now and start using it.