project; skip static route calc and prerender. Takes too much time.

into my own project.

	source tb_compat.tcl
	set ns [new Simulator]

	tb-elab-in-elab 1
	tb-set-inner-elab-eid two-simple
	tb-set-security-level Red

	$ns run

tbsetup/ns2ir/elabinelab.ns has all the goo, which is sourced from the
NS run subroutine, using "uplevel 1" so that the context is correct.
You can of course include you own goo, in which case the default goo
will be skipped.

ElabinElab experiments that wrap another experiment, either firewalled
or not. This instead of my security level stuff, that I decided was
too much of a pain the user, at least for now. New NS syntax:

	tb-set-inner-elab-eid two-simple

In the ElabinElab file, sets the name of an existing experiment in the
same project. Experiment is parsed, and after the parse we notice in
tbprerun that we have an inner eid, so we reparse the NS file, only
this time we pass in the maximum number of nodes needed by the inner
eid (tbprerun now computes min/max nodes at prerun time, instead of
later as first part of swapin). This number is used to allocatethe
appropriate number of inner experimental nodes. Why do it this way?
Cause the NS parser is the only tool we have for generating the virt
topology, and I do not want go down the path of inventing a new
frontend.

Anyway, after the reparse, we now have the proper number of nodes in
the wrapper experiment. Now its simply a matter of copying over the
type and fixnode info from the inner experiment to the outer
experiment.  Why? So that when the outer experiment is swapped in, it
gets the nodes (of the right type/fixnode) that the inner experiment
is going to want later, when it is swapped in by the inner emulab!

Another approach would be to make elabinelab and elabinelab_eid
options to batchexp (and thus the web form and XMLRPC interface) so
that we can avoid the double parse. I suspect people do not want more
crap on the web form, so I did not do it this way.

* Always run assign_wrapper using -t mode. This just runs the top file
  stuff, and writes the min/max nodes into the DB.

* Then look at the security level for the experiment, and if orange or
  red, create a parallel elabinelab experiment to run it in. This is a
  completely new experiement in addition to the original. The two
  experiments are linked with some DB state so we know what experiment
  to fire off inside the inner elab. I am using a template NS file and
  passing in the number of nodes computed in the previous step above.
  The template includes the firewall rules.

  This is quote hokey. It should be more invisible to the user.

  I have not dealt with yellow (just a firewall).

* I added some stats code so that we update the experiement_stats
  record with the elabinelab status and security level.

* Cleanup how errors were handled and get rid of silly duplicated
  code.

* The first involves swapmod. When a swapmod on an active experiment fails,
  tbswap will reswap the experiment back to the original configuration. The
  problem is that it is reswapping it with the *new* virtual state of the
  experiment in the DB. It is not until later when control returns to
  swapexp that the virtual state is restored. This is plainly wrong, and in
  fact was causing the event scheduler grief cause it was starting up,
  reading the the virtual topo, which was different, wrong, and about to be
  blown away.

  I reorganized the modify section of swapexp so that virtual state is
  restored only when its a swapmod on a swapped experiment. On an active
  experiment, I moved that code down into tbswap, which will now does all
  of the virtual and physical state retore before it does the reswap back
  to the original experiment. Just for kicks, its also done if tbswap
  decides to swap the experiment cause of a fatal error.

  Cleanups: I changed $NoRecover to $CanRecover. My feeble brain cannot
  deal with !$NoRecover. I know, two knots make a wright for most people.

  Renderer: I was annoyed by the fact that we rerun the renderer on a
  failed swapmod. The original reason is that the renderer runs in the
  background and so vis_nodes cannot be saved with the rest of the virtual
  state tables cause the renderer might still be running when the user
  fires off the swapmod. Well, the hell with that. We lock the vis_nodes
  table anyway in the renderer during update, so we are certain to get a
  consistent snapshot. We store the renderer pid in the experiments table,
  so if the renderer was running, just fire off another one; mostly this is
  not going to happen. In addition, tbprerun no longer starts a new
  renderer when doing the swapmod; I start the new renderer later after
  swapmod succeeds. I might end up tweaking this a bit depending on what
  people notice as being different.

* Termination changes to batchexp and swapexp: I've rearranged the
  termination code using an END block so that any uncontrolled exit from
  either batchexp or swapexp will go through the cleanup code, and
  hopefully insert a stats record, as well as not leave the experiment in
  some inbetween state. I've set the max DB retry count to zero in both
  cases, which means infinite retry. I've also added SIGTERM handlers to
  both so that again, we can kill a hung batch/swap and have it clean up
  things more or less. Note that END blocks are not caught when a signal
  causes the program to die; you have to catch it and then die() so that
  the END block is executed.

  Eventually, we need to clean up the various libraries so that we do not
  use DBQueryFatal(), but rather use DBQueryWarn(), and look for failure.
  Ditto for event system interface.

ipassign_wrapper no longer does that. Revert back to always invoking
the static route calculator (which figures out what to do). Remove the
stiff about generating a linkmap, which was an interim measure for
ddijk routing; now done with gentopfile.

gentopofile.  Note that the call to "staticroutes -j" is still there
while we transition over; eventually the -j option (which created an
input file for ddijkstra) will be removed, since the client side
rc.routes will generate that file on the remote node using the topomap
generated by gentopofile.

currently I run this all the time regardless of routertype or
use_ipassign flags, since its really quick. Will improve later.

from commvirtsig-branch into main trunk

Two testsuite tests fail: tbcmd and vtypes . Need to fix.

of virt_tables so that it is saved and restored like the rest of
the virtual state.

state machine (state). All of the stuff that was previously handled by
using batchstate is now embedded into the one state machine. Of
course, these mostly overlapped, so its not that much of a change,
except that we also redid the machine, adding more states (for
example, modify phases are now explicit. To get a picture of the
actual state machine, on boss:

		stategraph -o newstates EXPTSTATE
		gv newstates.ps

Things to note:

* The "batchstate" slot of the experiments table is now used solely to
  provide a lock for batch daemon. A secondary change will be to
  change the slot name to something more appropriate, but it can
  happen anytime after this new stuff is installed.

* I have left expt_locked for now, but another later change will be to remove
  expt_locked, and change it to active_busy or some such new state name in
  the state machine. I have removed most uses of expt_locked, except those
  that were necessary until there is a new state to replace it.

* These new changes are an implementation of the new state machine,
  but I have not done anything fancy. Most of the code is the same as
  it was before.

* I suspect that there are races with the batch daemon now, but they
  are going to be rare, and the end result is probably that a
  cancelation is delayed a little bit.

swapped out (non-recoverable) by tbswap. swapexp was leaving the
experiment in the running state instead of paused. We need to check
this after tbswap since we do not get reasonable error codes back.
Also some cleanup with respect to how aborted modifies are handled.
I think I understand what Chad did ...

A general comment; we need to be better about returning meaningful
error codes!

not have to wait 3 minutes for it to finish before he can watch his
experiment swapin fail for some other reason.

I adopted the same pid mechanism as in eventsys_control.in, which uses
a slot in the experiments table.

Running "prerender" puts the render into the background and stores
the pid. Running "prerender -r" kills a running prerender and removes
the existing info from the DB.

Fixed the problem with swapmod not restoring the old vis; swapmod now
kills any running prerender, and restarts one if the swapmod fails
(the prerun of the new NS file starts up another prerender in the
background).

Add setpriority() call in prerender to nice it and children to 15.

files.

Various Other changes to get Expt Modify ready for prime time.

 - If assign fails on a modify, experiment will
   be restored to old state, *not* swapped out.

 - Reboot option has been improved to reboot all
   nodes as part of os_setup, not in separate
   step.

 - Different assign error codes result in different
   retry behavior for assign_wrapper
   (Follow's Rob's change to assign to make it
    pass back special code for non-retriable faults)

 - '64' bit in assign_wrapper exit code indicates to tbswap
   that db/phys state hadn't been mucked with before
   the exit occurred
   (ergo, '65' and '1' are the common return codes,
    though the old 4,8,16,32 are still there for assign failing.)

 - (tbswap still returns codes from assign wrapper)

 - Added 5 sec pause between assign attempts.

 - Cleaned up tbswap code.

 - Physical state backup/restore removed from tbprerun,
   put into swapexp.

 - Interfaces table now getting cleaned up correctly
   (Mike noticed problem)

 - Changed menu display in showexp to show
   the "modify" menu option for swapped out experiments
   (like it used to.)

 - A couple other changes.

Note:
 Still admin-only, but I plan to change that soon.

To do:
 - Erase expt backups in /tmp after using them.
 - Re-viz failed experiments.

get an updated copyrights message.

Added new feature 'Experiment Modify'.
Now available (to admins only for now) from the showexp page.

Warning! doing a modify which alters the topology will probably
require a "reboot all nodes" afterwards.
(There will be a checkbox soon in the modify experiment page.)

Adding/removing delay nodes seems to work fine without reboots, though.

Warning! If the new version of the experiment cannot be mapped
 (not enough nodes available, for instance) the experiment will be
 swapped out! This will get fixed later.

Prerun backs up the experiment topology, so using a bad NS
file doesn't result in experiment termination.

As part of this, added library functions to libdb to
delete, backup, and restore both virtual and physical experiment state.

Stored in $TB/www/thumbs/,
with filenames generated by hashing random data with experiment name.
(Stored in vis_experiments table in db.)

New visualizer; uses "vis_nodes" table in database to store precomputed solutions.
Added "prerender" call to tbprerun and tbend.
"prerender_all" can be used to precompute vis solutions for all experiments.
Also, removed legacy vis cruft.

parser runs. The staticroutes script is a wrapper for Chad's route
solver. The network optimization is currently turned off; use -t to
turn it on, until I know if its correct.

Note that Chad gets credit for routecalc.cc; I'm just committing the
file for him, with a couple of trivial changes that I made.

	# Turn on manual routing.
	$ns rtproto Manual

	# Set manual routes
	$nodeA add-route $nodeC $nodeB
	$nodeC add-route $nodeA $nodeB

results in this information being returned from the tmcd routing
command:

	ROUTERTYPE=manual
	ROUTE DEST=192.168.2.3 DESTTYPE=host DESTMASK=255.255.255.0 \
		NEXTHOP=192.168.3.2 COST=0

The reason for DESTTYPE and DESTMASK is so that we can also support
routing to links and lans, since doing it on a per host basis if not
only hugely tedious, but plain impossible if the destination node has
multiple links; the add-route syntax takes a node, but we need the IP
of the relevant link in order to run the route add commands on the
nodes. So, I've "extended" the syntax of add-route so that you can
give it a Link or a Lan as the dest:

	$nodeA add-route $link0 $nodeB
	$nodeA add-route [$ns link $nodeB $nodeC] $nodeB

In this case, the DESTTYPE=net, and the netmask is no longer ignored;
it is used in the route add command. Currently, the mask is hardwired
in the DB to 255.255.255.0, but by providing it in the tmcd command,
we change it later if needed.

I did not implement add-route-to-adj-node since that is not really
useful in our context, and we definitely do not want the user to
change the default routes on his nodes. But, its easy to add if we
need to.

The client side stuff is not done yet.

provides a mapping from an object ($cbr0) to the node on which the
agent is running ($nodeA). It also includes the type of agent
(TRAFGEN, LINK, etc). There were a number of reasons for adding this
table:

* To avoid a series of specialized table lookups in the event
  scheduler to map from a name (link0) to the node on which the agent
  is running. Previously, it was looking the delays table and the
  virt_trafgens table. Well, now it just needs to look at this one
  table and store the mapping internally. When a dynamic event comes
  in, we can figure out where to send it easily.

* For NSE traffic generation. Unlike the simple TG based CBRs, Shashi
  wants to be able to send events to any of the objects in the config
  (udp0, telnet0, ftp0, etc). Well, the virt_trafgens table certainly
  does not store that info, and it would have been painful to work
  this into it. For every agent/application, just add an entry in the
  virt_agents table and the scheduler knows where to send the events.

* It nicely supports the new program object; just plug an entry in the
  virt_agents table.

Okay, there is one messy aspect; delays nodes! Delay nodes are not
computed until after assign wrapper runs, so in addition to munging
the static event list in assign_wrapper, we now munge the virt_agents
table as well. Not much to do about this; delay nodes are handled
outside the normal path everywhere.

Telnet sources)

assign_wrapper.in: Hack in a change that ensures a delay node is
created for any link on which an event is posted (up,down,modify),
no matter what its initial parameters are. ie: If a link is created
with no delay, but there is an event that adds a delay later, then we
must drop in a delay node. Same for up/down on a link. We do this in
the delay node. I am reasonably confident that this change is fine for
duplex links, but I am less sure of the effect on lans!

eventsys_control.in: Checkpoint latest changes. Add "replay" option,
which right now just stops and starts the event scheduler so that it
reloads the entire event list. Add check for existing experiment, and
that the experiment is either active or swapping (do not want to start
a scheduler for a swapped out experiment!). Add check to see if there
are any events, and skip startup if there are not events in the DB.
Lastly, get very serious about preventing more than one scheduler from
being started, either by accident or intentionally. My protocol is to
lock the table, grab and set the pid to -pid, test the pid for a
positive value, and if positive, send the scheduler a kill(TERM) so
that it can cleanup, clear the pid to zero in the DB, and exit. This
approach ensures that we do not try to send a kill to a pid that is no
longer active or owned by the user (this last part is not really
necessary cause of how pids are reused, but it was easy to add so why
not).

exports_setup.in: Trivial change to make it easier to turn this on
temporarily in devel trees.
named_setup.in: Ditto.

node_reboot.in: Add call to TBdbfork() in child cause of apparent DB
connection problems across forks. In the child, set the eventstatus
for the node to REBOOT if successful (not this event status stuff is
temporary, will be recast in next set of revisions).

GNUmakefile:  Add new controlling program, eventsys_control.
power.in:     Ditto previous comment about REBOOT.
os_setup.in:  Non event system cleanups.
tbend.in:     Add DB cleanup of the new virt_trafgens and eventlist tables.
tbprerun.in:  Ditto.
tbreport.in:  Print out the event list in a pretty print format.
tbswapin.in:  Add call to start the event system. Also a big fix; move
              the named script up above the os_setup so that the named
              tables have been updated by the time the first node
              reboots. I noticed that nodes were failing on gethostbyname().
tbswapout.in: Add call to stop the event system.

an infinite loop rather easily via the NS file TCL hooks. Added a
perl wrapper around parse.tcl called parse-ns, which forks a child to
run the parser. The parser is invoked "nice +10" and the CPU limit for
the child is set to 60 seconds, which should be enough for any parse.
If the limit is exceeded, send email to tbops since this indicates a
big problem or a user being dumb/malicious.

to use the DB library access routines, which also changed in response
to what the tb scripts needed. Added some functions and mor constants.
Removed the -nologfile option from all the scripts (startexp and
endexp too), since there is no reason for these scripts to worry about
log files. Thats handled in the wrappers. Tested with the testsuite
and live in my own tree.

Lots of small changes for turning our 'require lib*' lines into 'use lib*' lines. Proper modules declare themselves as a package, and use Exporter to export the names of the subroutines that should be visible from the outside world. Many of ours didn't do that, it was just a file with a bunch of subs in it. So now I've fixed many of them to be proper, and removed the requires and 'push(@INC,...)' hacks and changed it to the proper 'use lib @prefix@/lib/;' and use lib*.

Fixed some warnings in assign_wrapper that were subsequently caught.

exitonwarn is a library that, when used in a script, causes it to exit 255
when warnings are sent.  It still shows all warnings, just does an ugly
perl hack to error on script termination.

with the testing stuff.

them on the first run to accurate predictions.

fixes we missed.

tbprerun - Fixed estimations to be slightly more accurate.  Hard to get
complete accuracy without duplicating much of assign_wrapper.  This'll
probably change in someway in the near future.