Blew it up with the current set of 145 remote nodes. I increased the
limits and added checks so that it does not return bogus output.
Needs a pass to turn them into dynamically sized arrays, at some
point.

other than the first object name.

1) IP address based routes can now be added
   - The IP address is set on a link object
   - An "$ns rlink" is used to instantiate links
     that get cut and cross physical partitions
   - Traffic agents that are across physical
     partitions (i.e. different instances of nse)
     are connected by a new "$ns ip-connect"
     mechanism
   - A new Node instproc "add-route-to-ip" adds
     IP address based routes.
   - Changed ns multicast addressing to use 3 bits
     instead of the default 1
   - Currently, the classifier does a lookup on a
     complete 32 bit IP and if a target to route to
     is not found, uses a 24 bit IP mask. It does not
     try to match IP prefixes of all lengths. I'll add
     that later if necessary
2) NS packets that cross partitions are encapsulated in
   IPPROTO_ENCAP IP packets.
3) RAW IP sockets used to inject packets into the network
   now use a rtabid paramater so that packets can be
   routed according to different routing tables

Tested with 2 test cases, one with UDP/CBR traffic
and another with default NS TCP/FTP traffic. Setup was done
manually. As I do testbed integration, there may be more changes.
Here's the test setup:

   2.2    2.3   1.2      1.3   3.2      3.3
n0 --------- n1 ----------- n2 ------------ n3

n0,n1 are on one physical node and n2,n3 are on another. The n1-n2 link
is cut.

A TCP example:

---------------------physnode0---------------------
set ns [new Simulator]
$ns use-scheduler RealTime

set n0 [$ns node]
set n1 [$ns node]

$ns duplex-link $n0 $n1 10Mb 5ms DropTail
[$ns link $n0 $n1] set-ip 10.1.2.2
[$ns link $n1 $n0] set-ip 10.1.2.3

set rl0 [$ns rlink $n1 10.1.1.3 2Mb 40ms DropTail]
$rl0 set-ip 10.1.1.2

set tcp0 [new Agent/TCP]
# The last parameter specifies the port
$ns attach-agent $n0 $tcp0 20
$ns ip-connect $tcp0 10.1.3.3 20
set ftp0 [new Application/FTP]
$ftp0 attach-agent $tcp0

$n0 add-route-to-ip 10.1.3.3 10.1.2.3
$n1 add-route-to-ip 10.1.3.3 10.1.1.3

$ns at 1.0 "$ftp0 start"
$ns at 10.0 "$ftp0 stop"
-----------------end physnode0---------------------

---------------------physnode1---------------------
set ns [new Simulator]
$ns use-scheduler RealTime

set n2 [$ns node]
set n3 [$ns node]

$ns duplex-link $n2 $n3 10Mb 5ms DropTail
[$ns link $n2 $n3] set-ip 10.1.3.2
[$ns link $n3 $n2] set-ip 10.1.3.3

set rl1 [$ns rlink $n2 10.1.1.2 2Mb 40ms DropTail]
$rl1 set-ip 10.1.1.3

set tcpsink0 [new Agent/TCPSink]
$ns attach-agent $n3 $tcpsink0 20
$ns ip-connect $tcpsink0 10.1.2.2 20

$n3 add-route-to-ip 10.1.2.2 10.1.3.2
$n2 add-route-to-ip 10.1.2.2 10.1.1.2
-----------------end physnode1---------------------

Add install target to stick into the sbin directory.

number of connections to tmcd, and the resulting number of DB queries.
Currently thats about 24 per node when it boots. Each vnode adds
another 24 or so. The new approach is to use the "fullconfig" command,
which dumps the entire config in one shot, saving about 20 of those
connections. We still need to do the status/state commands for real of
course. When a node boots, it requests the fullconfig; the client side
takes this fullconfig, and dumps the individual sections to
/var/emulab/boot/tmcc/section_name. Subsequent requests first look for
it locally in the above named files, falling back to real tmcc if none
exists. The update command also refreshes the cache.

Tested for jails and plab node vservers as well.

ordering of events is not obvious to anyone except me (on a good day).

swapped out (non-recoverable) by tbswap. swapexp was leaving the
experiment in the running state instead of paused. We need to check
this after tbswap since we do not get reasonable error codes back.
Also some cleanup with respect to how aborted modifies are handled.
I think I understand what Chad did ...

A general comment; we need to be better about returning meaningful
error codes!

install file to reflect this.

All to try to avoid loopback lockups.

from /var/log/messages.

Fixed up the list of logfiles on the boss node.

as defined in the defs-* file (e.g. "TBLOGFACIL=local2").  The default is
"local5" which is what we are setup to use so you shouldn't need to mess
with your defs- file!

perl scripts just get this value configured in when configure is run.
C programs get the value in two ways.  For programs that are intimate with
the testbed infrastructure, and include "config.h", they just get it from
that file.  For programs that we sometimes use outside the Emulab build
environment (e.g., frisbee, capture) and that don't include config.h,
the value is set via a "-DLOG_TESTBED=..." in the GNUmakefile build line.
If the value isn't set, it defaults to what it used to be (usually LOG_USER).

Still to do: healthd, hmcd (whose build doesn't seem to be completely
integrated) and plabdaemon.in (since its icky python :-)

This is much more robust than the old method.

slightly different things for physnodes than for vnodes.

We have seen cases where dmesg has info from multiple boots and sometime
even garbage.

Update to libplab.plab.renew:

  * Make renewal robust against various kinds of failures.  These changes
    will augment my larger set of libplab and plab* updates/fixes coming
    soon to an Emulab near you.

that slot, and domounts checks that slot now. The older USESFS=1 is
still supported for now, but will chucked eventually.

More work on supporting client side caching of the full configuration.

users.

via the Mod User Info page.

also includes updated tb_compat.tcl include file and ns patch.

I have implemented the suggestion Jay made a couple of weeks ago
about allowing partial allocation in assign_wrapper, and retrying with a
modified set of "fixed" nodes.

My basic approach was to change nalloc to optionally allow partial
allocations, returning the number of nodes that could not be allocated as
its return value. In assign_wrapper, I determine which nodes we were able
to get (in each loop), set their allocstate to INIT_DIRTY, augment the
fixed_node set, and recreate the top file. Then I try again, up to the
current number of maxtries. If assign fails with an unretryable error, or
if we could not nalloc a user directed fixed node, then I stop right away
since the experiment is not going to map (in the near term) if the fixed
node list cannot be allocated.

I am confident that this works okay, although testing is a little
difficult. The main problem is how this interacts with experiment modify.
Chad's implementation is that a modify can be reverted (recovered from)
only as long as the DB is not modified by assign_wrapper. Well, a partial
allocation, followed by failure, obviously modifies the DB, and so is
deemed not recoverable. I am still trying to figure out the effects of
this, and whether I can relax this requirement, but in the meantime
lets install it and see what happens (won't affect many people).

but its nice to have it in the DB too so that we do not have to read
that file!