GitLab

(eventual) support for NFS servers without race conditions!

This means no NFS between nodes and ops/fs. There are still NFS mounts of
ops on boss however.

Added new defs-* variable NOSHAREDFS, which when set non-zero will disable
the export of NFS filesystems to nodes.  Involved lots of little changes:

 * /users, /proj, and /share filesystems are not exported to nodes.

 * Returned mount info now includes an FSTYPE key which will be set to "LOCAL"
   if NOSHAREDFS is in effect (by default it is set to "NFS-RACY"; more on
   this later).  In the case where it is set to LOCAL, the other mount lines
   no longer contain REMOTE=foo settings.  Because of this change,
   THE TMCD VERSION NUMBER HAS BEEN BUMPED TO 32.

 * The client rc.mounts script will now create local versions of /users/*,
   /proj/<pid>, and /share when FSTYPE=LOCAL.  It first runs mkextrafs to
   create a large partition for these, since someday we will likely want
   to pre-populate these with a non-trivial amount of data.  Right now,
   the only thing that is put in the user's homedir is the standard dotfiles
   for the OS and the Emulab authorized_keys file (so you can login).

 * Linktest had to be modified to fetch the various results files (via
   loghole) rather than just assuming they were in /proj.  And also changed
   to invoke tevc with the local copy of the event key so it won't try to
   read it over NFS.

 * create_image was modified to ssh to the node and run the imagezip
   command, capturing the output of ssh.  This is controlled via the "-s"
   option which defaults to on for a NOSHAREDFS system, but can also be
   used on a normal system.

 * elabinelab's can be configured with/without a shared FS via the
   CONFIG_SHAREDFS attribute (note polarity change) which defaults to 1.

Another new defs-* variable, NFSRACY, will some day allow you to specify
(by setting to 0) that your NFS server does NOT have the nefarious mountd
race condition when changing /etc/exports.  Currently, this defaults to 1
since all versions of FreeBSD supported as an "fs" node have this "feature."
Rumor has it that FreeBSD 8 does not have this problem nor, presumably,
would a Linux NFS server.

The only use of this variable right now is to set the FSTYPE returned by the
tmcd "mounts" call, which in turn is used by one client script, rc.topomap
(via a libsetup function) to determine whether it should try copying
the topo file multiple times.

Random: add python2.6 to list of python's checked for in configure.
Random: resync defs-example-privatecnet with defs-example.
Random: did a little code-pissin here and there.

In elab_linktest.pl, put explicit command line arguments after the defaults
so that they will override them in linktest.pl.  Also, don't conflate LOGDIR
and PROJDIR.

Half-assed attempt to free the script from the notion of a pid/eid, by
defining EVENTID param.  However, we still wind up reading a pid/eid from a
file though this could also be eliminated.

This will used the local copy of the keyfile (/var/emulab/boot) instead
of the across-NFS copy (/proj/.../tbdata).

 * added new tmcd directory with Ubuntu (really, Debian) specifics
 * fixed up GNUmakefiles to not do "-g wheel" when creating directories
 * other, relatively minor tweaks

Fix intermittent one-way loss test failure.  Let bind() choose the src port in rude; any constant port like 3001 may be already in use.

Two-day boondoggle to support "/scratch", an optional large, shared filesystem
for users.  To do this, I needed to find all the instances where /proj is used
and behave accordingly.  The boondoggle part was the decision to gather up all
the hardwired instances of shared directory names ("/proj", "/users", etc.)
so that they are set in a common place (via unexposed configure variables).
This is a boondoggle because:

1. I didn't change the client-side scripts.  They need a different mechanism
   (e.g., tmcd) to get the info, configure is the wrong way.

2. Even if I had done #1 it is likely--no, certain--that something would
   fail if you tried to rename "/proj" to be "/mike".  These names are just
   too ingrained.

3. We may not even use "/scratch" as it turns out.

Note, I also didn't fix any of the .html documentation.  Anyway, it is done.
To maintain my illusion in the future you should:

1. Have perl scripts include "use libtestbed" and use the defined PROJROOT(),
   et.al. functions where possible.  If not possible, make sure they run
   through configure and use @PROJROOT_DIR@, etc.

2. Use the configure method for python, C, php and other languages.

3. There are perl (TBValidUserDir) and php (VALIDUSERPATH) functions which
   you should call to determine if an NS, template parameter, tarball or
   other file are in "an acceptable location."  Use these functions where
   possible.  They know about the optional "scratch" filesystem.  Note that
   the perl function is over-engineered to handles cases that don't occur
   in nature.

accurate.  Not sure I improved it dramatically, but I sure did move the
code around a lot!

commit to rc.topomap).

Other, minor nits.

Still not complete, we really have to disable the LAN-optimization if the
experiment involves a node running an old linktest.

Change the rude.cfg script we generate to have a final "wait around" period
(MODIFY with 0 rate) before exiting.  We have it wait for the one-way latency
of the link.  This way, rude will not exit--and linktest will not do the
barrier synch--til all packets have had a fair chance to get to the other side.
This way we don't need a randomly chosen sleep after the barrier returns
and before we read the crude results.  Though rude already supported the
MODIFY with zero rate, I had to fix it since it didn't work if it was the
last command for the flow.

Adjust the timeout period for ping when doing the latency tests.  Previously
we waited for 3 seconds, now wait for max(RTT+1, 3) seconds.

Much more work will have to be done to support multi-second latencies
all around, this was just the low-hanging fruit.

It is amazing the amount of stuff you have to do, even if you are not going
to do anything!

Comment out a dubious piece of code that "adjusted" the recorded runtime
of iperf if the packet loss was greater than 10%.  I think this code was
another way of dealing with the lost/retransmitted FIN problem that has
been addressed with a new iperf option.

FIN timeout thus getting more accurate BW measurements.  From the code:

                  #
                  # Even without packet loss, there is a good chance that
                  # iperf's end-of-stream (FIN) will be lost since we are
                  # over-driving the link.  The ACK for that FIN needs to
                  # be received before iperf's timer stops, so a long
                  # timeout (for resending the FIN) on a short duration
                  # run will lower the BW reading considerably.  So we
                  # attempt to keep the timeout as short as possible,
                  # taking into account the round-trip latency of the link.
                  # We may also need to adjust the run time of the test
                  # upward, but we do not exceed the indicated max duration.
                  #
                  # 250 ms is the default timeout value for iperf.  We
                  # start with a candidate value of 50 ms which is chosen
                  # as it is sufficient for the 3% BW error on min duration
                  # (3 second) tests.  You should not pick a base acktime
                  # less than the resolution of the clock (10ms or 1ms).
                  #

See event/linktest/CHANGES for details.

   both directions at the same time again. I had put in a delay, attempting
   to see if that might help the packet loss problems; I do not think it
   does. So, loss tests will complete faster.

2) Changed the duration for the bw tests as we discussed. I did a quickie
   trial, and ended up with:

	    x > 33Mbs	3 seconds
    33Mbs > x > 10Mbs	5 seconds
    10Mbs > x > 64Kbs   7 seconds

   Then add 3 seconds if the loss if over 10%

   The above numbers are very conservative; we can probably do better, but
   it would take a longer set of trials (across a variety bw/delay
   products). Remember, all bw tests ran for 10 seconds (in each direction).

3) Add in the ethernet/ip/udp overhead to the bw results; Much nicer. I
   reduced the allowed error factor fom 6% to 2%, although its actually
   within less then 1% most of the time.

4) Allow BW checks when there is loss on the link; previously linktest
   skipped any link with loss on it. Since we UDP for the tests, we
   can fairly well estimate what the BW will be. This is not perfect
   yet, especially with low BW, high loss links. We need to run some
   trials to see where accuracy falls off too far, or how to
   compensate for it.

5) Added some more time stamps.

This is strictly information right now; it is not worth trying to make
linktest calculate the expected loss for a red or gred link, but at
least the error report will indicate it.

Also add a linktest version tag and send that to the output log.

* When adjusting the latency on very low bandwidth links, lets use 98
  bytes for the number of bytes on the wire. Where did 82 come from?

* Attempt to deal with session routing failures. Reworked the test
  code to do multiple trials (when session routing is active) in the
  hopes that routes will stabilize soon after trying to actually use
  the routes. Currently trying a second trial, with a delay of 60
  seconds + (0.25 * numberofnodes). I have no experimental basis for
  this number!

Also reworked error formatting as per Jay's request.

bandwidth side effects. Slow down by factor of 4, sending for 4
seconds instead of 1 second.

wait_all, just add one to the error_count instead of the exit code,
otherwise sync gets unhappy.

properly ordered.

multihomed nodes, picking the wrong set of edges cause it was not
looking at the name of the link, just the name of the node.

Also allow routing test when the routing type is ospf.