Commit 04f6583d authored by Jonathon Duerig's avatar Jonathon Duerig

Added various pelab helper scripts which hadn't been checked in.

parent d7387e6b
----------------------------------------------------------------------
Flexlab software release - Beta
----------------------------------------------------------------------
Disclaimer:
This software is currently in beta. It still needs a lot of polishing, and is
somewhat brittle in the face of PlanetLab node failure, which is frequent.
----------------------------------------------------------------------
Basic information:
** iperf.ns - sample ns file
We suggest that you read the paper "The Flexlab Approach to Realistic
Evaluation of Networked Systems", by Ricci, Duerig, et al. in the proc. of NSDI
2007 for an explanation of how Flexlab works.
** send-streams
This program starts an iperf server, and initiates simultaneous iperf
client connections to host specified using -d for duration specified
by -t.
You might see references to 'pelab' (PlanetLab + Emulab) in this software -
this was the internal name for this project, and not all references to it have
been purged.
usage: $0 [-h] [-m num] [-d ip -t time] [-w directory]
----------------------------------------------------------------------
What is supported:
-h : this (help) message
-v : verbose output
-m num : become master node for synchronization purpose of num nodes
-d ip : destination ip address or full name
-t time : iperf duration in secs
-w dir : working directory
* Automatic creation of an experiment of any number of nodes
example: $0 -d 192.168.1.1 -t 30 -d 192.168.4.2 -t 45
* Running experiments on the real PlanetLab or a 'fake' PlanetLab inside
of Emulab - or both in the same experiment
* Automatic software installation, and running and instrumenting server
code
* Automatic tcpdumps of your traffic on the Emulab and PlanetLab side
* Automatic logfile gathering
* Automatic collection of log files at the end of an experiment
* Easy starting/restarting of stubs and monitors
* Asking for specific PlanetLab nodes or sites
* Setting initial conditions for nodes based on flexmon data
----------------------------------------------------------------------
What is not supported:
* Running test clients (ie. 'iperf -c') for you (planned)
* Instrumenting programs that use preemptive (ie. kernel-level) threads
(planned)
----------------------------------------------------------------------
Running an experiment with the auto-pelab scripts:
Note: Depending in how you got the source, it will either be in a directory
called 'pelab' made by untarring the tarball, or in the 'pelab' subdirectory
of the testbed CVS repository.
1) Build the flexlab software:
Swap in an experiment using the 'PLAB-DEVBOX' image. Note that this disk
image corresponds to PlanetLab v4.0, and that binaries built on it
will not run on v3.3 nodes, due to incompatible system libraries. Note
that if you already have a Flexlab experiment swapped in, you can use
the nodes in the Emulab side of it for this purpose.
Run 'make' in the root directory of the Flexlab source
2) Create a tarball of the version of the Flexlab software (monitor, agent,
etc.) that you'd like to test:
From the root of your Flexlab source tree, run:
sh ./make-tarball /proj/<PID>/my-pelab.tar.gz
Of course, don't name it 'my-plab.tar.gz'! You can put this file in
a subdirectory if you want, but it must be somewhere in
/proj/<PID>/, where <PID> is the project ID of the project you're
going to run the experiment in (for example, 'tbres')
Put the path to your tarball in the pelab_tar variable in auto-pelab.ns
2) Read the user-settable options at the top of auto-pelab.ns and change them
to your preferences.
3) Use your modified auto-pelab.ns to create an experiment and swap it in
4) Once it has swapped in, and you're ready to start an experiment, run the
'start-experiment' script from your source directory on ops. It takes as
arguments the pid and eid of your experiment. It will:
Restart monitors and agents
Reset ALL link shaping to the defaults from the NS file
Set initial conditions for all paths from the flexmon database
Clean logfiles
Restart the user-specified servers on all 'elab-*' nodes
start-experiment has a few useful options for passing additional
arguments to the agents and monitors, e.g.
start-experiment -s '-d' pid eid
would turn on debugging in the magents
start-experiment -m '-d' pid eid
would do the same for the monitors (assuming it has a '-d' option).
Important: If you are using real planetlab nodes, instead of a 'fake'
planetlab inside of Emulab, run start-experiment with '-p'.
Important: If you want to have the initial conditions for your paths set
to measured values, rather than 0ms 100Mbps (and you should...), give
start-experiment the '-i' flag
Important: Watch for errors from start-experiment - errors about 'no such
file or directory' when cleaning out logfiles are harmless, but other
errors can be important.
4a) If you modify something in the flexlab (e.g., the agent) and need to
deploy it to the nodes, first recreate the tarball as in step 1,
and then
install-tarball pid eid
this triggers a program agent on all the nodes telling it to update
the tarballs and RPMs. You will need to run start-experiment again
after this to restart agents/monitors.
5) Log into the 'elab-*' node or nodes, and run your experiment (ie. one run
of iperf). Any programs that you run should be prefixed with
'/local/pelab/monitor/instrument.sh'. This shell script sets up environment
variables so that your program has all of its network calls instrumented.
All command line arguments are passed through to the program - ie:
/local/pelab/monitor/instrument.sh iperf -c elab-2
6) When you're done with your experiment, run 'stop-experiment' on ops. It
will:
Stop the stubs, monitors, and iperf servers
Collect logfiles from all of the nodes in the experiment, including
stub/monitor logs, tcpdumps, delay agent logs, and so on
Run tcptrace on all tcpdump logs to produce xplot graphs
Put all of these things in a zip file for you in the experiment's log
directory - you will get a message like 'Created log hole archive:
EXP.5.zip. This file can be found in (/proj/PID/exp/EID/logs). A new
zipfile is kept for each experiment, and old ones are not automatically
deleted.
7) If you want to look at the logs, unzip the .zip file somewhere, and see
the description of the files it contains below. If you want to look at logs
as the experiment runs, they are typically in /local/logs/ on the nodes.
----------------------------------------------------------------------
Important places:
* The contents of the flexlab tarball are installed in /local/pelab on each
node
* Logfiles from the monitors, agents, and the user-specified server program
can all be found in /local/logs on the node while the experiment runs. You
can tail the .out or .err files to see the stdout and stderr of these
processes
* Inside the archive zip file you will find:
elab-*/ - Logifles from the emulator hosts
event-sched.log - Log of all events (including dummynet control events) sent
during the experiment
plab-*/ - Logfiles from the FAKE plabnetlab nodes
planet-*/ - Logifles from the REAL PlanetLab nodes, including tcpdump traces
of traffic to and from the PlanetLab nodes
tbsdelay*/ - Logs from the delay nodes, including tcpdump traces from the
emulab nodes
** runiperfmesh
* to be run at node-startup
* takes care of FC2/FC4 dependency of iperf
* currently starts all pair iperf for 10 secs
-- result files are stored in /local/logs/iperfmesh directory
(you can use loghole to get them)
cd stub
gmake
cd ../libnetmon
gmake
cd ../magent
gmake
cd ../..
tar czvf /proj/tbres/duerig/pelab.tar.gz pelab
# Usage: analyze-multiplex.pl [filename [...]]
%nodeToPlab = ();
%plabToCap = ();
%plabToCapExempt = ();
%hostToPlab = ();
%plabToSite = ();
sub readNodeMap
{
open NODE_MAP, "<"."ltpmap";
my $line;
while ($line = <NODE_MAP>)
{
if ($line =~ /H node-([0-9]+) [^ ]+ (plab[0-9]+)/)
{
$nodeToPlab{$1} = $2;
}
}
}
sub readPlabMap
{
open PLAB_MAP, "<"."plab-map.txt";
my $line;
while ($line = <PLAB_MAP>)
{
if ($line =~ /(plab[0-9]+)\s+([a-zA-Z0-9.\-]+)\s+([a-zA-Z0-9\-]+)\s+([0-9]*)/)
{
if ($4 == "")
{
$plabToCap{$1} = 0;
}
else
{
$plabToCap{$1} = $4;
}
$plabToSite{$1} = $3;
$hostToPlab{$2} = $1;
}
}
}
sub readExempt
{
open EXEMPT, "<"."i2.nodes";
my $line;
while ($line = <PLAB_EXEMPT>)
{
$line =~ /\s*([a-zA-Z0-9.\-]+)\s*/;
my $plab = $hostToPlab{$1};
$plabToCapExempt{$plab} = 1;
}
}
@bandwidths = ();
sub regressSlope
{
my $begin = shift(@_);
my $end = shift(@_);
my $x_i = 0.0;
my $y_i = 0.0;
my $numA = 0.0;
my $numB = 0.0;
my $numC = 0.0;
my $denomA = 0.0;
my $denomB = 0.0;
my $denomC = 0.0;
for (my $i = $begin; $i < $end && $i < scalar(@bandwidths); ++$i)
{
$x_i = $i + 1;
$y_i = $bandwidths[$i];
$numA += $x_i * $y_i;
$numB += $x_i;
$numC += $y_i;
$denomA += $x_i * $x_i;
$denomB += $x_i;
$denomC += $x_i;
}
my $numD = $end - $begin;
my $denomD = $end - $begin;
my $num = ($numA * $numD) - ($numB * $numC);
my $denom = ($denomA * $denomD) - ($denomB * $denomC);
my $slope = 0;
if (abs($denom) > 0.0001)
{
$slope = $num/$denom;
}
# print("Result: $num / $denom\n");
return $slope;
}
sub regressIntercept
{
my $begin = shift(@_);
my $end = shift(@_);
my $x_i = 0.0;
my $y_i = 0.0;
my $numA = 0.0;
my $numB = 0.0;
my $numC = 0.0;
my $numD = 0.0;
my $denomA = 0.0;
my $denomB = 0.0;
my $denomC = 0.0;
for (my $i = $begin; $i < $end && $i < scalar(@bandwidths); ++$i)
{
$x_i = $i + 1;
$y_i = $bandwidths[$i];
$numA += $x_i * $x_i;
$numB += $x_i * $y_i;
$numC += $x_i;
$numD += $y_i;
$denomA += $x_i * $x_i;
$denomB += $x_i;
$denomC += $x_i;
}
my $denomD = $end - $begin;
my $num = ($numA * $numD) - ($numB * $numC);
my $denom = ($denomA * $denomD) - ($denomB * $denomC);
my $intercept = 0;
if (abs($denom) > 0.0001)
{
$intercept = $num/$denom;
}
# print("Result: $num / $denom\n");
return $intercept;
}
sub regressError
{
my $begin = shift(@_);
my $end = shift(@_);
my $slope = shift(@_);
my $intercept = shift(@_);
my $deviation = 0.0;
for (my $i = $begin; $i < $end && $i < scalar(@bandwidths); ++$i)
{
$lineY = $slope * ($i+1) + $intercept;
$deviation += ($bandwidths[$i] - $lineY) * ($bandwidths[$i] - $lineY);
}
my $mean = 0.0;
for (my $i = $begin; $i < $end && $i < scalar(@bandwidths); ++$i)
{
$mean += $bandwidths[$i];
}
$mean = $mean / ($end - $begin);
# ssm => sum of squares of difference with mean
my $ssm = 0.0;
for (my $i = $begin; $i < $end && $i < scalar(@bandwidths); ++$i)
{
$ssm += ($bandwidths[$i] - $mean) * ($bandwidths[$i] - $mean);
}
if ($ssm == 0)
{
return -9999999999;
}
else
{
# print "dev: " . $deviation . ", ssm: " . $ssm . "\n";
return 1 - ($deviation / $ssm);
# return ($ssm - $deviation) / $ssm;
}
}
readNodeMap();
readPlabMap();
readExempt();
@normal = ();
@interesting = ();
@problem = ();
$counter = 0;
for ($arg = 0; $arg < scalar(@ARGV); ++$arg)
{
@bandwidths = ();
open FILE, "<".$ARGV[$arg];
$good = 1;
while ($line = <FILE>)
{
if ($line =~ /([0-9.]+) ([0-9.]+) /)
{
$x = $1;
$y = $2;
push(@bandwidths, $y);
}
else
{
$good = 0;
}
}
$size = scalar(@bandwidths);
if ($good == 1 && $size == 10)
{
$bestLeftSlope = 0;
$bestRightSlope = 0;
$bestDivider = 0;
$bestError = 0;
for ($divider = 1; $divider < 9; ++$divider)
{
$leftSlope = regressSlope(0, $divider + 1);
$leftIntercept = regressIntercept(0, $divider + 1);
$rightSlope = regressSlope($divider, $size);
$rightIntercept = regressIntercept($divider, $size);
$error = (regressError(0, $divider + 1, $leftSlope, $leftIntercept)
+ regressError($divider, $size, $rightSlope, $rightIntercept))/2;
if ($error > $bestError)
{
$bestLeftSlope = $leftSlope;
$bestRightSlope = $rightSlope;
$bestDivider = $divider;
$bestError = $error;
}
}
# Sync up with the # of streams where 1 stream is at position 0.
$bestDivider += 1;
# print ($bestError . "\n");
# ++$counter;
# Initial slope positive flat or straight
# straight normalized in terms of bandwidth
# Accuracy in terms of percentage?
# Percent increase in y value?
# should increase by k per flow while window size limited
# Two lines, each of which are increasing decreasing or flat. Divide into 9 buckets.
# 20 node experiment
# keep tcpdump
# check in tonight
# ping
# Find ones where the max value == cap
# Find ones where the max value > cap
if ($bestError < 0.7
|| (abs($bestLeftSlope - $bestRightSlope) > 1
&& $bestLeftSlope < 1
&& ($bestRightSlope > 0.5 || $bestRightSlope < -0.5)))
{
# print("---\n");
# print($ARGV[$arg]."\n");
# print("---\n");
# print("Error: $bestError\n");
# print("Left-Slope: $bestLeftSlope\n");
# print("Right-Slope: $bestRightSlope\n");
# print("Divider: $bestDivider\n\n");
push(@interesting, $ARGV[$arg]);
}
else
{
if (abs($bestLeftSlope - $bestRightSlope) <= 1)
{
print("Straight:\t");
}
else
{
print("Plateau:\t");
}
$ARGV[$arg] =~ /([0-9]+)-to-([0-9]+)/;
$sourcePlab = $nodeToPlab{$1};
$destPlab = $nodeToPlab{$2};
if ($plabToCap{$sourcePlab} > 0
&& (! exists($plabToCapExempt{$sourcePlab})
|| ! exists($plabToCapExempt{$destPlab})))
{
print("Capped\t");
}
else
{
print("Free\t");
}
printf("%0.2f %0.2f %0.2f ", $bestError, $leftSlope, $rightSlope);
print($ARGV[$arg]."\n");
push(@normal, $ARGV[$arg]);
}
}
else
{
push(@problem, $ARGV[$arg]);
}
}
#print("Normal Count: ".scalar(@normal)."\n");
#print("Normal: ".join(" ",@normal)."\n\n");
#print("Interesting Count: ".scalar(@interesting)."\n");
#print("Interesting: ".join(" ",@interesting)."\n\n");
#print("Problem Count: ".scalar(@problem)."\n");
#print("Problem: ".join(" ",@problem)."\n");
# planetlab.ns - NS script to allocate PlanetLab nodes on Emulab/Netbed
#
# September 17, 2003
#
# Questions and comments to testbed-ops@flux.utah.edu
#
# Boilerplate
#
source tb_compat.tcl
set ns [new Simulator]
tb-set-colocate-factor 1
#
# Estimated resource use on each node by this experiment, used to determine
# if there are enough available resources on PlanetLab for this experiment.
# The scale is from 1 to 5, with 1 being negligible usage on a node,
# and 5 an experiment that, ideally, should have nodes to itself.
# If omitted, defaults to 3.
#
tb-set-cpu-usage 3
tb-set-mem-usage 3
#
# How many nodes to ask for
#
set num_nodes 3
#
# Possibly use a fixed set of nodes (if set, overrides num_nodes setting!)
#
set nodelist {plab736 plab111 plab14}
if { [llength $nodelist] > 0 } {
set num_nodes [llength $nodelist]
}
#
# Type of PlanetLab nodes to request. Current choices, with counts:
# pcplab Any PlanetLab node (127)
# The following are mutually exclusive sets:
# pcplabdsl Plab nodes on DSL lines (3)
# pcplabinet Plab nodes on the commodity Internet, in North America (12)
# pcplabintl Plab nodes outside North America (24)
# pcplabinet2 Plab end-hosts (not colo sites) on Internet2 (88)
#
# Can mix and match these types all you want with, say, multiple loops below.
#
# Instead of by type, you could also request specific nodes; for example:
# tb-fix-node $nodeA plab15
# tb-fix-node $nodeB plab33
#
set hwtype "pcplab"
set node_version "Production"
set basePath "/bw-bottleneck"
set port "1690"
#
# Select the N approximately least-loaded nodes of the given type. Fails if
# insufficient nodes are found due to excessive loads or inadequate disk space.
#
for {set i 1} {$i <= $num_nodes} {incr i} {
set node($i) [$ns node]
tb-set-hardware $node($i) $hwtype
if { [llength $nodelist] > 0 } {
tb-fix-node $node($i) [lindex $nodelist [expr $i - 1]]
}
# Allow experiment setup to succeed even if setup of some vnodes fails.
# Your choice, but currently recommended due to flaky Plab nodes.
tb-set-node-failure-action $node($i) "nonfatal"
# Select nodes with a specific software status.
# If you have selected the default, "Production", no desire will be set.
if { $node_version != "Production" } {
$node($i) add-desire "plabstatus-$node_version" 1.0
}
for {set j 1} {$j <= $num_nodes} {incr j} {
set "node-$i-to-node-$j" [$node($i) program-agent -command "sh $basePath/run-client.sh node-$j.bw-bottleneck.tbres.emulab.net $port $basePath"]
}
set "node-$i-to-server" [$node($i) program-agent -command "sh $basePath/run-server.sh $port $basePath"]
# Entirely optional stuff; see comments below.
#tb-set-node-tarfiles $node($i) /somedir /proj/yourproj/tarfiles/yourtarball.tar.gz
#tb-set-node-rpms $node($i) /proj/yourproj/rpms/yourrpm.rpm
#tb-set-node-startup $node($i) /somepath/yourstartupcmd
}
# The above loop includes three optional features:
# 1) install tarballs, 2) install rpms, 3) Command execution at boot time.
#
# You can specify tarfiles and/or RPMs to install on the vnodes.
# These files must exist in your /proj directory on ops.emulab.net.
# 1) "tarfiles" syntax is an alternating space-separated list of the
# dir from which that untar should start, and the path to the tarball.
# 2) "rpms" syntax is simply a space-separated list of paths to RPMs.
# 3) The "startup" command will be executed every time the vnode boots:
# at experiment startup, swapin, and vnode reboot. Of course, it needs
# to exist on the vnode, probably from a tarball or RPM you installed.
#
# Boilerplate
#
$ns run
\ No newline at end of file
%seen = ();
while ($line = <STDIN>)
{
if ($line =~ /[0-9a-z.]+ > [0-9a-z.]+: [A-Z.]+ ([0-9]+):[0-9]+\([0-9]+\) win ([0-9]+) .*/)
{
$sequence = $1;
$winSize = $2;
if ($winSize == 0)
{
print "Receive Window Size is 0\n";
}
if (exists($seen{$sequence}))
{
print "Retransmit detected\n";
}
$seen{$sequence} = 1;
}
}
#!/usr/bin/perl
$dumpPattern = '^([0-9.]*) IP [0-9.]+ > ([0-9.]+): [A-Za-z. ]+([0-9]+)';
while ($current = <STDIN>)
{
$current =~ /$dumpPattern/;
print "$1 $2 $3\n";
}
#!/usr/bin/perl
$path = $ARGV[0];
$count = $ARGV[1];
sub getBw
{
my $source = shift(@_);
my $dest = shift(@_);
my $result = 1000;
my $file = "$path/end-$source/local/logs/$source-to-$dest-run-1.graph";
open BW, "<".$file;
my $bw = <BW>;
chomp($bw);
if ($bw =~ /1 ([0-9.]+) /)
{
$result = $1*1000;
}
return $result;
}
sub getDelay
{
my $source = shift(@_);
my $dest = shift(@_);
my $result = 10000000;
open DELAY, "<$path/end-$source/local/logs/$source-$dest.ping";
my $delay;
while ($delay = <DELAY>)
{
if ($delay =~ /time=([0-9.]+) ms/)
{
if ($result > $1)
{
$result = $1;
}
}
}
return $result;
}
my $i = 1;
my $j = 1;
for ($i = 1; $i <= $count; ++$i)
{
for ($j = 1; $j <= $count; ++$j)
{
if ($i != $j)
{
my $bw = getBw($i, $j);
my $delay = getDelay($i, $j);
print "10.0.0.$i 10.0.0.$j $bw $delay 0\n";
}
}
}
perl /proj/tbres/duerig/testbed/pelab/bw-bottleneck/make-multiplex.pl /proj/tbres/duerig/truth 1 2,3,4,5,6 > /proj/tbres/duerig/truth/manual/multiplex-1.txt
perl /proj/tbres/duerig/testbed/pelab/bw-bottleneck/make-multiplex.pl /proj/tbres/duerig/truth 2 1,3,4,5,6 > /proj/tbres/duerig/truth/manual/multiplex-2.txt
perl /proj/tbres/duerig/testbed/pelab/bw-bottleneck/make-multiplex.pl /proj/tbres/duerig/truth 3 1,2,4,5,6 > /proj/tbres/duerig/truth/manual/multiplex-3.txt
perl /proj/tbres/duerig/testbed/pelab/bw-bottleneck/make-multiplex.pl /proj/tbres/duerig/truth 4 1,2,3,5,6 > /proj/tbres/duerig/truth/manual/multiplex-4.txt
perl /proj/tbres/duerig/testbed/pelab/bw-bottleneck/make-multiplex.pl /proj/tbres/duerig/truth 5 1,2,3,4,6 > /proj/tbres/duerig/truth/manual/multiplex-5.txt
perl /proj/tbres/duerig/testbed/pelab/bw-bottleneck/make-multiplex.pl /proj/tbres/duerig/truth 6 1,2,3,4,5 > /proj/tbres/duerig/truth/manual/multiplex-6.txt
#~!/usr/bin/perl
$path = $ARGV[0];
$source = $ARGV[1];
$clusterString = $ARGV[2];
sub printRow
{
my $dest = shift(@_);
print "10.0.0.$dest ";
my $filename = "$path/end-$source/local/logs/$source-to-$dest-run-1.graph";
open FILE, "<$filename";
my $line;
while ($line = <FILE>)
{
if ($line =~ /^[0-9]+ ([0-9.]+) $/)
{
my $bw = $1 * 1000;
print $bw." ";
}
}
print "\n";
}
my @clusters = split(/:/, $clusterString);
my $i = 0;
for ($i = 0; $i < scalar(@clusters); ++$i)
{
my @nodes = split(/,/, $clusters[$i]);
my $j = 0;
for ($j = 0; $j < scalar(@nodes); ++$j)
{
printRow($nodes[$j]);
}
print "%%\n";
}
perl /proj/tbres/duerig/testbed/pelab/bw-bottleneck/make-multiplex.pl /proj/tbres/duerig/truth 1 2,3,4:5,6 > /proj/tbres/duerig/truth/multiplex-1.txt
perl /proj/tbres/duerig/testbed/pelab/bw-bottleneck/make-multiplex.pl /proj/tbres/duerig/truth 2 1,4:3,5:6 > /proj/tbres/duerig/truth/multiplex-2.txt
perl /proj/tbres/duerig/testbed/pelab/bw-bottleneck/make-multiplex.pl /proj/tbres/duerig/truth 3 1,4:2:5:6 > /proj/tbres/duerig/truth/multiplex-3.txt
perl /proj/tbres/duerig/testbed/pelab/bw-bottleneck/make-multiplex.pl /proj/tbres/duerig/truth 4 1:2,3,5,6 > /proj/tbres/duerig/truth/multiplex-4.txt
perl /proj/tbres/duerig/testbed/pelab/bw-bottleneck/make-multiplex.pl /proj/tbres/duerig/truth 5 1,4:2:3:6 > /proj/tbres/duerig/truth/multiplex-5.txt
perl /proj/tbres/duerig/testbed/pelab/bw-bottleneck/make-multiplex.pl /proj/tbres/duerig/truth 6 1,4:2:3:5 > /proj/tbres/duerig/truth/multiplex-6.txt
......@@ -14,9 +14,10 @@ import select
import re
import traceback
import errno
import math
from optparse import OptionParser