Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Flexlab software release - Beta

This software is currently in beta. It still needs a lot of polishing, and is
somewhat brittle in the face of PlanetLab node failure, which is frequent.
Basic information:

We suggest that you read the paper "The Flexlab Approach to Realistic
Evaluation of Networked Systems", by Ricci, Duerig, et al. in the proc. of NSDI
2007 for an explanation of how Flexlab works.

You might see references to 'pelab' (PlanetLab + Emulab) in this software -
this was the internal name for this project, and not all references to it have
been purged.

20 21 22 23 24 25 26
What is supported:

* Automatic creation of an experiment of any number of nodes

* Running experiments on the real PlanetLab or a 'fake' PlanetLab inside
  of Emulab - or both in the same experiment

27 28 29 30 31 32
* Automatic software installation, and running and instrumenting server

* Automatic tcpdumps of your traffic on the Emulab and PlanetLab side

* Automatic logfile gathering
33 34 35 36 37

* Automatic collection of log files at the end of an experiment

* Easy starting/restarting of stubs and monitors

38 39 40 41
* Asking for specific PlanetLab nodes or sites

* Setting initial conditions for nodes based on flexmon data

42 43 44 45 46
What is not supported:

* Running test clients (ie. 'iperf -c') for you (planned)

47 48
* Instrumenting programs that use preemptive (ie. kernel-level) threads
49 50 51 52

Running an experiment with the auto-pelab scripts:

Note: Depending in how you got the source, it will either be in a directory
called 'pelab' made by untarring the tarball, or in the 'pelab' subdirectory
55 56 57 58 59 60 61 62 63 64 65 66 67 68
of the testbed CVS repository.

1) Build the flexlab software:
    Swap in an experiment using the 'PLAB-DEVBOX' image. Note that this disk
        image corresponds to PlanetLab v4.0, and that binaries built on it
        will not run on v3.3 nodes, due to incompatible system libraries. Note
        that if you already have a Flexlab experiment swapped in, you can use
        the nodes in the Emulab side of it for this purpose.
    Run 'make' in the root directory of the Flexlab source

2) Create a tarball of the version of the Flexlab software (monitor, agent,
   etc.) that you'd like to test:
     From the root of your Flexlab source tree, run:
         sh ./make-tarball /proj/<PID>/my-pelab.tar.gz
69 70
     Of course, don't name it 'my-plab.tar.gz'! You can put this file in
       a subdirectory if you want, but it must be somewhere in
71 72
         /proj/<PID>/, where <PID> is the project ID of the project you're
         going to run the experiment in (for example, 'tbres')

74 75 76 77 78
     Put the path to your tarball in the pelab_tar variable in auto-pelab.ns 

2) Read the user-settable options at the top of auto-pelab.ns and change them
   to your preferences.
3) Use your modified auto-pelab.ns to create an experiment and swap it in
80 81

4) Once it has swapped in, and you're ready to start an experiment, run the
82 83 84
   'start-experiment' script from your source directory on ops. It takes as
   arguments the pid and eid of your experiment.  It will:
     Restart monitors and agents
     Reset ALL link shaping to the defaults from the NS file
     Set initial conditions for all paths from the flexmon database
     Clean logfiles
     Restart the user-specified servers on all 'elab-*' nodes

Mike Hibler's avatar
Mike Hibler committed
   start-experiment has a few useful options for passing additional
91 92 93
   arguments to the agents and monitors, e.g.
	 start-experiment -s '-d' pid eid
   would turn on debugging in the magents 
Mike Hibler's avatar
Mike Hibler committed
94 95 96
	start-experiment -m '-d' pid eid
   would do the same for the monitors (assuming it has a '-d' option).

97 98 99 100 101 102
   Important: If you are using real planetlab nodes, instead of a 'fake' 
   planetlab inside of Emulab, run start-experiment with '-p'.

   Important: If you want to have the initial conditions for your paths set
   to measured values, rather than 0ms 100Mbps (and you should...), give
   start-experiment the '-i' flag
Mike Hibler's avatar
Mike Hibler committed

104 105 106 107 108
   Important: Watch for errors from start-experiment - errors about 'no such
   file or directory' when cleaning out logfiles are harmless, but other
   errors can be important.

4a) If you modify something in the flexlab (e.g., the agent) and need to
Mike Hibler's avatar
Mike Hibler committed
109 110 111 112 113
   deploy it to the nodes, first recreate the tarball as in step 1,
   and then
	install-tarball pid eid
   this triggers a program agent on all the nodes telling it to update
   the tarballs and RPMs.  You will need to run start-experiment again
   after this to restart agents/monitors.
Mike Hibler's avatar
Mike Hibler committed

5) Log into the 'elab-*' node or nodes, and run your experiment (ie. one run
117 118 119 120 121
   of iperf). Any programs that you run should be prefixed with
   '/local/pelab/monitor/'. This shell script sets up environment
   variables so that your program has all of its network calls instrumented.
   All command line arguments are passed through to the program - ie:
   /local/pelab/monitor/ iperf -c elab-2

6) When you're done with your experiment, run 'stop-experiment' on ops. It
124 125 126 127 128 129
    Stop the stubs, monitors, and iperf servers
    Collect logfiles from all of the nodes in the experiment, including
      stub/monitor logs, tcpdumps, delay agent logs, and so on
    Run tcptrace on all tcpdump logs to produce xplot graphs
    Put all of these things in a zip file for you in the experiment's log
130 131 132 133
      directory - you will get a message like 'Created log hole archive: This file can be found in (/proj/PID/exp/EID/logs). A new
      zipfile is kept for each experiment, and old ones are not automatically
134 135

7) If you want to look at the logs, unzip the .zip file somewhere, and see
136 137
   the description of the files it contains below. If you want to look at logs
   as the experiment runs, they are typically in /local/logs/ on the nodes.
138 139 140 141

Important places:

142 143 144 145 146 147 148
* The contents of the flexlab tarball are installed in /local/pelab on each

* Logfiles from the monitors, agents, and the user-specified server program
  can all be found in /local/logs on the node while the experiment runs. You
  can tail the .out or .err files to see the stdout and stderr of these
149 150

* Inside the archive zip file you will find:
151 152
  elab-*/ - Logifles from the emulator hosts
  event-sched.log - Log of all events (including dummynet control events) sent
153 154
      during the experiment
  plab-*/ - Logfiles from the FAKE plabnetlab nodes
155 156 157 158
  planet-*/ - Logifles from the REAL PlanetLab nodes, including tcpdump traces
      of traffic to and from the PlanetLab nodes
  tbsdelay*/ - Logs from the delay nodes, including tcpdump traces from the
      emulab nodes