The Emulab event system provides a means for automating your experiments. The event system consists of several types of "agents" that implement some sort of functionality, such as running programs or generating traffic, and a scheduler that triggers the events at the appropriate time. When your experiment is swapped in, any agents specified in your NS file are automatically setup on the experimental nodes and the ops node. A short time after the experiment becomes active, "event time" begins to flow. As event time progresses, any events scheduled in the NS file for a particular time offset are sent to the appropriate agents. Alternatively, events can be sent at runtime using the tevc command from ops or an experimental node. For a detailed walkthrough of using the event system, see the advanced example.
Recently, we have added some experimental extensions to make the event system even more capable. Note that many of these features are subject to change and are only available when using the latest versions of the FBSD410-STD and RHL90-STD disk images. The NS file below gives an example of using these extensions to automate the process of creating disk images. First, it downloads a network traffic analyzer, iftop, then proceeds to build and install the software. Next, the source directory is removed and a snapshot is taken of the node's disk. Finally, after the snapshot completes and the node has finished rebooting, the experiment is swapped out.
set opt(VERSION) 0.16
set ns [new Simulator]
source tb_compat.tcl
set node [$ns node]
tb-set-node-tarfiles $node \
/tmp http://www.ex-parrot.com/~pdw/iftop/download/iftop-$opt(VERSION).tar.gz
set builder [$node program-agent -dir "/tmp/iftop-$opt(VERSION)"]
set cleaner [$node program-agent]
set build [$ns event-sequence {
$builder run -command "./configure"
$builder run -command "gmake"
$builder run -command "sudo gmake install"
}]
set clean [$ns event-sequence {
$cleaner run -command "sudo rm -rf /tmp/iftop-$opt(VERSION)"
}]
set doit [$ns event-sequence {
$build run
$clean run
$node snapshot-to RHL90-CUSTOMIZED
$ns swapout
}]
$ns at 0.0 "$doit start"
$ns run
We also have a small package containing a more complicated experiment that runs BitTorrent on a bunch of nodes, collects their output, and generates a simple report on how they performed.
BitTorrent experiment package
The rest of this document is intended as a reference manual for the available set of agents and the events they can handle.
new Simulator
The simulator agent provides control over your Emulab experiment as a whole.
The simulator agent listens for the following events:
set ns [new Simulator]
...
set doit [$ns event-sequence {
$ns message "Testing one way, then the other..."
$thisway run
$thatway run
$ns report
}]
$ns event-sequence [body]
append
method.
An event sequence agent is an ordered list of events, each of which is sent when the previous event in the list has reported its completion. For example, in a sequence consisting of a pair of events that run programs, the first event will be sent immediately and the second will be sent when the run of the first program completes. While running two programs in a row may be trivial using conventional means, this capability works across machines and can interact with other operations like reloading disks and rebooting machines.
The semantics of when an event "completes" depend on the type of agent and event. Many events complete instantaneously, such as those used to set a property, so the next event in the sequence is sent immediately. Other events that take a variable amount of time to complete, such as running a program. Some agents provide two types of events to support non-blocking and blocking operation, usually called start and run. Whereas the start event completes instantly, the run blocks the sequence until the agent is finished.
Event sequences listen for the following events:
set doit [$ns event-sequence {
$prog0 run -command "setup.sh"
$node0 reboot
$prog0 run -command "test.sh"
}]
set doit [$ns event-sequence {
$serverprog start; # Start the server,
$clientprogs run; # run the clients to completion, then
$serverprog stop; # stop the server.
}]
set testseq [$ns event-sequence]
foreach test $tests {
$testseq append "$prog0 run -command \"$test\""
}
$ns event-timeline
An event timeline agent sends other events at a relative offset to the overall start time of the timeline. In other words, a timeline is a first class version of the existing "$ns at" syntax.
Event timelines listen for the following events:
set tl [$ns event-timeline]
$tl at 0s "$prog0 start"
$tl at 15s "$prog0 stop"
set seq [$ns event-sequence {
$tl run
$ns swapout
}]
$node program-agent [-command cmdline]
[-dir dir] [-timeout seconds] [-tag string]
[-expected-exit-code code]
Program agents listen for the following events:
The output from the command is stored in the "/local/logs" directory on the node. Each invocation of the agent is stored in a separate file tagged with a unique id, in addition, the stdout and stderr data are stored separately in ".out" and ".err" files. To make it easier to locate the last invocation of the agent, soft links are created with file names that lack the unique id (e.g. "prog0.out" -> "prog0.out.5"). If a "tag" is specified, a soft link will also be created that refers to the actual file (e.g. "prog0.baseline.out" -> "prog0.out.5").
The command will be executed with the following environment variables set:
Variable | Description |
---|---|
PATH | The default path for binaries is set to the standard path (e.g. /usr/bin, /bin, /usr/sbin, /sbin), the binary directories in /usr/local, and the directory containing Emulab specific binaries. |
EXPDIR | The experiment's directory in NFS space (e.g. /proj/foo/exp/bar). |
LOGDIR | The preferred directory for log files on the local machine. |
USER | The name of the user that swapped in this experiment. |
HOME | The path to the user's home directory. |
GROUP | The name of the unix group for the user that swapped in this experiment. |
PID | The project ID for the experiment this agent is running within. |
EID | The experiment ID for the experiment this agent is running within. |
NODECNET | The fully-qualified name of the node this program agent is running on. This name resolves to the IP address of the control network interface of the node. |
NODECNETIP | The IP address of the control network interface. This address should not be advertised to, or used by, applications within an experiment as it will cause all traffic to flow over the control network rather than the experimental network. |
NODE | The unqualified name of the node this program agent is running on. For nodes with experimental interfaces, this name resolves to the IP address of an experimental interface on the node. For nodes with more than one experimental interface, there is no guarantee which one it will resolve to. For nodes with no experimental interfaces, the name will not resolve. |
NODEIP | The IP address of the experiment network interface that NODE resolves to. For nodes with no experimental interfaces, this variable will not be set. |
set opt(VAR) values | Any entries in the "opt" array of the NS file will automatically be added to the environment. For example, to set a variable named "DURATION" with a value of "100", you would add "set opt(DURATION) 100" to the top of your NS file. See captured parameters. |
The command line is executed with "csh -c." Yes, that is the Berkeley C-shell and not the Bourne shell or bash. Sorry, it is an historical thing. So be aware of differences in redirection and expansion syntax (e.g., ">&" and "{}"). When in doubt, put your command in a script and set the command line to "sh -c myscript.sh".
Quoting is fragile and happens at a couple of levels:
A sick example might look like this:
where the
... -command {echo arg{1,2} "arg3 has spaces" arg4\ has\ \'\ \'\ too}
echo
command would have four arguments:
arg1
arg2
arg3 has spaces
arg4 has ' ' too
To summarize: put your commands in a script.
set prog0 [$node0 program-agent]
set prog1 [$node0 program-agent -command "/usr/bin/env"]
set prog2 [$node0 program-agent -command "inf_loop_bug" -timeout 10]
set prog3 [$node0 program-agent -command "ls" -dir "/foo/bar"]
$ns event-group [list-of-agents]
The event group agent is used to broadcast events to a group of agents of the same type. For example, if you wanted to start a program on a large number of nodes at the same time, you can create a group consisting of those program-agents and send a single start event to the group. An event group can also act as a simple synchronization method when used inside an event-sequence. In this case, the next event in the sequence won't be sent until all of the agents in the group have signalled completion.
set group [$ns event-group]
for {set i 0} {$i < 4} {incr i} {
set nodes($i) [$ns node]
set progs($i) [$nodes(i) program-agent]
$group add $progs($i)
}
set doit [$ns event-sequence {
$group run -command "setup.sh"
$group run -command "client.sh"
}]
set group [$ns event-group [list $rnode $lnode]]
set doit [$ns event-sequence {
$group reboot
$ns log "Reboot finished"
}]
$ns node
In addition to allocating an actual machine, the "$ns node" constructor will
create a node agent so the node can be controlled from the event system.
Node agents listen for the following events:
- reboot - Reboot the node. When used in a sequence, this event will complete when the node has
finished booting and is considered "up".
- snapshot-to imagename - Snapshot the node's disk into the
given disk image.
Before the snapshot is taken, the node's logs will be sync'd back to ops using
the loghole utility and the
"/local/logs" directory will be cleaned out. When used in a sequence, this event will complete when the snapshot has
been taken and the node has finished booting and is considered "up".
- reload [-image imagename] - Reload the node's disk with the
default image or the given image. When used in a sequence, this event will complete when the node has
finished booting and is considered "up".
- setdest x y speed [-orientation degrees] - (mobile nodes only) This event will set the next
physical destination for the node. When used in a sequence, this event will complete when the node has
reached its destination. If another setdest event is sent to a node
before it has reached its current destination, the new destination will
overwrite the old one.
Console Agent
Constructor: $node console
Console agents operate on the serial consoles attached to some Emulab nodes.
Currently, they only support capturing a slice of the output received on the
serial line.
Console agents listen for the following events:
- start - Start recording the serial console output from a node.
- stop id - Stop recording the serial console output from a
node and save it to a file named "agentname-id.log" in the
experiment's log directory.
Traffic Generator
Traffic generation agents output network traffic at a constant bit rate over a
link. Consult the advanced
example for more information and examples of their use.
Traffic generators listen for the following events:
- start - Start sending traffic.
- stop - Stop sending traffic.
- set - Change characteristics of the traffic.