Emulab has integrated network simulation using NS Emulation (NSE) enabling an experimenter to combine simulation and real hardware emulation. The advantages of such an integrated experiment are:
The primary advantage of using Emulab to perform such experiments is the complete automation that we have built when we integrated NSE. Here is a list of issues that we have addressed since our first version of Emulab NSE integration:
$ns at
statements. This is necessary since NSE is distributed
and events need to be delivered to different instances. A central event scheduler
is responsible for the delivery of such events at the right time. Some of the issues addressed were common with multiplexed virtual nodes. Thus our solutions such as multiple-routing-table and virtual-ethernet-device support are used in both simulation and virtual-node subsystems. This enables large scale integrated experiments that includes PC nodes, virtual nodes and simulated resources all in the same experiment. Note that simulated resources such as nodes are abstractions that do not have nearly the same level of virtualization as real PCs or virtual nodes. There is no equivalent of logging in, running unmodified applications, etc. on a simulated node.
The number of simulated nodes that are multiplexed on each PC depends mainly on how much traffic passes through a node. Since this is difficult to estimate statically, we initially use a simple co-location factor to optimistically map the simulated nodes. The initial mapping if it causes an overload (i.e. one or more NSE instances falls behind real-time), will cause the system to re-map more conservatively. Read the technical details section for the gory details.
An experimenter can also guide initial mapping by specifying the maximum co-location factor. The number of simulation objects that can be supported without falling behind real time depends on the amount of external traffic and the number of internal simulation events that need to be processed. Please read nse scaling and nse accuracy and capacity to get some idea on what values to specify for co-location factors.
The output from the simulation including errors such as the ones that
report inability to keep up with real time are logged into a file
/proj/<project_name>/exp/<experiment_name>/logs/nse-simhost-0...n.log
To create an experiment with simulated resources in it, a user simply has to
enclose a block of NS Tcl code in $ns make-simulated {
}
. You specify connections between simulated and physical nodes as usual.
Multiple make-simulated
blocks are allowed in a single experiment
which results in the concatenation of all such blocks.
Figure 1. A hybrid experiment with simulation and emulation
The following code gives an example of an experiment with a dumbell topology comprised of both real PC nodes and simulated nodes. This is illustrated in Figure 1.
set ns [new Simulator]
# Enable automatic static routing
$ns rtproto Static
set real1 [$ns node]
set real2 [$ns node]
# Emulab folks all like FreeBSD more!
tb-set-node-os $real1 FBSD-STD
tb-set-node-os $real2 FBSD-STD
$ns make-simulated {
# All the code here run in the simulation
set sim1 [$ns node]
set sim2 [$ns node]
set simrouter1 [$ns node]
set simrouter2 [$ns node]
# Bottleneck link inside simulation. Simulated
# and real traffic share this link and interfere
# with each other
$ns duplex-link $simrouter1 $simrouter2 1.544Mb 40ms DropTail
# More duplex links inside the simulation
$ns duplex-link $sim1 $simrouter1 10Mb 2ms DropTail
$ns duplex-link $sim2 $simrouter2 10Mb 2ms DropTail
# TCP agent in simulation on node sim1
set tcp1 [new Agent/TCP]
$ns attach-agent $sim1 $tcp1
# FTP application object in simulation on node sim1
set ftp1 [new Application/FTP]
$ftp1 attach-agent $tcp1
# TCPSink object in simulation on node sim2
set tcpsink1 [new Agent/TCPSink]
$ns attach-agent $sim2 $tcpsink1
# Do a connect to tell the system that
# $tcp1 and $tcpsink1 agents will talk to each other
$ns connect $tcp1 $tcpsink1
# Starting at time 1.0 send 75MB of data
$ns at 1.0 "$ftp0 send 75000000"
# connecting real and simulated nodes.
# It is now okay to specify links between real and simulated
# nodes inside the make-simulated block
$ns duplex-link $real1 $simrouter1 100Mb 1ms DropTail
}
# connecting real and simulated nodes.
$ns duplex-link $real2 $simrouter2 100Mb 1ms DropTail
# A real TCP traffic agent on PC real1
set tcpreal1 [new Agent/TCP]
$ns attach-agent $real1 $tcpreal1
set cbr0 [Application/Traffic/CBR]
$cbr0 attach-agent $tcpreal1
# A real TCP sink traffic agent on PC real2
set tcprealsink1 [new Agent/TCPSink]
$ns attach-agent $real2 $tcprealsink1
# Tell the system that tcpreal1 will talk to
# tcprealsink1
$ns connect $tcpreal1 $tcprealsink1
# Start traffic generator at time 10.0
$ns at 10.0 "$cbr0 start"
# Drastically reduce colocation factor for simulated nodes
# to demonstrate distributed NSE. With this, the 4 simulated
# nodes will be mapped to 2 PCs. Simulator packets from sim1
# to sim2 will be encapsulated and transported over a physical
# link
tb-set-colocate-factor 2
$ns run
The above dumbell topology of 6 nodes will be mapped to 4 PCs
in emulab. Note that this is a very low multiplexing factor
chosen to keep the example simple. Two simulation host PCs
are automatically allocated by the system. The code in the
make-simulated
block will be automatically
re-parsed into two Tcl sub-specifications, of which each is
fed into an instance of NSE running on the simulation host.
Depending on how the mapping happens, there can either be one
or two simulated links that cross PCs. In Figure 1, we have
one such link that cross PCs. Simulator packet flows
that cross such links are automatically compressed, encapsulated
in an IP packet and shipped over the physical link.
nse support is still under further development. Please let us know if you face
problems in this system. Here are some caveats:
The scaling, accuracy and capacity data reported here are from our OSDI publication
An instance of nse simulated 2Mb constant bit rate UDP flows between pairs of nodes on 2Mb links with 50ms latencies. To measure nse's ability to keep pace with real time, and thus with live traffic, a similar link was instantiated inside the same nse simulation, to forward live TCP traffic between two physical Emulab nodes, again at a rate of 2Mb. On an 850MHz PC, we were able to scale the number of simulated flows up to 150 simulated links and 300 simulated nodes, while maintaining the full throughput of the live TCP connection. With additional simulated links, the throughput dropped precipitously. We also measured nse's TCP model on the simulated links: the performance dropped after 80 simulated links due to a higher event rate from the acknowledgment traffic in the return path.
As a capacity test, we generated streams of UDP round-trip traffic between two nodes, with an interposed 850 Mhz PC running nse on a FreeBSD 4.5 1000HZ kernel. A maximum stable packet rate of 4000 packets per second was determined over a range of packet rates and link delays using 64-byte and 1518-byte packets. Since these are round trip measurements, the packet rates are actually twice the numbers reported. With this capacity, we performed experiments to measure the delay, bandwidth and loss rates for representative values. The results are summarized in Tables 1, 2 and 3
Emulab's integration of nse is much less mature than its support for dummynet based emulation. This is reflected in the large percentage error values in bandwidth and loss rates. Integrating nse has already uncovered a number of problems that have since been solved; as we continue to gain experience with nse, we expect the situation to improve.
delay (ms) |
packet size (bytes) |
observed | adjusted | |||
---|---|---|---|---|---|---|
RTT (ms) | stddev | %err | RTT (ms) | %err | ||
0 | 64 | 0.238 | 0.004 | N/A | N/A | N/A |
1518 | 1.544 | 0.025 | N/A | N/A | N/A | |
5 | 64 | 10.251 | 0.295 | 2.51 | 10.013 | 0.13 |
1518 | 11.586 | 0.067 | 15.86 | 10.032 | 0.32 | |
10 | 64 | 20.255 | 0.014 | 1.28 | 20.017 | 0.09 |
1518 | 21.675 | 0.093 | 8.38 | 20.121 | 0.61 | |
50 | 64 | 100.474 | 0.029 | 0.47 | 100.236 | 0.24 |
1518 | 102.394 | 3.440 | 2.39 | 100.840 | 0.84 | |
300 | 64 | 601.690 | 0.546 | 0.28 | 601.452 | 0.24 |
1518 | 602.999 | 0.093 | 0.49 | 601.445 | 0.24 |
TODO