tutorial.html 45.6 KB
Newer Older
Leigh B. Stoller's avatar
Leigh B. Stoller committed
1 2
   Copyright (c) 2000-2003 University of Utah and the Flux Group.
Leigh B. Stoller's avatar
Leigh B. Stoller committed
4 5
   All rights reserved.
<h1>Emulab Tutorial</h1>
8 9

10 11
<li> <a href="#GettingStarted">Getting Started</a>
14 15 16 17 18 19
    <li> <a href="#LoggingIn">Logging into the Web Interface</a>
    <li> <a href="#Designing">Designing a Network Topology</a>
    <li> <a href="#Beginning">Beginning the Experiment</a>
    <li> <a href="#UsingNodes">Using your Nodes</a>
    <li> <a href="#RootAccess">I need <b>root</b> access!</a>
    <li> <a href="#Wedged">My node is wedged!</a>
    <li> <a href="#Scrogged">I've scrogged my disk!</a>
    <li> <a href="#Finished">I've finished my experiment</a>
    <li> <a href="#Halting">Scheduling experiment swapout/termination</a>
    <li> <a href="#Help">Getting Help!</a>
<li> <a href="#Advanced">Advanced Topics</a>
    <li> <a href="#ADVEX">A more advanced example</a>
    <li> <a href="#RPMS">Installing RPMS automatically</a>
    <li> <a href="#TARBALLS">Installing Tar files automatically</a>
    <li> <a href="#Startupcmd">
            Starting your application automatically</a>
    <li> <a href="#ReadyBits">
            How do I know when all my nodes are ready?</a>
34 35
    <li> <a href="#Routing">
            Setting up IP routing between nodes</a>
    <li> <a href="#Simem">
37 38
           Hybrid Experiments with Simulation and Emulation</a>
	   <img src="../new.gif" alt="&lt;NEW&gt;">
40 41
<li> <a href="#BatchMode">Batch Mode Experiments</a>
<li> <a href="#CustomOS">Creating your own disk image</a>
42 43

44 45 46 47 48 49 50 51

<a NAME="GettingStarted"></a>
<h2>Getting Started</h2>

This section of the tutorial describes how to run your first Emulab
53 54 55 56 57
experiment. We cover basic NS syntax and various operational issues
that you will need to know in order conduct experiments to completion.
Later sections of the tutorial will cover more advanced topics such as
loading your own RPMs automatically, running programs automatically,
running batch jobs, creating your own disk images and loading those
58 59 60
images on your nodes. For your convenience, there is also a 
<a href="../buildui/bui.php3">Java GUI</a>
interface you can use to create NS files for your experiments.
61 62 63 64 65 66 67 68 69

<li> <a href="#LoggingIn">Logging into the Web Interface</a>
<li> <a href="#Designing">Designing a Network Topology</a>
<li> <a href="#Beginning">Beginning the Experiment</a>
<li> <a href="#UsingNodes">Using your Nodes</a>
<li> <a href="#RootAccess">I need <b>root</b> access!</a>
<li> <a href="#Wedged">My node is wedged!</a>
<li> <a href="#Scrogged">I've scrogged my disk!</a>
71 72 73 74 75
<li> <a href="#Finished">I've finished my experiment</a>
<li> <a href="#Help">Getting Help!</a>

76 77
<a NAME="LoggingIn"></a>
<li> <h3>Logging Into the Web Interface</h3>
78 79

If you already have an account on Emulab or Netbed, all you need to do is
81 82 83
go to <a href="https://www.emulab.net"> Emulab Home
Page</a>, enter your login name and your password, and then click on
the "Login" button. If you don't have an account, click on the "Join
Project" or "Start Project" links. For an overview of how you go about
85 86 87 88 89 90 91
getting an Emulab account, go to the <b><a href =
"http://www.emulab.net/auth.html"> "How To Get Started"</a></b> page.

<li> <a NAME="Designing"></a>
     <h3>Designing a Network Topology</h3>
Part of Emulab's power lies in its ability to assume many different
topologies; the description of a such a topology is a necessary part
94 95 96
of an experiment. (Note: You may want to take a look at our 
<a href="../buildui/bui.php3">Java GUI</a> to build experiments 
without directly editing NS files.)
97 98 99 100 101

Emulab uses the "NS" ("Network Simulator") format to describe network
topologies.  This is substantially the same <a
href="http://www.scriptics.com/software/tcltk/">Tcl</a>-based format
102 103
used by <a href="http://www.isi.edu/nsnam/ns/">ns-2</a>.  Since 
Emulab offers emulation, rather than simulation, these files are
104 105
interpreted in a somewhat different manner than ns-2.  Therefore, some
ns-2 functionality may work differently than you expect, or may not be
106 107 108 109 110 111 112
implemented at all. Please look for warnings of the form:
	*** WARNING: Unsupported NS Statement!
	    Link type BAZ, using DropTail!</code></pre>

If you feel there is useful functionality missing, please let us know.
Also, some <a href="docwrapper.php3?docname=nscommands.html">
113 114 115
testbed-specific syntax</a> has been added, which with the inclusion
of compatibility module (<a href="tb_compat.tcl">tb_compat.tcl</a>),
will be ignored by the NS simulator. This
allows the same NS file to work on both Emulab and ns-2, most of
117 118 119
the time.

120 121 122 123 124
For those unfamiliar with the NS format, here is a small example
(<em>We urge all new Emulab users to begin with a small 3-5 node experiment
such as this, so that you will become familiar with NS syntax and the
practical aspects of Emulab operation</em>). Let's say we are trying to
create a test network which looks like the following:
125 126
    <img src="abcd.png"><br><br>
    (A is connected to B, and B to C and D with a LAN.)
129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151

An NS file which would describe such a topology is as follows. First
off, all NS files start with a simple prolog, declaring a simulator 
and including a file that allow you to use the special 
<code>tb-</code> commands:
	# This is a simple ns script. Comments start with #.
	set ns [new Simulator]			
        source tb_compat.tcl                   </code></pre>

Then define the 4 nodes in the topology.
	set NodeA [$ns node]
	set NodeB [$ns node]
	set NodeC [$ns node]
	set NodeD [$ns node]			</code></pre>

Mac Newbold's avatar
Mac Newbold committed
152 153 154 155 156 157
Next define the link and the LAN that connect the nodes. NS syntax
permits you to specify the bandwidth, latency, and queue type. Note
that since NS can't impose artificial losses like Emulab can, there is
a separate command to add loss on a link.  For our example, we will
define a full speed LAN between B, C, and D, and a shaped link from
node A to B.
159 160 161
	set link0 [$ns duplex-link $nodeB $nodeA 30Mb 50ms DropTail]
	tb-set-link-loss $link0 0.01
	set lan0 [$ns make-lan "$nodeD $nodeC $nodeB " 100Mb 0ms]</code></pre>
162 163 164

In addition to the standard NS syntax above, a number of
<a href="docwrapper.php3?docname=nscommands.html">
166 167 168
extensions</a> have been added that allow you
to better control your experiment. For example, you may specify what
Operating System is booted on your nodes. We currently support FreeBSD
4.7 and Linux RedHat 7.3, as well as
<a href="http://www.cs.utah.edu/flux/oskit/">OSKit</a> kernels on the
Emulab PCs. By default, our most recent Linux image is selected.
172 173

	tb-set-node-os $NodeA FBSD-STD
	tb-set-node-os $NodeC RHL-STD</code></pre>
176 177

178 179 180 181 182 183 184 185
In a topology like this, you will likely want to communicate between
all the nodes, including nodes that aren't directly connected,
like A and C. In order for that to happen, we must enable
routing in our experiment, so B can route packets for the other
nodes. The typical way to do this is with Static routing. (Other
options are detailed below, in the 
<a href="http://www.emulab.net/~newbold/www/tutorial/docwrapper.php3?docname=tutorial.html#Routing"
>Routing section</a>.)
186 187

	$ns rtproto Static</code></pre>
189 190

Jay Lepreau's avatar
Jay Lepreau committed
Lastly, all NS files end with an epilogue that instructs the simulator
192 193 194 195 196 197 198 199 200
to start.

	# Go!
	$ns run					</code></pre>

If you would like to try the above example, the completed <a
href="basic.ns" target=stuff>NS file</a> can be run as an experiment in your

203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230
Because NS is based on TCL, the full power of the TCL language is
available for you to use in your NS files, including loops,
control structures, and even procedures/functions. Here's an
example of a simple loop: 
(Download this example: <a href="loop.ns" target=stuff>loop.ns</a>)

	# This is a simple ns script that demonstrates loops.
	set ns [new Simulator]                  
	source tb_compat.tcl

	set maxnodes 3
	set lanstr ""

	for {set i 1} {$i <= $maxnodes} {incr i} {
	    set node($i) [$ns node]
	    set name node-${i}
	    append lanstr "$name "
	    tb-set-node-os $node($i) FBSD-STD

	# Put all the nodes in a lan
	set big-lan [$ns make-lan "$lanstr" 100Mb 0ms]

	# Go!
	$ns run</pre></code>                            
231 232 233 234 235
<li> <a NAME="Beginning"></a>
     <h3>Beginning the Experiment</h3>

After logging on to the Emulab Web Interface, choose the "Begin
237 238 239 240 241 242 243
Experiment" option from the menu. First select which project you want
the experiment to be configured in. Most people will be a member of just
one project, and will not have a choice. If you are a member of
multiple projects, be sure to select the correct project from the

244 245
Next fill in the `Name' and `Description fields. The Name should be a
single word (no spaces) identifier, while the Description is a multi
246 247 248 249 250 251
word description of your experiment. In the "Your NS file" field,
place the <i>local path</i> of a NS file which you have created to
describe your network topology. This file will be uploaded through
your browser when you choose "Submit."

After submission, Emulab will begin processing your
253 254 255 256 257 258 259 260 261 262 263 264
request. This will likely take several minutes, depending on how large
your topology is, and what other features (such as delay nodes and
bandwidth limits) you are using. Assuming all goes well, you will
receive an email message indicating success or failure, and if
successful, a listing of the nodes and IP address that were allocated
to your experiment. 

For the NS file described above, you would receive a listing that looks
similar to this:

265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309
	Virtual Node Info:
	ID              Type         OS              Qualified Name
	--------------- ------------ --------------- --------------------
	nodeA           pc			     nodeA.myexp.myproj.emulab.net
	nodeB           pc                           nodeB.myexp.myproj.emulab.net
	nodeC           pc                           nodeC.myexp.myproj.emulab.net
	nodeD           pc                           nodeD.myexp.myproj.emulab.net
	Physical Node Mapping:
	ID              Type         OS              Physical    
	--------------- ------------ --------------- ------------
	tbsdelay0       pc850        FBSD47-STD      pc61
	nodeB           pc850        RHL73-STD       pc63
	nodeC           pc600        RHL73-STD       pc31
	nodeD           pc600        RHL73-STD       pc34
	nodeA           pc600        RHL73-STD       pc13
	Virtual Lan/Link Info:
	ID              Member          IP              Delay     BW (Kbs)  Loss Rate
	--------------- --------------- --------------- --------- --------- ---------
	lan0            nodeC:0     0.00      100000    0.000    
	                                                0.00      100000    0.000    
	lan0            nodeB:1     0.00      100000    0.000    
	                                                0.00      100000    0.000    
	lan0            nodeD:0     0.00      100000    0.000    
	                                                0.00      100000    0.000    
	link0           nodeB:0     25.00     30000     0.005    
	                                                25.00     30000     0.005    
	link0           nodeA:0     25.00     30000     0.005    
	                                                25.00     30000     0.005    
	Physical Lan/Link Info:
	ID              Member          Delay Node   Delay    BW (Kbs) PLR    Pipe     
	--------------- --------------- ------------ -------- -------- ------ ---------
	link0           nodeA           tbsdelay0    50.00    30000    0.010  100      
	link0           nodeB           tbsdelay0    50.00    30000    0.010  110      
	Route List:
	Node            Interface       Dest            Nexthop         Type  Cost 
	--------------- --------------- --------------- --------------- ----- ----
	nodeA      host  0    
	nodeA      net   0    
	nodeC      net   0    
	nodeD      net   0    
310 311 312 313 314 315 316

A few points should be noted:
<li> A single delay node was allocated and inserted into the link
     between NodeA and NodeB. This link is invisible from your
     perspective, except for the fact that it adds latency, error,
317 318 319 320
     or reduced bandwidth. However, the information for the delay links
     are included so that you can
     <a href="../faq.php3?#HDS-6">modify the delay parameters</a>
     after the experiment has been created. 

<li> Delays of less than 2ms (per trip) are too small to be
     accurately modeled at this time, and will be silently ignored.
324 325 326
     A delay of 0ms can be used to indicate that you do not want
     added delay; the two interfaces will be "directly" connected to
     each other. 
327 328 329 330 331 332 333 334

<li> Each link in the "Virtual Lan/Link" section has its delay, etc.,
     split between two entries. One is for traffic coming into the
     link from the node, and the other is for traffic leaving the
     link to the node. In the case of links, the four entries often
     get optimized to two entries in the "Physical Lan/Link"

335 336 337 338
<li> The names in the "Qualified Name" column refer to the control
     network interfaces for each of your allocated nodes. These names
     are added to the Emulab nameserver map on the fly, and are
     immediately available for you to use so that you do not have to
     worry about the actual physical node names that were chosen. In
340 341 342
     the names listed above, `myproj' is the name of the project that
     you chose to work in, and `myexp' is the name of the experiment
     that you provided in the "Begin Experiment" page.
343 344 345 346 347 348 349 350 351 352

<li> Please don't use the "Qualified Name" from within nodes in your
     experiment, since it will contact them over the control network,
     bypassing the link shaping we configured.
     (See also these two FAQ entries: 
<a href="http://www.emulab.net/docwrapper.php3?docname=faq.html#TR-2">
here</a> and
<a href="http://www.emulab.net/docwrapper.php3?docname=faq.html#Naming">

353 354 355 356 357 358 359 360 361

<li> <a NAME="UsingNodes"></a>
     <h3>Using your Nodes</h3>

By the time you receive the email message listing your nodes, the
Emulab configuration system will have ensured that your nodes are
fully configured and ready to use. If you have selected one of the
Emulab-supported operating system images (FreeBSD or Linux),
365 366
this configuration process includes:

368 369 370 371 372 373 374 375 376 377 378 379 380
<li> loading fresh disk images so that each node is in a known clean
<li> rebooting each node so that it is running the OS specified in the
     NS script;
<li> configuring each of the network interfaces so that each one is
     "up" and talking to its virtual LAN (VLAN);
<li> creating user accounts for each of the project members;
<li> mounting the projects NFS directory in /proj so that project
     files are easily shared amongst all the nodes in the experiment;
<li> creating a /etc/hosts file on each node so that you may refer to the
     experimental interfaces of other nodes by name instead of IP number;
<li> configuring all of the delay parameters;
381 382
<li> configuring the serial console lines so that project members may access the
     console ports from users.emulab.net or directly from their desktop.
383 384 385 386 387 388 389 390 391 392 393 394 395

As this point you may log into any of the nodes in your experiment.
You will need to use Secure Shell (ssh), and you should use the
`qualified name' from the nodes mapping table so that you do not form
dependencies on any particular physical node. Your login name and
password will be the same as your Web Interface login and password.

The /etc/hosts file on each node will provide a local name mapping for
the other nodes in your experiments. You should take care to use these
names (or IP numbers) and <b>not</b> the .emulab.net names listed in
the node mapping, since the Emulab names refer to the control network
397 398
LAN that is shared amongst all nodes in all experiments. It is only
the experimental interfaces that are entirely private to your
399 400 401 402 403
experiment. (See also these two FAQ entries: 
<a href="http://www.emulab.net/docwrapper.php3?docname=faq.html#TR-2">
here</a> and
<a href="http://www.emulab.net/docwrapper.php3?docname=faq.html#Naming">
404 405 406 407 408 409 410

<b> NOTE:</b> The configuration process just described occurs only on
Emulab constructed operating system images. If you are using an OSKit
kernel, or your own disk image with your own operating system, you
will be responsible for all of the configuration. At some point we
hope to provide tools to assist in the configuration, but for now you
411 412
are on your own. Ask <a href="mailto:testbed-ops@flux.cs.utah.edu">
Testbed Ops</a> for help or more information.
413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438

<li> <a NAME="RootAccess"></a>
     <h3>I need <b>root</b> access!</h3>

If you need to customize the configuration, or perhaps reboot nodes,
you can use the "sudo" command, located in <code>/usr/local/bin</code>
on FreeBSD and Linux, and <code>/usr/pkg/bin</code> on NetBSD. Our
policy is very liberal; you can customize the configuration in any way
you like, provided it does not violate Emulab's
<a href="../docwrapper.php3?docname=policies.html">
administrative policies</a>. As as example, to reboot a node that is
running FreeBSD:

	/usr/local/bin/sudo reboot			</code></pre>

<li> <a NAME="Wedged"></a>
     <h3>My node is wedged!</h3>

This is bound to happen when running experimental software and/or
experimental operating systems. Fortunately we have an easy way for
you to power cycle nodes without requiring Tested Operations to get
440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456
involved. If you must power cycle a node, log on to users.emulab.net
and use the "node_reboot" command:

	node_reboot <node> [node ... ]			</xmp></code></pre>

where `node' is the physical name, as listed in the node mapping
table. You may provide more than one node on the command line. Be
aware that you may power cycle only nodes in projects that you are
member of. Also, <tt>node_reboot</tt> does its very best to perform a
clean reboot before resorting to cycling the power to the node. This
is to prevent the damage that can occur from constant power cycling
over a long period of time.  For this reason, <tt>node_reboot</tt> may
delay a minute or two if it detects that the machine is still
responsive to network transmission.  In any event, please try to
reboot your nodes first (see above).

457 458 459 460 461 462 463 464 465 466 467
You may also reboot all the nodes in an experiment by using the <tt>-e</tt>
option to specify the project and experiment names. For example:

	node_reboot -e testbed,multicast		</xmp></code></pre>

will reboot all of the nodes reserved in the "multicast" experiment in
the "testbed" project. This option is provided as a shorthand method
for rebooting large groups of nodes.

468 469 470 471 472 473 474 475 476 477 478 479 480 481 482
<li> <a NAME="Scrogged"></a>
     <h3>I've scrogged my disk!</h3>

Scrogging your disk is certainly not as common, but it does happen.
You can either terminate your experiment, and recreate it (which will
allocate another group of nodes), or if you prefer you can reload the
disk image yourself. You will of course lose anything you have stored
on that disk; it is a good idea to store only data that can be easily
recreated, or else store it in your project directory in <tt>/proj</tt>.
Reloading your disk with a fresh copy of the default image is easy,
and requires no intervention by Emulab staff:

	os_load <node> [node ... ]			</xmp></code></pre>

485 486
os_load will wait (not exit) until the nodes have been reloaded,
so that you do not need to check the console lines of each node
487 488
to determine when the load is done.

489 490 491 492 493 494 495 496 497 498 499 500
<li> <a NAME="Finished"></a>
     <h3>I've finished my experiment</h3>

When your experiment is completed, and you no longer need the
resources that have been allocated to it, you will need to terminate
the experiment via the Emulab Web Interface. Click on the "End An
Experiment" link. You will be presented with a list of all of the
experiments in all of the projects for which you have the
authorization to terminate experiments. Select the experiment you want
to terminate by clicking on the button in the "Terminate" column on
the right hand side. You will be asked to <b>confirm</b> your choice.
The Emulab configuration system will then tear down your experiment,
502 503 504 505
and send you an email message when the process is complete. At this
point you are allowed to reuse the experiment name (say, if you wanted
to create a similar experiment with different parameters).

506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529
<li> <a NAME="Halting"></a>
     <h3>Scheduling experiment swapout/termination</h3>

If you expect that your experiment should run for a set period of
time, but you will not be around to terminate or swap the experiment
out, then you should use the scheduled swapout/termination feature.
This allows you to specify a maximum running time in your NS file so
that you will not hold scarce resources when you are offline. This can
also be used in conjunction with <a href="#BatchMode"><i>batch
mode</i></a> experiments to ensure that they do not run longer than is
reasonably expected. To schedule a swapout or termination in your NS
        $ns at 2000.0 "$ns terminate"
  	$ns at 2000.0 "$ns swapout"		</xmp></code></pre>

This will cause your experiment to either be terminated or swapped out
after 2000 seconds of wallclock time. <b> NOTE:</b> For batch mode
experiments, you can use the <em>terminate</em> option, but not the
<em>swapout</em> option.

530 531 532 533 534 535 536
<li> <a NAME="Help"></a>
     <h3>Getting Help!</h3>

If you have any questions or problems, or just want to comment on
Emulab's operation (maybe you want to suggest an improvement to one of
the Web pages), feel free to contact us by sending email to
537 538 539 540 541 542
<a href="../sendemail.php3">Testbed Operations</a>. Please send all
correspondence to this address, not to individual members of our
team. Messages that aren't on this list may not get responses.
Also note that much of the software is in development, and 
occasionally things might break or not work as you expect. 
Again, please feel free to contact us. 

544 545 546
<!-- This ends the Basic Tutorial Section -->

547 548
<a NAME="Advanced"></a>
550 551 552 553
<h2>Advanced Topics</h2>

<li> <a href="#ADVEX">A more advanced example</a>
<li> <a href="#RPMS">Installing RPMS automatically</a>
<li> <a href="#TARBALLS">Installing Tar files automatically</a>
557 558
<li> <a href="#Startupcmd">Starting your application automatically</a>
<li> <a href="#ReadyBits">How do I know when all my nodes are ready?</a>
<li> <a href="#Routing">Setting up IP routing between nodes</a>
560 561
<li> <a href="#Simem">Hybrid Experiments with Simulation and Emulation</a> 
     <img src="../new.gif" alt="&lt;NEW&gt;">
562 563 564 565

566 567 568 569 570 571 572

<li> <a NAME="ADVEX"></a>
     <h3>A more advanced example</h3>

We have a more <a href="docwrapper.php3?docname=advanced.html">
advanced example</a> demonstrating the use of RED queues, traffic
573 574
generators, the event system, program objects,  and the 
integration of network simulation (NS/NSE).
575 576


578 579 580 581
<li> <a NAME="RPMS"></a>
     <h3>Installing RPMS automatically</h3>

The Emulab NS extension <tt>tb-set-node-rpms</tt> allows you to
583 584 585 586 587 588 589 590 591
specify a (space separated) list of RPMs to install on each of your
nodes when it boots:

tb-set-node-rpms $nodeA /proj/pid/rpms/silly-freebsd.rpm
tb-set-node-rpms $nodeB /proj/pid/rpms/silly-linux.rpm	</code></pre>

The above NS code says to install the <tt>silly-freebsd.rpm</tt> file
on <tt>nodeA</tt>, and the <tt>silly-linux.rpm</tt> on <tt>nodeB</tt>.
592 593 594 595
RPMs are installed as root, must reside in either the project's
<tt>/proj</tt> directory, or if the experiment has been created in a
subgroup, in the <tt>/group</tt> directory. You may not place your
rpms in your home directory.
596 597 598


599 600 601 602
<li> <a NAME="TARBALLS"></a>
     <h3>Installing TAR files automatically</h3>

The Emulab NS extension <tt>tb-set-node-tarfiles</tt> allows you to
604 605 606 607 608 609 610 611 612 613 614 615 616 617
specify a set of tarfiles to install on each of your nodes when it
boots. This command is similar to the <a href="#RPMS">tb-set-node-rpms</a>
above, and is provided for those people who do not want to spend a month
trying to figure out how to build an RPM! The format of this command
is slightly different though in that you must specify a directory in
which to unpack the tar file. This avoids problems with having to
specify absolute pathnames in your tarfile, which many modern tar
programs balk at. 

tb-set-node-tarfiles $nodeA /usr/site /proj/pid/tarfiles/silly.tar.gz </code></pre>

The above NS code says to install the <tt>silly.tar.gz</tt> tar file
on <tt>nodeA</tt> from the working directory <tt>/usr/site</tt> when
618 619 620 621
the node first boots. The tarfile must reside in either the project's
<tt>/proj</tt> directory, or if the experiment has been created in a
subgroup, in the <tt>/group</tt> directory. You may not place your
tarfiles in your home directory. You may specify as many tarfiles as
622 623 624 625 626
you wish, as long as each one is proceeded by the directory it should
be unpacked in, all separated by spaces.


627 628 629 630 631
<li> <a NAME="Startupcmd"></a>
     <h3>Starting your application automatically</h3>

You can start your application automatically when your nodes boot by
using the <tt>tb-set-node-startup</tt> NS extension. The argument is
633 634 635 636 637 638 639 640 641 642 643 644
the pathname of a script or program that is run as the <tt>UID</tt> of
the experiment creator, after the node has reached multiuser mode. You
can specify the same program for each node, or a different program.
For example:

tb-set-node-startup $nodeA /proj/pid/runme.nodeA
tb-set-node-startup $nodeB /proj/pid/runme.nodeB	</code></pre>

will run <tt>/proj/pid/runme.nodeA</tt> on nodeA and
<tt>/proj/pid/runme.nodeA</tt> on nodeB. The programs must reside on
the node's local filesystem, or in a directory that can be reached via
645 646 647
NFS. This is either the project's <tt>/proj</tt> directory, in the
<tt>/group</tt> directory if the experiment has been created in a
subgroup, or a project member's home directory in <tt>/users</tt>.

649 650 651 652 653 654
The exit value of the startup command is reported back to the Web
Interface, and is made available to you via the "Experiment
Information" link. There is a listing for all of the nodes in the
experiment, and the exit value is recorded in this listing. The
special symbol <tt>none</tt> indicates that the node is still running
655 656 657
the startup command. A log file containing the output of the startup
command is created in the project's <tt>logs</tt> directory

659 660 661
The startup command is especially useful when
combined with <a href="#BatchMode"><i>batch mode</i></a> experiments.
662 663

664 665 666 667 668 669 670 671 672 673
<li> <a NAME="ReadyBits"></a>
     <h3>How do I know when all my nodes are ready?</h3>

It is often necessary for your startup program to determine when all
of the other nodes in the experiment have started, and are ready to
proceed. Sometimes called a <i>barrier</i>, this allows programs to
wait at a specific point, and then all proceed at once. Emulab
provides a primitive form of this mechanism using experiment <i>ready
bits</i>, which are set and read using the
<a href="../doc/docwrapper.php3?docname=tmcd.html">
675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705
TMCD/TMCC</a>. When an experiment is first configured, the ready bit
for each node is cleared. As each node starts its application and
reaches the point where it must be sure that all other nodes have
started up, it issues a TMCC <tt>ready</tt> command:

tmcc ready				</code></pre>

which tells Emulab's configuration system that the node is ready to
proceed. The node can then poll for the <i>ready count</i> to
determine how many nodes are ready (have issued a tmcc ready command):

tmcc readycount				</code></pre>

which will return the ready count as a string:

READY=N TOTAL=M				</code></pre>

where <tt>N</tt> is the number of nodes that are ready, and <tt>M</tt>
is the total number of nodes in the experiment. An application can
poll the ready count with a simple script, or it can encode the ready
bits check directly into its program. For example, here is a simple
Perl fragment that issues the ready command, and then polls for the
ready count, being sure to delay a small amount between each poll.

system("tmcc ready");
while (1) {
    my $bits = `tmcc readycount`;
    if ($bits =~ /READY=(\d*) TOTAL=(\d*)/) {
707 708 709 710 711 712 713
        if ($1 == $2) {
    # Please sleep to avoid swamping the TMCD!
715 716 717 718 719 720 721 722
}					</code></pre>

<i>Note that the ready count is essentially a use-once feature; The
ready count cannot be reinitialized to zero since there is no actual
synchronization happening.  If in the future it appears that a
generalized barrier synchronization would be more useful, we will
investigate the implementation of such a feature.</i>


725 726 727 728 729 730 731 732 733 734 735 736 737
<li> <a NAME="Routing"></a>
     <h3>Setting up IP routing between nodes</h3>

As Emulab strives to make all aspects of the network controllable by the
user, we do not attempt to impose any IP routing architecture or protocol
by default.
However, many users are more interested in end-to-end aspects and don't
want to be bothered with setting up routes.  For those users we provide
an option to automatically set up routes on nodes which run one of our
provided FreeBSD or Linux disk images.

You can use the NS <tt>rtproto</tt> syntax in your NS file to enable
$ns rtproto <i>protocol</i>
743 744 745 746 747 748 749 750 751 752 753 754
where the <i>protocol</i> option is limited to one of
<code>Session</code>, <code>Static</code>, or <code>Manual</code>.
<code>Session</code> routing provides fully automated routing support, 
and is implemented by enabling <code>gated</code> running the OSPF protocol
on all nodes in the experiment. <code>Static</code> routing also
provides automatic routing support, but rather than computing the
routes dynamically, the routes are precomputed when the experiment is
created, and then loaded on each node when it boots.

<code>Manual</code> routing allows you to explicitly specify per-node
routing information in the NS file. To do this, use the
Chad Barb's avatar
Chad Barb committed
<tt>Manual</tt> routing option to <tt>rtproto</tt>, 
followed by a list of routes using the <tt>add-route</tt> command:
$node add-route $dst $nexthop
760 761
where the <tt>dst</tt> can be either a node, a link, or a LAN. For
763 764 765
$client add-route $server $router
$client add-route [$ns link $server $router] $router
$client add-route $serverlan $router
767 768 769 770 771 772 773 774
Note that you would need a separate <code>add-route</code> command to
establish a route for the reverse direction; thus allowing you to
specify differing forward and reverse routes if so desired.
These statements are converted into appropriate <tt>route(8)</tt>
commands on your experimental nodes when they boot.

In the above
775 776 777 778 779 780 781 782 783 784 785 786
examples, the first form says to set up a manual route between
<tt>$client</tt> and <tt>$server</tt>, using <tt>$router</tt> as the
nexthop; <tt>$client</tt> and <tt>$router</tt> should be directly
connected, and the interface on <tt>$server</tt> should be
unambiguous; either directly connected to the router, or an edge node
that has just a single interface.

<img src=routing.png align=right>

If the destination has multiple interfaces configured, and it is not
connected directly to the nexthop, the interface that you are
787 788
intending to route to is ambiguous.
In the topology shown to the right,
789 790
<tt>$nodeD</tt> has two interfaces configured. If you attempted to 
set up a route like this:
$nodeA add-route $nodeD $nodeB
you would receive an error since it cannot be determined (easily, with
little programmer effort, by Emulab staff!) which of the two links on
<tt>$nodeD</tt> you are referring to. Fortunately, there is an easy
solution, courtesy of an Emulab extension. Instead of a node, specify the
link directly:
$nodeA add-route [$ns link $nodeD $nodeC] $nodeB
802 803
This tells us exactly which link you mean, enabling us to convert
that information into a proper <tt>route</tt> command on <tt>$nodeA</tt>.
804 805

The last form of <tt>add-route</tt> command is used when adding a
807 808
route to an entire LAN. It would be tedious and error prone to specify
a route to each node in a LAN by hand. Instead, just route to the
entire network:
set clientlan [$ns make-lan "$nodeE $nodeF $nodeG" 100Mb 0ms]
$nodeA add-route $clientlan $nodeB
813 814

815 816 817 818 819 820 821
While all this manual routing infrastructure sounds really nifty, its
probably a good idea to use either <tt>Session</tt> or <tt>Static</tt>
routing for all but small, simple topologies.  Explicitly setting up
all the routes in even a moderately-sized experiment is extremely
error prone.  Consider this: a recently created experiment with 17
nodes and 10 subnets required 140 hand-created routes in the NS
file. Yow!
822 823

Two final, cautionary notes on routing:
826 827 828
<li> You might be tempted to set the default route on your nodes
     to reduce the number of explicit routes used.  <b>Don't do it.</b>
     That would prevent nodes from contacting the outside world, i.e., you.
829 830
     The default route <em>must</em> be set to use the control network
834 835 836 837
<li> If you use your own routing daemon, you must avoid using the
     control network interface in the configuration.  Since every node
     in the testbed is directly connected to the control network LAN,
     a naive routing daemon configuration will discover that any node
838 839
     is just one hop away, via the control network, from any other node
     and <em>all</em> inter-node traffic will be routed via that interface.
841 842 843

<a NAME="Simem"></a>
844 845
<li><h3>Hybrid Experiments with Simulation and Emulation 
<img src="../new.gif" alt="&lt;NEW&gt;"></h3>
846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892

Emulab has integrated network simulation using 
<a href="http://www.isi.edu/nsnam/ns/doc/node487.html">NS Emulation (NSE)</a>
enabling an experimenter to combine simulation and real
hardware emulation. This allows scale beyond the limit of
physical resources as well as interaction of real 
application traffic with simulated traffic. The latter makes it 
possible to do validation of simulation models against the real 
world or use simulation cross traffic when the particular model 
is still experimental and not available in a real implementation. 

To create an experiment with simulated resources in it, a user simply has to
enclose a block of NS Tcl code in <code>$ns make-simulated {
}</code>. You specify connections between simulated and physical nodes as usual,
with the current restriction that they must be lexically outside the
<code>make-simulated</code> block. The following code gives an example:


set ns [new Simulator]

set realnode1 [$ns node]
set realnode2 [$ns node]

$ns make-simulated {

    # All the code here run in the simulation
    set simnode1 [$ns node]
    set simnode2 [$ns node]

    # A duplex link inside the simulation
    $ns duplex-link $simnode1 $simnode2 1.5Mb 40ms DropTail

# connecting real and simulated nodes. outside make-simulated
$ns duplex-link $realnode1 $simnode1 5Mb 10ms DropTail
$ns duplex-link $realnode2 $simnode2 5Mb 10ms DropTail


A hybrid experiment like this causes the simulation to run in
best effort real time. The number of simulation objects that
can be supported without falling behind real time depends on
894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931
the amount of external traffic and the number of internal
simulation events that need to be processed. Please read 
<a href="docwrapper.php3?docname=nse.html">nse scaling, accuracy and capacity</a> 
 to get a better picture.

The output from the simulation including errors such as the ones that
report inability to keep up with real time are logged into a file 

nse support is still under development. Please let us know if you face 
problems in this system. Here are some caveats:

Currently, all simulated nodes in an experiment
are mapped to one physical pc in emulab. This limits the scalability
of this system. Support to automatically map simulated resources
on to multiple physical nodes is coming soon. 

Enabling NS tracing causes huge file I/O overhead resulting in 
nse not keeping up with real time. Therefore, do not enable tracing.

Remember that each link between the simulated nodes and the physical
nodes is a real physical link, so there can't be more of them than 
there are ethernet links on a physical node (currently 4).



934 935 936 937 938 939 940 941 942

<!-- Batch Mode -->

<a NAME="BatchMode"></a>
<h2>Batch Mode</h2>

943 944 945 946 947 948 949 950 951 952
<li> <a href="#BatchIntro">Batch Mode Introduction</a>
<li> <a href="#BatchExample">A Batch Mode Example</a>

<li> <a NAME="BatchIntro"></a>
     <h3>Batch Mode Introduction</h3>

Batch Mode experiments can be created on Emulab via the "Create
954 955 956 957
an Experiment" link in the operations menu to your left. There is a
checkbox near the bottom of the form that indicates you want to use
the batch system. There are several important differences between a
regular experiment and a batch mode experiment:
958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976

<li> The experiment is run when enough resources (ie: nodes) are
     available. This might be immediately, or it might be sometime in
     the future.
<li> Once your NS file is handed off to the system, the batch system
     is responsible for setting up the experiment and tearing it down
     once the experiment has completed. You will receive email
     notifying you when the experiment has been scheduled and when it
     has been terminated.
<li> Your NS file must define a <i>startup</i> command to run on each
     node using the <a href="#Startupcmd"><tt>tb-set-node-startup</tt></a>
     NS extension. It is the exit value(s) of the startup command(s) that
     indicates that the experiment is completed; when all of the
     nodes have run their respective startup commands and exited, the
     batch system will then tear down the experiment. The output of
977 978
     the startup command is stored in the project <tt>logs</tt> directory
     (<tt>/proj/pid/logs</tt>) so you can follow what has happened.
979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011

<li> <a NAME="BatchExample"></a>
     <h3>A Batch Mode Example</h3>

Consider example NS file <a href="batch.ns" target=stuff>batch.ns</a>.
First off, we have to arrange for the experimental software to be
automatically installed when the nodes boot. This is done with the <a
href="#RPMS"><tt>tb-set-node-rpms</tt></a> NS extension:

tb-set-node-rpms $nodeA /proj/testbed/rpms/silly-1.0-1.i386-freebsd.rpm
tb-set-node-rpms $nodeB /proj/testbed/rpms/silly-1.0-1.i386-freebsd.rpm

The next two lines of the NS file specify what program should be run
on each of the nodes. Using the <a href="#Startupcmd">
<tt>tb-set-node-startup</tt></a> NS extension, we say that the program
<tt>run-silly</tt> (installed by the <tt>silly-1.0</tt> RPM) is to be
run on both nodes:

tb-set-node-startup $nodeA /usr/site/bin/run-silly
tb-set-node-startup $nodeB /usr/site/bin/run-silly

After you have been notified via email that the batch experiment is
running, you can track the progress of your experiment by looking in
the "Experiment Information" page. As each node completes the startup
command, the listing for that node will be updated to reflect the exit
status of the command (you may need to hit the Reload button to see
the changes). Once all of the nodes hare reported in an exit status,
1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023
the batch system will tear down the experiment and send you email.  If
your experiment is such that one node is the controller, and runs
commands on all the other nodes, then simply run a dummy startup
command on the other nodes so that the batch system will receive an
exit value for that node. Since the batch is not terminated until
<em>all</em> nodes have reported in, be sure that the controlling node
does not exit from its startup command until all of the nodes have
finished. A dummy startup command can be setup like this:

tb-set-node-startup $nodeC /bin/echo
1024 1025 1026 1027 1028

The status of your batch experiment can be viewed via the "Experiment
Information" link in the Web Interface Options menu. You may also
cancel a batch after you have submitted it using the "Terminate"
1029 1030 1031 1032
option in the information display. As noted in the section on the <a
href="#Startupcmd">Startupcmd</a>, the output of the startup command
on each node is written to separate files in your project log
directory. You can use these log files to debug your batch experiment.
1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047

The batch system is still under development. It appears to be
functional, but there are bound to be kinks in the system. Please help
us debug and improve it by letting us know what you think and if you
have problems with it. Currently, the batch system tries every 10
minutes to run your batch. It will send you email every 5 or so
attempts to let you know that it is trying, but that resources are not
available. It is a good idea to glance at the message to make sure
that the problem is lack of resources and not an error in your NS

<!-- This ends the Basic Tutorial Section -->
1048 1049 1050 1051 1052 1053 1054 1055 1056

<!-- Custom OS Images -->

<a NAME="CustomOS"></a>
<h2>Custom OS Images</h2>

Leigh B. Stoller's avatar
Leigh B. Stoller committed
1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070
If your set of operating system customizations cannot be easily
contained within an RPM/TAR (or multiple RPM/TARs), then you can
create your own custom OS image; Emulab allows you to create your own
disk images and load them on your experimental nodes, automatically
when your experiment is created or swapped in. Once you have created a
custom disk image (and the associated <a
href="https://www.emulab.net/newimageid_ez.php3"> image/osid
descriptor</a> for it, you can use that OSID in your NS file. When
your experiment is swapped in, the testbed system will arrange for
your disks to be loaded in parallel using a locally written multicast
disk loading protocol. Experience has shown that it is much faster to
load a disk image on 10 nodes at once, then it is to load a bunch of
RPMS or tarballs on each node as it boots. So, while it may seem like
overkill to create your own disk image, we can assure you it is not!
1071 1072

1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084
The most common approach is to use the
<a href="https://www.emulab.net/newimageid_ez.php3">New Image Descriptor</a>
form to create a disk image that contains a customized version of the
standard Redhat Linux partition or the FreeBSD partition. Or, you can
start from scratch and load your own operating system in any of the
DOS partitions, and then capture that partition when you are
done. Either way, all you need to do is enter the node name in the
form, and the testbed system will create the image for you
automatically, notifying you via email when it is finished.  You can
then use that image in subsequent experiments by specifying the
descriptor name in your NS file with the
<a href="docwrapper.php3?docname=nscommands.html#OS">
1085 1086
<tt>tb-set-node-os</tt></a> directive. When the experiment is
configured, the proper image will be loaded on each node automatically by
the Emulab system.
1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149

A typical approach to creating your own disk image using one of the
default images as a base, goes like this:

<li> Create a single-node Linux or FreeBSD experiment. In your NS
     file, use the appropriate <tt>tb-set-node-os</tt> directive.
     For example, one of the following statements:

     tb-set-node-os $NodeA FBSD-STD
     tb-set-node-os $NodeA RHL-STD		</code></pre>

<li> After your experiment has swapped in (you have received the email
     saying it is running), log into the node and load all of the RPMS
     and software packages that you wish to load. Be sure to arrange
     for any programs that need to be started at boot time. It is a
     good idea to reboot the node and make sure that everything is
     running as you want it to when it comes up.

<li> Note the physical (pcXX) name of the machine used!

<li> Create an image descriptor and image using the
     <a href="https://www.emulab.net/newimageid_ez.php3">New Image
     Descriptor</a> form. For "which DOS partition" select 1 if you
     are customizing the FreeBSD slice, or 2 if you are customizing
     the Linux slice. Since you are customizing one of the base
     images, you should select all four OS features, operational mode
     "NormalV1", and all four node types (pc600, etc.). Leave the last
     two check boxes alone. <br>

     Where it asks for "Node to Create Image from," fill in the node
     you have just customized ((pcXX noted in previous step). Then
     click the submit button.

<li> Wait for the email saying the image creation is done.

<li> Now you can create a second single-node experiment to test your
     new image. In your .ns file, use <tt>tb-set-node-os</tt> to
     select the OSID that you just created. Be sure to remove any RPM
     or tarball directives. Submit that NS file and wait for the email
     notification. Then log into the new node and check to make sure
     everything is running normally.

<li> If everything is going well, terminate both of these single-node
     experiments. If not, release the experiment created in the
     previous step, and then go back and fix the original node (pcXX
     above). Recreate the image as needed:

     create_image -p &lt;pid&gt &lt;imageid&gt &lt;node&gt  </pre></code>

<li> Once your image is working properly, you can use it in any NS
     file by using the <tt>tb-set-node-os</tt>. 


1150 1151 1152
If you ever want to reload a node in your experiment, either with one
of your images or with one of the default images, you can use the
<tt>os_load</tt> command. Log into <tt>users</tt> and run:

1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164
	os_load -p &lt;pid&gt -i &lt;imageid&gt &lt;node&gt </pre></code>
This program will run in the foreground, waiting until the image has
been loaded. At that point you should log in and make sure everything
is working oaky. You might want to watch the console line as well (see
the <a href="../faq.php3?#UTT-3">FAQ</a>). If you want to load the
default image, then simply run:

	os_load &lt;node&gt </pre></code>