tutorial.html 37.7 KB
Newer Older
1 2
<html>
<head>
3 4
	<title>Emulab.Net - Tutorial</title>
	<link rel='stylesheet' href='../tbstyle-plain.css' type='text/css'>
5 6
</head>
<body>
7

8 9

<center>
10
<h1>Emulab Tutorial</h1>
11 12
</center>

13 14
<h2>Contents</h2>
<ul>
15
<li> <a href="#GettingStarted">Getting Started</a>
16
    <ul>
17 18 19 20 21 22
    <li> <a href="#LoggingIn">Logging into the Web Interface</a>
    <li> <a href="#Designing">Designing a Network Topology</a>
    <li> <a href="#Beginning">Beginning the Experiment</a>
    <li> <a href="#UsingNodes">Using your Nodes</a>
    <li> <a href="#RootAccess">I need <b>root</b> access!</a>
    <li> <a href="#Wedged">My node is wedged!</a>
23
    <li> <a href="#Scrogged">I've scrogged my disk!</a>
24 25
    <li> <a href="#Finished">I've finished my experiment</a>
    <li> <a href="#Help">Getting Help!</a>
26
    </ul>
27
<li> <a href="#Advanced">Advanced Topics</a>
28
    <ul>
29 30
    <li> <a href="#RPMS">Installing RPMS automatically</a>
    <li> <a href="#Startupcmd">
31
            Starting your application automatically</a>
32
    <li> <a href="#ReadyBits">
33
            How do I know when all my nodes are ready?</a>
34
    <li> <a href="#Delta">
35
            Customizing an OS (How to create a <i>delta</i>)</a>
36 37
    <li> <a href="#Routing">
            Setting up IP routing between nodes</a>
38
    </ul>
39 40
<li> <a href="#BatchMode">Batch Mode Experiments</a>
<li> <a href="#CustomOS">Creating your own disk image</a>
41 42
</ul>

43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66

<hr>
<a NAME="GettingStarted"></a>
<center>
<h2>Getting Started</h2>
</center>

<p>
This section of the tutorial describes how to run your first Testbed
experiment. We cover basic NS syntax and various operational issues
that you will need to know in order conduct experiments to completion.
Later sections of the tutorial will cover more advanced topics such as
loading your own RPMs automatically, running programs automatically,
running batch jobs, creating your own disk images and loading those
images on your nodes. 
</p>

<ul>
<li> <a href="#LoggingIn">Logging into the Web Interface</a>
<li> <a href="#Designing">Designing a Network Topology</a>
<li> <a href="#Beginning">Beginning the Experiment</a>
<li> <a href="#UsingNodes">Using your Nodes</a>
<li> <a href="#RootAccess">I need <b>root</b> access!</a>
<li> <a href="#Wedged">My node is wedged!</a>
67
<li> <a href="#Scrogged">I've scrogged my disk!</a>
68 69 70 71 72
<li> <a href="#Finished">I've finished my experiment</a>
<li> <a href="#Help">Getting Help!</a>
</ul>

<ul>
73 74
<a NAME="LoggingIn"></a>
<li> <h3>Logging Into the Web Interface</h3>
75 76 77 78 79 80 81 82 83 84 85 86 87 88

<p>
If you already have an account on the Testbed, all you need to do is
go to <a href="https://www.emulab.net"> Emulab Home
Page</a>, enter your login name and your password, and then click on
the "Login" button. If you don't have an account, click on the "Join
Project" or "Start Project" links. For an overiew of how you go about
getting an Emulab account, go to the <b><a href =
"http://www.emulab.net/auth.html"> "How To Get Started"</a></b> page.

<p>
<li> <a NAME="Designing"></a>
     <h3>Designing a Network Topology</h3>
<p>
Jay Lepreau's avatar
nits  
Jay Lepreau committed
89
Part of the Testbed's power lies in its ability to assume many different
90 91 92 93 94 95 96 97 98 99 100 101 102
topologies; the description of a such a topology is a necessary part
of an experiment.

<p>
Emulab uses the "NS" ("Network Simulator") format to describe network
topologies.  This is substantially the same <a
href="http://www.scriptics.com/software/tcltk/">Tcl</a>-based format
used by <a href="http://www.isi.edu/nsnam/ns/">ns-2</a>.  Since the
Testbed offers emulation, rather than simulation, these files are
interpreted in a somewhat different manner than ns-2.  Therefore, some
ns-2 functionality may work differently than you expect, or may not be
implemented.  If you feel there is useful functionality missing,
please let us know.  Also, some
103
<a href="docwrapper.php3?docname=nscommands.html">
104 105 106 107 108 109 110
testbed-specific syntax</a> has been added, which with the inclusion
of compatibility module (<a href="tb_compat.tcl">tb_compat.tcl</a>),
will be ignored by the NS simulator. This
allows the same NS file to work on both the Testbed and ns-2, most of
the time.

<p>
111 112 113 114 115
For those unfamiliar with the NS format, here is a small example
(<em>We urge all new Emulab users to begin with a small 3-5 node experiment
such as this, so that you will become familiar with NS syntax and the
practical aspects of Emulab operation</em>). Let's say we are trying to
create a test network which looks like the following:
116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153
  <br>
  <center>
    <img src="simple.jpg"><br><br>
    (A is connected to B, B to C, and B to D.)
  </center>
  <br>

<p>
An NS file which would describe such a topology is as follows. First
off, all NS files start with a simple prolog, declaring a simulator 
and including a file that allow you to use the special 
<code>tb-</code> commands:
	<code><pre>
	# This is a simple ns script. Comments start with #.
	set ns [new Simulator]			
        source tb_compat.tcl                   </code></pre>

<p>
Then define the 4 nodes in the topology.
	<code><pre>
	set NodeA [$ns node]
	set NodeB [$ns node]
	set NodeC [$ns node]
	set NodeD [$ns node]			</code></pre>
</p>

<p>
Next define the 3 links between the nodes. NS syntax permits you to
specify the bandwidth, latency, and queue type. For our example, we
will define full speed links between B and C,D and a delayed link
from node A to B.
	<code><pre>
	$ns duplex-link $NodeA $NodeB 100Mb 50ms DropTail
	$ns duplex-link $NodeB $NodeC 100Mb .1ms DropTail
	$ns duplex-link $NodeB $NodeD 100Mb .1ms DropTail</code></pre>

<p>
In addition to the standard NS syntax above, a number of
154
<a href="docwrapper.php3?docname=nscommands.html">
155 156 157
extensions</a> have been added that allow you
to better control your experiment. For example, you may specify what
Operating System is booted on your nodes. We currently support FreeBSD
158
4.3 and Linux RedHat 7.1, as well as
159
<a href="http://www.cs.utah.edu/flux/oskit/">OSKit</a> kernels on the
160
testbed PCs. By default, Linux RedHat 7.1 is selected.
161 162

	<code><pre>
163 164
	tb-set-node-os $NodeA FBSD-STD
	tb-set-node-os $NodeC RHL-STD		</code></pre>
165 166 167 168 169 170 171 172 173 174 175 176 177

<p>
You may also control what IP addresses are assigned to the
experimental interfaces on your nodes. The experiment configuration
software will select IP addresses for you, but if your experiment
depends on particular IP addresses, you may specify them at each
link. The following example sets the IP address of node B on the port
going to node C:

	<code><pre>
	tb-set-ip-interface $NodeB $NodeC 192.168.42.42	</code></pre>

<p>
Jay Lepreau's avatar
nits  
Jay Lepreau committed
178
Lastly, all NS files end with an epilogue that instructs the simulator
179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224
to start.

	<code><pre>
	# Go!
	$ns run					</code></pre>

<p>
If you would like to try the above example, the completed <a
href="basic.ns" target=stuff>NS file</a> can be run as an experiment in your
project. Another example ns script that shows off using the power of
Tcl to generate topologies is <a href="loop.ns" target=stuff>here</a>.

<p>
<li> <a NAME="Beginning"></a>
     <h3>Beginning the Experiment</h3>

<p>
After logging on to the Testbed Web Interface, choose the "Begin
Experiment" option from the menu. First select which project you want
the experiment to be configured in. Most people will be a member of just
one project, and will not have a choice. If you are a member of
multiple projects, be sure to select the correct project from the
menu.

<p>
Next fill in the `Name' and `Long Name' fields. The Name should be a
single word (no spaces) identifier, while the Long Name is a multi
word description of your experiment. In the "Your NS file" field,
place the <i>local path</i> of a NS file which you have created to
describe your network topology. This file will be uploaded through
your browser when you choose "Submit."

<p>
After submission, the Testbed interface will begin processing your
request. This will likely take several minutes, depending on how large
your topology is, and what other features (such as delay nodes and
bandwidth limits) you are using. Assuming all goes well, you will
receive an email message indicating success or failure, and if
successful, a listing of the nodes and IP address that were allocated
to your experiment. 

<p>
For the NS file described above, you would receive a listing that looks
similar to this:

	<code><pre>
225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255
        Node Info:
        ID              Type       OSID           
        --------------- ---------- ---------------
        NodeA           pc         FBSD-STD       
        NodeB           pc                        
        NodeC           pc         RHL-STD        
        NodeD           pc                        

        Node Mapping:
        Virtual         Physical        Qualified Name
        --------------- --------------- --------------------
        NodeA           pc18            NodeA.myexp.myproj.emulab.net
        NodeB           pc29            NodeB.myexp.myproj.emulab.net
        NodeC           pc28            NodeC.myexp.myproj.emulab.net
        NodeD           pc35            NodeD.myexp.myproj.emulab.net
        tbsdelay0       pc15            tbsdelay0.myexp.myproj.emulab.net

        Lan/Link Info:
        ID              Member          IP              Delay     BW (Kbs)  Loss Rate
        --------------- --------------- --------------- --------- --------- ---------
        l6              NodeB:eth0      192.168.42.42   0.00      100000    0.000    
        l5              NodeB:eth2      192.168.1.3     25.00     100000    0.000    
        l5              NodeA:eth0      192.168.1.2     25.00     100000    0.000    
        l6              NodeC:eth0      192.168.42.2    0.00      100000    0.000    
        l7              NodeB:eth1      192.168.2.2     0.00      100000    0.000    
        l7              NodeD:eth0      192.168.2.3     0.00      100000    0.000    

        Delay Node Info:
        LinkID          Virtual         Physical        Pipe Numbers   
        --------------- --------------- --------------- --------------- 
        l5              tbsdelay0       pc15            100,110		</code></pre>
256 257 258 259 260 261 262

<p>
A few points should be noted:
<ul>
<li> A single delay node was allocated and inserted into the link
     between NodeA and NodeB. This link is invisible from your
     perspective, except for the fact that it adds latency, error,
263 264 265 266
     or reduced bandwidth. However, the information for the delay links
     are included so that you can
     <a href="../faq.php3?#HDS-6">modify the delay parameters</a>
     after the experiment has been created. 
267
<p>
268
<li> Delays of less than 2ms (per trip) are too small to be
269 270
     accurately modeled at this time, and will be silently ignored.
     Please see the
271
     <a href="docwrapper.php3?docname=nscommands.html#LOSS">
272
     <i>Link Loss Commands</i></a> section in the
273
     <a href="docwrapper.php3?docname=nscommands.html">
274 275 276
     Extensions</a> reference.
     
<p>
277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303
<li> The names in the "Qualified Name" column refer to the control
     network interfaces for each of your allocated nodes. These names
     are added to the Emulab nameserver map on the fly, and are
     immediately available for you to use so that you do not have to
     worry about the actual physical node names that were choosen. In
     the names listed above, `myproj' is the name of the project that
     you chose to work in, and `myexp' is the name of the experiment
     that you provided in the "Begin Experiment" page.
<p>
<li> Since the IP address for the link from NodeB to NodeC was set in
     the NS file, the system selected an appropriate IP address for
     the other end of the link. All of the other links were configured
     by the system to use 192.168.XXX.XXX subnets.
</ul>


<p>
<li> <a NAME="UsingNodes"></a>
     <h3>Using your Nodes</h3>

<p>
By the time you receive the email message listing your nodes, the
Testbed configuration system will have ensured that your nodes are
fully configured and ready to use. If you have selected one of the
Testbed supported operating system images (FreeBSD, Linux, NetBSD),
this configuration process includes:

304
<p>
305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389
<ul>
<li> loading fresh disk images so that each node is in a known clean
     state;
<li> rebooting each node so that it is running the OS specified in the
     NS script;
<li> configuring each of the network interfaces so that each one is
     "up" and talking to its virtual LAN (VLAN);
<li> creating user accounts for each of the project members;
<li> mounting the projects NFS directory in /proj so that project
     files are easily shared amongst all the nodes in the experiment;
<li> creating a /etc/hosts file on each node so that you may refer to the
     experimental interfaces of other nodes by name instead of IP number;
<li> configuring all of the delay parameters;
<li> configuring the tip lines so that project members may access the
     console ports from users.emulab.net.
</ul>

<p>
As this point you may log into any of the nodes in your experiment.
You will need to use Secure Shell (ssh), and you should use the
`qualified name' from the nodes mapping table so that you do not form
dependencies on any particular physical node. Your login name and
password will be the same as your Web Interface login and password.

<p>
The /etc/hosts file on each node will provide a local name mapping for
the other nodes in your experiments. You should take care to use these
names (or IP numbers) and <b>not</b> the .emulab.net names listed in
the node mapping, since the emulab names refer to the control network
LAN that is shared amongst all nodes in all experiments. It is only
the experimental interfaces that are entirely private to your
experiment.

<p>
<b> NOTE:</b> The configuration process just described occurs only on
Emulab constructed operating system images. If you are using an OSKit
kernel, or your own disk image with your own operating system, you
will be responsible for all of the configuration. At some point we
hope to provide tools to assist in the configuration, but for now you
are on your own.


<p>
<li> <a NAME="RootAccess"></a>
     <h3>I need <b>root</b> access!</h3>

<p>
If you need to customize the configuration, or perhaps reboot nodes,
you can use the "sudo" command, located in <code>/usr/local/bin</code>
on FreeBSD and Linux, and <code>/usr/pkg/bin</code> on NetBSD. Our
policy is very liberal; you can customize the configuration in any way
you like, provided it does not violate Emulab's
<a href="../docwrapper.php3?docname=policies.html">
administrative policies</a>. As as example, to reboot a node that is
running FreeBSD:

	<code><pre>
	/usr/local/bin/sudo reboot			</code></pre>

	
<p>
<li> <a NAME="Wedged"></a>
     <h3>My node is wedged!</h3>

<p>
This is bound to happen when running experimental software and/or
experimental operating systems. Fortunately we have an easy way for
you to power cycle nodes without requiring Testbed Operations to get
involved. If you must power cycle a node, log on to users.emulab.net
and use the "node_reboot" command:

	<code><pre><xmp>
	node_reboot <node> [node ... ]			</xmp></code></pre>

where `node' is the physical name, as listed in the node mapping
table. You may provide more than one node on the command line. Be
aware that you may power cycle only nodes in projects that you are
member of. Also, <tt>node_reboot</tt> does its very best to perform a
clean reboot before resorting to cycling the power to the node. This
is to prevent the damage that can occur from constant power cycling
over a long period of time.  For this reason, <tt>node_reboot</tt> may
delay a minute or two if it detects that the machine is still
responsive to network transmission.  In any event, please try to
reboot your nodes first (see above).

390 391 392 393 394 395 396 397 398 399 400
<p>
You may also reboot all the nodes in an experiment by using the <tt>-e</tt>
option to specify the project and experiment names. For example:

	<code><pre><xmp>
	node_reboot -e testbed,multicast		</xmp></code></pre>

will reboot all of the nodes reserved in the "multicast" experiment in
the "testbed" project. This option is provided as a shorthand method
for rebooting large groups of nodes.

401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422
<p>
<li> <a NAME="Scrogged"></a>
     <h3>I've scrogged my disk!</h3>

<p>
Scrogging your disk is certainly not as common, but it does happen.
You can either terminate your experiment, and recreate it (which will
allocate another group of nodes), or if you prefer you can reload the
disk image yourself. You will of course lose anything you have stored
on that disk; it is a good idea to store only data that can be easily
recreated, or else store it in your project directory in <tt>/proj</tt>.
Reloading your disk with a fresh copy of the default image is easy,
and requires no intervention by Emulab staff:

	<code><pre><xmp>
	os_load -w <node> [node ... ]			</xmp></code></pre>

The <tt>-w</tt> option causes os_load to wait (not exit) until the
nodes have been reloaded. This is the preferred mode of operation
since otherwise you will need to check the console lines of each node
to determine when the load is done.

423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446
<p>
<li> <a NAME="Finished"></a>
     <h3>I've finished my experiment</h3>

When your experiment is completed, and you no longer need the
resources that have been allocated to it, you will need to terminate
the experiment via the Emulab Web Interface. Click on the "End An
Experiment" link. You will be presented with a list of all of the
experiments in all of the projects for which you have the
authorization to terminate experiments. Select the experiment you want
to terminate by clicking on the button in the "Terminate" column on
the right hand side. You will be asked to <b>confirm</b> your choice.
The Testbed configuration system will then tear down your experiment,
and send you an email message when the process is complete. At this
point you are allowed to reuse the experiment name (say, if you wanted
to create a similar experiment with different parameters).

<p>
<li> <a NAME="Help"></a>
     <h3>Getting Help!</h3>

If you have any questions or problems, or just want to comment on
Emulab's operation (maybe you want to suggest an improvement to one of
the Web pages), feel free to contact us by sending email to
447 448 449
<a href="../sendemail.php3">Testbed Operations</a>. Also note that much
of the software is in development, and occasionally things might break
or not work as you expect. Again, please feel free to contact us.
450

451 452 453
<!-- This ends the Basic Tutorial Section -->
</ul>

454 455
<hr>
<a NAME="Advanced"></a>
456
<center>
457 458 459 460 461 462 463 464
<h2>Advanced Topics</h2>
</center>

<ul>
<li> <a href="#RPMS">Installing RPMS automatically</a>
<li> <a href="#Startupcmd">Starting your application automatically</a>
<li> <a href="#ReadyBits">How do I know when all my nodes are ready?</a>
<li> <a href="#Delta">Customizing an OS (How to create a <i>delta</i>)</a>
465
<li> <a href="#Routing">Setting up IP routing between nodes</a>
466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495
</ul>
<p>

<ul>
<li> <a NAME="RPMS"></a>
     <h3>Installing RPMS automatically</h3>
<p>

The Testbed NS extension <tt>tb-set-node-rpms</tt> allows you to
specify a (space separated) list of RPMs to install on each of your
nodes when it boots:

<code><pre>
tb-set-node-rpms $nodeA /proj/pid/rpms/silly-freebsd.rpm
tb-set-node-rpms $nodeB /proj/pid/rpms/silly-linux.rpm	</code></pre>

The above NS code says to install the <tt>silly-freebsd.rpm</tt> file
on <tt>nodeA</tt>, and the <tt>silly-linux.rpm</tt> on <tt>nodeB</tt>.
RPMs are installed as root when the node first boots, and must reside
on the node's local filesystem, or in a directory that can be reached
via NFS. This is either the project's <tt>/proj</tt> directory, or a
project member's home directory in <tt>/users</tt>.

</p>

<li> <a NAME="Startupcmd"></a>
     <h3>Starting your application automatically</h3>
<p>

You can start your application automatically when your nodes boot by
496
using the <tt>tb-set-node-startup</tt> NS extension. The argument is
497 498 499 500 501 502 503 504 505 506 507 508 509 510 511
the pathname of a script or program that is run as the <tt>UID</tt> of
the experiment creator, after the node has reached multiuser mode. You
can specify the same program for each node, or a different program.
For example:

<code><pre>
tb-set-node-startup $nodeA /proj/pid/runme.nodeA
tb-set-node-startup $nodeB /proj/pid/runme.nodeB	</code></pre>

will run <tt>/proj/pid/runme.nodeA</tt> on nodeA and
<tt>/proj/pid/runme.nodeA</tt> on nodeB. The programs must reside on
the node's local filesystem, or in a directory that can be reached via
NFS. This is either the project's <tt>/proj</tt> directory, or a
project member's home directory in <tt>/users</tt>.

512 513 514 515 516 517
<p>
The exit value of the startup command is reported back to the Web
Interface, and is made available to you via the "Experiment
Information" link. There is a listing for all of the nodes in the
experiment, and the exit value is recorded in this listing. The
special symbol <tt>none</tt> indicates that the node is still running
518 519 520
the startup command. A log file containing the output of the startup
command is created in the project's <tt>logs</tt> directory
(<tt>/proj/pid/logs</tt>).
521

522 523 524
<p>
The startup command is especially useful when
combined with <a href="#BatchMode"><i>batch mode</i></a> experiments.
525 526

<p>
527 528 529 530 531 532 533 534 535 536
<li> <a NAME="ReadyBits"></a>
     <h3>How do I know when all my nodes are ready?</h3>
<p>

It is often necessary for your startup program to determine when all
of the other nodes in the experiment have started, and are ready to
proceed. Sometimes called a <i>barrier</i>, this allows programs to
wait at a specific point, and then all proceed at once. Emulab
provides a primitive form of this mechanism using experiment <i>ready
bits</i>, which are set and read using the
537
<a href="../doc/docwrapper.php3?docname=tmcd.html">
538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576
TMCD/TMCC</a>. When an experiment is first configured, the ready bit
for each node is cleared. As each node starts its application and
reaches the point where it must be sure that all other nodes have
started up, it issues a TMCC <tt>ready</tt> command:

<code><pre>
tmcc ready				</code></pre>

which tells Emulab's configuration system that the node is ready to
proceed. The node can then poll for the <i>ready count</i> to
determine how many nodes are ready (have issued a tmcc ready command):

<code><pre>
tmcc readycount				</code></pre>

which will return the ready count as a string:

<code><pre>
READY=N TOTAL=M				</code></pre>

where <tt>N</tt> is the number of nodes that are ready, and <tt>M</tt>
is the total number of nodes in the experiment. An application can
poll the ready count with a simple script, or it can encode the ready
bits check directly into its program. For example, here is a simple
Perl fragment that issues the ready command, and then polls for the
ready count, being sure to delay a small amount between each poll.

<code><pre>
system("tmcc ready");
while (1) {
    my $bits = `tmcc readycount`;
    if ($bits ~= /READY=(\d*) TOTAL=(\d*)/) {
        if ($1 == $2) {
            last;
	}
    }
    #
    # Please sleep to avoid swamping the TMCD!
    # 
577
    sleep(5);
578 579 580 581 582 583 584 585
}					</code></pre>

<i>Note that the ready count is essentially a use-once feature; The
ready count cannot be reinitialized to zero since there is no actual
synchronization happening.  If in the future it appears that a
generalized barrier synchronization would be more useful, we will
investigate the implementation of such a feature.</i>

586
</p>
587

588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625
<li> <a NAME="Delta"></a>
     <h3>Customizing an OS (How to create a <i>delta</i>)</h3>
<p>

If your set of operating system customizations cannot be easily
contained within an RPM (or multiple RPMs), or if you are just not
familiar with the RPM mechanism, then you can create your own
operating system <i>delta</i>. A delta is like an RPM or Tar file in
that it contains a bunch of files to be unpacked onto the node.  The
difference is that with a delta you do not have to figure what files
you changed, and how to automate the installation process.  Instead,
you just allocate a node, change it anyway you like, and then issue
the <tt>create-delta</tt> command. The resulting delta file can then
be specified in your NS file using the Testbed NS extension
<tt>tb-set-node-deltas</tt>. You can create one delta to install on
all of your nodes, or several different deltas for various nodes in
your experiment. When the nodes in your new experiment boot for the
first time, the delta will be installed (all of the files unpacked)
very early in the boot phase, and the node rebooted again (in case you
have installed daemons that need to be started during initialization).
Your experiment can then proceed. 

<p>
The key point is that the Testbed configuration software deals with
figuring out what files you changed, installing the delta on your
nodes, rebooting the nodes that have new software installed, and
ensuring that any particular delta is installed only once on each
node.

<p>
Lets step through an example. The first thing you need to do is
create an experiment with a single node in it. The following NS file
can be submitted to the "Begin Experiment" page. 

<code><pre>
set ns [new Simulator]			
source tb_compat.tcl
set nodeA [$ns node]
626
tb-set-node-os $nodeA FBSD-STD
627 628 629 630 631 632 633 634 635 636 637 638 639 640 641
$ns run					</code></pre>

When you have received email notification that the experiment has
configured, log into the node with <tt>ssh</tt>. Install whatever
software you like, making sure to update the necessary files if you
have installed daemons that need to be started automatically at boot
time. After all of your software is installed, create the delta file
with:

<code><pre>
sudo /usr/local/bin/create-delta /proj/testbed/foo.delta	</code></pre>

The argument to the <tt>create-delta</tt> command is a complete
pathname, which <b>must</b> reside someplace in your /proj directory
(a subdirectory is fine).  You cannot write the delta file to any
642 643 644
other filesystem. This restriction is enforced so that diskspace (and
resources in general) can be accounted for on a per-project basis.
It should be noted that a delta created on one OS
645 646 647 648
cannot be installed on another. In other words, a delta created on a
FreeBSD machine can only be installed on a FreeBSD machine. If you
need the same software installed on a Linux machine as well, you will
need to repeat this process with a node running Linux. See the section
649
on <a href="docwrapper.php3?docname=nscommands.html#OS">
650
<tt>tb-set-node-os</tt></a> in the
651
<a href="docwrapper.php3?docname=nscommands.html">
652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667
Extensions</a> reference.

<p>
After you have created your delta, you can then use it in subsequent
experiments by using the Testbed NS extension
<tt>tb-set-node-deltas</tt>.  For example, here is an NS file that
creates a two node experiment, installs a different delta on each
node, and then runs a program automatically on each node. Presumably,
the startup program is installed by the delta, and encapsulates the
experiment being performed.

<code><pre>
set ns [new Simulator]			
source tb_compat.tcl
set nodeA [$ns node]
set nodeB [$ns node]
668 669
tb-set-node-os $nodeA FBSD-STD
tb-set-node-os $nodeB RHL-STD
670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686
tb-set-node-deltas $nodeA /proj/testbed/deltas/silly-freebsd.delta
tb-set-node-deltas $nodeB /proj/testbed/deltas/silly-linux.delta
tb-set-node-startup $nodeA /usr/site/bin/run-my-experiment
tb-set-node-startup $nodeB /usr/site/bin/run-my-experiment
$ns run					</code></pre>

<i>Implementation Notes:</i>
<ul>
<li> Deltas are created and installed with the unix filesystem backup
     utilities <tt>dump</tt> and <tt>restore</tt>.
<li> Beware of changing too many critical systems and/or too many
     changes to the <tt>/etc/rc</tt> scripts.
<li> If you find that your customizations are too much for the Delta
     mechanism, feel free to contact us so that we can arrange to
     create a complete snapshot of your system.
</ul>

687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859
<li> <a NAME="Routing"></a>
     <h3>Setting up IP routing between nodes</h3>
<p>

As Emulab strives to make all aspects of the network controllable by the
user, we do not attempt to impose any IP routing architecture or protocol
by default.
However, many users are more interested in end-to-end aspects and don't
want to be bothered with setting up routes.  For those users we provide
an option to automatically set up routes on nodes which run one of our
provided FreeBSD or Linux disk images.

<p>
You can use the <tt>tb-set-ip-routing</tt> command in your NS file to
enable automatic routing:
<code><pre>
tb-set-ip-routing <i>protocol</i>
</pre></code>
where <i>protocol</i> is either <code>none</code>, the default for no
automatic routing, or <code>ospf</code> to enable <code>gated</code>
running the OSPF protocol on all nodes in the experiment.
In the future we may add support for automatic pre-computation of static
routes as well as the ability for users to specify per-node routing
information in the NS file.

<p>
If you do want to setup and manage routing yourself, you can use the
<tt>tb-set-node-startup</tt> command in your NS file to specify a
per-node script in your home directory to set up routing at boot time.
You can use this startup script to setup either static routes or to
fire up a routing daemon, as the following examples illustrate.

<p>
The simplest topology requiring routing is one in which two nodes,
<i>client</i> and <i>server</i> are each connected via a single link
to <i>router</i>.  In order to get router to correctly forward packets
between client and server, you would add this line to your NS file:

<code><pre>
tb-set-node-startup $router /proj/pid/router-startup
</pre></code>

This will make router run the <code>router-startup</code> script
from you project directory.  That script looks like:

<code><pre>
#!/bin/sh
if [ `uname` = "FreeBSD" ]
then
    sudo sysctl -w net.inet.ip.forwarding=1
    sudo sysctl -w net.inet.ip.fastforwarding=1
else
    sudo sysctl -w net.ipv4.conf.all.forwarding=1
fi
exit 0
</pre></code>

These commands ensure that the router node will forward IP packets.
Note the use of <code>sudo</code> which is necessary since startup scripts
run as you and not as root.

Now to get client and server to talk to each other through this router,
you would add these lines to your NS file:

<code><pre>
tb-set-node-startup $client /proj/pid/clientroutecmd
tb-set-node-startup $server /proj/pid/serverroutecmd
</pre></code>

This will have the client and the server each call a small script
to set up routes. To add a route (on client) to interface 0 of the
server through router, you would run a script called
<code>clientroutecmd</code> that looks like this:

<code><pre>
#!/bin/sh
if [ `uname` = "FreeBSD" ]
then
    sudo route add server-0 router
else
    sudo route add server-0 gw router
fi
exit 0
</pre></code>

Similarly, to add a route (on server) to interface 0 of the client
through router, you would use this <code>serverroutecmd</code> script:

<code><pre>
#!/bin/sh
if [ `uname` = "FreeBSD" ]
then
    sudo route add client-0 router
else
    sudo route add client-0 gw router
fi
exit 0
</pre></code>

That should do it. You will now have a router node that really
routes and forwards packets, and a client and a server that know
how to talk to each other through a gateway router.
Note that this setup can be generalized to work for any topology
where all nodes are directly connected to a single router node.
However, every node would need a custom script with a route add command
for every other node or you would need to use a subnet route as in
the next example.

<p>
In complex topologies with multiple routers, each end node will need
a route to the entire experimental network, through its local router.
For FreeBSD, this is done with:

<code><pre>
#!/bin/sh
if [ `uname` = "FreeBSD" ]
then
    sudo route add -net 192.168.0.0 -netmask 255.255.0.0 myrouter
else
    sudo route add -net 192.168.0.0 netmask 255.255.0.0 gw myrouter
fi
exit 0
</pre></code>

You might be tempted to just set the default route for each node to
<code>myrouter</code>, but be aware that such a route would prevent
you from contacting the outside world.  The default route <em>must</em>
be set to use the control network interface.

<p>
You can make a single script that will handle all end nodes, by replacing
<code>myrouter</code> in the above commands with <code>$1</code>,
and specifying the router in your NS file:

<code><pre>
tb-set-node-startup $clientA {/proj/pid/router-startup router0}
tb-set-node-startup $clientB {/proj/pid/router-startup router1}
</pre></code>

For multiple routers, the startup script for each router will need to
contain a set of routes for all subnets it is not directly connected
to. This will differ, depending on the topology you are using. To make
this task easier, you can use the <code>tb-set-ip</code> command in your NS
file, so that you know which subnets will be assigned to which nodes.

<p>
If you want to enable dynamic routing,
you can also use the startup script to fire off a routing daemon.
A startup script might look like:

<code><pre>
#!/bin/sh
if [ `uname` = "FreeBSD" ]
then
    sudo sysctl -w net.inet.ip.forwarding=1
    sudo sysctl -w net.inet.ip.fastforwarding=1
else
    sudo sysctl -w net.ipv4.conf.all.forwarding=1
fi
sudo gated -f /proj/pid/gated.conf
exit 0
</pre></code>

One complication of using a routing daemon, is that you need to
avoid using the control network interface in the configuration.
Since every node in the testbed is directly connected to the
control network LAN, a naive routing daemon configuration will
discover that any node is just one hop away via the control
network and <em>all</em> inter-node traffic will be routed via
that interface.  
See one of the installed <tt>/etc/testbed/gated*.conf</tt>
files for an example of how to avoid this pitfall.

860
<!-- This ends the Advanced Tutorial Section -->
861 862 863 864 865 866 867 868 869 870 871
</ul>


<!-- Batch Mode -->

<hr>
<a NAME="BatchMode"></a>
<center>
<h2>Batch Mode</h2>
</center>

872 873 874 875 876 877 878 879 880 881
<ul>
<li> <a href="#BatchIntro">Batch Mode Introduction</a>
<li> <a href="#BatchExample">A Batch Mode Example</a>
</ul>

<ul>
<li> <a NAME="BatchIntro"></a>
     <h3>Batch Mode Introduction</h3>

<p>
882 883 884 885 886
Batch Mode experiments can be created on the Testbed via the "Create
an Experiment" link in the operations menu to your left. There is a
checkbox near the bottom of the form that indicates you want to use
the batch system. There are several important differences between a
regular experiment and a batch mode experiment:
887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940

<p>
<ul>
<li> The experiment is run when enough resources (ie: nodes) are
     available. This might be immediately, or it might be sometime in
     the future.
     <p>
<li> Once your NS file is handed off to the system, the batch system
     is responsible for setting up the experiment and tearing it down
     once the experiment has completed. You will receive email
     notifying you when the experiment has been scheduled and when it
     has been terminated.
     <p>
<li> Your NS file must define a <i>startup</i> command to run on each
     node using the <a href="#Startupcmd"><tt>tb-set-node-startup</tt></a>
     NS extension. It is the exit value(s) of the startup command(s) that
     indicates that the experiment is completed; when all of the
     nodes have run their respective startup commands and exited, the
     batch system will then tear down the experiment. The output of
     the startup command is stored in a file in your home directory so
     you can follow what has happened.
</ul>
<p>

<li> <a NAME="BatchExample"></a>
     <h3>A Batch Mode Example</h3>

Consider example NS file <a href="batch.ns" target=stuff>batch.ns</a>.
First off, we have to arrange for the experimental software to be
automatically installed when the nodes boot. This is done with the <a
href="#RPMS"><tt>tb-set-node-rpms</tt></a> NS extension:

<code><pre>
tb-set-node-rpms $nodeA /proj/testbed/rpms/silly-1.0-1.i386-freebsd.rpm
tb-set-node-rpms $nodeB /proj/testbed/rpms/silly-1.0-1.i386-freebsd.rpm
</code></pre>

The next two lines of the NS file specify what program should be run
on each of the nodes. Using the <a href="#Startupcmd">
<tt>tb-set-node-startup</tt></a> NS extension, we say that the program
<tt>run-silly</tt> (installed by the <tt>silly-1.0</tt> RPM) is to be
run on both nodes:

<code><pre>
tb-set-node-startup $nodeA /usr/site/bin/run-silly
tb-set-node-startup $nodeB /usr/site/bin/run-silly
</code></pre>

After you have been notified via email that the batch experiment is
running, you can track the progress of your experiment by looking in
the "Experiment Information" page. As each node completes the startup
command, the listing for that node will be updated to reflect the exit
status of the command (you may need to hit the Reload button to see
the changes). Once all of the nodes hare reported in an exit status,
941 942 943 944 945 946 947 948 949 950 951 952
the batch system will tear down the experiment and send you email.  If
your experiment is such that one node is the controller, and runs
commands on all the other nodes, then simply run a dummy startup
command on the other nodes so that the batch system will receive an
exit value for that node. Since the batch is not terminated until
<em>all</em> nodes have reported in, be sure that the controlling node
does not exit from its startup command until all of the nodes have
finished. A dummy startup command can be setup like this:

<code><pre>
tb-set-node-startup $nodeC /bin/echo
</code></pre>
953 954 955 956 957

<p>
The status of your batch experiment can be viewed via the "Experiment
Information" link in the Web Interface Options menu. You may also
cancel a batch after you have submitted it using the "Terminate"
958 959 960 961
option in the information display. As noted in the section on the <a
href="#Startupcmd">Startupcmd</a>, the output of the startup command
on each node is written to separate files in your project log
directory. You can use these log files to debug your batch experiment.
962 963 964 965 966 967 968 969 970 971 972 973 974 975 976

<p>
<i>
The batch system is still under development. It appears to be
functional, but there are bound to be kinks in the system. Please help
us debug and improve it by letting us know what you think and if you
have problems with it. Currently, the batch system tries every 10
minutes to run your batch. It will send you email every 5 or so
attempts to let you know that it is trying, but that resources are not
available. It is a good idea to glance at the message to make sure
that the problem is lack of resources and not an error in your NS
file.</i>

<!-- This ends the Basic Tutorial Section -->
</ul>
977 978 979 980 981 982 983 984 985

<!-- Custom OS Images -->

<hr>
<a NAME="CustomOS"></a>
<center>
<h2>Custom OS Images</h2>
</center>

Leigh B. Stoller's avatar
Leigh B. Stoller committed
986 987 988 989
Sorry, this section of the documentation is still under construction.
In the meantime, please check the documentation for creating your own
custom disk images at <a href="https://www.emulab.net/newimageid_explain.php3">
https://www.emulab.net/newimageid_explain.php3.</a>