tutorial.html 28.9 KB
Newer Older
1 2
3 4
	<title>Emulab.Net - Tutorial</title>
	<link rel='stylesheet' href='../tbstyle-plain.css' type='text/css'>
5 6

8 9

<h1>Emulab Tutorial</h1>
11 12

13 14
<li> <a href="#GettingStarted">Getting Started</a>
17 18 19 20 21 22 23 24
    <li> <a href="#LoggingIn">Logging into the Web Interface</a>
    <li> <a href="#Designing">Designing a Network Topology</a>
    <li> <a href="#Beginning">Beginning the Experiment</a>
    <li> <a href="#UsingNodes">Using your Nodes</a>
    <li> <a href="#RootAccess">I need <b>root</b> access!</a>
    <li> <a href="#Wedged">My node is wedged!</a>
    <li> <a href="#Finished">I've finished my experiment</a>
    <li> <a href="#Help">Getting Help!</a>
<li> <a href="#Advanced">Advanced Topics</a>
28 29
    <li> <a href="#RPMS">Installing RPMS automatically</a>
    <li> <a href="#Startupcmd">
            Starting your application automatically</a>
    <li> <a href="#ReadyBits">
            How do I know when all my nodes are ready?</a>
    <li> <a href="#Delta">
            Customizing an OS (How to create a <i>delta</i>)</a>
36 37
<li> <a href="#BatchMode">Batch Mode Experiments</a>
<li> <a href="#CustomOS">Creating your own disk image</a>
38 39

40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68

<a NAME="GettingStarted"></a>
<h2>Getting Started</h2>

This section of the tutorial describes how to run your first Testbed
experiment. We cover basic NS syntax and various operational issues
that you will need to know in order conduct experiments to completion.
Later sections of the tutorial will cover more advanced topics such as
loading your own RPMs automatically, running programs automatically,
running batch jobs, creating your own disk images and loading those
images on your nodes. 

<li> <a href="#LoggingIn">Logging into the Web Interface</a>
<li> <a href="#Designing">Designing a Network Topology</a>
<li> <a href="#Beginning">Beginning the Experiment</a>
<li> <a href="#UsingNodes">Using your Nodes</a>
<li> <a href="#RootAccess">I need <b>root</b> access!</a>
<li> <a href="#Wedged">My node is wedged!</a>
<li> <a href="#Finished">I've finished my experiment</a>
<li> <a href="#Help">Getting Help!</a>

69 70
<a NAME="LoggingIn"></a>
<li> <h3>Logging Into the Web Interface</h3>
71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106

If you already have an account on the Testbed, all you need to do is
go to <a href="https://www.emulab.net"> Emulab Home
Page</a>, enter your login name and your password, and then click on
the "Login" button. If you don't have an account, click on the "Join
Project" or "Start Project" links. For an overiew of how you go about
getting an Emulab account, go to the <b><a href =
"http://www.emulab.net/auth.html"> "How To Get Started"</a></b> page.

<li> <a NAME="Designing"></a>
     <h3>Designing a Network Topology</h3>
The Testbed's power lies in its ability to assume many different
topologies; the description of a such a topology is a necessary part
of an experiment.

Emulab uses the "NS" ("Network Simulator") format to describe network
topologies.  This is substantially the same <a
href="http://www.scriptics.com/software/tcltk/">Tcl</a>-based format
used by <a href="http://www.isi.edu/nsnam/ns/">ns-2</a>.  Since the
Testbed offers emulation, rather than simulation, these files are
interpreted in a somewhat different manner than ns-2.  Therefore, some
ns-2 functionality may work differently than you expect, or may not be
implemented.  If you feel there is useful functionality missing,
please let us know.  Also, some
<a href="../docwrapper.php3?docname=tutorial/nscommands.html">
testbed-specific syntax</a> has been added, which with the inclusion
of compatibility module (<a href="tb_compat.tcl">tb_compat.tcl</a>),
will be ignored by the NS simulator. This
allows the same NS file to work on both the Testbed and ns-2, most of
the time.

107 108 109 110 111
For those unfamiliar with the NS format, here is a small example
(<em>We urge all new Emulab users to begin with a small 3-5 node experiment
such as this, so that you will become familiar with NS syntax and the
practical aspects of Emulab operation</em>). Let's say we are trying to
create a test network which looks like the following:
112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153
    <img src="simple.jpg"><br><br>
    (A is connected to B, B to C, and B to D.)

An NS file which would describe such a topology is as follows. First
off, all NS files start with a simple prolog, declaring a simulator 
and including a file that allow you to use the special 
<code>tb-</code> commands:
	# This is a simple ns script. Comments start with #.
	set ns [new Simulator]			
        source tb_compat.tcl                   </code></pre>

Then define the 4 nodes in the topology.
	set NodeA [$ns node]
	set NodeB [$ns node]
	set NodeC [$ns node]
	set NodeD [$ns node]			</code></pre>

Next define the 3 links between the nodes. NS syntax permits you to
specify the bandwidth, latency, and queue type. For our example, we
will define full speed links between B and C,D and a delayed link
from node A to B.
	$ns duplex-link $NodeA $NodeB 100Mb 50ms DropTail
	$ns duplex-link $NodeB $NodeC 100Mb .1ms DropTail
	$ns duplex-link $NodeB $NodeD 100Mb .1ms DropTail</code></pre>

In addition to the standard NS syntax above, a number of
<a href="../docwrapper.php3?docname=tutorial/nscommands.html">
extensions</a> have been added that allow you
to better control your experiment. For example, you may specify what
Operating System is booted on your nodes. We currently support FreeBSD
4.3 and Linux RedHat 7.1, as well as
<a href="http://www.cs.utah.edu/flux/oskit/">OSKit</a> kernels on the
testbed PCs. By default, Linux RedHat 7.1 is selected.
157 158

159 160
	tb-set-node-os $NodeA FBSD-STD
	tb-set-node-os $NodeC RHL-STD		</code></pre>
161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247

You may also control what IP addresses are assigned to the
experimental interfaces on your nodes. The experiment configuration
software will select IP addresses for you, but if your experiment
depends on particular IP addresses, you may specify them at each
link. The following example sets the IP address of node B on the port
going to node C:

	tb-set-ip-interface $NodeB $NodeC	</code></pre>

Lastly, all NS files end with an epilog that instructs the simulator
to start.

	# Go!
	$ns run					</code></pre>

If you would like to try the above example, the completed <a
href="basic.ns" target=stuff>NS file</a> can be run as an experiment in your
project. Another example ns script that shows off using the power of
Tcl to generate topologies is <a href="loop.ns" target=stuff>here</a>.

<li> <a NAME="Beginning"></a>
     <h3>Beginning the Experiment</h3>

After logging on to the Testbed Web Interface, choose the "Begin
Experiment" option from the menu. First select which project you want
the experiment to be configured in. Most people will be a member of just
one project, and will not have a choice. If you are a member of
multiple projects, be sure to select the correct project from the

Next fill in the `Name' and `Long Name' fields. The Name should be a
single word (no spaces) identifier, while the Long Name is a multi
word description of your experiment. In the "Your NS file" field,
place the <i>local path</i> of a NS file which you have created to
describe your network topology. This file will be uploaded through
your browser when you choose "Submit."

After submission, the Testbed interface will begin processing your
request. This will likely take several minutes, depending on how large
your topology is, and what other features (such as delay nodes and
bandwidth limits) you are using. Assuming all goes well, you will
receive an email message indicating success or failure, and if
successful, a listing of the nodes and IP address that were allocated
to your experiment. 

For the NS file described above, you would receive a listing that looks
similar to this:

	Node Mapping:
	Virtual         Physical        Qualified Name
	--------------- --------------- --------------------
	nodeA           pc12            nodeA.myproj.myexp.emulab.net
	nodeB           pc14            nodeB.myproj.myexp.emulab.net
	nodeC           pc16            nodeC.myproj.myexp.emulab.net
	nodeD           pc18            nodeD.myproj.myexp.emulab.net
	delay0          pc7             delay0.myproj.myexp.emulab.net
	IP Addresses:
	Node            IFC   Destination          IP                  
	--------------- -------------------- --------------------
	nodeB           eth0  nodeC             
	nodeB           eth2  nodeA               
	nodeA           eth0  nodeB               
	nodeC           eth0  nodeB              
	nodeD           eth0  nodeB               
	nodeB           eth1  nodeD      	</code></pre>

A few points should be noted:
<li> A single delay node was allocated and inserted into the link
     between NodeA and NodeB. This link is invisible from your
     perspective, except for the fact that it adds latency, error,
     or reduced bandwidth.
248 249 250 251 252 253 254 255 256
<li> Delays of less than 3ms (per trip) are too small to be
     accurately modeled at this time, and will be silently ignored.
     Please see the
     <a href="../docwrapper.php3?docname=tutorial/nscommands.html#LOSS">
     <i>Link Loss Commands</i></a> section in the
     <a href="../docwrapper.php3?docname=tutorial/nscommands.html">
     Extensions</a> reference.
257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283
<li> The names in the "Qualified Name" column refer to the control
     network interfaces for each of your allocated nodes. These names
     are added to the Emulab nameserver map on the fly, and are
     immediately available for you to use so that you do not have to
     worry about the actual physical node names that were choosen. In
     the names listed above, `myproj' is the name of the project that
     you chose to work in, and `myexp' is the name of the experiment
     that you provided in the "Begin Experiment" page.
<li> Since the IP address for the link from NodeB to NodeC was set in
     the NS file, the system selected an appropriate IP address for
     the other end of the link. All of the other links were configured
     by the system to use 192.168.XXX.XXX subnets.

<li> <a NAME="UsingNodes"></a>
     <h3>Using your Nodes</h3>

By the time you receive the email message listing your nodes, the
Testbed configuration system will have ensured that your nodes are
fully configured and ready to use. If you have selected one of the
Testbed supported operating system images (FreeBSD, Linux, NetBSD),
this configuration process includes:

285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369
<li> loading fresh disk images so that each node is in a known clean
<li> rebooting each node so that it is running the OS specified in the
     NS script;
<li> configuring each of the network interfaces so that each one is
     "up" and talking to its virtual LAN (VLAN);
<li> creating user accounts for each of the project members;
<li> mounting the projects NFS directory in /proj so that project
     files are easily shared amongst all the nodes in the experiment;
<li> creating a /etc/hosts file on each node so that you may refer to the
     experimental interfaces of other nodes by name instead of IP number;
<li> configuring all of the delay parameters;
<li> configuring the tip lines so that project members may access the
     console ports from users.emulab.net.

As this point you may log into any of the nodes in your experiment.
You will need to use Secure Shell (ssh), and you should use the
`qualified name' from the nodes mapping table so that you do not form
dependencies on any particular physical node. Your login name and
password will be the same as your Web Interface login and password.

The /etc/hosts file on each node will provide a local name mapping for
the other nodes in your experiments. You should take care to use these
names (or IP numbers) and <b>not</b> the .emulab.net names listed in
the node mapping, since the emulab names refer to the control network
LAN that is shared amongst all nodes in all experiments. It is only
the experimental interfaces that are entirely private to your

<b> NOTE:</b> The configuration process just described occurs only on
Emulab constructed operating system images. If you are using an OSKit
kernel, or your own disk image with your own operating system, you
will be responsible for all of the configuration. At some point we
hope to provide tools to assist in the configuration, but for now you
are on your own.

<li> <a NAME="RootAccess"></a>
     <h3>I need <b>root</b> access!</h3>

If you need to customize the configuration, or perhaps reboot nodes,
you can use the "sudo" command, located in <code>/usr/local/bin</code>
on FreeBSD and Linux, and <code>/usr/pkg/bin</code> on NetBSD. Our
policy is very liberal; you can customize the configuration in any way
you like, provided it does not violate Emulab's
<a href="../docwrapper.php3?docname=policies.html">
administrative policies</a>. As as example, to reboot a node that is
running FreeBSD:

	/usr/local/bin/sudo reboot			</code></pre>

<li> <a NAME="Wedged"></a>
     <h3>My node is wedged!</h3>

This is bound to happen when running experimental software and/or
experimental operating systems. Fortunately we have an easy way for
you to power cycle nodes without requiring Testbed Operations to get
involved. If you must power cycle a node, log on to users.emulab.net
and use the "node_reboot" command:

	node_reboot <node> [node ... ]			</xmp></code></pre>

where `node' is the physical name, as listed in the node mapping
table. You may provide more than one node on the command line. Be
aware that you may power cycle only nodes in projects that you are
member of. Also, <tt>node_reboot</tt> does its very best to perform a
clean reboot before resorting to cycling the power to the node. This
is to prevent the damage that can occur from constant power cycling
over a long period of time.  For this reason, <tt>node_reboot</tt> may
delay a minute or two if it detects that the machine is still
responsive to network transmission.  In any event, please try to
reboot your nodes first (see above).

370 371 372 373 374 375 376 377 378 379 380
You may also reboot all the nodes in an experiment by using the <tt>-e</tt>
option to specify the project and experiment names. For example:

	node_reboot -e testbed,multicast		</xmp></code></pre>

will reboot all of the nodes reserved in the "multicast" experiment in
the "testbed" project. This option is provided as a shorthand method
for rebooting large groups of nodes.

381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409
<li> <a NAME="Finished"></a>
     <h3>I've finished my experiment</h3>

When your experiment is completed, and you no longer need the
resources that have been allocated to it, you will need to terminate
the experiment via the Emulab Web Interface. Click on the "End An
Experiment" link. You will be presented with a list of all of the
experiments in all of the projects for which you have the
authorization to terminate experiments. Select the experiment you want
to terminate by clicking on the button in the "Terminate" column on
the right hand side. You will be asked to <b>confirm</b> your choice.
The Testbed configuration system will then tear down your experiment,
and send you an email message when the process is complete. At this
point you are allowed to reuse the experiment name (say, if you wanted
to create a similar experiment with different parameters).

<li> <a NAME="Help"></a>
     <h3>Getting Help!</h3>

If you have any questions or problems, or just want to comment on
Emulab's operation (maybe you want to suggest an improvement to one of
the Web pages), feel free to contact us by sending email to
<a href=\"mailto:testbed-ops@flux.cs.utah.edu\">Testbed
Operations (testbed-ops@flux.cs.utah.edu)</a>. Also note that much of
the software is in development, and occasionally things might break or
not work as you expect. Again, please feel free to contact us.

410 411 412
<!-- This ends the Basic Tutorial Section -->

413 414
<a NAME="Advanced"></a>
416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453
<h2>Advanced Topics</h2>

<li> <a href="#RPMS">Installing RPMS automatically</a>
<li> <a href="#Startupcmd">Starting your application automatically</a>
<li> <a href="#ReadyBits">How do I know when all my nodes are ready?</a>
<li> <a href="#Delta">Customizing an OS (How to create a <i>delta</i>)</a>

<li> <a NAME="RPMS"></a>
     <h3>Installing RPMS automatically</h3>

The Testbed NS extension <tt>tb-set-node-rpms</tt> allows you to
specify a (space separated) list of RPMs to install on each of your
nodes when it boots:

tb-set-node-rpms $nodeA /proj/pid/rpms/silly-freebsd.rpm
tb-set-node-rpms $nodeB /proj/pid/rpms/silly-linux.rpm	</code></pre>

The above NS code says to install the <tt>silly-freebsd.rpm</tt> file
on <tt>nodeA</tt>, and the <tt>silly-linux.rpm</tt> on <tt>nodeB</tt>.
RPMs are installed as root when the node first boots, and must reside
on the node's local filesystem, or in a directory that can be reached
via NFS. This is either the project's <tt>/proj</tt> directory, or a
project member's home directory in <tt>/users</tt>.


<li> <a NAME="Startupcmd"></a>
     <h3>Starting your application automatically</h3>

You can start your application automatically when your nodes boot by
using the <tt>tb-set-node-startup</tt> NS extension. The argument is
455 456 457 458 459 460 461 462 463 464 465 466 467 468 469
the pathname of a script or program that is run as the <tt>UID</tt> of
the experiment creator, after the node has reached multiuser mode. You
can specify the same program for each node, or a different program.
For example:

tb-set-node-startup $nodeA /proj/pid/runme.nodeA
tb-set-node-startup $nodeB /proj/pid/runme.nodeB	</code></pre>

will run <tt>/proj/pid/runme.nodeA</tt> on nodeA and
<tt>/proj/pid/runme.nodeA</tt> on nodeB. The programs must reside on
the node's local filesystem, or in a directory that can be reached via
NFS. This is either the project's <tt>/proj</tt> directory, or a
project member's home directory in <tt>/users</tt>.

470 471 472 473 474 475
The exit value of the startup command is reported back to the Web
Interface, and is made available to you via the "Experiment
Information" link. There is a listing for all of the nodes in the
experiment, and the exit value is recorded in this listing. The
special symbol <tt>none</tt> indicates that the node is still running
476 477 478
the startup command. A log file containing the output of the startup
command is created in the project's <tt>logs</tt> directory

480 481 482
The startup command is especially useful when
combined with <a href="#BatchMode"><i>batch mode</i></a> experiments.
483 484

485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543
<li> <a NAME="ReadyBits"></a>
     <h3>How do I know when all my nodes are ready?</h3>

It is often necessary for your startup program to determine when all
of the other nodes in the experiment have started, and are ready to
proceed. Sometimes called a <i>barrier</i>, this allows programs to
wait at a specific point, and then all proceed at once. Emulab
provides a primitive form of this mechanism using experiment <i>ready
bits</i>, which are set and read using the
<a href="../docwrapper.php3?docname=doc/tmcd.html">
TMCD/TMCC</a>. When an experiment is first configured, the ready bit
for each node is cleared. As each node starts its application and
reaches the point where it must be sure that all other nodes have
started up, it issues a TMCC <tt>ready</tt> command:

tmcc ready				</code></pre>

which tells Emulab's configuration system that the node is ready to
proceed. The node can then poll for the <i>ready count</i> to
determine how many nodes are ready (have issued a tmcc ready command):

tmcc readycount				</code></pre>

which will return the ready count as a string:

READY=N TOTAL=M				</code></pre>

where <tt>N</tt> is the number of nodes that are ready, and <tt>M</tt>
is the total number of nodes in the experiment. An application can
poll the ready count with a simple script, or it can encode the ready
bits check directly into its program. For example, here is a simple
Perl fragment that issues the ready command, and then polls for the
ready count, being sure to delay a small amount between each poll.

system("tmcc ready");
while (1) {
    my $bits = `tmcc readycount`;
    if ($bits ~= /READY=(\d*) TOTAL=(\d*)/) {
        if ($1 == $2) {
    # Please sleep to avoid swamping the TMCD!
}					</code></pre>

<i>Note that the ready count is essentially a use-once feature; The
ready count cannot be reinitialized to zero since there is no actual
synchronization happening.  If in the future it appears that a
generalized barrier synchronization would be more useful, we will
investigate the implementation of such a feature.</i>


546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583
<li> <a NAME="Delta"></a>
     <h3>Customizing an OS (How to create a <i>delta</i>)</h3>

If your set of operating system customizations cannot be easily
contained within an RPM (or multiple RPMs), or if you are just not
familiar with the RPM mechanism, then you can create your own
operating system <i>delta</i>. A delta is like an RPM or Tar file in
that it contains a bunch of files to be unpacked onto the node.  The
difference is that with a delta you do not have to figure what files
you changed, and how to automate the installation process.  Instead,
you just allocate a node, change it anyway you like, and then issue
the <tt>create-delta</tt> command. The resulting delta file can then
be specified in your NS file using the Testbed NS extension
<tt>tb-set-node-deltas</tt>. You can create one delta to install on
all of your nodes, or several different deltas for various nodes in
your experiment. When the nodes in your new experiment boot for the
first time, the delta will be installed (all of the files unpacked)
very early in the boot phase, and the node rebooted again (in case you
have installed daemons that need to be started during initialization).
Your experiment can then proceed. 

The key point is that the Testbed configuration software deals with
figuring out what files you changed, installing the delta on your
nodes, rebooting the nodes that have new software installed, and
ensuring that any particular delta is installed only once on each

Lets step through an example. The first thing you need to do is
create an experiment with a single node in it. The following NS file
can be submitted to the "Begin Experiment" page. 

set ns [new Simulator]			
source tb_compat.tcl
set nodeA [$ns node]
tb-set-node-os $nodeA FBSD-STD
585 586 587 588 589 590 591 592 593 594 595 596 597 598 599
$ns run					</code></pre>

When you have received email notification that the experiment has
configured, log into the node with <tt>ssh</tt>. Install whatever
software you like, making sure to update the necessary files if you
have installed daemons that need to be started automatically at boot
time. After all of your software is installed, create the delta file

sudo /usr/local/bin/create-delta /proj/testbed/foo.delta	</code></pre>

The argument to the <tt>create-delta</tt> command is a complete
pathname, which <b>must</b> reside someplace in your /proj directory
(a subdirectory is fine).  You cannot write the delta file to any
600 601 602
other filesystem. This restriction is enforced so that diskspace (and
resources in general) can be accounted for on a per-project basis.
It should be noted that a delta created on one OS
603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625
cannot be installed on another. In other words, a delta created on a
FreeBSD machine can only be installed on a FreeBSD machine. If you
need the same software installed on a Linux machine as well, you will
need to repeat this process with a node running Linux. See the section
on <a href="../docwrapper.php3?docname=tutorial/nscommands.html#OS">
<tt>tb-set-node-os</tt></a> in the
<a href="../docwrapper.php3?docname=tutorial/nscommands.html">
Extensions</a> reference.

After you have created your delta, you can then use it in subsequent
experiments by using the Testbed NS extension
<tt>tb-set-node-deltas</tt>.  For example, here is an NS file that
creates a two node experiment, installs a different delta on each
node, and then runs a program automatically on each node. Presumably,
the startup program is installed by the delta, and encapsulates the
experiment being performed.

set ns [new Simulator]			
source tb_compat.tcl
set nodeA [$ns node]
set nodeB [$ns node]
626 627
tb-set-node-os $nodeA FBSD-STD
tb-set-node-os $nodeB RHL-STD
628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644
tb-set-node-deltas $nodeA /proj/testbed/deltas/silly-freebsd.delta
tb-set-node-deltas $nodeB /proj/testbed/deltas/silly-linux.delta
tb-set-node-startup $nodeA /usr/site/bin/run-my-experiment
tb-set-node-startup $nodeB /usr/site/bin/run-my-experiment
$ns run					</code></pre>

<i>Implementation Notes:</i>
<li> Deltas are created and installed with the unix filesystem backup
     utilities <tt>dump</tt> and <tt>restore</tt>.
<li> Beware of changing too many critical systems and/or too many
     changes to the <tt>/etc/rc</tt> scripts.
<li> If you find that your customizations are too much for the Delta
     mechanism, feel free to contact us so that we can arrange to
     create a complete snapshot of your system.

<!-- This ends the Advanced Tutorial Section -->
646 647 648 649 650 651 652 653 654 655 656

<!-- Batch Mode -->

<a NAME="BatchMode"></a>
<h2>Batch Mode</h2>

657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746
<li> <a href="#BatchIntro">Batch Mode Introduction</a>
<li> <a href="#BatchExample">A Batch Mode Example</a>

<li> <a NAME="BatchIntro"></a>
     <h3>Batch Mode Introduction</h3>

Batch Mode experiments can be created on the Testbed via the "Create a
Batch Experiment" link in the operations menu to your left. A batch
mode experiment is a lot like a regular experiment, but with a few
important differences:

<li> The experiment is run when enough resources (ie: nodes) are
     available. This might be immediately, or it might be sometime in
     the future.
<li> Once your NS file is handed off to the system, the batch system
     is responsible for setting up the experiment and tearing it down
     once the experiment has completed. You will receive email
     notifying you when the experiment has been scheduled and when it
     has been terminated.
<li> Your NS file must define a <i>startup</i> command to run on each
     node using the <a href="#Startupcmd"><tt>tb-set-node-startup</tt></a>
     NS extension. It is the exit value(s) of the startup command(s) that
     indicates that the experiment is completed; when all of the
     nodes have run their respective startup commands and exited, the
     batch system will then tear down the experiment. The output of
     the startup command is stored in a file in your home directory so
     you can follow what has happened.

<li> <a NAME="BatchExample"></a>
     <h3>A Batch Mode Example</h3>

Consider example NS file <a href="batch.ns" target=stuff>batch.ns</a>.
First off, we have to arrange for the experimental software to be
automatically installed when the nodes boot. This is done with the <a
href="#RPMS"><tt>tb-set-node-rpms</tt></a> NS extension:

tb-set-node-rpms $nodeA /proj/testbed/rpms/silly-1.0-1.i386-freebsd.rpm
tb-set-node-rpms $nodeB /proj/testbed/rpms/silly-1.0-1.i386-freebsd.rpm

The next two lines of the NS file specify what program should be run
on each of the nodes. Using the <a href="#Startupcmd">
<tt>tb-set-node-startup</tt></a> NS extension, we say that the program
<tt>run-silly</tt> (installed by the <tt>silly-1.0</tt> RPM) is to be
run on both nodes:

tb-set-node-startup $nodeA /usr/site/bin/run-silly
tb-set-node-startup $nodeB /usr/site/bin/run-silly

After you have been notified via email that the batch experiment is
running, you can track the progress of your experiment by looking in
the "Experiment Information" page. As each node completes the startup
command, the listing for that node will be updated to reflect the exit
status of the command (you may need to hit the Reload button to see
the changes). Once all of the nodes hare reported in an exit status,
the batch system will tear down the experiment and send you email.

The status of your batch experiment can be viewed via the "Experiment
Information" link in the Web Interface Options menu. You may also
cancel a batch after you have submitted it using the "Terminate"
option in the information display.

The batch system is still under development. It appears to be
functional, but there are bound to be kinks in the system. Please help
us debug and improve it by letting us know what you think and if you
have problems with it. Currently, the batch system tries every 10
minutes to run your batch. It will send you email every 5 or so
attempts to let you know that it is trying, but that resources are not
available. It is a good idea to glance at the message to make sure
that the problem is lack of resources and not an error in your NS

<!-- This ends the Basic Tutorial Section -->
747 748 749 750 751 752 753 754 755 756

<!-- Custom OS Images -->

<a NAME="CustomOS"></a>
<h2>Custom OS Images</h2>

Sorry, this section of the documentation has not been finshed yet.
758 759 760
Please check back again later!