elabinelab.php3 8.88 KB
Newer Older
Leigh B. Stoller's avatar
Leigh B. Stoller committed
1 2 3
<?php
#
# EMULAB-COPYRIGHT
4
# Copyright (c) 2005, 2007 University of Utah and the Flux Group.
Leigh B. Stoller's avatar
Leigh B. Stoller committed
5 6 7 8 9
# All rights reserved.
#
chdir("..");
require("defs.php3");

10 11 12 13
#
# Verify page arguments.
#
$optargs = OptionalPageArguments("printable",  PAGEARG_BOOLEAN);
Leigh B. Stoller's avatar
Leigh B. Stoller committed
14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

if (!isset($printable))
    $printable = 0;

#
# Standard Testbed Header
#
if (!$printable) {
    PAGEHEADER("Emulab Tutorial - Emulab in Emulab");
}

if (!$printable) {
    echo "<b><a href=$REQUEST_URI?printable=1>
             Printable version of this document</a></b><br>\n";
}
?>
30 31 32
<center>
<h2>Emulab Tutorial - Emulab in Emulab</h2>
</center>
Leigh B. Stoller's avatar
Leigh B. Stoller committed
33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57

<p>
Emulab in Emulab (henceforth known as "ElabInElab") is a new feature
that allows the creation of an entire (inner) testbed as an experiment
running on the (outer) testbed. The "inner" Emulab is functionally
equivalent to the "outer" Emulab, except for how it interacts with
certain aspects of the hardware infrastructure (switches, power
controllers, etc), which must be mediated by the outer Emulab to avoid
improper access to devices that an experiment is not normally allowed
to access directly. For example, in order for an inner Emulab to power
cycle a node, it must ask the outer Emulab to do it via a proxy that
ensures that the node in question is actually part of the inner Emulab
(a node in the experiment that comprises the inner Emulab).

<br>
ElabInElab serves several purposes:

<ul>
<li> Can be used to provide an isolated environment (in conjunction
     with firewalling) for running "dangerous" experiments that
     include the use of worms and other malware. Instead of running
     the experiment on the outer Emulab, the experiment is run on the
     inner Emulab, and thus has access to all of the Emulab's
     services, but in a context that does not put the outer Emulab at
     risk from attack. 
58 59 60 61 62 63 64

<li> Allows testing and development of Emulab itself in a controlled
     environment, without needing a dedicated testbed. New features
     can be tested without affecting users of the main testbed.
     In fact, multiple independent inner Emulabs can be constructed
     with each one being used for the testing and development of
     different features. 
Leigh B. Stoller's avatar
Leigh B. Stoller committed
65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82
</ul>

<p>
There are a few things to keep in mind about ElabInElab:

<ul>
<li> While it is tempting to think of ElabInElab as "recursive", it should
     be noted that real recursion is not supported; you cannot create
     an inner Emulab inside of an inner Emulab.

<li> The inner emulab has its own "boss" and "users" nodes, its own
     web server, its own file server, etc.

<li> From the outer Emulab's perspective, all of the nodes that make
     up the inner emulab are simply nodes in an experiment.

<li> All of the nodes consume one of their experimental network
     interfaces to use for the innner Emulab "control" network.
83
     Therefore, inner experimental nodes have one fewer experimental
Leigh B. Stoller's avatar
Leigh B. Stoller committed
84 85 86 87 88 89 90 91 92 93 94 95 96
     interface to use in experiments.
</ul>

Here is a simple example that sets up a tiny ElabInElab experiment,
with just a single inner experimental node. For the purposes of this
discussion, the project is "testbed" and the experiment is called
"myemulab." 
	<code><pre>
	source tb_compat.tcl
	set ns [new Simulator]

	tb-elab-in-elab 1

Russ Fish's avatar
Russ Fish committed
97 98 99
	namespace eval TBCOMPAT {
	    set elabinelab_maxpcs 1
	}
Leigh B. Stoller's avatar
Leigh B. Stoller committed
100 101

	$ns run
Russ Fish's avatar
Russ Fish committed
102
	</pre></code>
Leigh B. Stoller's avatar
Leigh B. Stoller committed
103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122
which is instantiated as shown in the visualization in Figure 1.

  <br>
  <center>
  <table cellpadding='0' cellspacing='0' border='0' class='stealth'>
    <tr><td class='stealth'>
            <img class=stealth src="elabinelab-pic1.png" align=center>
	</td>
    </tr>
    <tr>
        <td align=center class='stealth'>
          <b>Figure 1.</b>
	</td>
    </tr>
  </table>
  </center>

<br>
As you can see in Figure 1, most of the details are handled for you;
the experiment includes a boss node, an ops node, and a single pc600,
Russ Fish's avatar
Russ Fish committed
123 124 125 126 127 128
all of which are connected via a lan.  (You may specify 0 to
<tt>elabinelab_maxpcs</tt> to get only an inner boss and ops, or a positive
integer for multiple experimental pc nodes.)

<br><br>
Once the experiment swaps in,
Leigh B. Stoller's avatar
Leigh B. Stoller committed
129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144
you can log into <tt>myops.myemulab.testbed.emulab.net</tt>, or you
can log into the web server at <tt>myboss.myemulab.testbed.emulab.net</tt>.
There is also a single experimental node that can be used to create an
experiment. In all aspects, the inner Emulab can be used the same way
that the outer Emulab can be used.

<br><br>
Another example:

	<code><pre>
	source tb_compat.tcl
	set ns [new Simulator]

	tb-elab-in-elab 1
        tb-set-inner-elab-eid myexp
	$ns run
Russ Fish's avatar
Russ Fish committed
145
	</pre></code>
Leigh B. Stoller's avatar
Leigh B. Stoller committed
146 147 148
In this example, we have included a <tt>tb-set-inner-elab-eid</tt>
directive, which says to automatically launch an experiment within the
inner Emulab once it is set up. The "myexp" experiment must already
149 150
exist in the same project; it must have already
been created, but not swapped in. The system uses the NS file associated
Leigh B. Stoller's avatar
Leigh B. Stoller committed
151 152 153 154 155 156 157 158
with the "myexp" experiment to construct an experiment on the inner
Emulab and swap it in. You will be notified via email, first when the
inner Emulab has been fully swapped in, and then again when the inner
experiment has been swapped in. You can interact with the inner
experiment normally, albeit from the inner boss (myboss) web interface
and the inner users node (myops), or you can log into the inner
experimental node (mypc1) directly.

Russ Fish's avatar
Russ Fish committed
159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208
<br><br>
You may specify a different Emulab source tarfile to be used in setting up the
inner Elab, for example:
	<code><pre>
	namespace eval TBCOMPAT {
	    set elabinelab_source_tarfile "/proj/yourpid/emulab-src.tar.gz"
	}
	</pre></code>

This source tarfile is created in your object tree (preferably one that is
pure, without any other hacks.)  Run "make elabinelab" and it will create the
tarfile for you.

<br><br>
Or if you want to check out a specific tag of the Emulab source code:

	<code><pre>
	tb-set-elabinelab-cvstag SomeTag
	</pre></code>

which will generate a checkout from the CVS repository.

<br><br>
You may specify tarfiles to modify your inner inner boss and ops, for example:
	<code><pre>
  	namespace eval TBCOMPAT {
  	    set elabinelab_tarfiles("boss") "/usr/site /proj/yourpid/patch.tar.gz"
  	}
	</pre></code>
  
You can specify multiple tarfiles in the string, just as 
<a href="docwrapper.php3?docname=nscommands.html#tb-set-node-tarfiles"
<tt>tb-set-node-tarfiles</tt> </a> allows.

<br><br>
Using similar sytax, there are also <tt>set elabinelab_</tt> versions of some
of the <tt>tb-set-</tt> commands that control how the inner boss, ops, and
experimental nodes are set up:
<ul>
  <li> <tt>elabinelab_nodeos</tt> - Choose a node OS ID, similar to
       <a href="docwrapper.php3?docname=nscommands.html#tb-set-node-os"
       <tt>tb-set-node-os</tt> </a>. </li>
  <li> <tt>elabinelab_hardware</tt> - Choose a node hardware type, similar to
       <a href="docwrapper.php3?docname=nscommands.html#tb-set-hardware"
       <tt>tb-set-hardware</tt> </a>. </li>
  <li> <tt>elabinelab_fixnodes</tt> - Choose an exact node, similar to
       <a href="docwrapper.php3?docname=nscommands.html#tb-fix-node"
       <tt>tb-fix-node</tt> </a>. </li>
</ul>

Leigh B. Stoller's avatar
Leigh B. Stoller committed
209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239
<h3> Implementation Notes</h3>

Our goal was to make the inner Emulab look as much like a real Emulab
as possible. To do that, we decided to use one of the experimental
interfaces on each node as the (inner) control network lan (see Figure
1). This lan connects all of the nodes (myboss, myops, and mypc1) in
much the same way boss, ops, and experimental nodes are attached in
the outer Emulab. The main difference is that there are no firewalls
or subnets on the inner control network since the security concerns
are not as strict; breaking into the inner boss is not going to do any
more damage then does having root on any experimental node within the
(outer) testbed.

<br><br>
The (inner) control network is used for all of the same traffic that
nodes in the outer Emulab would; DHCP traffic, multicast disk
reloading, experiment setup, etc. While each node still has its outer
control network interface, that interface is not even configured,
except on inner boss and inner ops.

<br><br>
Since we want to be able to create experiments on the inner Emulab
normally, it is also necessary to "proxy" access to the Emulab
infrastructure. For example, when setting up an inner experiment, a
number of vlans will need to be created on the switches. Obviously, we
cannot let the inner boss access the switches (or any other protected
resources) directly; it must be mediated via a proxy on the outer
boss. The proxy on the outer boss checks to make sure that the actions
are allowed (and make sense), and then proceeds to do them itself. 


240 241 242 243 244 245 246 247
<br><br>
<b>Setup time</b>: Largely due to dynamic construction of extensive
parts of the inner Emulab environment, it currently takes about 20
minutes to set up ElabinElab on pc850s.  In the future we will reduce
this time considerably by more caching of pre-built components.
Some faster nodes (2 and 3 GHz) will soon be available, which will also help.


Leigh B. Stoller's avatar
Leigh B. Stoller committed
248 249 250 251 252 253 254 255
<?php
#
# Standard Testbed Footer
# 
if (!$printable) {
    PAGEFOOTER();
}
?>