firewall.html 14 KB
Newer Older
1 2
   Copyright (c) 2004, 2005 University of Utah and the Flux Group.
4 5 6
   All rights reserved.
<h1>Firewalled Experiment Support in Emulab</h1>
8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

<li> <a href="#Overview">Overview</a>
<li> <a href="#Use">Use</a>
<li> <a href="#Limitations">Current Limitations</a>
<li> <a href="#KnownBugs">Known Bugs</a>
<li> <a href="#Example">A Complete Example</a>

<a NAME="Overview"></a><h2>Overview</h2>
Emulab allows the ability to setup a
<a href="docwrapper.php3?docname=tutorial.html#ControlNet">control-network</a>
firewall between an experiment and the outside world.
This is <b>not</b> a firewall
between nodes within an experiment, that is, the firewall is not part of
your NS-specified network topology.
The purpose of an Emulab firewall is to prevent experimental traffic from
Mike Hibler's avatar
Mike Hibler committed
escaping, via the control network, to the Internet due to a mis-configured
30 31
application or router within an experiment.  Control net firewalls are also
a key component of Emulab ``high-security'' experiment environments.
32 33
The firewall is implemented by allocating an additional node to the experiment.
34 35 36 37 38 39 40
Through switch-enforced VLANs, the experiment is given its own private
control network connecting all nodes in the experiment.  Network traffic
between experiment nodes and any hosts outside the experiment (Emulab
infrastructure or Internet hosts in general) must pass through the firewall
node.  The firewall is setup as a filtering layer2 bridge using IPFW2
on FreeBSD and can be configured in a number
of ways to block or allow certain types of traffic.
Mike Hibler's avatar
Mike Hibler committed
41 42 43
<a NAME="Swapin"></a><h3>Swapin actions</h3>
Mike Hibler's avatar
Mike Hibler committed
When a firewalled experiment is swapped in, the firewall is setup and
Mike Hibler's avatar
Mike Hibler committed
45 46 47 48 49 50
activated before any experiment nodes are allowed to setup.  Currently
this adds approximately five more minutes to swapin time, as the firewall
node must first be loaded with the FBSD-IPFW2 image.
<a NAME="Swapout"></a><h3>Swapout actions</h3>
Mike Hibler's avatar
Mike Hibler committed
51 52
When a firewalled experiment is swapped out, extra precautions are taken
to ensure that the nodes are decontaminated before the firewall is taken
Mike Hibler's avatar
Mike Hibler committed
53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72
down.  The nodes, including the firewall itself, are scheduled to boot into
a network-loaded, memory-based version of FreeBSD (the "MFS") which will run
a script to zero all the known bootblocks on the disk.  This prevents a node
from accidentally booting from its disk prior to being reloaded.
The "boot to MFS" process is done not by issuing a reboot command or even by
doing a power cycle, but rather by simultaneously powering off all nodes
and then later powering them back on.  Reboot is not used since the reboot
command could be spoofed on a contaminated node.  Power cycling is not used
since cycling is normally performed in batches to avoid network (or power)
overload on restart.  Skewed reboots like this open a window of vulnerability
where nodes rebooted later might be able, before they are rebooted, to spoof
the reload server for nodes that have just rebooted.
So we first turn everyone off so that later batched power-ons are safe.
Only when all nodes are known to be running in the MFS and their disks
neutered, will the nodes be released to the reloading experiment.
Any failure in the above process results in all nodes for the experiment
being left powered off, with the control net disabled, and marked as "paniced"
so that only a testbed admin can swap it out.
73 74 75 76 77 78 79 80
<a NAME="Use"></a><h2>Use</h2>
To add a firewall to an Emulab experiment, you specify a Firewall object
in your NS file (there is currently no way to add a firewall via the Emulab
client or experiment creation GUI):
	set fw [new Firewall $ns]
	$fw set-type &lt;type&gt;
82 83 84
	$fw set-style &lt;style&gt;
This tells Emulab to add a node called "fw" to your experiment, and provides
85 86 87 88 89 90 91
a handle for adding rules to the firewall.
The "type" of the firewall should be <code>ipfw2-vlan</code>.
Another type, <code>ipfw</code>, <a href="#YeOleImpl">has been deprecated</a>
and should not be used.
In the future, there may be additional types for different firewall
implementations (e.g., Linux with ipchains).
The "style" of the firewall is one of:
Mike Hibler's avatar
Mike Hibler committed
<a NAME="Styles"></a>
94 95 96 97 98 99
<li><code>open</code>: A completely open firewall allowing all traffic.
This gives you a hook for setting up custom firewall rules (below).
<li><code>closed</code>: A closed firewall allowing no communication with
the outside world.  Nodes can still communicate in a limited fashion with
the Emulab infrastructure.
<li><code>basic</code>: A mostly closed firewall allowing only <code>ssh</code>
100 101
connections with the outside world, and <code>HTTP/HTTPS</code> connections
from inside (for Windows Update).
102 103 104 105 106
The firewall can be augmented with user-specified rules as well:
	$fw add-rule &lt;IPFW format rule string&gt;
In the basic form, rules are numbered starting at 100.  Thus, rules are
interpreted in the firewall in the order in which they appear in the NS file.
109 110 111 112
The default rules for a firewall style have numbers either less than 100
or greater than 60000, so
user-specified rules are interpreted ahead of all but the most important
default rules.
113 114 115 116 117 118 119 120 121
There is also a method allowing explicit numbering of rules:
	$fw add-numbered-rule &lt;ruleno&gt; &lt;IPFW format rule string&gt;
allowing more precise control over their interpretation and allowing for
linked rules.  A <a href="#Example">complete example</a> is shown below.
<a NAME="Limitations"></a><h2>Limitations</h2>
One should note carefully the following issues.
123 124 125 126
<li><b>The firewall is just another node in the topology</b>.
It is setup just like all other nodes, this includes enabling accounts
and NFS access.  Thus anyone who can login to a node, can login to the
127 128 129 130
firewall.  This is not as bad as it may sound since, in all but the open
firewall style, the firewall node rejects all packets sent to it from the
inside.  However, the nodes and the firewall do share common filesystems
in /users and /proj.
In the future, firewalls will have much more constrained access.
<li><b>The firewall must run FreeBSD with IPFW</b>.
133 134 135 136
However, this is not fundamental to the design and we hope that the
firewall syntax will be general enough to support at least Linux with
ipchains as well.
<li><b>The firewall rule syntax is fixed</b>.
137 138 139 140 141 142
In particular, it allows only fixed strings with no form of per-user
or per-experiment variable substitution.
There is a limited form of substitution used to plug in Emulab-wide
parameters such as the Utah control network subnet.
Since IPFW allows hostnames, you can at least
use symbolic host names in place of IP addresses in your rules.
143 144 145 146 147 148 149 150 151 152 153 154 155 156 157
<li><b>Firewall setup is static</b>.
Rules are specified in the NS file at experiment creation time.
Rules can be changed using experiment modify, or by logging into the firewall
and changing rules by hand with IPFW.
A more dynamic interface may be desirable,
in particular integration with the event system.
On the other hand, it is not necessarily
a good thing to have a firewall that is too easy to (mis)configure.
<li><b>There is always a firewall for experiments</b>.
Keep in mind that Emulab has an external firewall already which imposes
some site-wide restrictions (yes, there should be a link here...)
So just allowing some ports through your experimental firewall doesn't
guarantee that the affected traffic will make it to the outside world.
Mike Hibler's avatar
Mike Hibler committed
<a NAME="KnownBugs"></a><h2>Known Bugs</h2>
159 160
There is a mighty fine line between a "limitation" and a "bug".
161 162 163 164
But this one probably crosses the line:
<li><b>The firewall exposes more of the infrastructure than it should</b>.
The Emulab node self-configuration and monitoring infrastructure uses a
lot of different services on the boss and ops nodes.  At the current time,
166 167 168 169 170 171 172 173
we allow all those through.  Moreover, many of these, such as TFTP, cannot
be pinned down too precisely in firewall rules, so the rules are more open
than we would prefer.  Additionally, we also currently preserve the shared
filesystem access model inside firewalled experiments, which permits not
only attacks on NFS but allows for trojans to be placed in the filesystems.
For particularly anti-social applications, you should check out
<a href="elabinelab.php3">Emulab in Emulab</a>.
174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189
<a NAME="Example"></a><h2>A Complete Example</h2>
Here is the NS code for a simple two node experiment with a firewall:
	source tb_compat.tcl
	set ns [new Simulator]

	set n1 [$ns node]
	tb-set-node-os $n1 FBSD-STD
	set n2 [$ns node]
	tb-set-node-os $n2 RHL-STD
	set link [$ns duplex-link $n1 $n2 100Mb 0ms DropTail]

	# create a firewall node
	set fw [new Firewall $ns]
	$fw set-type ipfw2-vlan
191 192
	$fw set-style basic

193 194 195
	# allow traceroute through
	$fw add-rule "allow udp from EMULAB_CNET to any 33434-33524"
	$fw add-rule "allow udp from any 33434-33524 to EMULAB_CNET"
196 197 198

	$ns run
199 200 201 202 203
<code>EMULAB_CNET</code> is an example of the limited variable replacement
capability.  In the Utah case, it expands to "155.98.36/22".  Note that
even though this rule names the entire control net space of all experiments,
it effects only those nodes within this experiment since the firewall is
per-experiment.  This NS specification yields a topology that looks like:
204 205 206 207 208 209 210 211 212 213 214 215 216 217 218
<img src="firewall-experiment.png" alt="&lt;Three nodes, two connected&gt;">
No surprises.  Two nodes connected by a link, with a "disconnected" firewall
off to the side.
What isn't shown is that all three are connected via the control net.
The firewall is accessible via a DNS name of
similarly to
other nodes.  You can login to the firewall, reboot it, etc. just as any
other node.
The <code>Experiment Details</code> web page for the experiment lists
all the rules for the firewall:
Firewall information:
219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250
ID              Type       Style    Rule# Rule
--------------- ---------- -------- ----- -----------------------------------
fw              ipfw2-vlan basic        2 check-state
                                       10 allow all from me to me
                                       11 deny all from any to me via vlan0
                                       12 deny all from any to EMULAB_CNET via vlan0
                                       13 allow mac-type arp
                                       14 allow all from any to any frag
                                       50 allow udp from any to EMULAB_NS 53 keep-state
                                      100 allow udp from EMULAB_CNET to any 33434-33524
                                      101 allow udp from any 33434-33524 to EMULAB_CNET
                                    60000 allow tcp from any to any 22 setup keep-state
                                    60010 allow ip from any to ntp1,ntp2 123 keep-state
                                    60020 allow udp from any 514 to ops 514
                                    60030 allow ip from any to fs 111 keep-state
                                    60031 allow udp from any not 0-700 to fs keep-state
                                    60032 allow udp from any to fs 900 keep-state
                                    60033 allow udp from any to fs 2049 keep-state
                                    60040 allow tcp from any to boss 5999 setup keep-state
                                    60050 allow ip from any to ops 2917 keep-state
                                    60060 allow udp from any to boss 8509
                                    60080 allow udp from any to EMULAB_MCADDR
                                    60081 allow udp from boss EMULAB_MCPORT to any EMULAB_MCPORT
                                    60082 allow igmp from any to any
                                    60090 allow icmp from any to any
                                    61000 allow udp from any 68 to 67 recv vlan0
                                    61001 allow udp from any 67 to any 68 in not recv vlan0
                                    61010 allow udp from any to boss,ops 69 keep-state
                                    61011 allow udp from boss,ops not 0-1023 to any not 0-1023 keep-state
                                    61020 allow udp from any to boss 6969 keep-state
                                    61021 allow ip from any to boss 7777 keep-state
                                    65534 deny all from any to any
252 253 254 255 256 257 258 259 260 261
Again, notice the rules with <code>EMULAB_</code> variables.  <code>NS</code>
is the Emulab name server IP address, <code>CNET</code> the control network
subnet, <code>MCADDR</code> and <code>MCPORT</code>, the multicast addresses
and ports used by the Frisbee disk loader.  Other rules involving the hosts
<code>boss</code>, <code>ops</code>, <code>fs</code>, <code>ntp1</code>,
and <code>ntp2</code>, are Emulab infrastructure related.
<a NAME="YeOleImpl"></a><h2>The deprecated <code>ipfw</code> firewall</h2>
An earlier, less secure, firewall implementation did not require support
from the switching infrastructure.  This "software" firewall solution
263 264 265
still allocated an extra node to act as an IP firewall.
This node was then set as the default route for all other nodes in the
experiment.  Thus, all outgoing, non-experimental traffic was passed through
the firewall node.  Inbound traffic directed to the nodes did not pass through the firewall.
So in addition to <a href="#Limitations">the limitations above</a>
you can add the following for this deprecated version:
<li><b>The firewall was implemented using OS-provided routing</b>.
271 272
Specifically, every node has its default route changed to point to the
firewall node.  Sufficiently powerful applications could accidentally
or intentionally change the default route back to the Emulab router,
thus circumventing all protection.
<li><b>Intra-Emulab traffic was not firewalled</b>.
276 277 278 279 280 281 282 283
Traffic between nodes over the control net
(155.98.36.x addresses) is not filtered since the shared control net is a LAN
and all other nodes are directly reachable.
Additionally, traffic between the nodes and the Emulab
infrastructure (boss and ops) does not pass through the firewall.  Host
routes are explicitly setup to avoid the firewall.