Firewalled Experiment Support in Emulab

Contents


Overview

Emulab allows the ability to setup a control-network firewall between an experiment and the outside world. This is not a firewall between nodes within an experiment, that is, the firewall is not part of your NS-specified network topology. The purpose of an Emulab firewall is to prevent experimental traffic from escaping, via the control network, to the Internet due to a mis-configured application or router within an experiment. Control net firewalls are also a key component of Emulab ``high-security'' experiment environments.

The firewall is implemented by allocating an additional node to the experiment. Through switch-enforced VLANs, the experiment is given its own private control network connecting all nodes in the experiment. Network traffic between experiment nodes and any hosts outside the experiment (Emulab infrastructure or Internet hosts in general) must pass through the firewall node. The firewall is setup as a filtering layer2 bridge using IPFW2 on FreeBSD and can be configured in a number of ways to block or allow certain types of traffic.

Swapin actions

When a firewalled experiment is swapped in, the firewall is setup and activated before any experiment nodes are allowed to setup. Currently this adds approximately five more minutes to swapin time, as the firewall node must first be loaded with the FBSD-IPFW2 disk image.

Swapout actions

When a firewalled experiment is swapped out, extra precautions are taken to ensure that the nodes are decontaminated before the firewall is taken down. The nodes, including the firewall itself, are scheduled to boot into a network-loaded, memory-based version of FreeBSD (the "MFS") which will run a script to zero all the known bootblocks on the disk. This prevents a node from accidentally booting from its disk prior to being reloaded.

The "boot to MFS" process is done not by issuing a reboot command or even by doing a power cycle, but rather by simultaneously powering off all nodes and then later powering them back on. Reboot is not used since the reboot command could be spoofed on a contaminated node. Power cycling is not used since cycling is normally performed in batches to avoid network (or power) overload on restart. Skewed reboots like this open a window of vulnerability where nodes rebooted later might be able, before they are rebooted, to spoof the reload server for nodes that have just rebooted. So we first turn everyone off so that later batched power-ons are safe.

Only when all nodes are known to be running in the MFS and their disks neutered, will the nodes be released to the reloading experiment. Any failure in the above process results in all nodes for the experiment being left powered off, with the control net disabled, and marked as "paniced" so that only a testbed admin can swap it out.

Use

To add a firewall to an Emulab experiment, you specify a Firewall object in your NS file (there is currently no way to add a firewall via the Emulab client or experiment creation GUI):

	set fw [new Firewall $ns]
	$fw set-type <type>
	$fw set-style <style>
	
This tells Emulab to add a node called "fw" to your experiment, and provides a handle for adding rules to the firewall. The "type" of the firewall should be ipfw2-vlan. Another type, ipfw, has been deprecated and should not be used. In the future, there may be additional types for different firewall implementations (e.g., Linux with ipchains). The "style" of the firewall is one of: The firewall can be augmented with user-specified rules as well:
	$fw add-rule <IPFW format rule string>
	
In the basic form, rules are numbered starting at 100. Thus, rules are interpreted in the firewall in the order in which they appear in the NS file. The default rules for a firewall style have numbers either less than 100 or greater than 60000, so user-specified rules are interpreted ahead of all but the most important default rules. There is also a method allowing explicit numbering of rules:
	$fw add-numbered-rule <ruleno> <IPFW format rule string>
	
allowing more precise control over their interpretation and allowing for linked rules. A complete example is shown below.

It should be noted that user-specified rules will only apply to IP traffic passing through the firewall. Non-IP traffic is not allowed through the firewall, and traffic to the firewall node itself is controlled by a different set of rules which cannot be overridden with this mechanism. For example, in a closed firewall that does not allow ssh, you can add a rule to allow ssh access from a particular machine to one of the firewalled nodes, but not to allow access to the firewall itself.

Limitations

One should note carefully the following issues.

Known Bugs

There is a mighty fine line between a "limitation" and a "bug". But this one probably crosses the line:

For particularly anti-social applications, you should check out Emulab in Emulab.

A Complete Example

Here is the NS code for a simple two node experiment with a firewall:

	source tb_compat.tcl
	set ns [new Simulator]

	set n1 [$ns node]
	tb-set-node-os $n1 FBSD-STD
	set n2 [$ns node]
	tb-set-node-os $n2 RHL-STD
	set link [$ns duplex-link $n1 $n2 100Mb 0ms DropTail]

	# create a firewall node
	set fw [new Firewall $ns]
	$fw set-type ipfw2-vlan
	$fw set-style closed

	# allow traceroute through
	$fw add-rule "allow udp from EMULAB_CNET to any 33434-33524"
	$fw add-rule "allow udp from any 33434-33524 to EMULAB_CNET"

	# and ssh sessions from "users" to one of the nodes
	$fw add-rule "allow tcp from users.emulab.net to n1 setup keep-state"

	$ns run
	
EMULAB_CNET is an example of the limited variable replacement capability. In the Utah case, it expands to "155.98.36/22". Note that even though this rule names the entire control net space of all experiments, it effects only those nodes within this experiment since the firewall is per-experiment. The last add-rule invocation demonstrates using a symbolic node name, "n1", to reference the node identified by the object "$n1" in the NS file. This rule also takes advantage of the dynamic rule capability in IPFW2. The rule itself only allows through TCP packets in the "setup" state (SYN but no ACK), but instructs IPFW to construct a limited lifetime dynamic rule allowing bidirectional traffic between the source and destination IP addresses and TCP ports from the packet.

This NS specification yields a topology that looks like:
<Three nodes, two connected>
No surprises. Two nodes connected by a link, with a "disconnected" firewall off to the side. What isn't shown is that all three are connected via the control net. The firewall is accessible via a DNS name of fw.experiment.project.emulab.net similarly to other nodes. You can login to the firewall (if allowed by the rules), reboot it, etc. just as any other node. The Experiment Details web page for the experiment lists all the rules for the firewall:

Firewall information:
ID              Type       Style    Rule# Rule
--------------- ---------- -------- ----- -----------------------------------
fw              ipfw2-vlan closed       4 check-state
                                        9 skipto 80 all from any to any layer2 in
                                       10 deny all from any to me via vlan0
                                       11 allow all from me to me
                                       20 allow udp from me to EMULAB_NS 53 keep-state
                                       22 allow tcp from boss to me 22 setup keep-state
                                       24 allow ip from me to ntp1,ntp2 123 keep-state
                                       26 allow udp from me 514 to ops 514
                                       30 allow ip from me to fs 111 keep-state
                                       31 allow udp from me not 0-700 to fs keep-state
                                       32 allow udp from me to fs 900 keep-state
                                       33 allow udp from me to fs 2049 keep-state
                                       34 allow ip from me to fs frag
                                       35 allow ip from fs to me frag
                                       36 allow tcp from me to boss 5999 setup keep-state
                                       38 allow ip from me to ops 2917 keep-state
                                       40 allow udp from me to boss 8509
                                       50 allow icmp from boss to me icmptypes 6,8
                                       51 allow icmp from me to boss icmptypes 0
                                       70 allow ip from me to boss 7777 keep-state
                                       79 deny all from any to any 
                                       80 deny not mac-type ip
                                       83 allow ip from EMULAB_GWIP to any in not via vlan0
                                       84 deny ip from any to EMULAB_CNET in via vlan0
                                       85 deny ip from EMULAB_CNET to any in not via vlan0
                                       88 deny ip from not 0.0.0.0,255.255.255.255,EMULAB_CNET to any in via vlan0
                                      100 allow udp from EMULAB_CNET to any 33434-33524
                                      101 allow udp from any 33434-33524 to EMULAB_CNET
                                      102 allow tcp from users.emulab.net to n1 setup keep-state
                                    60020 allow udp from any to EMULAB_NS 53 keep-state
                                    60022 allow tcp from boss to any 22 setup keep-state
                                    60024 allow ip from any to ntp1,ntp2 123 keep-state
                                    60026 allow udp from any 514 to ops 514
                                    60030 allow ip from any to fs 111 keep-state
                                    60031 allow udp from any not 0-700 to fs keep-state
                                    60032 allow udp from any to fs 900 keep-state
                                    60033 allow udp from any to fs 2049 keep-state
                                    60034 allow ip from any to fs frag
                                    60035 allow ip from fs to any frag
                                    60036 allow tcp from any to boss 5999 setup keep-state
                                    60038 allow ip from any to ops 2917 keep-state
                                    60040 allow udp from any to boss 8509
                                    60046 allow udp from any to EMULAB_MCADDR EMULAB_MCPORT in via vlan0
                                    60047 allow udp from boss EMULAB_MCPORT to any EMULAB_MCPORT
                                    60048 allow igmp from any to any
                                    60050 allow icmp from boss to any icmptypes 6,8
                                    60051 allow icmp from any to boss icmptypes 0
                                    60064 allow udp from any 68 to 255.255.255.255 67 recv vlan0
                                    60065 allow udp from any 67 to any 68 in not recv vlan0
                                    60066 allow udp from any to boss,ops 69 keep-state
                                    60067 allow udp from boss,ops not 0-1023 to any not 0-1023 keep-state
                                    60068 allow udp from any 9696 to boss 6969 keep-state
                                    60069 allow udp from boss 6970 to any 9696
                                    60070 allow ip from any to boss 7777 keep-state
                                    65534 deny all from any to any 
The rules from 10 to 79 are for the firewall itself, from 80 to 65534 are for bridged ("layer2") packets, i.e., those that pass through the firewall. As user rules start at 100, Again, notice the rules with EMULAB_ variables. NS is the Emulab name server IP address, CNET the control network subnet, MCADDR and MCPORT, the multicast addresses and ports used by the Frisbee disk loader. Other rules involving the hosts boss, ops, fs, ntp1, and ntp2, are Emulab infrastructure related.

The deprecated ipfw firewall

An earlier, less secure, firewall implementation did not require support from the switching infrastructure. This "software" firewall solution still allocated an extra node to act as an IP firewall. This node was then set as the default route for all other nodes in the experiment. Thus, all outgoing, non-experimental traffic was passed through the firewall node. Inbound traffic directed to the nodes did not pass through the firewall. So in addition to the limitations above you can add the following for this deprecated version: