The Emulab installation process requires very specific information about your network topology. This page will help you to make important design decisions so you can put them in your definitions file in the next section.
First a bit of background: Emulab utilizes two main types of networks; the control network and the experimental network. On the control network will be your control nodes (boss, ops and fs servers, tipservers, etc.), control hardware (SNMP-controllable devices such as power controllers and switch IP interfaces), your connection to the outside world, and the control interfaces of each of your nodes. The control network can be expanded beyond just one switch by using trunk lines (802.1q, or proprietary, like Cisco ISL).
The experimental network will also consist of one or more switches connected with trunk lines. Most of the configuration of these switches will be taken care of by our software.
A note about subnets
The Emulab software requires that the control network be its own subnet, distinct from your production networks. This is because Emulab runs its own DHCP server (which responds to dynamic requests) and because the nodes should not receive DHCP replies from any other DHCP server. In either case, confusion will result, and you might even annoy people when their desktop machines fail to boot because the Emulab DHCP server gave them bogus information.
Your uplink switch/router should also not have proxy arp enabled.
Splitting up the control network (Security)
This is something you will have to decide based on the needs of your site. Basically, everything on the control network can either be part of the same subnet/VLAN, or you can separate specific types of hardware (such as servers, power controllers, and nodes) from each other and place them in separate VLANs, for greater security.
No matter what you decide, you should at the very least set up a firewall between your testbed and the outside world. The benefits of splitting up the control network provide mostly internal security improvements (i.e., prevent malicious users from attacking your switches or other users from a running experiment). Because it is assumed that users are not malicious, we recommend placing the entire control network in the same VLAN, and naming it 'Control' for compatibility reasons.
If, however, you are interested in splitting up your control network, see the bottom of this page for hints on how we do it at Utah. We've put a lot of thought into this, so if you're wondering how your choice will affect other aspects of security, ask Utah, and we can probably help you.
Since our firewall rules change once in a while as we add new services, it's best to contact Utah and ask us for the current set.
Protecting SNMP on your switches
Emulab controls the testbed switches via SNMP. The default SNMP community string (i.e. the password) for most switches is very predictable and should be changed. This is especially important if you decided not to partition your Control network, thus potentially allowing nodes access to your switches. To change your community string in CatOS:
snmp-server community <string> rw
Make sure to remember this string, as you will need it later on.
Connecting the control network to the experimental network
In order to control the experimental switches (i.e., create new VLANs, etc.) you need a way to talk to them. So, you'll need one line that goes from your control network to your experimental network. However, you will need to be very careful that no experimental traffic can leak across this line, and that it cannot be used as a 'back door' by the nodes into the control network. If you have divided your control network, this link should be placed in the 'control-hardware' VLAN on the control switch.
On the experimental switch, it should be the same VLAN as the switch's IP interface (if you are running CatOS; for IOS just make sure the vlan has a valid IP address as there is no "switch interface"). The best thing to do is create a new VLAN, and move the switch's IP interface into it (CatOS). For CatOS only, you accomplish this with:
set interface sc0 <vlan #>
You will also need to give the experimental switch(es) IP addresses. With CatOS, you do it like this:
set interface sc0 <IP> <netmask> set interface sc0 up
or on IOS:
interface vlan <vlan #> ip address <IP> <netmask> no shutdown
As an alternative (when security is not a concern), you may also directly connect boss to both the control and experimental switches. On the experimental switch, you would put boss' port into the 'control-hardware' vlan, and this vlan would exist only on the experimental switch, not across both.
Basic Emulab Topology
Okay, lets look at a simple example of what all this means:
Salient points about the above topology:
- All nodes and servers are on a single routeable (public) subnet (184.108.40.206). All of these control ports should be in the "default" vlan (typically Vlan 1) on your control switch and the ports enabled. These are the black colored lines (wires). This setup allows all of the nodes and servers to access the public internet without having to use NAT, but a completely private testbed is setup similarly.
- The red wires comprise the "control-hardware" vlan and is attached to boss using a secondary Ethernet interface. This vlan is used to control the switches via snmp and is intended to be private to boss; only processes on boss can access this vlan and thus the switches. To achieve this, the ports on both the control and experiment switch should be placed into a new vlan called "control-hardware" (typically we use number 10).
- The blue wires comprise the "experimental" network. These connect the experiment switch to all of the additional Ethernet interfaces on your PCs. As with the control switch, all of these ports should be in the default vlan and enabled.
- You will need to configure IP on the control-hardware Vlan. For example, you should statically configure the ethernet interface on boss as 10.10.10.1/24, Vlan 10 on the control switch as 10.10.10.2/24 and Vlan 10 on the experiment switch as 10.10.10.3/24.
A minor variation on the above topology is to connect each switch directly to boss, but that requires more interfaces on boss, which you might not have. But as above, all the ports should be in the control-hardware vlan.
Using a Single switch for both Control and Experimental
Some Emulab sites have just a single switch and use it for both the control and experiment functions. While we do not really recommend this arrangement, it does work. However, instead of putting all of the control ports into the default vlan, you will want to create a distinct vlan called "Control" (say, vlan 2), and place all of the control ports in it. This will prevent traffic leaking from the experiment ports to the control vlan.
IGMP snooping on the control network
In order for multicast to work correctly, you need to make sure that IGMP snooping is enabled on the control switch. This is needed for frisbee, our disk-loading system. It's up to you whether you want to enable this on the experimental switches. In general, we recommend it so that your experimenters can use multicast, but it does seem that unexpected or malformed multicast packets have an easier time DOSing the switch control processor than unicast traffic. On CatOS, the command is:
set igmp enable
or on IOS:
ip igmp snooping
Multicast routing on boss
If your boss is multi-homed, your multicast traffic from boss may be routed out the wrong interface. You will need to create an explicit route to ensure it goes out the control network. Run:
route get 220.127.116.11
on boss and see which interface it tells you it would go out. If it isn't the control net interface, then you will have to add a route:
route add -net 234/8 -interface XXX
where XXX is whatever the control net interface is called.
Our suggestion is to leave VTP off on all switches in your testbed, unless you have a non-Emulab reason to want it. Emulab software (snmpit) takes care of getting the right VLANs created on the right switches. Just make sure that when you create the entry in the switch_stack_types database table (discussed later in the install guide), to set the single_domain column to 0. To turn VTP off, you can do this in CatOS with:
set vtp mode transparent
or on IOS:
vtp mode transparent
On some Cisco switches, it might be:
vtp mode off
If you have multiple experimental switches connected by trunk lines, and you decide/need to use VTP, pick a domain name (we call ours simply 'Testbed'), and run the following on all of your switches (CatOS):
set vtp domain <domainname>
or on IOS:
vtp domain <domainname>
Pick one switch to be the master - it doesn't really matter which one. (See the Initial Switch Setup instructions in the database chapter and make sure you name the stack after the master switch.) On this one, run (CatOS; for IOS run the same commands omitting the word "set"):
set vtp mode server
And on all the others:
set vtp mode client
Dividing your Control Network (not required)
At Utah, we have a more complicated arrangement that divides the control network into different segments. As mentioned earlier, the reason for doing this is to provide more security; we have students and researchers from all over the world, most of whom we do not know directly. While we do not assume malicious users, this is always a possibility to be considered. In addition, we have found that new users tend to do things purely by accident, that can have a negative impact on other users. By using separate VLANS, we can route (using an 'L3 switching' module in our control network switch) between these VLANs, and do some fire walling between each of them. You could also do the routing with a real router or PC. Okay, lets look at a picture to make this more concrete:
The 5 VLANs on the control network:
- External (green): Contains our connection to the public (outside) world.
- Private (purple): Our boss node and our tape backup server.
- Public (black): Our ops (users) node and the console tip servers.
- Control (brownish): All of the control network interfaces of the experimental nodes.
- Control-Hardware (red): All our IP-controllable devices (namely, power controllers and switch IP interfaces, as well as a second interface on the boss node). This VLAN uses private IP addresses, and does not contain a router interface.
Using this method provides the following levels of security:
- Protect both control and experimental nodes from the outside world (and vice-versa - we don't want people attacking the outside world from our nodes).
- Protect the control nodes from the experimental nodes.
- Protect the control hardware (power controllers, switches, etc.) from nodes and the outside world.
- Protect the boss node (which is not publicly accessible) from the ops node (which all experimenters have shells on).
Without dividing up the control network, #1 (closed) can be achieved with a firewall between the testbed and the outside world. By satisfying #2 (closed) and #3 (closed), experimenters and testbed infrastructure are protected by other experimenters, who potentially have root access on their machines. #4 (closed) is probably the least important part to worry about. As noted earlier, you should contact Utah for the current firewall rule set. For compatibility reasons, you should probably keep any extra VLANs named the same as we do, for ease of communication between testbed admins if nothing else.
If you divide your control network into separate VLANs, you will need to make sure that DHCP traffic can get from the control network to your boss node, since normally DHCP is not forwarded through routers. On Cisco routers, this is done with the 'ip helper-address'. For example, here, the name of the router's interface in the node control network is 'Vlan3'. So, you would log into the router and run the following (CatOS):
configure terminal interface Vlan3 ip helper-address 18.104.22.168
Of course, replace 'Vlan3' with the name of your router's node control-net interface, and replace the IP address with that of your boss node.