setup-network.txt 7.16 KB
Newer Older
1 2
#
# EMULAB-COPYRIGHT
Mike Hibler's avatar
Mike Hibler committed
3
# Copyright (c) 2002-2005, 2007 University of Utah and the Flux Group.
4 5 6
# All rights reserved.
#

7
#####
Robert Ricci's avatar
Robert Ricci committed
8
##### Laying Out Your Network
9
##### 
10 11 12 13

The purpose of this document is to aid in designing and setting up the control
and experimental networks on other emulabs.

14
First a bit of background: On the control network will be your control
15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
nodes (boss, ops, any tipservers, etc.), control hardware (SNMP-controllable
devices such as power controllers and switch IP interfaces), your connection to
the outside world, and the control interfaces of your nodes. So far, we haven't
tried to distribute the control net across multiple switches, but this should
be theorically possible. You'll need to set up the VLANs, etc. on the control
net by hand.

The experimental network will consist of one or more (this, we have tested)
switches connected with trunk (802.1q, or proprietary, like Cisco ISL) lines.
Most configuration of these switches will be taken care of by our software.
(See setup-cisco.txt for some configruation options you many want to apply to
these switches if they are Ciscos.)

##### Splitting up the control net

30
We basically have 5 VLANs on the control network:
31 32 33 34 35
'External' contains our connection to the outside world
'Private' contains the boss node, and our tape backup server
'Public' contains our ops node
'Control' contains the control net interfaces of all experimental nodes
'Control-hardware' contains all IP-controllable devices (namely, power
36 37 38
	controllers and switch IP interfaces, as well as a second
	interface on the boss node. This VLAN uses private IP addresses,
	and does NOT contain a router interface.
39

40 41 42
This is done for security - we route (using an 'L3 switching' module in our
control-net switch) between these VLANs, and do some firewalling between each
of them. You could also do the routing with a real router or a PC. The main
43 44 45 46 47 48
goals are:
1) Protect both control and experimental nodes from the outside world (and
   vice-versa - we don't want people attacking the outside world from our nodes)
2) Protect the control nodes from the experimental nodes
3) Protect the control hardware (power controllers, etc.) from nodes
   and the outside world
49
4) Protect the boss node (which is _not_ publicly accessible) from the ops
50 51 52 53 54 55 56 57 58 59 60 61
   node (which all experimenters have shells on.)

Now, it's entirely possible to combine these VLANs into one big one - this is
what we've done on our mini-testbed here. But, there are some serious security
implications with doing it this way - namely, that the nodes can theoretically
impersonate each other, power cycle each other, and all kinds of nasty things.

At the very least, you should have a firewall between your testbed and the
outside world, to satisfy #1.

It is also a good idea to separate the nodes' control net into a separate VLAN,
which satisfies #2 and #3. After all, you are giving people root access to the
62
experimental nodes. In situations where you are only giving access to a small
63 64 65 66 67 68 69 70 71 72 73 74
number of trusted people, this is probably not too big a deal, but once access
gets outside the small circle of your friends, or if you are allowing students
access, then taking these precautions are a very good idea.

If possible, putting boss and ops on separate networks is also a good idea
(#4), but is probably the least important part to worry about.

Basically, it's up to you to decide how much security you want to worry about.
I'd recommend at least going after #1, #2, and #3. We've put a lot of thought
into this, so if you're wondering how your choice here will affect other
aspects of security, ask Utah, and we can probably help you.

75 76
Since our firewall rules change frequently as we add new services, it's best to
contact Utah and ask us for the current set.
77

78 79 80 81
Note: If you plan to use our control-network firewalling code, you should make
sure to name the control network 'Control' (case sensitive) so that our code
can find it.

82 83 84 85 86 87 88
Note: For compatibility with Emulab's current control-network firewalling
code, and possible future improvements such as inter-experiment control
network isolation, you should make sure to name the control network 
'Control' (case sensitive).  In fact, we recommend keeping all 5 VLANs
named as we do, for ease of communication among testbed admins,
if nothing else.

Mike Hibler's avatar
Mike Hibler committed
89
##### Connecting the control net to the experimental net
90 91 92 93 94 95 96

In order to be able to control the experimental switches (ie. create new VLANs,
etc.) you need a way to talk to them. So, you'll need one line that goes from
your control net to the experimental net. However, you will need to be _very_
careful that no experimental traffic can leak across this line, and that it
cannot be used as a 'back door' into the private net by nodes.

97
On the control switch, it should be put into the 'control-hardware' VLAN.
98 99

On the experimental switch, it should be in the same VLAN as the switch's IP
100 101 102
interface. The best thing to do is create a new VLAN, and move the switch's IP
interface into it. On CatOS, you accomplish this with:
  set interface sc0 <vlan #>
103

104 105 106 107 108
You will also need to give the experimental switch(es) IP addresses. With
CatOS, you do it like this:
  set interface sc0 <IP> <netmask>
  set interface sc0 up

109 110
##### DHCP through the router

111
If your boss node is on a separate VLAN from the node control net, you'll need
112 113 114
to make sure that DHCP traffic can get from the control net to your boss node,
since normally, DHCP is not forwarded through routers. On Cisco routers, this
is done with the 'ip helper-address'. For example, here, the name of the
115
router's interface in the node control net is 'Vlan3'.  So, I'd log into the
116 117 118 119 120 121
router, and run the following:                                 
  configure terminal                                                      
  interface Vlan3                                                         
  ip helper-address 155.101.128.70                                   

Of course, replace 'Vlan3' with the name of your router's node
122 123
control-net interface, and replace the IP address with that of your boss node.

124 125 126 127
##### IGMP snooping on the control net

In order for multicast to work correctly, you need to make sure that IGMP
snooping is enabled on the control switch. This is needed for frisbee, our
128
disk-loading system. It's up to you whether you want to enable this on the
129 130 131 132
experimental switches.  In general, we recommend it so that your experimenters
can use multicast, but it does seem that unexpected or malformed multicast
packets have an easier time DOSing the switch control processor than unicast
traffic.  On CatOS, the command is:
133 134 135 136 137 138 139 140 141 142
  set igmp enable

#### VTP domains

If you have only one Cisco experimental-net switch, you don't need to have it
using VTP, which is a mechanism for keeping VLANs in sync across multiple
switches. You do this (in CatOS) with:
  set vtp mode transparent

If you have multiple experimental switches connected by trunk lines, you should
143
use VTP. Pick a domain name (we call ours simply 'Testbed',) and run the
144 145 146 147
following on all of your switches:
  set vtp domain <domainname>

Pick one switch to be the master - it doesn't really matter which one. (See the
148
switch setup instructions in setup-db.txt, and make sure you name the stack
149 150 151 152
after the master switch.) On this one, run
  set vtp mode server
On all the others, run
  set vtp mode client