Commit 30e72494 authored by Grant Ayers's avatar Grant Ayers

Moved old setup documentation to setup-archive.

parent a563f099
# Copyright (c) 2002-2008 University of Utah and the Flux Group.
# All rights reserved.
The Emulab installation documentation has been moved to the main Emulab
All information contained in this directory should be considered deprecated
and for reference only.
This diff is collapsed.
This file documents the process of adding a new node to the testbed.
A. Information about the node
1. MAC address. For each port in the new node you need to find out the
MAC address, and which port (eth0/fxp0) it is in software. You need
to know both the Linux (eth) and BSD (fxp,xl,dc,etc.) names.
2. Wiring. We need to know which physical port on the back of the
machine maps to eth0, eth1, etc., and where each port is connected
to the cisco (get module/port, ie 3/21).
3. Power. Plug it into a power controller, and make note of which one
it is (name or IP) and which port you plug it into (1-8).
4. Serial line(s). When you plug in the serial lines, make sure which
ports on the serial expander they are plugged into.
B. If the node is of a new type
1. You'll need specs on the nodes for the node_types and node_type_attributes
tables. For node_types, you'll just need a name for the type:
insert into `node_types` (class,type) values ("pc", "pc2800d");
For node_type_attributes you will create a row for each of several
attributes. For this, you will need a name for the processor class
(e.g. Core Duo), speed (MHz), RAM size (in MB), boot hard disk type and unit
('ad' and '0' for IDE, 'da' and '0' for SCSI, 'ad' and '4' for SATA),
boot disk size (in GB), max # of physical cards it holds (including the
motherboard as a card if it has built-in ethernet), and the approximate
amount of time it takes the machine to "power cycle" (in seconds).
You'll also need to give it a default OS id (the default OS to boot)
and image ID (disk image the default OS comes from), which port is the
control net (e.g. 4) and what its Linux name is (e.g. "eth4"), how many
links this node can delay (usually: num_experimental_links / 2), and
how many virtual nodes ("jails") the machine can support.
insert into `node_type_attributes` values
('pc2800d','processor','Pentium D','string'),
There are also assorted other attributes you need not change, just
use these:
insert into `node_type_attributes` values
2. There are several scripts that limit searches to certain classes.
If the new type you have added does not have class "pc", you may need
to include this new class as appropriate.
Some of the scripts that might need to be updated are:
C. What to do on boss:
1. Insert entries into interfaces table using info from A(1). Try:
insert into interfaces (node_id,card,port,MAC,IP,interface_type,iface)
2. Insert entries into wires table using info from A(2). Try:
insert into wires (node_id1,card1,port1,node_id2,card2,port2) values
For the control interface do:
insert into wires (type,node_id1,card1,port1,node_id2,card2,port2)
values ("Control","pcN",0,1,"ciscoX",5,1)
Check to make sure your cards and ports match up with what you
entered in the interfaces table.
3. Insert entry into outlets table, using info from A(3). Try:
insert into outlets (node_id, power_id, outlet) values
4. Add entries to the nodes table for each node. Try:
insert into nodes (node_id,type,phys_nodeid,role,def_boot_osid,priority,op_mode)
P (priority) is the where it gets printed out. These need to be
ascending numbers, and in the right region. See the table.
4a. Add entries into the tiplines table. The "server" field is where
the actual capture process runs:
INSERT INTO tiplines VALUES ('pc1','pc1','',0,0,'');
INSERT INTO tiplines VALUES ('pc111','pc111','',0,0,'');
You need to add the usual lines in /etc/remote on the machine
where the capture process runs. In addition, add a line on users
to that users can use tip to connect to a console on a remote tip
server. So, on users:
The device field is ignored, but something must be there.
5. Until you are ready to put it in service, reserve it to an expt,
either with nalloc or by adding an entry to the reserved table
directly. You'll probably also want to put its ports in a vlan to
enable them.
6. Add the node to the system files:
- DNS: on boss, cd /etc/namedb
co -l
add these lines with all the others:
pcN IN A 155.101.132.N
IN MX 10 ops
IN MX 20
ci -u
cd reverse/
co -l 155.101.132.db
in 155.101.132.db, make these changes:
update serial number on line 10
add entry for node, like this:
ci -u 155.101.132.db
run /usr/testbed/sbin/named_setup to update.
- DHCP: on boss, cd /usr/local/etc/
if you added a new node type, then you need to add a line
of the form:
(where <type> is the new type is called) to dhcpd.conf.template.
Then as root run:
dhcpd_makeconf dhcpd.conf.template > Ndhcpd.conf
you can diff dhcpd.conf with the new file to verify nothing
catostrophic happened. Finally:
sudo cp Ndhcpd.conf dhcpd.conf
sudo /usr/local/etc/rc.d/ restart
- tip: on ops or tipserv1, edit /etc/remote
add a line like this:
pcN-tty:dv=/dev/cua<port #>:br#115200:nt:pa=none:
then do these:
sudo touch /var/log/tiplogs/pcN.log
sudo touch /var/log/tiplogs/
- capture: on ops or tipserv1, edit /usr/site/etc/capture.rc:
add a line like this:
/usr/site/bin/capture -r -s 115200 pcN tty<port#> >/dev/null 2>&1 &
D. How to get the first image on it:
1. If everything is set up right, you can use the magic PXE Flash Floppy
to put the right thing on the PXE card. Edit the BIOS to put the
boot order to Floppy, PXE, Hard Drive, then reboot it.
2. If everything goes right, you should see it PXE boot and find its
DHCP info, then contact the ProxyDHCP server to get its bootinfo
data, then it should decide according to that what to boot.
3. If process 3 went okay to that point, do an os_load to try to
install the standard testbed images for the node.
4. If it doesn't seem to be working just like the others, talk to
Leigh and Mike.
E. What next
1. Test it out and see if it works well enough to put into service. If
its ready, release it into the wild with nfree or by deleting its
entry in the reserved table.
2. Do some more tests to find any obvious problems. Fix them, if any.
3. Sit back and relax for a few minutes until the bug reports start
flowing in.
# Copyright (c) 2002-2006 University of Utah and the Flux Group.
# All rights reserved.
##### Configuration suggestions for Cisco switches
This file contains some configuration guidelines that we (Utah) have found
useful to improve the performance of our Cisco switches.
All commands given are to be typed at the (enable) prompt on your cisco
switches. They are for CatOS - switches that run IOS may not have these
<ports> means a list of ports, which on the CatOS command line, can include
lists and rages, such as "3/1,3/2" or "3/1-48" or "3/1-48,4/1-48,5/1-48"
##### Allowing ports to come up quicker
This one is useful on both the experimental and control nets:
set spantree portfast <ports> enable
Use this on all ports that are directly connected to nodes, servers, power
controllers - anything that is not another switch. Normally, the switch waits
a while (several seconds) when a port first comes up before forwarding traffic
from this port - it does so to prevent loops in the switch topology. The main
place you will see the benefit of this is on the control net - with portfast
disabled, the first few DHCP packets sent by booting nodes will get dropped,
causing the DHCP to take much longer than necessary.
##### Reducing stray traffic
Disable spanning tree (STP.) If on, STP sends out packets approximately every
two seconds on every port. You can disable it on all VLANs with the command
set spantree disable all
There are two major consequences (for our purposes) of disabling STP:
1) You cannot have _any_ loops in your switch topology, or bad things will
2) VLAN pruning on trunks won't work, causing broadcast traffic to be
forwarded across trunks that it does not need to cross. We've added
features to snmpit to manually do STP's job in this case, so this
problem is taken care of.
You must have STP disabled on _all_ switches that are trunked together! If it
is enabled on even one, STP traffic will be seen on all of them.
The switch doesn't trust you to use portfast responsibly. So, it has a
'bpdu-guard' feature that helps guard against loops. Turn off this feature
with the command:
set spantree portfast bpdu-guard disable
Cisco uses a protocol called 'CDP' to discover other Cisco devices. This sends
out small packets every two minutes. You can disable it with:
set cdp disable <ports>
Ideally, you should only disable CDP on ports that don't have other Cisco
devices attached, but in practice, running with CDP disabled on all ports is
Switch ports will, by default, try to negotiate trunking and channeling.
Cisco provides a handy macro:
set port host <ports>
to disable both of these. Also enables portfast on the ports.
##### Setting MAC address aging time
We have found that some experimenters use applications, kernels, etc. that only
receive traffic, not send it. This presents a problem, because it prevents the
switch from learning which port the node is on, and thus broadcasting traffic
for it to every port in the VLAN. This can be solved by 'priming' - ie. having
the receive-only node send some traffic (like an ARP response) at the beginning
of the experiment. However, the default aging time of 300 seconds makes this
impractical. So, we have disabled this aging, making learned MACs permanent
(until the VLAN is torn down.)
You must do this for each VLAN, with the command:
set cam agingtime <vlan> 0
For convenience, we've supplied a file (in this directory) called
'no-cam-aging.cfg' that disables aging on VLANs 2-999 (the ones potentially
used by our software.) Transfer this file to the switch using the:
copy tftp config
We also suggest that you do this on your control network as well - part of the
booting process leaves the nodes sitting dormant at a boot loader for extended
periods of time, so the switch will tend to forget their MACs. Turning off
aging is not critical, but we suggest it, because it will reduce stray traffic
while the switch re-learns MAC addresses.
##### Setting up multicast between multiple switches
If you have more than one switch on the experimental or control networks, you
may need to do a little setup to get multicast between them. The symptom of
this problem is that multicast doesn't work between two nodes on different
switches, and if you run 'show multicast groups' on each switch, some will show
the group as existing, and others will not.
Run the following command for both sides (ie. on both switches) of every trunk
set multicast router 1/1
(assuming that port 1/1 is your trunk link). If you are using EtherChannel to
bond together multiple links to form a single trunk, you only need to run this
command for the first port in the channel.
We had some problems running this command on the trunk on one of our switches:
it failed with the error:
Failed to add port 2/1 to multicast router port list.
What I finally did to resolve this was to tear down the trunk link and
EtherChannel that port was a part of, run the command on it (which succeeded
this time), and then build the EtherChannel and trunk back up.
##### Setting the clock
Since bos is an NTP server, you should set your switches to sync time with it.
On CatOS, this is accomplised with:
set ntp server
set ntp timezone MST -7
set ntp summertime MDT
set ntp summertime enable
set ntp summertime recurring
set ntp client enable
show time
Of course, you'll need to replace with the IP address your boss node
uses to talk to the switches (usually its control-hardware interface), and
'MST', -7, and 'MDT' with the names of your timezone and its offset from GMT.
If you don't use daylight-savings time, leave out the 'summertime' steps, and
instead do:
set ntp summertime enable
Watch the output of 'show time' for a while to make sure the clock syncs up.
It may take a few minutes.
On IOS, these commands are:
configure terminal
ntp server
clock timezone MST -7
clock summer-time MDT recurring
... and to see the current time, run 'show clock'
##### IOS commands
The above commands are given under the assummption that your switches are
running CatOS. If you are running IOS, here are a few notes that may help you
'translate' the above commands.
Interfaces in CatOS are named as module/port, while interfaces in IOS are named
as TypeModule/Port - For example, if module 1 has gigabit interfaces, what you
call 1/1 in CatOS is Gi1/1 in IOS. 100Mbit Ethernet is 'Fa'. (Really, these are
'GigabitEthernet' and 'FastEthernet' respectively, but you can abbreviate them.)
In order to operate on many interfaces at once, you can issue configuration
commands like this:
range gi1/1 - 48, gi2/1 - 48, gi3/1 - 48
... which would configure all 48 Gigabit interfaces on modules 1, 2, and 3.
The equivalent of 'set port host' (which sets portfast, disabled BPDU guard,
etc.) is:
switchport host
... applied to an interface or a range of interfaces. As in:
interface range gi1/1 - 48, gi2/1 - 48, gi3/1 - 48
switchport host
In order to disable spanning tree, you would use:
no spanning-tree vlan 1-1005
In order to create a VLAN and set its name:
vlan 10
name control-hardware
In order to set the IP address of the interface in VLAN 10:
interface vlan 10
ip address
In order to enable an interface:
interface vlan10
no shutdown
In order to remove a VLAN:
no vlan 1000
To put an interface into a VLAN:
interface gi0/1
switchport access vlan 10
In order to turn on trunking for an interface:
interface gi0/1
switchport mode trunk
In order to turn off trunking for an interface:
interface gi0/1
switchport mode access
In order to put interfaces into an EtherChannel:
interface range gi1/41 - 48
channel-group 1 mode on
(Notes: If you want to make more than one channel, give each set of ports
a different channel number. And, now, you will configure the whole channel
as 'interface port-channel 1')
To set the native VLAN on a trunk:
interface gi0/1
switchport trunk native vlan 1
To set the read-write SNMP community string to 'public':
snmp-server community public rw
To globally disable the Cisco Router Discovery (cdp) protocol:
no cdp run
This diff is collapsed.
This diff is collapsed.
# Copyright (c) 2002-2005, 2007 University of Utah and the Flux Group.
# All rights reserved.
##### Laying Out Your Network
The purpose of this document is to aid in designing and setting up the control
and experimental networks on other emulabs.
First a bit of background: On the control network will be your control
nodes (boss, ops, any tipservers, etc.), control hardware (SNMP-controllable
devices such as power controllers and switch IP interfaces), your connection to
the outside world, and the control interfaces of your nodes. So far, we haven't
tried to distribute the control net across multiple switches, but this should
be theorically possible. You'll need to set up the VLANs, etc. on the control
net by hand.
The experimental network will consist of one or more (this, we have tested)
switches connected with trunk (802.1q, or proprietary, like Cisco ISL) lines.
Most configuration of these switches will be taken care of by our software.
(See setup-cisco.txt for some configruation options you many want to apply to
these switches if they are Ciscos.)
##### Splitting up the control net
We basically have 5 VLANs on the control network:
'External' contains our connection to the outside world
'Private' contains the boss node, and our tape backup server
'Public' contains our ops node
'Control' contains the control net interfaces of all experimental nodes
'Control-hardware' contains all IP-controllable devices (namely, power
controllers and switch IP interfaces, as well as a second
interface on the boss node. This VLAN uses private IP addresses,
and does NOT contain a router interface.
This is done for security - we route (using an 'L3 switching' module in our
control-net switch) between these VLANs, and do some firewalling between each
of them. You could also do the routing with a real router or a PC. The main
goals are:
1) Protect both control and experimental nodes from the outside world (and
vice-versa - we don't want people attacking the outside world from our nodes)
2) Protect the control nodes from the experimental nodes
3) Protect the control hardware (power controllers, etc.) from nodes
and the outside world
4) Protect the boss node (which is _not_ publicly accessible) from the ops
node (which all experimenters have shells on.)
Now, it's entirely possible to combine these VLANs into one big one - this is
what we've done on our mini-testbed here. But, there are some serious security
implications with doing it this way - namely, that the nodes can theoretically
impersonate each other, power cycle each other, and all kinds of nasty things.
At the very least, you should have a firewall between your testbed and the
outside world, to satisfy #1.
It is also a good idea to separate the nodes' control net into a separate VLAN,
which satisfies #2 and #3. After all, you are giving people root access to the
experimental nodes. In situations where you are only giving access to a small
number of trusted people, this is probably not too big a deal, but once access
gets outside the small circle of your friends, or if you are allowing students
access, then taking these precautions are a very good idea.
If possible, putting boss and ops on separate networks is also a good idea
(#4), but is probably the least important part to worry about.
Basically, it's up to you to decide how much security you want to worry about.
I'd recommend at least going after #1, #2, and #3. We've put a lot of thought
into this, so if you're wondering how your choice here will affect other
aspects of security, ask Utah, and we can probably help you.
Since our firewall rules change frequently as we add new services, it's best to
contact Utah and ask us for the current set.
Note: For compatibility with Emulab's current control-network firewalling
code, and possible future improvements such as inter-experiment control
network isolation, you should make sure to name the control network
'Control' (case sensitive). In fact, we recommend keeping all 5 VLANs
named as we do, for ease of communication among testbed admins,
if nothing else.
##### Connecting the control net to the experimental net
In order to be able to control the experimental switches (ie. create new VLANs,
etc.) you need a way to talk to them. So, you'll need one line that goes from
your control net to the experimental net. However, you will need to be _very_
careful that no experimental traffic can leak across this line, and that it
cannot be used as a 'back door' into the private net by nodes.
On the control switch, it should be put into the 'control-hardware' VLAN.
On the experimental switch, it should be in the same VLAN as the switch's IP
interface. The best thing to do is create a new VLAN, and move the switch's IP
interface into it. On CatOS, you accomplish this with:
set interface sc0 <vlan #>
You will also need to give the experimental switch(es) IP addresses. With
CatOS, you do it like this:
set interface sc0 <IP> <netmask>
set interface sc0 up
##### DHCP through the router
If your boss node is on a separate VLAN from the node control net, you'll need
to make sure that DHCP traffic can get from the control net to your boss node,
since normally, DHCP is not forwarded through routers. On Cisco routers, this
is done with the 'ip helper-address'. For example, here, the name of the
router's interface in the node control net is 'Vlan3'. So, I'd log into the
router, and run the following:
configure terminal
interface Vlan3
ip helper-address
Of course, replace 'Vlan3' with the name of your router's node
control-net interface, and replace the IP address with that of your boss node.
##### IGMP snooping on the control net
In order for multicast to work correctly, you need to make sure that IGMP
snooping is enabled on the control switch. This is needed for frisbee, our
disk-loading system. It's up to you whether you want to enable this on the
experimental switches. In general, we recommend it so that your experimenters
can use multicast, but it does seem that unexpected or malformed multicast
packets have an easier time DOSing the switch control processor than unicast
traffic. On CatOS, the command is:
set igmp enable
#### VTP domains
If you have only one Cisco experimental-net switch, you don't need to have it
using VTP, which is a mechanism for keeping VLANs in sync across multiple
switches. You do this (in CatOS) with:
set vtp mode transparent
If you have multiple experimental switches connected by trunk lines, you should
use VTP. Pick a domain name (we call ours simply 'Testbed',) and run the
following on all of your switches:
set vtp domain <domainname>
Pick one switch to be the master - it doesn't really matter which one. (See the
switch setup instructions in setup-db.txt, and make sure you name the stack
after the master switch.) On this one, run
set vtp mode server
On all the others, run
set vtp mode client
This diff is collapsed.
#### Setting up a Nortel 5500-series switch as an experimental switch
The Emulab snmpit support works for at least the Nortel 5500 series
switches; at DETER, we use 5510-48Ts and 5530-24Ts. Keith Sklower
of UCB wrote the actual code, and can provide suggestions about
which other Nortel switches will work with the snmpit code.
There are at least two different lines of switches with different SNMP
MIBs, so there are definitely some Nortel switches which will NOT
work. The switches that use the RAPID-CITY mib should be okay, including
the 8000-series switches, although they have not been tested.
Apparently, *none* of the Nortel switches that are provided as part
of IBM blade centers will work with the current code; if you're
interested in support, definitely contact Keith.
All of these commands are supposed to be typed from the command
line (use the "Command Line Interface" menu option). Make sure that
you go into "config term" mode to make the changes. We usually
leave our switches configured to come up with the menu, as there
are some common functions that are much easier to do from the menu.
#### Access to the switches
At DETER, we've had the switches lock up infrequently. When they
lock up, they require a power cycle to come back, and often have
managed to lose all or part of their configuration.
For this reason, we have permanent connections to the serial ports
on the switches from serial port servers, so that we can get in and
reset the configuration by hand. We also have the switches on power
controllers, so that we can power off the switches remotely.
#### Configuration
While I've set up Nortels at DETER several times, and they seem
to work just fine, I may have forgotten some of the details in my
rush to get things working. Here are my suggestions -- please
let me know what details I've forgotten!
You can ask us at DETER for our 300KB+ Nortel configuration files;
for obvious reasons, we don't include them here. Oh, yeah, and
we're running SW v4.2.1.005.
#### ToS bits
By default, the Nortels appear to set the *IP* ToS bits to zero
for most packets. To prevent this, turn off QoS on the switch
altogether. This make sense if we want the network to appear as
though the switch is a direct connection:
no qos if-assign port ALL
#### Spanning tree
I believe that both Keith and I just turned off STP via the menu
options, but from the command line it should go something like
this (YMMV):
interface FastEthernet ALL
spanning-tree port ALL learning disable
#### Extra packets
As above, we also want to turn off all topology discovery packets,
which will otherwise interfere with idle detection:
no autotopology
#### IGMP packets
Make sure that multicast can be flooded around the switch correctly;
the Nortel documentation for this command is particularly cryptic,
but it works for us. Doing IGMP snooping apparently (as per Keith)
locked up our switches. Ugh.
vlan igmp unknown-mcast-no-flood disable
#### mac address aging
While we don't have this set at DETER right now, Keith recommended
turning off mac address aging, as per the various Cisco setup
suggestions. The Nortel maximum (as per Keith) is apparently
a million; there is no way to turn it off altogether.
mac-address-table aging-time 1000000
# Copyright (c) 2002-2008 University of Utah and the Flux Group.
# All rights reserved.
##### Setting up the Utah Network Testbed software
##### Most recently tested on FreeBSD 6.2 .
##### Step 1 - OS installation and setup
Install FreeBSD on the machine you'll be using for your ops node, using the
standard FreeBSD installation process. When asked by the installer, it's best
to choose the 'Developer' distribution set - this gets you full sources. When
it asks if you want to install the ports collection, answer *no*. Do not