setup-cisco.txt 3.85 KB
Newer Older
##### Configuration suggestions for Cisco switches
4 5

This file contains some configuration guidelines that we (Utah) have found
useful to improve the performance of our Cisco switches.
7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

All commands given are to be typed at the (enable) prompt on your cisco
switches. They are for CatOS - switches that run IOS may not have these

<ports> means a list of ports, which on the CatOS command line, can include
lists and rages, such as "3/1,3/2" or "3/1-48" or "3/1-48,4/1-48,5/1-48"

##### Allowing ports to come up quicker

This one is useful on both the experimental and control nets:
    set spantree portfast <ports> enable
Use this on all ports that are directly connected to nodes, servers, power
controllers - anything that is not another switch. Normally, the switch waits
a while (several seconds) when a port first comes up before forwarding traffic
from this port - it does so to prevent loops in the switch topology. The main
23 24 25 26 27 28 29 30 31 32
place you will see the benefit of this is on the control net - with portfast
disabled, the first few DHCP packets sent by booting nodes will get dropped,
causing the DHCP to take much longer than necessary.

##### Reducing stray traffic

Disable spanning tree (STP.) If on, STP sends out pakcets approximately every
two seconds on every port. You can disable it on all VLANs with the command
    set spantree disable all
There are two major consequences (for our purposes) of disabling STP:
    1) You cannot have _any_ loops in your switch topology, or bad things will
34 35
    2) VLAN pruning on trunks won't work, causing broadcast traffic to be
	forwarded across trunks that it does not need to cross. We've added
37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57
	features to snmpit to manually do STP's job in this case, so this
	problem is taken care of.
You must have STP disabled on _all_ switches that are trunked together! If it
is enabled on even one, STP traffic will be seen on all of them. 

The switch doesn't trust you to use portfast responsibly. So, it has a
'bpdu-guard' feature that helps guard against loops. Turn off this feature
with the command:
    set spantree portfast bpdu-guard disable

Cisco uses a protocol called 'CDP' to discover other Cisco devices. This sends
out small packets every two minutes. You can disable it with:
    set cdp disable <ports>
Ideally, you should only disable CDP on ports that don't have other Cisco 
devices attached, but in practice, running with CDP disabled on all ports is

Switch ports will, by default, try to negotiate trunking and channeling.
Cisco provides a handy macro:
    set port host <ports>
to disable both of these. Also enables portfast on the ports.
58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76

##### Setting MAC address aging time

We have found that some experimenters use applications, kernels, etc. that only
receive traffic, not send it. This presents a problem, because it prevents the
switch from learning which port the node is on, and thus broadcasting traffic
for it to every port in the VLAN. This can be solved by 'priming' - ie. having
the receive-only node send some traffic (like an ARP response) at the beginning
of the experiment. However, the default aging time of 300 seconds makes this
impractical. So, we have disabled this aging, making learned MACs permanent
(until the VLAN is torn down.)

You must do this for each VLAN, with the command:
    set cam agingtime <vlan> 0
For convenience, we've supplied a file (in this directory) called
'no-cam-aging.cfg' that disables aging on VLANs 2-999 (the ones potentially
used by our software.) Transfer this file to the switch using the:
    copy tftp config
77 78 79 80 81 82

We also suggest that you do this on your control network as well - part of the
booting process leaves the nodes sitting dormant at a boot loader for extended
periods of time, so the switch will tend to forget their MACs.  Turning off
aging is not critical, but we suggest it, because it will reduce stray traffic
while the switch re-learns MAC addresses.