cisco-voodoo.txt 9.12 KB
Newer Older
1
Some notes on how to diagnose problems on Cisco switches running CatOS
2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

1. Looking at switch stats/status for a node interface.

   Mac may have a wrapper for this, but here is the guts of what you have
   to do.  To look at the switch port for a particular enet card, first
   figure out what switch port it is connected to!

      mysql tbdb
      select * from wires where node_id1='tbpcXX';

   What you care about are node_id2/card2/port2.  Node_id2 tells you which
   cisco: "cisco" is the testbed cisco (``tip test'') and "cisco2" is the
   control net cisco (``tip control'').  card2/port2 give you the info you
   need for the cisco-style card/port name (e.g., 3/39).

17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
[Note:
 There are also a couple of other easier ways to do this:

 Rob has created a command called if2port:
 108 paper:~> if2port
 Usage: /usr/testbed/sbin/if2port <node | node:if | node if>
 109 paper:~> if2port tbpc06
 +--------+-----+--------+-----+-----+
 |node_id1|card1|node_id2|card2|port2|
 +--------+-----+--------+-----+-----+
 |tbpc06  |    0|cisco   |    3|   42|
 |tbpc06  |    1|cisco   |    3|   44|
 |tbpc06  |    2|cisco   |    3|   46|
 |tbpc06  |    3|cisco   |    3|   48|
 |tbpc06  |    4|cisco2  |    3|   11|
 +--------+-----+--------+-----+-----+
 5 rows processed

 Another useful way is with 'snmpit -l -debug'. This will give all the 
 tbpcXX:Y <==> ciscoport translations for any port currently in use,
 as well as listing the VLANs currently set up:

111 paper:~> snmpit -l -debug
DEBUG MODE ON: Set to level 1
Command line was: snmpit -l -debug
Use of uninitialized value at /usr/testbed/bin/snmpit line 1281.
READING TRANSLATIONS
Opening SNMP session to 155.101.128.175...Succeeded
Getting VLAN info...
Got default     vtpVlanName     1.1 (1) default
Got tact-reserve2-l0-0          vtpVlanName     1.2 (2) tact-reserve2-l0-0
Got brandeis-BuddyCache-l0-0    vtpVlanName     1.3 (3) brandeis-BuddyCache-l0-0
Got tact-reserve2-l0-1          vtpVlanName     1.4 (4) tact-reserve2-l0-1
...
Got 3           vlanPortVlan    6.34    3       ('6.34' == tbpc23:0)
Got 19          vlanPortVlan    6.41    19      ('6.41' == tbpc21:0)
Got 19          vlanPortVlan    6.42    19      ('6.42' == tbpc24:0)
Got 19          vlanPortVlan    7.25    19      ('7.25' == tbpc25:0)
Got 19          vlanPortVlan    7.34    19      ('7.34' == tbpc29:0)
Got 3           vlanPortVlan    7.41    3       ('7.41' == tbpc27:0)
Got 19          vlanPortVlan    7.42    19      ('7.42' == tbpc30:0)
Got 19          vlanPortVlan    8.33    19      ('8.33' == tbpc32:0)
Got 5           vlanPortVlan    8.42    5       ('8.42' == tbpc36:0)
Got 8           vlanPortVlan    9.33    8       ('9.33' == tbpc38:0)
ID  Name                            Members of VLAN
--------------------------------------------------
1   default
2   tact-reserve2-l0-0              tbpc04:0  tbpc10:0
3   brandeis-BuddyCache-l0-0        tbpc20:0  tbpc22:1  tbpc23:0  tbpc27:0
4   tact-reserve2-l0-1              tbpc04:1  tbpc13:0
5   agile-test001-l0                tbpc36:0
6   janos-moab-l0                   tbpc02:0  tbpc03:1
7   janos-moab-l1                   tbpc01:0  tbpc03:0
8   agile-afreenet-l0               tbpc06:0  tbpc38:0
12  magi-test1-l0                   tbpc07:0  tbpc11:0
18  _mylan                          tbpc15:0  tbpc16:0  tbpc18:0  tbpc19:0
19  __mylan                         tbpc21:0  tbpc24:0  tbpc25:0  tbpc29:0  tbpc30:0  tbpc32:0

 The interesting translations are just above the vlan table, in the
 far right hand columns. For example, the last line before the table
 indicates that in vlan 8 is port 9.33, which belongs to tbpc38:0.

]

81 82 83 84 85 86 87 88 89 90 91 92 93
   Armed with this info, tip to the correct cisco, login and enable.
   Then you can do:

      show port status card/port

   or

      show port card/port

   for everything.  If, say, the port is disabled, you can do:

      set port enable card/port

94 95 96 97


2. Checking on the firewall rules

98 99
   You have to login to the "control" Cisco and then "session 15" to
   connect to the Router module.
100 101 102 103 104 105 106 107 108 109 110

   While at the Router> prompt, you will get any "access denied" type
   messages that the router produces, ala:

   23w2d: %SEC-6-IPACCESSLOGP: list control-shark denied \
	  udp 0.0.0.0(0) -> 255.255.255.255(0), 602 packets

   If you suspect that some rule is preventing your traffic from getting
   through, then try generating your traffic while you are connected to
   the router and see if you get errors.

111 112
   To see the whole lists in all their ugliness, type:
	show ip access-lists
113 114 115 116
   The rules are pretty straightforward. First match wins. Netmasks are
   bass-ackwards (intead of 255.255.255.0, you would use 0.0.0.255).
   Each list is applied both on entrance and exit to the like-named control
   network VLAN.
117

118 119
3. Changing the firewall rules

120 121 122
   ROB: Working on this part

4. Finding MAC address information
123 124 125 126 127 128 129 130

   To find which port a given MAC address is on type (on the switch console):
	show cam <MAC>
   where MAC is colon-seperated, like 08:00:2b:81:62:d3.

   To show all MAC addresses in a given VLAN, type:
	show cam dynamic <VLAN>
   where VLAN is the number, not the name.
131
	
132 133 134 135 136 137 138 139 140 141 142 143 144 145
5. Deleting a "sticky" ARP entry

   If you should ever be so unfortunate as to have to replace a faulty
   shark, in addition to recording the new MAC address in the DB and DHCP
   config file, you may also need to clear it from the router module.
   If you fire up a new shark, and it says that it cannot get its DHCP info,
   this is likely the problem.  To find out, login to the control Cisco and
   "session 15" to get to the router module.  You should start seeing periodic
   
   24w2d: %IP-3-STCKYARPOVR: Attempt to overwrite Sticky ARP entry: \
   	155.101.130.73, hw: 0800.2b81.62d3 by hw: 0800.2b81.611b

   messages.  To clear the arp entry (actually the whole cache), enable at
   the Router> prompt and then do "clear arp".
146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167

6. Replacing a node/NIC

   If you replace a node, you'll need to change the secure MAC address for that port.
   The following command should work:
   	Console> (enable) set port security 3/1 enable 01-02-03-04-05-06
   Of course, use the real port number and MAC address (noting the funky MAC
   syntax). Note that you will proably also need to use the 'Sticky ARP Entry' 
   clearing procedure covered above.

7. Checking on port security
   To find out what MAC address(es) are associated with a given port. use:
   	Console> (enable) show port security <port>
   To find out if a given port has been disabled for being a Bad Boy (tm) w/
   respect to MAC addresses, use:
   	Console> (enable) show port <port>
   - the state will be 'disabled', and you should see some information on the
   security violation
   To re-enable a port after it has been disabled due to security violations:
   	Console> (enable) set port enable <port>
   To disable security for a port:
   	Console> (enable) set port security <port> disable
168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185

8. Manual VLAN configuration - From the switch command line
   To see a list of all configured VLANs, use:
     Console> (enable) show vlan
   On the control net, all of the VLAN names should be self-explanatory
   Adding a port to a VLAN is very easy. Just type:
     Console> (enable) set vlan <NUM> <PORT>
   ... where <NUM> is the number of the VLAN, and <PORT> is the port (you can
     use the 'if2port' script to get the port number)
   To 'remove' a port from a VLAN, set it to VLAN 1.
   To create a new VLAN, use:
     Console> (enable) set vlan <NUM> name <NAME>
   ... where <NUM> is some unused VLAN number (use 'show vlan' to find one), and
     <NAME> is some descriptive string
   To delte a VLAN use:
      Console> (enable) clear vlan <NUM>
   ... where <NUM> is the VLAN number (duh!) NOTE: This puts all of the VLANs ports
     back into VLAN 1, and disables them. Use 'set port enable <PORT>' to re-enable it
Robert Ricci's avatar
Robert Ricci committed
186 187 188 189 190 191 192

9. Cloning all traffic from a port or VLAN to another port
   Pick a port to recieve the traffic - let's call it <monitor>
   To forward the traffc from one port:
      Console> (enable) set span <port> <monitor>
   To forward the traffic from an entire VLAN:
      Console> (enable) set span <vlan #> <monitor>
Mike Hibler's avatar
Mike Hibler committed
193
   NOTE: You might want to append 'rx' or 'tx' to the VLAN command line, or
Robert Ricci's avatar
Robert Ricci committed
194
      you'll get doubles of everything (incoming and outgoing both)
195 196 197
   NOTE2: If you want to be able to send traffic from the monitor port,
      you'll need to append 'inpkts enable' ala:
      Console> (enable) set span 18 3/9 rx inpkts enable
Robert Ricci's avatar
Robert Ricci committed
198 199 200
   To stop cloning:
      Console> (enable) set span disable <monitor>

201 202 203 204 205 206 207 208
   NOTE: as of 11/5/01 we have fxp1 on plastic attached to the control net
   so that it can be used for cloning.  Its cisco2 port is 3/23, use that
   for <monitor>.  Also note that you will have to have fxp1 configured
   in order to run tcpdump on it.  I just do:
      ifconfig fxp1 inet 192.168.1.1 netmask 0xffffff00
   and then:
      ifconfig fxp1 down
   when I am done.
Mike Hibler's avatar
Mike Hibler committed
209 210 211

10. Good reference for making nodes boot quickly (ensuring that port spanning,
   trunking, etc. are off on the switch port):
212

Mike Hibler's avatar
Mike Hibler committed
213
   http://www.cisco.com/warp/public/473/12.html
214 215 216 217 218 219 220 221

   The gist is that we should use the convenient:

   set port host <port ...>

   To turn on fastport (disable spanning tree) and turn off channeling
   (combining multiple ports to make a fast link) and trunking (a single
   port serving multiple VLANs).