cisco-voodoo.txt 11.6 KB
Newer Older
1
Some notes on how to diagnose problems on Cisco switches running CatOS
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

1. Looking at switch stats/status for a node interface.

   Mac may have a wrapper for this, but here is the guts of what you have
   to do.  To look at the switch port for a particular enet card, first
   figure out what switch port it is connected to!

      mysql tbdb
      select * from wires where node_id1='tbpcXX';

   What you care about are node_id2/card2/port2.  Node_id2 tells you which
   cisco: "cisco" is the testbed cisco (``tip test'') and "cisco2" is the
   control net cisco (``tip control'').  card2/port2 give you the info you
   need for the cisco-style card/port name (e.g., 3/39).

17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
[Note:
 There are also a couple of other easier ways to do this:

 Rob has created a command called if2port:
 108 paper:~> if2port
 Usage: /usr/testbed/sbin/if2port <node | node:if | node if>
 109 paper:~> if2port tbpc06
 +--------+-----+--------+-----+-----+
 |node_id1|card1|node_id2|card2|port2|
 +--------+-----+--------+-----+-----+
 |tbpc06  |    0|cisco   |    3|   42|
 |tbpc06  |    1|cisco   |    3|   44|
 |tbpc06  |    2|cisco   |    3|   46|
 |tbpc06  |    3|cisco   |    3|   48|
 |tbpc06  |    4|cisco2  |    3|   11|
 +--------+-----+--------+-----+-----+
 5 rows processed

 Another useful way is with 'snmpit -l -debug'. This will give all the 
 tbpcXX:Y <==> ciscoport translations for any port currently in use,
 as well as listing the VLANs currently set up:

111 paper:~> snmpit -l -debug
DEBUG MODE ON: Set to level 1
Command line was: snmpit -l -debug
Use of uninitialized value at /usr/testbed/bin/snmpit line 1281.
READING TRANSLATIONS
Opening SNMP session to 155.101.128.175...Succeeded
Getting VLAN info...
Got default     vtpVlanName     1.1 (1) default
Got tact-reserve2-l0-0          vtpVlanName     1.2 (2) tact-reserve2-l0-0
Got brandeis-BuddyCache-l0-0    vtpVlanName     1.3 (3) brandeis-BuddyCache-l0-0
Got tact-reserve2-l0-1          vtpVlanName     1.4 (4) tact-reserve2-l0-1
...
Got 3           vlanPortVlan    6.34    3       ('6.34' == tbpc23:0)
Got 19          vlanPortVlan    6.41    19      ('6.41' == tbpc21:0)
Got 19          vlanPortVlan    6.42    19      ('6.42' == tbpc24:0)
Got 19          vlanPortVlan    7.25    19      ('7.25' == tbpc25:0)
Got 19          vlanPortVlan    7.34    19      ('7.34' == tbpc29:0)
Got 3           vlanPortVlan    7.41    3       ('7.41' == tbpc27:0)
Got 19          vlanPortVlan    7.42    19      ('7.42' == tbpc30:0)
Got 19          vlanPortVlan    8.33    19      ('8.33' == tbpc32:0)
Got 5           vlanPortVlan    8.42    5       ('8.42' == tbpc36:0)
Got 8           vlanPortVlan    9.33    8       ('9.33' == tbpc38:0)
ID  Name                            Members of VLAN
--------------------------------------------------
1   default
2   tact-reserve2-l0-0              tbpc04:0  tbpc10:0
3   brandeis-BuddyCache-l0-0        tbpc20:0  tbpc22:1  tbpc23:0  tbpc27:0
4   tact-reserve2-l0-1              tbpc04:1  tbpc13:0
5   agile-test001-l0                tbpc36:0
6   janos-moab-l0                   tbpc02:0  tbpc03:1
7   janos-moab-l1                   tbpc01:0  tbpc03:0
8   agile-afreenet-l0               tbpc06:0  tbpc38:0
12  magi-test1-l0                   tbpc07:0  tbpc11:0
18  _mylan                          tbpc15:0  tbpc16:0  tbpc18:0  tbpc19:0
19  __mylan                         tbpc21:0  tbpc24:0  tbpc25:0  tbpc29:0  tbpc30:0  tbpc32:0

 The interesting translations are just above the vlan table, in the
 far right hand columns. For example, the last line before the table
 indicates that in vlan 8 is port 9.33, which belongs to tbpc38:0.

]

81
82
83
84
85
86
87
88
89
90
91
92
93
   Armed with this info, tip to the correct cisco, login and enable.
   Then you can do:

      show port status card/port

   or

      show port card/port

   for everything.  If, say, the port is disabled, you can do:

      set port enable card/port

94
95
96
97


2. Checking on the firewall rules

98
99
   You have to login to the "control" Cisco and then "session 15" to
   connect to the Router module.
100
101
102
103
104
105
106
107
108
109
110

   While at the Router> prompt, you will get any "access denied" type
   messages that the router produces, ala:

   23w2d: %SEC-6-IPACCESSLOGP: list control-shark denied \
	  udp 0.0.0.0(0) -> 255.255.255.255(0), 602 packets

   If you suspect that some rule is preventing your traffic from getting
   through, then try generating your traffic while you are connected to
   the router and see if you get errors.

111
112
   To see the whole lists in all their ugliness, type:
	show ip access-lists
113
114
115
116
   The rules are pretty straightforward. First match wins. Netmasks are
   bass-ackwards (intead of 255.255.255.0, you would use 0.0.0.255).
   Each list is applied both on entrance and exit to the like-named control
   network VLAN.
117

118
119
3. Changing the firewall rules

120
121
122
   ROB: Working on this part

4. Finding MAC address information
123
124
125
126
127
128
129
130

   To find which port a given MAC address is on type (on the switch console):
	show cam <MAC>
   where MAC is colon-seperated, like 08:00:2b:81:62:d3.

   To show all MAC addresses in a given VLAN, type:
	show cam dynamic <VLAN>
   where VLAN is the number, not the name.
131
	
132
133
134
135
136
137
138
139
140
141
142
143
144
145
5. Deleting a "sticky" ARP entry

   If you should ever be so unfortunate as to have to replace a faulty
   shark, in addition to recording the new MAC address in the DB and DHCP
   config file, you may also need to clear it from the router module.
   If you fire up a new shark, and it says that it cannot get its DHCP info,
   this is likely the problem.  To find out, login to the control Cisco and
   "session 15" to get to the router module.  You should start seeing periodic
   
   24w2d: %IP-3-STCKYARPOVR: Attempt to overwrite Sticky ARP entry: \
   	155.101.130.73, hw: 0800.2b81.62d3 by hw: 0800.2b81.611b

   messages.  To clear the arp entry (actually the whole cache), enable at
   the Router> prompt and then do "clear arp".
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167

6. Replacing a node/NIC

   If you replace a node, you'll need to change the secure MAC address for that port.
   The following command should work:
   	Console> (enable) set port security 3/1 enable 01-02-03-04-05-06
   Of course, use the real port number and MAC address (noting the funky MAC
   syntax). Note that you will proably also need to use the 'Sticky ARP Entry' 
   clearing procedure covered above.

7. Checking on port security
   To find out what MAC address(es) are associated with a given port. use:
   	Console> (enable) show port security <port>
   To find out if a given port has been disabled for being a Bad Boy (tm) w/
   respect to MAC addresses, use:
   	Console> (enable) show port <port>
   - the state will be 'disabled', and you should see some information on the
   security violation
   To re-enable a port after it has been disabled due to security violations:
   	Console> (enable) set port enable <port>
   To disable security for a port:
   	Console> (enable) set port security <port> disable
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185

8. Manual VLAN configuration - From the switch command line
   To see a list of all configured VLANs, use:
     Console> (enable) show vlan
   On the control net, all of the VLAN names should be self-explanatory
   Adding a port to a VLAN is very easy. Just type:
     Console> (enable) set vlan <NUM> <PORT>
   ... where <NUM> is the number of the VLAN, and <PORT> is the port (you can
     use the 'if2port' script to get the port number)
   To 'remove' a port from a VLAN, set it to VLAN 1.
   To create a new VLAN, use:
     Console> (enable) set vlan <NUM> name <NAME>
   ... where <NUM> is some unused VLAN number (use 'show vlan' to find one), and
     <NAME> is some descriptive string
   To delte a VLAN use:
      Console> (enable) clear vlan <NUM>
   ... where <NUM> is the VLAN number (duh!) NOTE: This puts all of the VLANs ports
     back into VLAN 1, and disables them. Use 'set port enable <PORT>' to re-enable it
Robert Ricci's avatar
Robert Ricci committed
186
187
188
189
190
191
192

9. Cloning all traffic from a port or VLAN to another port
   Pick a port to recieve the traffic - let's call it <monitor>
   To forward the traffc from one port:
      Console> (enable) set span <port> <monitor>
   To forward the traffic from an entire VLAN:
      Console> (enable) set span <vlan #> <monitor>
Mike Hibler's avatar
Mike Hibler committed
193
   NOTE: You might want to append 'rx' or 'tx' to the VLAN command line, or
194
195
      you'll get doubles of everything (incoming and outgoing both).
      <rx> means into the switch on <port>, <tx> means out of the switch.
196
197
198
   NOTE2: If you want to be able to send traffic from the monitor port,
      you'll need to append 'inpkts enable' ala:
      Console> (enable) set span 18 3/9 rx inpkts enable
Robert Ricci's avatar
Robert Ricci committed
199
200
201
   To stop cloning:
      Console> (enable) set span disable <monitor>

Mike Hibler's avatar
Mike Hibler committed
202
   NOTE: as of 10/10/05 we have bge0 on oboss attached to the control net
Mike Hibler's avatar
Mike Hibler committed
203
   so that it can be used for cloning.  Its cisco2 port is 2/13, use that
Mike Hibler's avatar
Mike Hibler committed
204
205
206
   for <monitor>.  Also note that you will have to have the interface
   configured in order to run tcpdump on it.  I just do:
      ifconfig bge0
207
   no IP info is necessary.  Then:
Mike Hibler's avatar
Mike Hibler committed
208
      ifconfig bge0 down
209
   when I am done.
Mike Hibler's avatar
Mike Hibler committed
210

211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
10. Multiple span sessions
   You can setup multiple span sessions on a switch.  Just add 'create'
   to the end of your setup line:
      Console> (enable) set span <port> <monitor> create
   To tear down a specific span session, name it by the <monitor> port
   as above:
      Console> (enable) set span disable <monitor>

11. Span on trunk ports
   You can also span traffic on a trunk port or even a set of bonded
   trunk ports.  For example, on our experimental switches we have four
   bonded Gb ports, 2/1-4, as an inter switch link, so you could see all
   traffic coming in from the trunk with:
      Console> (enable) set span 2/1-4 5/5 rx create
   You can even pick out a specific VLAN or VLANs from the trunk.  So
   if you only care about VLAN 300 and 400:
      Console> (enable) set span 2/1-4 5/5 rx filter 300,400 create
   I haven't tried this, but according to the document cited below (#12),
   you can preserve the VLAN tagging when spanning a trunk by putting the
   monitor port into trunk mode first.

12. Good reference for span:

    http://www.netplusinc.com/misc/CiscoSPAN.pdf

Mike Hibler's avatar
Mike Hibler committed
236
237
238
239
    How to setup span for IOS on the 6500 series:

    http://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/12_1e/swconfig/span.htm

240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
13. Stuck 1000/100/10 ports [ ISI - faber@isi.edu ]

    Occasionally Gigabit switch ports seem to get stuck in a state where
    they do not output packets to the host connected to them.  We're not
    sure why this happens (we're looking), but there is a fix.  Simply
    toggle the speed using 
    
    	set port speed module/port 1000

   and then back to whatever speed it was originally set for.  Of course
   if it was originally 1000 Mb/s the first set should be to 100 and
   then back to 1000.  We have seen ports lock up this way configured to
   both 1000 and 100 Mb/sec.

   You may have to both toggle the switch speed and re-ifconfig the host
   NICs to re-establish connectivity.

14. Good reference for making nodes boot quickly (ensuring that port spanning,
Mike Hibler's avatar
Mike Hibler committed
258
   trunking, etc. are off on the switch port):
259

Mike Hibler's avatar
Mike Hibler committed
260
   http://www.cisco.com/warp/public/473/12.html
261
262
263
264
265
266
267
268

   The gist is that we should use the convenient:

   set port host <port ...>

   To turn on fastport (disable spanning tree) and turn off channeling
   (combining multiple ports to make a fast link) and trunking (a single
   port serving multiple VLANs).
269

270
15. Good reference for how IGMP snooping (used for multicast) works:
271
272
273
274
275
276

    http://www.cisco.com/warp/public/473/22.html

    The most interesting bits are near the bottom in the part with the
    heading "IGMP Snooping", and some more in "Practical Example of IGMP
    Snooping"
277

278
16. IOS for guys who know CatOS: (ie. me)
279
280

    http://www.cisco.com/warp/public/cc/pd/si/casi/ca6000/tech/cat65_wp.htm
281

282
17. For into on how the switches do load balancing on etherchannel
283
284
285
    (aka "trunk") links:

    http://www.cisco.com/warp/public/473/4.html