Exoc_module 15 Network Troubleshooting

Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.

Extreme Networks Customer Training ExtremeXOS 12.1 Operations and Configuration 9

Network Troubleshooting



Extreme Networks Customer Training ExtremeXOS 12.1 Operations and Configuration10

Student ObjectivesThe Network Troubleshooting module presents a systematic methodology for troubleshooting, how to use the troubleshooting tools provided in Extreme Networks® switches, and examples of how to identify common problems.

Upon completion of this module, the successful student will be able to:

● Identify the network management and maintenance systems that prepare you to troubleshoot.

● Describe the Layered approach used for troubleshooting.

● Identify useful commands for finding errors in OSI Layers 1, 2, and 3.

● Interpret the output of show commands.

● Use correct to troubleshoot problems at the Physical, Data lLnk, and Network Layers.

● List the steps required to perform systematic troubleshooting.



Figure 1: Student Objectives




OverviewTroubleshooting is an important part of managing any network. This module presents a systematic methodology for troubleshooting, how to use the troubleshooting tools provided in Extreme Networks switches, and examples of how to identify common problems. The troubleshooting topics covered are:

● Gathering and maintaining the information and tools necessary to troubleshoot the network.

● Organizing and applying a systematic method of troubleshooting.

● Identifying which commands to use to systematically troubleshoot a network problem.

● Describing how to test the operation of the switch.



Figure 2: Overview




Maintenance before TroubleshootingThis page presents steps you should complete prior to having to troubleshoot a network problem.

Document the Network

Documenting the network topology, configuration, and normal operation so you have the information needed to recognize and locate a network problem. The following documentation is recommended:

● Create an illustration of the physical connectivity of the network and keep it up to date.

● Create an illustration of the logical connectivity of the network and keep it up to date.

● Set up a Trivial File Transfer Protocol (TFTP) server and:

■ Store copies of the active and previous versions of configuration files and images, so that you can access them quickly in case of a problem.

■ Make a backup of each switch configuration file on a regularly scheduled basis.

■ Update the stored configuration files after making any changes.

● Maintain a contact list of vendors, service suppliers, and users to inform in case of a problem.

● Store user names, passwords, and SNMP default community strings in a safe location and change (or remove) the default values.

● Record network problems, symptoms, and the resolution.

Monitor the Network

To identify a problem as soon as possible, monitor the condition of the network.

● Establish baselines for network response time, and traffic statistics.

● Test connectivity and response time on a regularly scheduled basis and compare the response time to the baseline.

● Monitor traffic statistics on a regularly scheduled basis and compare the statistics to the baseline.

● Use Extreme Networks EPICenter to

■ Monitor alarms and events.

■ Verify the physical and logical topology.

■ Monitor network, VLAN, and port statistics.

■ Set traffic thresholds.

■ Store a syslog of switch activity.

Maintain the Network

When building a network, the importance of a well structured network maintenance system is often underestimated. No network is error free. Therefore an important aspect of the performance of the network depends on proper maintenance of the network. Maintenance includes:

● Provide a clean/conditioned environment (air and power) and provide proper ventilation depending on hardware type. Some switches vent front to rear, others vent side to side.

● Software maintenance such as installing new software to fix bugs or add functionality.



Figure 3: Maintenance before Troubleshooting




Use a Layered Approach when TroubleshootingThe layered approach is the way to thoroughly check a network.

This layered model also helps in isolating the problem.

If you experience problems in the communication between devices, check the communication layer by layer, using the per layer description as presented on the following pages.

When, for example, an end-to-end ping test at Layer 3 succeeds, the problem is probably in the upper four layers.



Figure 4: Use a Layered Approach when Troubleshooting




Troubleshooting at the Physical LayerSeveral kinds of common problems can affect the Physical Layer.

Your hardware may be faulty. Defective hardware can be cables, patch panels, connectors, ports, or modules in your switch.

● First check the physical connectivity.

● Is the connector seated properly?

● Ensure that the physical medium connecting to the interface port is fault free.

● Isolate the problem by swapping cables with ports. Does the problem follow?

● Check the status of the LEDs.

● Use loop back plugs or a loop back cable to verify the link.

● Be aware of the surrounding environment for potential electro-mechanical interference (EMI).

Verify that the correct cables are being used. Distinguish between:

● Media type

● Length of the cables

● Crossover cable versus straight cable

Verify that the port settings are configured properly. Common problems in port settings are:

● Auto negotiation of speed and duplex is set incorrectly.

● The port is administratively disabled.

● Speed and duplex settings do not match the settings of the connected device.

● Load sharing configuration is set improperly.



Figure 5: Troubleshooting at Layer 1: Physical Layer




Port LED IndicatorsThis page describes the port Light Emitting Diodes (LEDs) for switches that run ExtremeXOS.

The port LEDs provide an indication of the port status. Always verify the status of the port link integrity by checking port activity LEDs when troubleshooting Layer-1 problems.

Table 1: Summit Port LEDs

The description for port LEDs in Table 1 applies to ports on the SummitX450 family of switches, Summit 200, Summit 300, and Summit 400.

Table 2: BlackDiamond Port LEDs

The description for port LEDs in Table 2 applies to ports on the BlackDiamond 8800 series, BlackDiamond 10808 series, and non-POE ports on the BlackDiamond 12804 switches.

Table 3: BlackDiamond 12804 POE Port LEDs

The description for port LEDs in Table 3 applies to POE ports with power enabled on the BlackDiamond 12804.

LED Color Description

Port Solid green Link up. Link Integrity signal is received on copper ports or light is detected on fiber ports.

Green blinking Activity. Packets received and transmitted.

Off Link Down. No link Integrity signal or the port is disabled.

Solid Amber Power over Ethernet (POE) port has power applied.

Amber slow blinking POE port has power applied and link is down or port is disabled.

Amber fast blinking POE port has power applied and port has activity.

Blinking amber/green POE port has a power fault or insufficient power.

Off POE port has no link or is disabled and power is not applied.


Port Solid green Link up. Link Integrity signal is received on copper ports. Light is detected on fiber ports.

Slow green blinking Port disabled by CLI.

Amber blinking Activity. Packets received and transmitted.

Off Link Down. No link Integrity signal is received on copper ports.

Blinking amber/green POE port with power enabled has a power fault or insufficient power.


Port Solid amber Link up. Link Integrity signal is received.

Amber slow blinking Link Down or Port disabled by CLI.

Amber fast blinking Activity. Packets received and transmitted.

Blinking amber/green Power over Ethernet port with power enabled has a power fault or insufficient power.



Figure 6: Port LED Indicators




Troubleshooting Commands for the Physical LayerA systematic approach to troubleshooting involves beginning with Layer 1 and working up the OSI model towards Layer. Use show commands for troubleshooting.

The illustration lists the most common show commands used to troubleshoot Layer 1.



Figure 7: Commands for Layer 1 Troubleshooting




Displaying Port Configuration StatisticsTo verify the port configuration, enter the following command:

show ports {<portlist>} configuration

Compare the port settings with the port settings at the other end of the cable.

Primary InformationThe show ports configuration command output shows:

● The port number and virtual router.

● The port state as enabled (E) or disabled (D).

● The link status as ready (R), active (A), or not present (NP).

● If auto negotiation of speed and duplex is enabled (ON) or disabled (OFF).

● The link speed configured and if auto negotiation is enabled, the actual speed.

● The duplex mode configured and if auto negotiation is enabled, the actual duplex as half or full.

● Link media type for the primary and if configured, redundant port. Examples of media types are:

■ Unshielded Twisted Pair (UTP)

■ Multimode Gigabit Interface Connector (MGBIC)

■ Small Formfactor Pluggable (SFP)

Use this information to verify that the intended ports are enabled and active. Also verify that the speed, duplex and frame size match the other end of the physical connection.

If the switch supports user-created virtual routers, verify that the VLAN router interface (RIF) is configured on the correct VR. Commands such as ping and tftp require that the VR be specified.

The media type is useful if redundant fiber and copper ports are supported. In this case the port number is the same for both the fiber and copper port.

Secondary InformationThe command output also shows:

● If Ethernet flow control is enabled symmetrically in both directions (SYM) or asymmetrically in one direction (ASYM). 10Gbps ports can be asymmetrical.

● The load sharing master port if the port is participating in link aggregation.

Use this information to verify that the port settings and media type match the other end of the physical connection.



Figure 8: Displaying Port Configuration Statistics




Displaying Real Time Transmitted Packet Errors To monitor the real-time transmitted packet errors for a group of ports, enter the following command:

show ports {<port_list>} txerrors {no-refresh}

The display automatically refreshes unless you enter the no-refresh option.

For each port the display shows:

● Transmit Collisions (TX Coll). The total number of collisions seen by the port, regardless of whether a device connected to the port participated in any of the collisions.

● Transmit Late Collisions (TX Late Coll). The total number of collisions that have occurred after the ports transmit window has expired. This could indicate a duplex setting mismatch.

● Transmit Deferred Frames (TX Deferred). The total number of frames that were transmitted by the port after the first transmission attempt was deferred by other network traffic.

● Transmit Errored Frames (TX Errors). The total number of frames that were not completely transmitted by the port because of network errors (such as late collisions or excessive collisions).

● Transmit Lost Frames (TX Lost). The total number of transmit frames that did not get completely transmitted because of buffer problems (FIFO underflow).

● Transmit Parity Frames (TX Parity). The bit summation has a parity mismatch.

Displaying Real Time Received Packet ErrorsTo monitor the real-time received packet errors for a group of ports, enter the following command:

show ports {<port_list>} rxerrors {no-refresh}

The display automatically refreshes unless you enter the no-refresh option.

For each port the display shows:

● Receive Bad CRC Frames (RX CRC). The total number of frames received by the port that were of the correct length, but contained a bad FCS value. This could indicate a duplex setting mismatch.

● Receive Oversize Frames (RX Over). The total number of good frames received by the port greater than the supported maximum length of 1,522 bytes.

● Receive Undersize Frames (RX Under). The total number of frames received by the port that were less than 64 bytes long.

● Receive Fragmented Frames (RX Frag). The total number of frames received by the port were of incorrect length and contained a bad FCS value.

● Receive Jabber Frames (RX Jabber). The total number of frames received by the port that was of greater than the support maximum length and had a Cyclic Redundancy Check (CRC) error.

● Receive Alignment Errors (RX Align). The total number of frames received by the port that occurs if a frame has a CRC error and does not contain an integral number of octets. In a shared Ethernet environment, alignment errors could be the result of collisions.

● Receive Frames Lost (RX Lost). The total number of frames received by the port that were lost because of buffer overflow in the switch.



Figure 9: Displaying Real Time Transmit Errors

Figure 10: Displaying Real Time Received Packet Errors




Displaying Real Time Collision StatisticsTo monitor the real-time retransmissions due to successive collisions for a group of ports, enter the following command:

show ports {mgmt | <port_list>} collisions {no-refresh}

The display i automatically refreshes unless you enter the no-refresh option.

The mgmt option specifies the management Ethernet port.

The display shows:

● Collision statistics for each port.

● The numbers 1 to 16 represent the number of collisions and subsequent retransmissions encountered prior to successfully transmitting the packet. Collision statistics are applicable only for half-duplex links.

Collisions are primarily a problem for shared Ethernet environments where devices are attached through a hub. If you see collisions, it indicates that someone has connected a hub to the switch port.



Figure 11: Displaying Real Time Collision Statistics




Displaying Real Time Port Utilization InformationEnter the following command to monitor the current and peak packets-per-second transmitted and received for a group of ports:

show ports {<portlist>} utilization

This command gives a good indication of the port load. You can toggle with the space bar between packet-per-second, bytes-per-second, and bandwidth utilization. You can use this information to decide when to invest in extra link capacity to avoid congestion in the network.

NOTE

Monitoring port utilization is useful to identify broadcast storms caused by loops in the network. Extreme Networks switches are non-blocking. So, links can be utilized up to 100% when there is a network loop.

Enter the show ports utilization command then toggle the screen using the space bar.

The first two screens in the output show transmitted and received port byte and packet counts.

Use this information to verify that the ports are passing traffic.

The third screen in the show port utilization command output shows:

● The port number, link status, and link speed.

● The current and peak port utilization received as a percentage of the maximum bandwidth.

● The current and peak port utilization transmitted as a percentage of the maximum bandwidth.

Use this information to verify that there are no Layer 2 forwarding loops.

If a Layer 2 forwarding loop exists it causes a broadcast storm. During a broadcast storm the port utilization reaches close to 100%.

NOTE

If a forwarding loop exists it may take a minute of two for the utilization to show the broadcast storm.

To clear the port utilization counters, enter the following command:

clear counters ports



Figure 12: Displaying Real Time Port Utilization Information




Displaying Connectivity and Configuration Information for Neighboring EXTR SwitchesEnter the following command to verify connectivity and configuration information received from neighboring Extreme Networks switches:

show edp ports all

The show edp command output shows:

● The EDP interval timers and which ports have EDP enabled.

Use this information to verify that EDP is enabled on the intended ports and that the timers match the other end of the connection.

The show edp ports all command output shows:

● The SNMP sysname of each EDP neighbor on every active port.

● The MAC address of each EDP neighbor.

● The slot and port number of the other end of the connection.

● The age of the EDP entry.

● The number of VLANs supported on this link.

Use this information to verify that the ports are physically connected to the intended EDP neighbor on the intended port.

You can also use this command for troubleshooting at the Data Link Layer because you get information about the VLANs on all neighboring switches.



Figure 13: Displaying Connectivity and Configuration Information for Neighboring EXTR Switches




Layer 1 Problem: Diagnosis and SolutionThe illustration shows a typical Layer 1 error often seen in live networks. There is connectivity between the two switches but a lot of errors occur. This has a negative impact on the performance of connections using this link.

This might be because of misconfiguration, but often this mismatch occurs because of problems with the auto negotiation of link speed and duplex mode. The easiest command to verify this error is the show ports configuration command.

This command displays the configured and actual link speed and duplex mode of all ports.

There appears to be a duplex mismatch between the two switches. One is forwarding frames in full duplex mode and the other one is running in half duplex mode.

In the example, Switch 1 is configured for autonegotiation off with speed set at 100mb and duplex at full. Switch 2 is left in autonegotiation. Switch 2 is unable to negotiate its duplex setting as Switch 1 has auto disabled.

The following port transmit error information is collected by the system:

● Port Number

● Link Status — The current status of the link. Options are:

■ Ready (the port is ready to accept a link)

■ Active (the link is present at this port)

■ Disabled (D): The link is disabled at this port.

■ Not Present (NP): The link is not present at this port.

● Transmit Collisions: The total number of collisions seen by the port, regardless of whether a device connected to the port participated in any of the collisions.

● Transmit Late Collisions: The total number of collisions that have occurred after the port’s transmit window has expired.

● Transmit Deferred Frames: The total number of frames that were transmitted by the port after the first transmission attempt was deferred by other network traffic.

● Transmit Errored Frames: The total number of frames that were not completely transmitted by the port because of network errors (such as late collisions or excessive collisions).

● Transmit Parity Frames: The bit summation has a parity mismatch.



Figure 14: Layer 1 Problem: Symptoms and Diagnosis




Layer 1 Problem: Further Symptoms and Diagnosis Often a duplex mismatch is not immediately recognized. Another problem might be that a user connected a small unmanageable switch to the port and it is unknown if it is running half or full duplex mode.

Use sh ports txerrors and sh ports rxerrors to examine the port error statistics.and identify if there are incorrect settings. A duplex mismatch causes transmission errors to occur on both sides.

Switch 1 SymptomsOn a switch running in half duplex mode there are a lot of late collisions. The attached switch, running in full duplex mode, sends frames if there is data. It does not detect if the media is already used or not.

Switch 2 SymptomsOn a switch running in full duplex mode there are a lot of CRC errors. The attached switch, running in half duplex mode stops transmitting data each time it detects a collision.

The illustration shows Switch 1, running in half duplex mode and Switch 2, running in full duplex mode. The show ports txerrors command on Switch 1 shows the late collisions. The show ports rxerrors command on Switch 2 shows the CRC errors.

The following port receive error information is collected by the system:

● Port Number

● Link Status — The current status of the link. Options are:

■ Ready (the port is ready to accept a link)

■ Active (the link is present at this port)

■ Disabled (D): The link is disabled at this port.

■ Not Present (NP): The link is not present at this port.

● Receive Bad CRC Frames (RX CRC)—The total number of frames received by the port that were of the correct length, but contained a bad FCS value.

● Receive Oversize Frames (RX Over)—The total number of good frames received by the port greater than the supported maximum length of 1,522 bytes. Ports with jumbo frames enabled do not increment this counter.

● Receive Undersize Frames (RX Under)—The total number of frames received by the port that were less than 64 bytes long.

● Receive Fragmented Frames (RX Frag)—The total number of frames received by the port were of incorrect length and contained a bad frame check sequence (FCS) value.

● Receive Jabber Frames (RX Jabber)—The total number of frames received by the port that were longer than the supported maximum length and had a Cyclic Redundancy Check (CRC) error.

● Receive Alignment Errors (RX Align)—The total number of frames received by the port that have a CRC error and do not contain an integral number of octets.

● Receive Frames Lost (RX Lost)—The total number of frames received by the port that were lost because of buffer overflow in the switch.



Figure 15: Layer 1 Problem: Further Symptoms and Diagnosis




Troubleshooting at Layer 2: Data Link LayerVarious kinds of issues have to be considered while troubleshooting at the Data Link Layer. Common sources of problems include:

● Protocol mismatches

● MAC addressing problems

● VLAN problems

There might be protocol mismatches because of the different L2 protocols. The Ethernet II, IEEE 802.3, or LLC/SNAP setting causes the mismatch. A wrong driver in the Ethernet card of a server might also cause a mismatch.

Addressing ProblemsStart by answering the following questions to identify possible addressing problems.

Are there permanent MAC address entries in the Forwarding Database (FDB) which are incorrect?

Is there a faulty or misconfigured device that is responding to all ARP requests on a VLAN or network?

When troubleshooting spanning tree, answer the following questions:

● Which mode of spanning tree are you using?

● Does your spanning tree domain layout and your VLAN layout work together?

● Is the VLAN in the correct spanning tree domain?

● Is spanning tree enabled for the VLAN?

VLAN ProblemsWhen troubleshooting VLAN problems verify the following:

● The ports are properly configured on the VLAN.

● Consistent VLAN tags are used.

■ If VLAN tags are not properly configured, your 802.1Q link cannot work. Remember that VLAN names are only locally significant but you should use a consistent naming scheme, to prevent confusion and obstruct troubleshooting.

● The VLAN name is not a reserved word or the name includes special characters that are not allowed or the switch will not accept the name.

■ VLAN names consist of alphanumeric characters, must begin with an alphabetical letter, no spaces are allowed, and special characters are only allowed if the VLAN name is enclosed by quotation marks.

● The protocol filter is set to ANY or the desired protocol filter when using protocol based VLANs. When the filter is set to NONE, protocol based VLANS will not work. The protocol filter is set to NONE when a special protocol filter is deleted.

● The port you are adding is not already added to a different VLAN untagged or you will receive an error message.



Figure 16: Troubleshooting at the Data Link Layer




Commands for Layer 2 TroubleshootingThe illustration lists the most common show commands used to troubleshoot Layer 2. The proper use and the interpretation of the information displayed by each of these commands will be explained in the following section.Use the clear command to clear dynamic forwarding database (FDB) entries.

The output from the show commands is later used to identify the problem in the Layer 2 example. Each relevant field in the output of the command is explained.



Figure 17: Commands for Layer 2 Troubleshooting




Displaying Forwarding Database (FDB) informationUse the following command to verify the MAC addresses are learned and in the correct VLAN:

show fdb {<mac_address> | vlan <vlan name> | <portlist> | permanent}

Primary InformationThe show fdb command output shows:

● The Ethernet MAC address of each learned device in the forwarding database.

● The VLAN where the device is a member.

● The port where the address is learned.

The flags show:

● If the entry is an ingress or egress black hole entry (b, B).

For troubleshooting Layer 2 and Layer 3, verify that the expected device is physically connected to the port and is being learned. Verify that the entry is not a black hole. Verify that the device is in the right VLAN(s).

Secondary InformationThe show fdb command output also shows:

● The age of the FDB entry.

● The FDB statistics on total learned, static, locked, permanent, dynamic, dropped, locked, and locked with a timer.

● The age timer and VPLS age timer.

The flags show:

● If the entry is learned dynamically (d) or configured statically (s).

● If the entry is permanent (p).

● If the entry is locked down or locked with a timer (l, L).

● If the entry is on a port being used as the mirror port (M).

● If the entry is created by NetLogin (n).

Use the following command to clear dynamic FDB entries:

clear fdb {<mac_address> | locked-mac <mac_address> | vlan <vlan name> | ports <portlist>}



Figure 18: Displaying Forwarding Database (FSB) information




Displaying information about every VLAN one-by-oneUse this command to display detailed information for all VLANs.

show vlan detail

This command displays the same information as for an individual VLAN, but shows every VLAN, one-by-one.

Primary show vlan detail InformationThe show vlan detail command output shows:

● If the VLAN is enabled or disabled.

● If the VLAN is tagged and what the tag is.

● The virtual router where this VLAN is assigned.

● The IP address and subnetwork mask of the router interface of this VLAN.

● If loopback mode is enabled.

● The tagged and untagged ports assigned to this VLAN.

● Flags indicating if each port is:

■ active (*).

■ disabled (!).

■ part of a load sharing group (g).

■ blocked for this VLAN (b).

For Layer 1 problems verify that the port is enabled, active, and part of or not part of a load sharing group.

For Layer 2 problems verify that the port is:

● assigned to the correct VLAN.

● correctly tagged.

● not blocked.

Verify that the VLAN is:

● enabled.

● correctly tagged.

● not in loopback mode.

For Layer 3 related problems, verify that the following are correct:

● The IP address

● The subnetwork mask

● The virtual router



Figure 19: Displaying Information about every VLAN one-by-one




Displaying Information about every VLAN (continued)The show vlan detail command output also shows:

● The IPv6 address if one is assigned.

● The Spanning Tree Protocol Domain (STPD) name if one is assigned.

● The VLAN protocol filter name if one is assigned.

● If the NetLogin security feature is enabled.

● The name of the QOS policy file used to assign quality of service to packets on this VLAN if one is assigned.

● Flags indicating if each port is authenticated (a) or unauthenticated (u) for this VLAN by Netlogin.

For Layer 1 problems verify the port is authenticated if applicable.

For Layer 2 problems verify that the VLAN:

● is in the correct Spanning Tree domain.

● has the correct protocol filter if applicable.

● is correctly configured for security and QOS features if applicable.



Figure 20: Displaying Information about every VLAN (continued)




Layer 2 Problem: SymptomsOn each switch in this example the configuration looks fine, but a ping from Switch 1 to Switch 2 does not work.

To find the problem, check the VLAN configuration on both switches and compare them using the show vlan command.

Layer 2 Problem: Diagnosis and SolutionThe show vlan command shows an error in adding the ports to a VLAN. Ports can be added as tagged or untagged, but it must be the same setting on both sides of the link.

The show vlan <vlan name> command helps to find this error. This command shows if the ports are added tagged or untagged.

The tag is the same on both switches, so this is not the problem. However, on Switch 1 the port is tagged, while on Switch 2 the port is untagged. To solve the problem, change the configuration on one of the two switches.

Enter the following command on Switch 2 to solve the problem and provide connectivity between the two switches:

configure vlan tagproblem add ports 2 tagged

NOTE

In general it is recommended to always add ports tagged on inter-switch links so the port can be used as 802.1Q links for several VLANs.



Figure 21: Layer 2 Problem: Symptoms

Figure 22: Layer 2 Problem: Diagnosis and Solution




Troubleshooting at Layer 3: Network LayerTo have Layer 3 connectivity, the following rules must be observed:

● Every device must have a unique host address.

● Devices with the same subnetwork address have to be within the same VLAN.

● When devices are in separate VLANs, those VLANs must have different subnetwork addresses and the devices providing connectivity between them must be configured to forward Layer 3 traffic.

● Every device involved in Layer 3 forwarding must have a route to reach the network of the destination IP address or have a default route to the right gateway.

Routing AdvertisementsExtreme Networks switches do not advertise routes if:

● The VLAN does not have an IP address assigned to it.

● IP Forwarding is not enabled for that VLAN.

● The VLAN has not been added to the Routing Protocol.

● The Routing Protocol has not been enabled globally.

Special RIP IssuesFor issues affecting RIP, answer the following:

● Are Split Horizon, Poison Reverse, and Triggered Updates enabled?

Special OSPF IssuesFor issues affecting OSPF, answer the following:

● Does the switch discover its neighbors and establish adjacencies?

● Are all routers in the same area?

● Is OSPF enabled on the VLAN and globally?

● Are the timers correct?

● Are the Link State Databases synchronized? Is a link state advertisement (LSA) missing?

● Is SPF running too often because of flapping links?

● Is authentication used, ECMP enabled, and the Router ID unique?

In a stable network, the Link State Database (LSDB) does not change much. One way to determine whether the entries in the LSDB are changing is to monitor the checksum and SPF runs. These are indicators of how often the LSDB is changing. Using multiple areas there are issues concerning stub area, NSSA, default routes, ABRs, ASBRs, Virtual Links, Route Summarization and more.



Figure 23: Troubleshooting at the Network Layer

Figure 24: Troubleshooting Commands for Layer 3




Displaying IP Forwarding and Routing ProtocolUse the following command to verify that IP forwarding and the correct routing protocol is enabled for a VLAN.

show vlan

For each VLAN the show vlan command output shows:

● The name of the VLAN.

● The IP address and subnetwork mask of the router interface.

● If IP forwarding (f) is enabled.

● If OSPF is enabled on the VLAN (o).

● If RIP is enabled on the VLAN (r).

● If the VLAN loopback is enabled (L).



Figure 25: Displaying IP forwarding and Routing Protocol




Displaying VLAN Configuration InformationUse the following command to verify that IP forwarding and the routing protocol is enabled for a VLAN.

show ipconfig {vlan <vlan name>} {detail}

For each VLAN the show ipconfig command output shows:

● The name of the VLAN.

● The IP address and subnetwork mask of the router interface.

● If IP forwarding (f) is enabled.

● If the interface is enabled (E) and active (U).

Use this information to verify the following:

● The interface is up

● IP forwarding is enabled

● The interface has the correct IP address and subnetwork mask



Figure 26: Displaying VLAN Configuration Information




Displaying Contents of IP Routing TableUse the following command to verify which destination networks are in the routing table and the source of the routing entry.

show iproute {vlan <vlan name> | permanent | <ip address> <netmask> | origin [direct | static | blackhole | rip | icmp | ospf-intra]} {sorted}

The show iproute command is one of the most important commands for Layer 3 troubleshooting.

The show command output shows:

● The destination network.

● The next hop gateway.

● The preferred route for unicast and multicast traffic.

● The duration of time this route has been in the routing table.

Use this information to verify that the destination network is in the routing table. Verify where the packets are being sent for a particular designating network. Also verify if the preferred route is the intended route.

If a destination network is unreachable, check the following:

● Every router in the path knows the destination network

● The default gateway is in the correct direction

● The return path is correct

Verify that directly connected routes are used when IP route sharing is enabled using Equal Cost Multi Paths (ECMP). For IP route sharing directly connected routes should be used instead of other routes even if the cost or relative route priority of the other routes has been manually set.

The type of route entry flags show:

● The route is Dynamic (D).

● The route is a Black hole (B).

● The route is a Gateway (G).

● The route is a Host Route (H).

● The route is a Label Distribution Protocol (LDP) Label Switch Path (LSP).

● The route is an indirect LDP LSP. (i)

● The route is a multicast (m) or unicast (u) route.

● The route is a Longest Prefix Match (LPM) route (P).

● The route is modified (R).

● The route is a Static route (S).

● The route is a Resource Reservation Protocol (RSVP) Traffic Engineering (TE) LSP. (T)

● The route is an indirect RSVP-TE LSP. (t)

● The route is up (U).



Figure 27: Displaying Contents of IP Routing Table




Verifying Contents of IP Routing TableThe command show iproute also shows the origin of the route. In addition to OSPF, routes can be learned from:

● Directly connected interfaces (d)

● Border Gateway Protocol (bg), External BGP (be), or Internal BGP (bi)

● BootP (bo)

● Core Based Tree (CBT) multicast routing protocol (ce)

● Down Interface (df)

● Distance-vector Multi Routing Protocol (dv)

● Intermediate System - Intermediate System (ISIS) level-1 external (e1) (ExtremeWare only)

● Intermediate System - Intermediate System level-2 external (e1) (ExtremeWare only)

● ISIS level-1 internal (i1) and ISIS level-2 internal (i2) (ExtremeWare only)

● Hardcoded (h)

● Internet Control Message Protocol (ICMP) (i)

● Multicast Border Gateway Protocol (MBGP) (mb), MBGP external (mbe), or MBGP internal (mbi)

● Multicast Open Shortest Path First (MOSPF) (mo)

● OSPF external LSA type 1 (o1) and OSPF external LSA type 2 (o2)

● OSPF intranetwork (oa) and OSPF internetwork (or)

● OSPF external Autonomous System (oe)

● Protocol Independent Multicast - Dense Mode (pd), PIM-Sparse Mode (ps)

● Routing Information Protocol (RIP) (r)

● Route Advertisement (ra)

● Static (s)

● Server Load Balance (SLB) virtual server IP (VIP) (sv) (ExtremeWare only)

● Unknown (un)

Use this information to determine the source of the routes. This is useful for the following:

● When using route redistribution

● When troubleshooting routing loops

● When unexpected routes are showing up in the routing table

Directly connected routes have a relative route priority of 10. Relative route priorities cannot be modified for directly connected routes or black hole routes.

Redistribution of RIP routes into OSPF results in OSPF external routes with different relative priorities.



Figure 28: Verifying Contents of IP Routing Table




Displaying the IP Address Resolution Protocol TableUse the following command to verify the contents of the IP Address Resolution Protocol (ARP) table:

show iparp {<ip address> | <mac_address> | vlan <vlan name> | permanent}

The show iparp command output shows:

● Each Ethernet MAC address that has been mapped by ARP.

● The IP address mapped to the MAC address. If the entry is marked incomplete, it means the switch is arping for an IP address that is not responding.

● The port on which the MAC address is located.

● The VLAN on which the MAC address is located.

● The virtual router associated with the VLAN on which the MAC address is located.

● A list of all duplicate IP addresses detected.

Use this information to verify that each IP address is mapped to a single MAC address. Duplicate IP addresses are a very common misconfiguration and can cause confusing and unpredictable behavior. Also verify that the IP address is in the correct VLAN and subnetwork.

To remove dynamic entries in the IP ARP table, enter the following command:

clear iparp {<ip address> | vlan <vlan name>}

Secondary InformationThe show iparp command output also shows:

● If the ARP mapping entry is statically configured.

● The age of the entry in seconds.

● Statistics for:

■ The current number of dynamic and static entries.

■ The current number of pending entries. The ARP request is sent but no response has been received so far.

■ The number of ARP requests received (in request), ARP responses transmitted (out response), ARP requests sent (out request), and ARP responses received (in response).

■ The number of failed requests. A request was sent but no response was ever received.

■ The number of received ARP responses that were rejected due to some protocol error in the packet. For all ARP responses that were rejected what is the count, port, IP address, and interface.

● The maximum number of ARP entries and ARP pending replies configured or supported on this switch.

● Whether ARP address checking and ARP refresh are enabled. Checking checks if the ARP Request source IP address is within the range of the local interface. Disabling ARP refresh stops the sending of any ARP requests before the ARP entry times out.

● The ARP entry age timeout.

Use this information to verify that ARP is working correctly. Also use this to verify that the proxy ARP server is responding



Figure 29: Displaying the IP Address Resolution Protocol Table




Displaying Global OSPF InformationUse the following command to verify the OSPF area ID, router state, and link cost.

show ospf

The show command output shows:

● If OSPF is enabled.

● The router ID and whether it is configured or automatically assigned.

● If this router is an ASBR or ABR.

● If the router ID is configured or automatically selected.

Use this information to verify that OSPF is enabled globally on the router. Verify that the router ID is configured correctly. Virtual links are built using the router ID.

Verify that the router is acting in the right role: Interior Router, ABR, or ASBR.



Figure 30: Displaying Global OSPF Information




Displaying RIP Specific Configuration InformationUse the following command to verify RIP specific configuration for all VLANs. enter the following command:

show rip

The command displays the following:

● The global status of:

■ RIP

■ Split Horizon

■ Poison Reverse

■ Triggered Updates

● RIP protocol timers



Figure 31: Displaying RIP Specific Configuration Information




Displaying IP Statistics for the CPUTo display IP statistics for the CPU for the switch or for a particular VLAN, enter the following command:

show ipstats {vlan <vlan name>}

The command displays the following:

● Packet statistics and error statistic for IP.

● Packet statistics and error statistic for the Internet Control Message Protocol (ICMP).

● Packet statistics and error statistic for the Internet Group Management Protocol (IGMP).

Use the IP statistics to identify if errors are causing the protocol to fail or converge slowly.

Use the ICMP statistics to identify if errors are causing connectivity failures or ping response failures.

Use the ICMP statistics to identify if errors are causing problems with multicast routing.

Displaying IP Statistics for the VLANIf errors are seen in the IP statistics information, use the VLAN IP statistics section to determine which VLAN is experiencing the errors.

The second part of the display shows the following for each VLAN:

● The IP address and network mask of the router interface

● Packets and Bytes in and out

● Multicast and Broadcast packets in and out

● Packets with errors and discarded packets

● Packets received of a unknown protocol type



Figure 32: Show IP Statistics for the CPU

Figure 33: Displaying IP Statistics for the VLAN




Using ICMP Commands for Layer 3 TroubleshootingUse the following command to test end-to-end connectivity:

ping {udp} {continuous} {size <start_size> {-<end_size}} [<ip_address> | <hostname>] {from <src_ipaddress>}

Ping is a good tool to use to start troubleshooting. If ping responses are returned, the problem must be at a higher layer. If the ping response is not received, the problem is at the IP or lower layer.

NOTE

If you ping a router interface that is configured on the switch and is active, the router interface responds with an ICMP echo response even if IP forwarding is not enabled. The router interface may appear to be forwarding packets at Layer 3 even though it is not.

Use the following command to trace the routed path between the switch and a destination end station.:

traceroute {vr <vrid>} {ipv4 <host>} {ipv6 <host>} {ttl <number>} {from <from>} {[port <port>] | icmp}

Traceroute works by increasing the time-to-live (TTL) value of each successive batch of packets sent. The first three packets have a time-to-live (TTL) value of one (implying that they make a single hop). The next three packets have a TTL value of 2, and so on.

The command output shows the IP address of the router interface at each hop along the path and the number of milliseconds it took each packet to receive an ICMP time exceeded packet from that interface.

The command continues until the user enters Ctrl-c to abort the command.

The traceroute command is useful in determining which route a packet is taking when multiple routes are available. It is also useful in identifying firewalls that may be blocking access to a site.

If the trace route cannot reach the destination the command output identifies the reason why. Failures can be due to:

● ICMP network unreachable - There is no path. Check the routing table. Make sure you are specifying the correct virtual router. Make sure IP forwarding is enabled.

● ICMP host unreachable - Make sure you entered the correct address.

● ICMP fragmentation needed - The packet is too large for one of the routers in the path. Enable fragmentation on the router.

● ICMP source route failed - Source routing is seldom supported.

● Transmit error - Check IP statistics for IP errors.



Figure 34: Additional Troubleshooting Commands for Layer 3




Layer 3 Problem: SymptomsIn this example the IP configuration looks correct and Switch 1 has connectivity with Host 1 and Switch 2. But Host 1 cannot reach Switch 2.

A closer look shows that network 10.1.1.0/24 is not in the routing table of Switch 2 although RIP is running. But even if we configure a static route for network 10.1.1.0/24 on Switch 2 there is still no connectivity. Of course configuring a static route would not be a proper solution, even if it helps, because we are running a dynamic routing protocol.

A typical example for a Layer 3 error is that IP forwarding is not enabled for one VLAN.

The next step is to take a closer look at Switch 1's IP configuration, perhaps IP forwarding is not enabled for all VLANs. There are several commands to use to display the relevant information.

Layer 3 Problem: DiagnosisUse the following command to find the error.

show ipconfig

The first screen displays the switch global settings. You see that IP routing and RIP are enabled correctly. However, this does not mean that these functions are enabled for all VLANs.

NOTE

Part of the output is omitted from the illustration. What is displayed is the overview of the router interfaces because that is what is important.

For the VLAN named noipforwarding, IP forwarding is not enabled. This is the cause of the problem.



Figure 35: Layer 3 Problem: Symptoms

Figure 36: Layer 3 Problem: Diagnosis




Layer 3 Problem: SolutionAnother way to find the error is entering the following command:

show vlan

When you use this command without any parameters it displays an overview of the configured VLANs. The display includes the most important IP information.

The flags indicate that IP forwarding is not enabled for the VLAN noipforwarding and the error is detected.



Figure 37: Layer 3 Problem: Solution




Interpreting a syslog FileThe illustration shows an example of system log messages. Examples of events that might generate a log message are:

● A link going down.

● A user logging in.

● A command entered on the command line.

● The software executing a debugging statement.

Each log messages contains the following:

● A timestamp of when the event was logged. The timestamp is in the format yyyy-mm-dd,hh:mm:ss.

● The sending device’s IP Address.

● The IP Port number used for receiving log messages. 23 in the example.

● The Syslog Facility number 7 is the default for Extreme Networks devices.

● The protocol reporting the event. OSPF in this example.

● The actual log message.

Use system logging to track a series of events. For example:

● Start Intra area SPF Area 0.0.0.10. (The start of an SPF run for area 0.0.0.10)

● Start IntraArea Route Table update SPF area 0.0.0.10. (The start of the routing table update)

● End IntraArea Route Table update area 0.0.0.10. (The end of the routing table update)

● End IntraArea Spf Area 0.0.0.10. (The end of an SPF run for area 0.0.0.10)

NOTE

The syslog facility can be set to different values and the receiver can be programmed to treat different facility levels uniquely.



Figure 38: Interpreting a syslog File




Sample syslog File: You Set ParametersSwitch configuration and fault information is filtered and saved to target logs, in a memory buffer, and in NVRAM.

Use the following command to display system log messages.

show log {messages [memory-buffer | nvram]} {events {<event-condition> | <event-component>]} {<severity> {only}} {starting [date <date> time <time> | date <date> | time <time>]} {ending [date <date> time <time> | date <date> | time <time>]} {match <regex>} {chronological}

The show log command displays the messages stored in either the internal memory buffer or in NVRAM depending on the switch type. The messages shown can be limited by specifying a severity level, a time range, or a match expression.

Each entry in the log contains the following information:

● Timestamp - records the month and day of the event, along with the time (hours, minutes, seconds, and hundredths of a second).

● Severity Level - indicates the urgency of a condition reported in the log. Severities include critical, error, warning, notice, info, debug-summary, debug-verbose, and debug-data.

● Component, Subcomponent, and Condition Name - describes the subsystem in the software that generates the event. This provides a good indication of where a fault might be.

● Message - A description of the event occurrence. If the event was caused by a user, the user name is also provided.

Use this information in the same way you use the remote system logging messages.

Logging of CLI configuration commands must be enabled. The switch log overwrites existing log messages in a wrap-around memory buffer.



Figure 39: Sample syslog File: You Set Parameters




Systematic Troubleshooting StepsThe following steps should be followed when troubleshooting any problem:

● Ask questions to clearly define the problem.

● Gather information about the network.

● Consider escalating the problem to someone with additional expertise. The goal is to reach a resolution as fast as possible.

● Determine if the problem can be consistently reproduced or if it is intermittent.

● Document the history of the problem. Identify when the problem was first noticed and what events preceded the problem occurrence.

● Locate information about similar problems. This may indicate possible sources of the problem.

● Develop theories about what could be wrong. Start with simple ideas and advance to more complex issues.

● Test theories starting with the most inexpensive and simple, then test the more expensive and complex theories until the solution is found.

● Always document the problem, theories, tests, and end resolution.

● Retain the problem documentation and make it available for future troubleshooting.



Figure 40: Systematic Troubleshooting Steps




Defining the ProblemMake sure you have all relevant information about the problem. Begin by getting answers to the following questions:

● What are the symptoms?

● Is the problem reproducible?

● How long has the problem been happening?

● How often does the problem happen?

● When does the problem happen?

● Is the issue affecting only one device or a large portion of the network?

● Is the issue isolated within a VLAN or does it also affect Layer 3 functions?

● How many users are affected?

● Which applications are affected?

● Does the problem seem to be related to the network load?

● Does the problem seem related to a new network installation or change?

● Was the network previously operating properly?

■ How long was the network operational prior to this issue arising?



Figure 41: Defining the Problem




Gathering Information Used for TroubleshootingCollect all relevant documentation about the network. Use the following list and accompanying flow chart:

● Physical network layout

■ A detailed network diagram with all connected devices, network addresses, and physical media types

● Logical network layout

● Traffic bandwidth baselines

■ You have to know your network in a stable situation, so that you know what to expect when looking at the status of the network.

● Availability reports

● Records of recent changes made to the network

● Current status of devices and connections

● Event and error logs

Keep manuals and release notes of implemented equipment ready to use.

Make sure that everybody involved in troubleshooting has access to this information.



Figure 42: Gathering Information Used for Troubleshooting




Consider EscalationBefore you escalate a problem to technical support, ask the following:

● What is the impact on the users?

● What is the impact on the resiliency of the network?

● What is the cost per day of the problem?

● Does the problem seem to be in an area you have expertise in?

● How much time has already been spent on troubleshooting?

● What is covered in your support contract?

Escalating does not mean you have to stop troubleshooting.

Escalation is most effective when you provide a good problem description and all relevant network information.

Extreme Networks TAC EscalationThe process for escalating a problem to Extreme Networks is documented in the file found at:

http://www.extremenetworks.com/services/tac-userguide.aspx

Prior to contacting Extreme Networks upgrade to the newest software release and gather the following:

● Your company name and the phone number of the contact

● Serial number(s) of the switch chassis and additional module(s)

● Service contract number

● Detailed problem description

● Output of the following commands:

■ show version

■ show switch

■ show configuration

■ show tech-support or show diagnostics

■ show log

■ run diagnostics extended

● Remote login information for Extreme Networks TAC

NOTE

The output of the show tech-support command can be very large.



Figure 43: Consider Escalation

Figure 44: Extreme Networks Escalation




Developing and Testing TheoriesDevelop theories based upon the information you have collected. Do this in a structured way, so that you do not repeat steps, but work your way trough all possible causes.

Document your theories to provide a checklist for testing.

Testing TheoriesA layered approach gives structure to the troubleshooting process.

Use the OSI model to structure tests. Start testing the lowest layers first then work your way up the OSI layers.

For example:

1. Use the link Light Emitting Diodes (LEDs), show commands, and physical inspection of the cable to verify the Physical Layer.

2. Use the ping command, verify packet statistics, verify port settings, and verify the Layer 2 forwarding tables to test Layer 2 connectivity.

3. Use an end-to-end ping command, trace route command, verify Layer 3 configuration, and verify Layer 3 route tables to test Layer 3 connectivity.

4. Use telnet, FTP, TFTP, or SNMP to test protocols at the top three layers.

If the test results support the theory, go on to implementing a solution.

If the test results do not support the theory, go back to test a new theory.



Figure 45: Developing and Testing Theories




Implementing a SolutionAfter you implement a solution, recheck the status of the problem. If there is still a problem, go back to either gather more information or test different theories.

The symptoms of the problem might change as a result of the troubleshooting actions. If you did not improve the situation by your action, consider undoing the actions.



Figure 46: Implementing a Solution




Documenting the SolutionTake care of your documentation. Logging of problems and solutions helps the next time you have to troubleshoot. It also provides a record of repetitive or intermittent failures and the uptime of your network.

If you make configuration changes, make sure that you update the network documentation so it is accurate.

After the correct solution is implemented, document the following:

● The symptoms

● The time and frequency of the symptom

● The theories tested

● The theory that proved to be correct

● The solution

● The root cause of the problem

● Any ideas of how to recognize this same problem in the future

● How long the network operation was affected



Figure 47: Documenting the Solution




SummaryThe Network Troubleshooting module presented basic concepts of network troubleshooting and the show commands to use.

You should now be able to:

● Identify the network management and maintenance systems that prepare you to troubleshoot.

● Define the steps to perform systematic troubleshooting.

● Describe the layered approach for troubleshooting.

● Identify useful commands for finding errors at each layer.

● Interpret the output of show commands.

Troubleshoot problems at the Physical, Data Link, and Network Layers.



Figure 48: Summary




Exoc_module 15 Network Troubleshooting

Documents

regularly scheduled basis

transmit errored frames

receive oversize frames

receive fragmented frames

receive undersize frames

link integrity signal

sample syslog file

receive jabber frames