Router Installation and Configuration Manual/Troubleshooting

From ImageStream Router Documentation

< Router Installation and Configuration Manual
Revision as of 19:56, 4 May 2011 by Jc (Talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Contents

Introduction

This chapter presents general troubleshooting information and a discussion of tools and techniques for troubleshooting serial connections.
The chapter includes the following topics:
  • Troubleshooting with the interface statistics detail screen
  • Serial Lines: Line Status Conditions
  • Serial Lines: Increasing Output Drops On Serial Link
  • Serial Lines: Increasing Input FIFO Buffer Drops on Serial Link
  • Serial Lines: Increasing Input Non-FIFO Buffer Drops on Serial Link
  • Serial Lines: Input Errors Of Over 1% Of Total Interface Traffic
  • Serial Lines: Troubleshooting Serial Line Input Errors
  • Serial Lines: Increasing Carrier Transitions Count on Serial Link
  • Troubleshooting Clocking Problems
  • Troubleshooting T1/E1 CSU/DSU Problems

Troubleshooting with the Interface Statistics Detail Screen

The output of the interface statistics detail screen displays information specific to the serial interface or subinterface. Please see the Chapter 27, Router Installation and Configuration Manual/Using The Interface Statistics (stats) Program, for more information on accessing this program. The example below shows the output of the detail page for a High-Level Data Link Control (HDLC) serial interface.
The following section describes how to use this screen to diagnose serial line connectivity problems in a wide area network (WAN) environment. The following sections describe some of the important fields of the command output.

Serial Lines: Line Status Conditions

Serial Lines: Line Status Conditions
18:08:37	Serial2 Detail	                  router
Serial2 is   up , protocol is   up
     Description	: Qwest T1
     Encapsulation	: Cisco HDLC
     IP address	: 63.148.112.74 255.255.255.252
     Broadcast address : 63.148.112.75
     Line has been up since Mon Oct 28 14:59:54 2002 (1w3h)
     Last input :	00:00:00	Last output:	00:00:00
     Bandwidth:	1.54 Mbit   Load in:	23% Load out:	37%
     3 second average input rate :	354.58 Kb/s,	44.32 KB/s, 112 packets/s
     3 second average output rate:	0.57 Mb/s,	71.64 KB/s, 142 packets/s
          Rx Packets	265,482,595	7,851,846 bytes
          Tx Packets	378,208,552	388,020,255 bytes
          Rx Errors: 568	(0 CRC	235 frame	0 fifo	333 dropped)
          Tx Errors: 0	(0 collisions	0 fifo	0 dropped)
          Carrier transitions 7 
          DCD =   up   DSR =   n/a DTR =   n/a RTS =   n/a CTS =   n/a 
c CSU | y Summary | n Next | p Previous | z Zero | h Help | q Quit 
You can identify four possible states in the interface status line (in bold) of the example display above:
# Serial2 is up, line protocol is up 
# Serial2 is down, line protocol is down 
# Serial2 is up, line protocol is down 
# Serial2 is administratively down, line protocol is down 
The table below shows the interface status conditions for the example "Serial2" device, possible problems associated with the conditions, and potential solutions to those problems.
Status Condition Possible Problem Solution
Serial2 is up, line protocol is up None 1. This is the normal line condition. No action Required
Serial2 is down, line protocol is down 1. Local router is misconfigured. 1. Check the DCD = section near the bottom of the output to see if the carrier detect signal is active, or insert a breakout box on the line to check for a CD signal
2. Typically indicates that the router is not sensing a carrier signal, except with X.21 serial interfaces and unstructured mode E1, where there is no carrier signal. 2. Verify that you are using the proper cable and interface (see the Hardware Installation Guide).
3. Telephone company problem - line is down or line is not connected to CSU/DSU 3. Insert Breakout box and check all control leads
4. Faulty or incorrect cabling 4. Contact your leased line provider or other carrier service to see if they are experiencing a problem
5. Hardware failure (CSU/DSU) 5. If you suspect faulty router hardware, change the serial line to another port. If the connection comes up, the previously connected interface has a problem.

6. Swap faulty suspected parts.

Serial2 is up, line protocol is down 1. Local or remote router is misconfigured 1. Insert a loopback plug into the CSU/DSU (or crossover the coaxial cable from the transmit side to the receive side) and see if the router receives its own packets normally. You should see a received packet for each transmitted packet.
2. Remote router is not using IEFT frame relay (when using frame relay encapsulation 2. If data is being received normally, a telephone company problem or a misconfigured/failed remote router is the likely problem
3. Remote router is not using proper ATM encapsulation (when using ATM encapsulation) 3. If problem appears to be on the remote end, repeat Step 1 on the remote CSU/DSU.
4. Keepalives are not being sent by remote router 4. Verify all cabling. Make sure that the cable is attached to the correct interface, the correct CSU/DSU, and the correct telephone company network terminal point
5. Leased line provider or other carrier service problem - Noisy line, or misconfigured or failed switch 5. Enable debug protocol command in the interface configuration. The debug protocol command displays all protocol negotiation packets to the router's debug log, available from the "Advanced Menu"

Caution: because debugging output is assigned a high priority in the CPU process, it can use considerable system resources. For this reason, the use of the debug command should be limited to troubleshooting specific problems or during troubleshooting sessions with ImageStream's technical support personnel during periods of low network traffic or when fewer users are expected.

6.Failed local or remote CSU/DSU 6. If the port does not receive its own packets back when looped, a router hardware problem is likely.
7. Router hardware failure (local or remote) 7. If you suspect faulty router hardware, change the serial line to an unused port. If the connection comes up, the previous connected interface is experiencing a problem.

8. If the line protocol comes up and the keepalive counter increments in the debug log, the problem is not in the local router. Troubleshoot the serial line as described in the section "Troubleshooting Clocking Problems" later in this chapter. 9. Swap faulty parts.

Serial2 is up, line protocol is down 1. Missing internal 'clock source interface configuration command 1. Add the service-module XX clock source internal interface configuration command on the serial interface. See Router Installation and Configuration Manual/Configuring an Integrated CSU/DSU WAN Interface or the Command Reference for the correct syntax.
2. Missing 'timeslot interface configuration command Add the service-module XX timeslot interface configuration command on the serial interface. See Router Installation and Configuration Manual/Configuring an Integrated CSU/DSU WAN Interface or the Command Reference for the correct syntax.
3. Failed remote CSU/DSU 3. Verify that the correct cable is being used
4. Failed or incorrect cable 4. If the line protocol is down there is a possible hardware failure or cabling problem. Insert a breakout box and observe leads.
5. Router hardware failure 5. if you suspect faulty router hardware, move the serial line to an unused port. Check to see if the connection comes up, the previous connected interface has a problem.

6. Swap faulty parts.

Serial2 is administratively down, line protocol is down 1. Router configuration includes shutdown interface configuration command 1. Check the wan.conf' file for the shutdown command.

2. Delete the shutdown command from the wan.conf file.


Serial Lines: Increasing Output Drops on Serial Link

     18:08:37	      Serial2 Detail	                           router 
     Serial2 is   up , protocol is   up 
          Description	        : Qwest T1 
          Encapsulation	: Cisco HDLC 
          IP address	        : 63.148.112.74 255.255.255.252 
          Broadcast address    : 63.148.112.75 
          Line has been up since Mon Oct 28 14:59:54 2002 (1w3h) 
          Last input :	00:00:00	Last output:	00:00:00 
          Bandwidth:	1.54 Mbit   Load in:	23% Load out:	37% 
          3 second average input rate :	354.58 Kb/s,	44.32 KB/s, 112 packets/s 
          3 second average output rate:	0.57 Mb/s,	71.64 KB/s, 142 packets/s 
               Rx Packets	265,482,595	7,851,846 bytes 
               Tx Packets	378,208,552	388,020,255 bytes 
               Rx Errors: 568	(0 CRC	235 frame	0 fifo	333 dropped) 
               Tx Errors: 0	(0 collisions	0 fifo	0 dropped)
               Carrier transitions 7 
               DCD =   up   DSR =   n/a DTR =   n/a RTS =   n/a CTS =   n/a 
     c CSU | y Summary | n Next | p Previous | z Zero | h Help | q Quit 
Output drops appear in the Tx Errors section (in bold) of the interface detail screen when the router is attempting to hand off a packet to a transmit buffer but no buffers are available. The table below shows the possible problems that may cause an increasing number of output drops on a serial link, and potential solutions to those problems.
Possible Problem Solution
Output (Tx) rate to serial interface exceeds bandwidth available on serial link 1. Minimize periodic broadcast traffic (such as routing updates, or spanning tree bridging algorithm updates) by using firewall/filtering rules or by disabling unneeded services. For example, to disable the spanning tree algorithm, use the command spanning-tree disabled in the interface configuration.

2. Implement queuing and prioritization using quality of service (QoS) tools. For information on configuring QoS, see Chapter 18, Router Installation and Configuration Manual/Configuring Services: Quality of Service Menu.

3. Note: Output drops are acceptable under certain conditions. For instance, if a link is known to be overused (with no way to remedy the situation), it is often preferable to drop packets than to queue them. This is true for protocols that support flow control and can retransmit data (such as TCP/IP). However, some protocols, such as UDP (ping, SNMP, etc.) are sensitive to dropped packets and often do not accommodate retransmission.

Serial Lines: Increasing Input Fifo Buffer Drops on Serial Link

     18:08:37	      Serial2 Detail	      router
     Serial2 is   up , protocol is   up
          Description	 : Qwest T1
          Encapsulation	: Cisco HDLC
          IP address	: 63.148.112.74 255.255.255.252
          Broadcast address : 63.148.112.75
          Line has been up since Mon Oct 28 14:59:54 2002 (1w3h)
          Last input :	00:00:00	Last output:	00:00:00
          Bandwidth:	1.54 Mbit   Load in:	23% Load out:	37%
          3 second average input rate :	354.58 Kb/s,	44.32 KB/s, 112 packets/s
          3 second average output rate:	0.57 Mb/s,	71.64 KB/s, 142 packets/s
               Rx Packets	265,482,595	7,851,846 bytes
               Tx Packets	378,208,552	388,020,255 bytes
               Rx Errors: 568	(0 CRC	235 frame	0 fifo	333 dropped)
               Tx Errors: 0	(0 collisions	0 fifo	0 dropped)
               Carrier transitions 7 
               DCD =   up   DSR =   n/a DTR =   n/a RTS =   n/a CTS =   n/a 
     c CSU | y Summary | n Next | p Previous | z Zero | h Help | q Quit 
Input FIFO buffer drops appear in the Rx Errors section (in bold) of the interface detail screen when too many packets from that interface are being processed by the router. The table below shows the possible problems that may cause an increasing number of input drops on a serial link, and potential solutions to those problems.
Possible Problem Solution
Input rate exceeds the capacity of the router or input queues exceed the size of output queues 1. Note: Input drop problems are typically seen when traffic is being routed between faster interfaces (such as Ethernet and serial interfaces). When traffic is light, no drops occur. As traffic rates increase, packet backlogs start occurring. Routers drop packets during these congested periods.

2. Implement queuing and prioritization using Quality of Service(QoS) tools. For information on configuring QoS, see Router Installation and Configuration/Configuring Services: Quality of Service Menu.

3. Note: Input drops are acceptable under certain conditions. For instance, if a link is known to be overused (with no way to remedy the situation), it is often preferable to drop packets than to queue them. This is true for protocols that support flow control and can retransmit data (such as TCP/IP). However, some protocols, such as UDP (ping, SNMP, etc.) are sensitive to dropped packets and often do not accommodate retransmission.


Serial Lines: Increasing Non-Fifo Input Drops on Serial Link

     18:08:37	      Serial2 Detail	      router
     Serial2 is   up , protocol is   up
          Description	: Qwest T1
          Encapsulation	: Cisco HDLC
          IP address	: 63.148.112.74 255.255.255.252
          Broadcast address : 63.148.112.75
          Line has been up since Mon Oct 28 14:59:54 2002 (1w3h)
          Last input :	00:00:00	Last output:	00:00:00
          Bandwidth:	1.54 Mbit   Load in:	23% Load out:	37%
          3 second average input rate :	354.58 Kb/s,	44.32 KB/s, 112 packets/s
          3 second average output rate:	0.57 Mb/s,	71.64 KB/s, 142 packets/s
               Rx Packets	265,482,595	7,851,846 bytes
               Tx Packets	378,208,552	388,020,255 bytes
               Rx Errors: 568	(0 CRC	235 frame	0 fifo	333 dropped)
               Tx Errors: 0	(0 collisions	0 fifo	0 dropped)
               Carrier transitions 7 
               DCD =   up   DSR =   n/a DTR =   n/a RTS =   n/a CTS =   n/a 
     c CSU | y Summary | n Next | p Previous | z Zero | h Help | q Quit 
Non-FIFO input drops appear in the Rx Errors section (in bold) of the interface detail screen when the router receives a packet that cannot be processed. The table below shows the possible problems that may cause an increasing number of non-FIFO input drops on a serial link, and potential solutions to those problems.
Possible Problem Solution
1. Non-IP/SNAP packets being processed by IP/SNAP interface 1. Check the remote router configuration to ensure that only supported data types are being transmitted across the serial link. Disable forwarding of non-supported data types.

2. Note: Input drops do not necessarily denote a problem that must be addressed if the drop percentage is small compared to the total interface traffic.

2. Proprietary diagnostic packet types being sent by remote router 1. Disable proprietary diagnostic packets such as Nortel's "Breath of Life" and Cisco Discovery Protocol.

2. Note: Input drops do not necessarily denote a problem that must be addressed if the drop percentage is small compared to the total interface traffic.


SERIAL LINES: INPUT ERRORS OF OVER 1% OF TOTAL INTERFACE TRAFFIC

     18:08:37	      Serial2 Detail	      router
     Serial2 is   up , protocol is   up
          Description	: Qwest T1
          Encapsulation	: Cisco HDLC
          IP address	: 63.148.112.74 255.255.255.252
          Broadcast address : 63.148.112.75
          Line has been up since Mon Oct 28 14:59:54 2002 (1w3h)
          Last input :	00:00:00	Last output:	00:00:00
          Bandwidth:	1.54 Mbit   Load in:	23% Load out:	37%
          3 second average input rate :	354.58 Kb/s,	44.32 KB/s, 112 packets/s
          3 second average output rate:	0.57 Mb/s,	71.64 KB/s, 142 packets/s
               Rx Packets	265,482,595	7,851,846 bytes
               Tx Packets	378,208,552	388,020,255 bytes
            Rx Errors: 568	(0 CRC	235 frame	0 fifo	333 dropped)
               Tx Errors: 0	(0 collisions	0 fifo	0 dropped)
               Carrier transitions 7 
               DCD =   up   DSR =   n/a DTR =   n/a RTS =   n/a CTS =   n/a 
     c CSU | y Summary | n Next | p Previous | z Zero | h Help | q Quit 
If input errors appear in the Rx Errors section (in bold) of the interface detail screen, there are several possible sources of these errors. An input error (Rx Errors) value for CRC (cyclic redundancy check) and/or frame above one percent of the total input (Rx Packets) traffic indicates a line problem that should be isolated and corrected. The table below shows the possible problems that may cause input drops to exceed 1% of total traffic on a serial link, and potential solutions to those problems.
Possible Problem Solution
The following problems can result due to the following:
  • Faulty telephone or company equipment
  • Noisy serial line
  • Incorrect clocking configuration
  • Incorrect cable or cable too long
  • Faulty cable or connection
  • Faulty CSU/DSU
  • Faulty router hardware
  • Improper or malfunctioning media converter
  • line tap or other device being used between router and CSU/DSU
1. Use a serial analyzer to isolate the source of the input errors. If you detect errors, it is likely that there is a hardware problem or a clock mismatch in a device that is external to the router.

2. Use a loopback plug (or cross over a coaxial cable) to isolate the router and CSU/DSU from the telephone company equipment or other carrier equipment.

3. Look for patterns. For example, if errors occur at consistent intervals, they could be related to a periodic function such as the sending of routing updates or broadcast traffic.

4. Check for any errors or alarms on the telephone company equipment or other carrier equipment.

5. If the problem can be isolated to the telephone company equipment or other carrier equipment, ask your provider to fully test the line. If simple automated tests do not locate the problem, ask the provider to run the following tests in order:

  • All 0's for 5-20 minutes with no errors:
  • All 1's for 5-20 minutes with no errors:
  • Quasi-random word (QRW) for 5-20 minutes with no errors


Serial Lines: Troubleshooting Serial Line Input Errors

     18:08:37	      Serial2 Detail	      router
     Serial2 is   up , protocol is   up
          Description	: Qwest T1
          Encapsulation	: Cisco HDLC
          IP address	: 63.148.112.74 255.255.255.252
          Broadcast address : 63.148.112.75
          Line has been up since Mon Oct 28 14:59:54 2002 (1w3h)
          Last input :	00:00:00	Last output:	00:00:00
          Bandwidth:	1.54 Mbit   Load in:	23% Load out:	37%
          3 second average input rate :	354.58 Kb/s,	44.32 KB/s, 112 packets/s
          3 second average output rate:	0.57 Mb/s,	71.64 KB/s, 142 packets/s
               Rx Packets	265,482,595	7,851,846 bytes
               Tx Packets	378,208,552	388,020,255 bytes
               Rx Errors: 568	(0 CRC	235 frame	0 fifo	333 dropped)
               Tx Errors: 0	(0 collisions	0 fifo	0 dropped)
               Carrier transitions 7 
               DCD =   up   DSR =   n/a DTR =   n/a RTS =   n/a CTS =   n/a 
     c CSU | y Summary | n Next | p Previous | z Zero | h Help | q Quit 
If input errors appear in the Rx Errors section (in bold) of the interface detail screen, there are several possible problems that may be causing these errors. The table below shows the possible problems that may cause input drops to exceed 1% of total traffic on a serial link, and potential solutions to those problems.
Input Error Type Possible Problem Solution
CRC errors (CRC) Cyclic Redundancy Check errors occur when the CRC calculation produces an error-indicating that data is corrupted-for one of the following reasons:
  • Noisy serial line
  • Serial cable is too long, or the cable is not shielded (interference or signal loss problems)
  • CSU/DSU clocking is incorrectly configured
  • "Ones density" problem on T1 link (incorrect framing or encoding configuration)
1. Ensure that the line is clean enough for transmission requirements. Shield the cable if necessary.

2. Make sure the cable is within the recommended length - ideally, no more than 50 feet (15.24 meters) from the telephone company or carrier equipment.

3. Make sure the cable between the CSU/DSU and the router is within the recommended length - ideally, no more than 25 feet (7.62 meters) for V.35.

4. Ensure that all devices are properly configured for a common and single line clock. Each line must have one and only one clock source configured on all devices. In most cases, the leased line provider will provide the clocking for the line.

5. Make certain that the local and remote CSU/DSU's are configured for the same framing and encoding scheme as that used by the leased-line or other carrier service (for example, ESF/B8ZS or CCS/HDB3).

6. Contact your leased-line or other carrier service and have it perform integrity tests on the line. If simple automated tests do not locate the problem, ask the provider to run the following tests in order:

  • All 0's for 5-20 minutes with no errors
  • All 1's for 5-20 minutes with no errors
  • Quasi-random word (QRW) for 5-20 minutes with no errors
Framing Errors (frame) A framing error occurs when a packet does not end on an 8-bit byte boundary for one of the following reasons:
  • Noisy serial line
  • Improperly designed cable; serial cable is too long; the cable is not shielded (interference or signal loss problems)
  • The CSU/DSU line clock is incorrectly configured; more than one clock source is configured on the line
  • "Ones density" problem on T1 link (incorrect framing or encoding specification)
1. Ensure that the line is clean enough for transmission requirements. Shield the cable if necessary. Make certain you are using the correct cable.

2. Make sure the cable is within the recommended length - ideally, no more than 50 feet (15.24 meters) from the telephone company or carrier equipment.

3. Make sure the cable between the CSU/DSU and the router is within the recommended length - ideally, no more than 25 feet (7.62 meters) for V.35.

4. Ensure that all devices are properly configured for a common and single line clock. Each line must have one and only one clock source configured on all devices. In most cases, the leased line provider will provide the clocking for the line.

5. Make certain that the local and remote CSU/DSU's are configured for the same framing and encoding scheme as that used by the leased-line or other carrier service (for example, ESF/B8ZS or CCS/HDB3).

6. Contact your leased-line or other carrier service and have it perform integrity tests on the line. If simple automated tests do not locate the problem, ask the provider to run the following tests in order:

  • All 0's for 5-20 minutes with no errors
  • All 1's for 5-20 minutes with no errors
  • Quasi-random word (QRW) for 5-20 minutes with no errors


Serial Lines: Increasing Carrier Transition Count on Serial Link

     18:08:37	      Serial2 Detail	      router
     Serial2 is   up , protocol is   up
          Description	: Qwest T1
          Encapsulation	: Cisco HDLC
          IP address	: 63.148.112.74 255.255.255.252
          Broadcast address : 63.148.112.75
          Line has been up since Mon Oct 28 14:59:54 2002 (1w3h)
          Last input :	00:00:00	Last output:	00:00:00
          Bandwidth:	1.54 Mbit   Load in:	23% Load out:	37%
          3 second average input rate :	354.58 Kb/s,	44.32 KB/s, 112 packets/s
          3 second average output rate:	0.57 Mb/s,	71.64 KB/s, 142 packets/s
               Rx Packets	265,482,595	7,851,846 bytes
               Tx Packets	378,208,552	388,020,255 bytes
               Rx Errors: 568	(0 CRC	235 frame	0 fifo	333 dropped)
               Tx Errors: 0	(0 collisions	0 fifo	0 dropped)
               Carrier transitions 7 
               DCD =   up   DSR =   n/a DTR =   n/a RTS =   n/a CTS =   n/a 
      c CSU | y Summary | n Next | p Previous | z Zero | h Help | q Quit 
If carrier transitions (in bold) appear in the interface detail screen, there are several possible problems that may be causing these errors. Carrier transitions appear whenever there is an interruption in the carrier signal (such as an interface reset at the remote end of a link, or a line problem). The table below shows the possible problems that may cause carrier transitions, and potential solutions to those problems.


Possible Problem Solution

1. Line interruptions due to an external source(such as physical separation of cabling red or yellow T1 alarms or lightning striking somewhere along the network)

2. Bad line causing carrier detection transitions

3. Congestion on link (typically associated with output drops)

4. Faulty switch, CSU/DSU, or router hardware

1. If there is a high number of output drops in the interface statistics detail screen, see the section Router Installation and Configuration Manual/Serial Lines: Increasing Output Drops on Serial Link earlier in this chapter.

2. If the carrier transitions are coupled with input errors, the problem is probably a bad link or bad CSU/DSU. Contact your leased line or other carrier service and swap faulty equipment as necessary.

3. Check hardware at both ends of the link. Attach a breakout box or a serial analyzer and test to determine source of problems.

4. If an analyzer or breakout box is unable to identify any external problems, check the router hardware.

5. Swap faulty equipment as necessary.


Troubleshooting Clocking Problems

Clocking conflicts in serial connections can lead either to chronic loss of connection service or to degraded performance. This section discusses the important aspects of clocking problems:
  • Clocking Problem Causes
  • Detecting Clocking Problems
  • Isolating Clocking Problems
  • Clocking Problem Solutions.


Clocking Overview

The CSU/DSU derives the data clock from the data that passes through it. In order to recover the clock, the CSU/DSU hardware must receive at least one 1-bit value for every 8 bits of data that pass through it; this is known as "ones density". Maintaining ones density allows the hardware to recover the data clock reliably.
Newer T1 implementations commonly use Extended Superframe Format (ESF) framing with binary eight-zero substitution (B8ZS) coding. B8ZS provides a scheme by which a special code is substituted whenever eight consecutive zeros are sent through the serial link. This code is then interpreted at the remote end of the connection. This technique guarantees ones density independent of the data stream.
Older T1 implementations use D4-also known as Superframe Format (SF) framing and Alternate Mark Inversion (AMI) coding. AMI does not utilize a coding scheme like B8ZS. This restricts the type of data that can be transmitted because ones density is not maintained independent of the data stream.
Obviously, the line can have only one timing source. If more than one data clock exists on a line, synchronization of frames is impossible and line errors will result. If no data clock exists, similar problems result.


Clocking Problem Causes

In general, clocking problems in serial WAN interconnections can be attributed to one of the following causes:
  1. Incorrect CSU/DSU configuration
  2. Multiple or no clock source
  3. Cables out of specification-that is, longer than 50 feet (15.24 meters) or unshielded
  4. Noisy or poor patch panel connections
  5. Several cables connected together in a row


Detecting Clocking Problems

To detect clocking conflicts on a serial interface, look for input errors as follows:
  1. Use the interface statistics program on the routers at both ends of the link.
  2. Examine the detail screen output for CRC, framing errors, and fifo drops.
  3. If either of these steps indicates errors exceeding approximately 1 percent of traffic on the interface, clocking problems likely exist somewhere in the WAN.
  4. Isolate the source of the clocking conflicts as outlined in the following section, "Isolating Clocking Problems."
  5. Repair or replace any faulty equipment or cabling.


Isolating Clocking Problems

After you determine that clocking conflicts are the most likely cause of input errors, the following procedure will help you isolate the source of those errors:
  1. Use the interface statistics detail screen to determine if input error counts are increasing and where they are accumulating.
  2. Determine the end of the connection that is the source of the problem, or if the problem is in the line. Use a loopback plug (or loop the coaxial cable) to verify proper transmittal and receipt of traffic. This step isolates the router and CSU/DSU hardware from the line.
If input errors are accumulating on both ends of the connection, clocking of the CSU is the most likely problem. If only one end is experiencing input errors, there is probably a CSU/DSU clocking or cabling problem. Drops on one end suggests that the other end is sending bad information or that there is a line problem.
Note: Always refer to the interface statistics detail screen output and note any changes in error counts or note if the error count does not change.


Clocking Problem Solutions

The table below shows the possible clocking problems that may cause service outages, and potential solutions to those problems.
Possible Problem Solution
Incorrect CSU/DSU Configuration 1. Determine if the CSU/DSU's at both ends agree on the clock source (local or line).

2. If the CSU/DSUs do not agree, configure them so that they do. Usually the line is the source.

3. Check the LBO setting on the CSU/DSU's to ensure that the impedance matches that of the physical line. On integrated CSU/DSU's, use the service-module XX lbo command. See Chapter 18, Router Installation and Configuration Manual/Configuring Services: Quality of Service Menu or the Command Reference for details on using this interface command. Note: In general, the line build out setting should remain at zero (the default).

4. Make sure that "ones density" is maintained. This requires that the DSU's use the same framing and encoding schemes (for example, ESF and B8ZS) used by the leased line or other carrier service. Check with your leased line provider for information on its framing and coding schemes. Many leased line providers use auto-sensing switch cards that will automatically match the framing and encoding schemes used by the router.

5. If your carrier service uses AMI encoding, try inverting the data encoding on both sides of the link. On integrated CSU/DSU's, use the service-module XX data-coding inverted interface command.

Cable to router is out of specification 1. If the cable is longer than 50 feet (15.24 meters), use a shorter cable. If the cable is unshielded, try replacing it with shielded cable.


Troubleshooting T1/E1 CSU/DSU Problems

     19:02:22	      Serial0.1 CSU Statistics	      router
     Firmware version: 0.14 
     CSU self test: no failures 
     Rx status: no alarms 
     Tx status: no alarms 
     Far end CSU status: normal 
     Loopback: csu will respond to loop up command, not currently looped up 
     Statistics for current interval (2 seconds elapsed): 
                    Errored seconds: 0	Controlled slip seconds: 0
             Bursty errored seconds: 0	Degraded minutes: 0
           Severely errored seconds: 0	Path code violations: 0
           Severely errored framing seconds: 0	Line errored seconds: 0
                Unavailable seconds: 0	Line code violations: 0
     Line status information: 
        The line appears to be up. 
     y Summary | d Detail | z Zero | h Help | q Quit 
This section outlines procedures for troubleshooting T1 and E1 circuits based on the error messages in the CSU/DSU detail screen of the interface statistics program. The information displayed is generally useful for diagnostic tasks performed by technical personnel only. The CSU/DSU detail screen displays statistics for the current 15 minute period and updates the display every 15 seconds. This detail screen provides information to logically troubleshoot physical layer and data link layer problems.
Note: Most T1 errors are caused by misconfigured lines. Ensure that line encoding, framing and clock source are configured according to what the service provider recommends.
The table below shows the possible integrated CSU/DSU messages and descriptions of each.
Detail Screen Message Explanation Solution
1. CSU is initializing 1. The CSU/DSU is still initializing. 1. The port has recently been enabled or reloaded.
2. ROM checksum failure or EEPROM checksum failure 2. The on-board CSU/DSU has failed its initial self-tests. 2. If the problem persists, it may indicate a hardware problem.

Contact ImageStream technical support for assistance.

3. Framer type is incorrect, Framer hardware failure or Internal loopback failure 3. The CSU/DSU has failed its internal loopback test. 3. If the problem persists, it may indicate a hardware problem.

Contact ImageStream technical support for assistance.

An internal loopback failure alone is generally not the cause of a line problem.

4. CSU will respond to loop up command, not currently looped up 4. The CSU/DSU has passed its initial self-tests and is operational. 4. The CSU/DSU is ready to accept loop up commands, but is not currently in a loopback state.

This is the most common state of the CSU/DSU.

5. Your CSU is currently looped up. No data can be received by your router while the CSU is looped up. 5. The CSU/DSU has received a "loop up" command from a remote device or test set. 5. The CSU/DSU is currently in a payload loopback mode and is reflecting all received data back to the network.

The router will receive no data from th integrated CSU/DSU while it is in payload loopback mode.

6. RAI - remote alarm indication (yellow alarm) on Rx Status 6. The CSU/DSU is receiving a signal from the remote end that the remote CSU/DSU has lost signal 6. If there are no local errors (AIS,LOS,OOF), this indicates a problem at the far end and not with the local CSU/DSU.

The line is generally down if a yellow alarm is encountered and indicates a remote problem.

7. RAI - remote alarm indication (yellow alarm) on Tx Status 7. The CSU/DSU transmitting a signal to the remote end that the local CSU/DSU has lost signal. 7. Other local errors (AIS, LOS, OOF) will accompany this Tx Status error.

The other local errors will indicate the nature of the problem that the local CSU/DSU is having with the remote end.

8. AIS - alarm indication signal (blue alarm) 8. The CSU/DSU is transmitting and/or receiving a continuity signal to indicate that there is a transmission fault upstream of the CSU/DSU. 8. The line is down if a blue alarm is encountered and generally indicates a line problem

Check to see that the framing format configuration on the CSU/DSU matches the format used on the line

If the problem persists, contact the leased line provider for additional assistance.

9. PDE - pulse density error 9. The CSU/DSU is not transmitting and/or receiving at least one pulse in every 8 bits from the line 9. This error often appears along with a yellow alarm, and indicates that the line is down due to a remote problem.
10. LOS - loss of signal 10. The CSU/DSU has not received a pulse in 175 consecutive pulse positions 10. This error, also referred to as a red alarm, usually indicates that the CSU/DSU has lost connectivity with the local loop of the leased line.

Often this error occurs because the CSU/DSU has been disconnected from the line. LOS is a local problem and not with the far end CSU/DSU.

Make sure that the cable between the CSU/DSU and the T1 or E1 terminal equipment is connected correctly.

Check to see that the cable is connected to the correct ports.

Check cable integrity. Look for breaks or other physical abnormalities in the cable. Over short distances, it is acceptable to use a standard, straight-through Cat3 or Cat5 Ethernet cable instead of a standard T1 or E1 cable.

11. OOF - out of frame 11. The CSU/DSU has received 4 consecutive frames out of alignment. This error generally indicates a clocking or framing problem. 11. Check to see that the framing format configuration on the CSU/DSU matches the format used on the line and on the remote CSU/DSU.

Check the line buildout setting in the interface configuration file (wan.conf).

12. PRM - performance report monitoring failure 12. The remote CSU/DSU is not reporting performance data. 12. this error by itself does not indicate a problem with the line, as some equipment does not provide PRM data, or has the service disabled.
13. Payload loopback 13. The remote CSU/DSU has received a loop up command and is reflecting received data back onto the link.


Troubleshooting Ethernet Problems

To view LAN station traffic on an interface the Interface Status utility (stats) has a special option in detail mode which will show traffic by Ethernet MAC adddress.
Run the interface status utility (option 2 from the main menu) and get detail on the interface you want to view LAN station traffic.
Hit v (the view command) to bring up the view traffic dialog box.
Use the arrow keys or hit L to select 'IPTraf LAN station monitor'
13:44:47              Ethernet0 Detail                             inside-envoy
-------------------------------------------------------------------------------
Ethernet0 is  up , protocol is  up
     Description       : 100Mb Ethernet
     Encapsulation     : 100baseTx-FD
     IP address        : 192.168.100.2 255.255.255.0
     Broadcast address : 192.168.100.255

     Link status: Link beat established
     Auto-negotiation: enabled (100baseTx-FD)
     Partner capabilities: 100baseTx-FD, 100baseTx, 10baseT-FD, 10baseT

     Bandwidth: 100.00 Mbit  Load in:   0% Load out:   1%
     3 second average input rate :    0.31 Mb/s,   38.32 KB/s, 198 packets/s
     3 second average output rate:    1.67 Mb/s,    0.21 MB/s, 214 packets/s
          Rx Packets          324,136       45,150,440 bytes
          Tx Packets          286,121      125,045,304 bytes
          Rx Errors: 0  (0 CRC  0 frame  0 fifo  0 dropped)
          Tx Errors: 0┌─────────────────────────────────┐
                      │ IPTraf                          │
----------------------│ IFTop                           │----------------------
c CSU | y Summary | n │─────────────────────────────────│
                      │ IPTraf detailed statistics      │
                      │ IPTraf TCP/UDP monitor          │
                      │ IPTraf packet size counts       │
                      │ IPTraf LAN station monitor      │
                      │─────────────────────────────────│
                      │ Exit                            │
                      └─────────────────────────────────┘

 IPTraf
┌──── PktsIn ─── IP In ─ BytesIn ─ InRate  PktsOut ── IP Out  BytesOut OutRate ┐
│ └       30         0      3120      0.8        0         0         0     0.0 │
│ Ethernet HW addr: 0019d1799ae9 on eth0                                       │
│ └       30         0     14531      0.0       36         0      8235     0.0 │
│ Ethernet HW addr: 0019d1758fa3 on eth0                                       │
│ └       11         0       881      0.0       11         0       805     0.0 │
│ Ethernet HW addr: 000b821e874a on eth0                                       │
│ └        0         0         0      0.0        2         0        92     0.0 │
│ Ethernet HW addr: 000c29a16de8 on eth0                                       │
│ └        0         0         0      0.0        3         0       635     0.0 │
│ Ethernet HW addr: ffffffffffff on eth0                                       │
│ └        3         0       635      0.0        0         0         0     0.0 │
│ Ethernet HW addr: 01005e000005 on eth0                                       │
│ └        6         0       384      0.2        0         0         0     0.0 │
│ Ethernet HW addr: 000b821f53ba on eth0                                       │
│ └        0         0         0      0.0        1         0        46     0.0 │
│ Ethernet HW addr: 000b821f518e on eth0                                       │
│ └        0         0         0      0.0        1         0        46     0.0 │
│ Ethernet HW addr: 003080b194d1 on eth0                                       │
│ └        3         0       168      0.0        3         0       138     0.0 │
│ Ethernet HW addr: 01005e00000d on eth0                                       │
│ └        1         0        30      0.0        0         0         0     0.0 │
│ Ethernet HW addr: 00e08140d40b on eth0                                       │
│ └        3         0      4476      7.0        3         0      1728     2.6 │
└ 19 entries ── Elapsed time:   0:00 ─── InRate and OutRate are in kbits/sec ──┘
 Up/Down/PgUp/PgDn-scroll window  S-sort  X-exit

Troubleshooting Service Specific Problems

ImageStream Linux includes a very powerful command line utility called tcpdump which can be used to troubleshoot many network and service related problems. To use tcpdump you must go to the bash shell using main menu option 3 (Advanced) then option 1 (Bash shell).
The tcpdump utility has many options which can be viewed on the tcpdump man page. Some of the more common options are:
  • -i <interface> (Capture traffic on the specified interface)
  • -n (Don't convert addresses to names, i.e. show IP addresses)
  • -s0 (Include the whole packet in the capture -- snaplen 0)
  • -v (Verbose protocol analysis/display)
  • -w <capture file> (Save the capture to the specified file for later viewing locally or via Wireshark)
  • port <port number> (Show traffic for tcp/udp port specified)
  • host <hostname or ip> (Show traffic for the specified source or destination host)
Example: tcpdump -n -i eth0 -s0 -v port 80 and host 192.168.1.15 | more
Pipe the output to the more utility to avoid overloading your telnet/ssh session with display output. The more utility allows you to view a page full of output at a time. Hit the spacebar to view the next page or the q key to quit.
If you do not pipe the output to more then use <ctrl>-c (hold down the control key and hit the c key) to exit the tcpdump utility.

Output from tcpdump

The tcpdump command output contains the following fields:
 15:06:29.465667 00:90:fb:25:64:47 > 00:1c:b0:b5:a8:c0, ethertype IPv4 (0x0800), 
 [1]             [2]                 [3]                [4]      
 length 508: 208.89.39.84.49289 > 67.192.92.227.80: P    2550652488:2550652942(454) ack 2743695195 win 16560
        [5]  [6]          [7]     [8]           [9] [10] [11]       [12]       [13] [14]
 [1] TimeStamp           [8]  Destination IP
 [2] SourceMac           [9]  Destination Port
 [3] DestinationMac      [10] TCP Flags
 [4] Network Protocol    [11] TCP Sequence Number
 [5] IP Packet Length    [12] TCP Last Sequence Number
 [6] Source IP           [13] TCP Length
 [7] Source Port         [14] ACK flag

SMTP / E-mail issues

SMTP uses TCP port 25 so we'll use a port 25 filter to view only SMTP traffic.
router:/usr/local/sand# tcpdump -n -i eth1 port 25 | more
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 68 bytes
19:38:06.130124 IP 10.169.20.247.50665 > 74.54.198.162.25: S 3311495967:3311495967(0) win 8192 <mss 1460,nop,wscale 2,nop,nop,sackOK>
19:38:06.193285 IP 74.54.198.162.25 > 10.169.20.247.50665: S 2052155192:2052155192(0) ack 3311495968 win 5840 <mss 1460,nop,wscale 6>
19:38:06.198957 IP 10.169.20.247.50665 > 74.54.198.162.25: . ack 1 win 16425
19:38:06.262482 IP 74.54.198.162.25 > 10.169.20.247.50665: P 1:184(183) ack 1 win 92
19:38:06.273158 IP 10.169.20.247.50665 > 74.54.198.162.25: P 1:12(11) ack 184 win 16379
19:38:06.336350 IP 74.54.198.162.25 > 10.169.20.247.50665: . ack 12 win 92
19:38:06.336358 IP 74.54.198.162.25 > 10.169.20.247.50665: P 184:242(58) ack 12 win 92
19:38:06.347693 IP 10.169.20.247.50665 > 74.54.198.162.25: P 12:49(37) ack 242 win 16364
This shows an SMTP conversation between 10.169.20.247 from port 50665 to 74.54.198.162 port 25. In this case 74.54.198.162 is the e-mail server.
If we wanted to prevent this client from sending e-mail we would insert a firewall rule either in the firewall script (Configuration menu -> Option 4 (Firewall and QOS configuration) -> Option 2 (Firewall configuration) -> Option 1 (Configure firewall rules)).
For a temporary block you can also issue the firewall rule on the command line.
iptables -I FORWARD -s 10.169.20.247 -p tcp --dport 25 -j DROP
To view traffic to and from certain hosts add a src or dst filter.
Example: Show traffic going to 205.159.243.5 port 25:
tcpdump -n -i eth1 dst port 25 and dst 205.159.243.5
Example: Show traffic going to either 205.159.243.5/port 25 or 205.159.243.10/port 25:
tcpdump -n -i eth1 dst port 25 and '(dst 205.159.243.5 or dst 205.159.243.10)' 
Note we need to use parenthesis ( to separate the or clause and this means we need to use the single quote ' to prevent Bash from misinterpreting the parenthesis.

PPPoE issues

PPPoE uses ethernet protocols 0x8863 for control messages and 0x8864 for data packets.
PPPoE clients initiate a connection with a PADI request. The server will respond with a PADO offer. The client then sends a PADR Active Discovery request to request a specific service. The server will acknowledge the request with a PADS confirmation packet. These requests are all sent using ethernet protocol 0x8863. Sessions are terminated with a PADT packet which can be sent from either the client or server.
To view the PPPoE session setup requests the following tcpdump filter can be used:
router:/usr/local/sand# tcpdump -n -i eth0 -e -s0 -v ether proto 0x8863 | more
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
14:51:52.163920 00:21:29:ae:46:4c > ff:ff:ff:ff:ff:ff, ethertype PPPoE D (0x8863), length 60: PPPoE PADI [Host-Uniq 0x00000210] [Service-Name]
14:51:52.171293 00:16:b6:b5:c9:ff > ff:ff:ff:ff:ff:ff, ethertype PPPoE D (0x8863), length 60: PPPoE PADI [Host-Uniq 0x909A0110] [Service-Name]
14:51:52.176291 00:16:b6:33:7b:83 > ff:ff:ff:ff:ff:ff, ethertype PPPoE D (0x8863), length 60: PPPoE PADI [Host-Uniq 0x00000001] [Service-Name]
14:55:52.610035 00:16:b6:b5:c9:ff > ff:ff:ff:ff:ff:ff, ethertype PPPoE D (0x8863), length 60: PPPoE PADT [Host-Uniq 0x909A0110] [Service-Name]
15:00:11.498568 00:90:fb:25:64:46 > 00:21:29:d9:be:66, ethertype PPPoE D (0x8863), length 71: PPPoE PADO [AC-Name "ImageStream"] [Service-Name] [AC-Cookie 0x1896169478790F590269141FF1AC78B5BF030000] [Host-Uniq 0x509F4700]
15:00:55.135946 00:90:fb:25:64:46 > 00:23:69:40:36:98, ethertype PPPoE D (0x8863), length 32: PPPoE PADS [ses 0x236] [Service-Name] [Host-Uniq 0xDB470000]
We use the -e option to show the source and destination ethernet MAC addresses.
To view PPPoE data the following tcpdump command for traffic with ether proto 0x8864 can be used:
router:/usr/local/sand# tcpdump -n -i eth0 -e -s0 -v ether proto 0x8864 | more
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
15:13:53.707337 00:90:fb:25:64:46 > 00:1c:10:f4:a0:39, ethertype PPPoE S (0x8864), length 62: PPPoE  [ses 0x4c1] IP (0x0021), length 42: (tos 0x0, ttl  51, id 31409, offset 0, flags [DF], proto: TCP (6), length: 40) 38.116.147.180.80 > 208.89.38.15.4329: ., cksum 0x247c (correct), ack 2944659365 win 65535
15:13:53.707343 00:90:fb:25:64:46 > 00:1c:10:f4:a0:39, ethertype PPPoE S (0x8864), length 62: PPPoE  [ses 0x4c1] IP (0x0021), length 42: (tos 0x0, ttl  51, id 31410, offset 0, flags [DF], proto: TCP (6), length: 40) 38.116.147.180.80 > 208.89.38.15.4329: ., cksum 0x1fbc (correct), ack 1217 win 65535
15:13:53.707467 4a:00:3e:d2:a6:93 > 00:90:fb:25:64:46, ethertype PPPoE S (0x8864), length 74: PPPoE  [ses 0x169] IP (0x0021), length 54: (tos 0x0, ttl  63, id 22162, offset 0, flags [DF], proto: TCP (6), length: 52) 208.89.37.49.33586 > 208.111.145.179.80: ., cksum 0xb81d (correct), ack 935292310 win 65535 <nop,nop,timestamp 751697 2268923799>
To narrow down the output for a specific host you can add an ether src or ether dst filter:
router:/usr/local/sand# tcpdump -n -i eth0 -e -s0 -v ether proto 0x8864 and ether host 4a:00:3e:d8:25:f8 | more
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
15:15:37.356470 00:90:fb:25:64:46 > 4a:00:3e:d8:25:f8, ethertype PPPoE S (0x8864), length 225: PPPoE  [ses 0x6c] IP (0x0021), length 205: (tos 0x0, ttl 125, id 3573, offset 0, flags [DF], proto: TCP (6), length: 203) 207.114.197.99.443 > 74.116.44.166.58044: P, cksum 0x1198 (correct), 863428291:863428454(163) ack 1873881143 win 65535

Failsafe Mode

The lilo boot menu contains a failsafe option for recovering the root password and performing other low level operations. To recover the root password choose the Clear root password option. Other configuration changes which prevent logging into the router can be fixed using the following instructions from the command prompt. These commands are only valid for the 2.x, 3.x, 4.0, 4.1, 4.2, 4.3 and 4.4 distributions. The 5.0 distribution uses a different method of configuration management.

In failsafe mode DO NOT USE THE backup flash COMMAND as it will overwrite your configuration with the failsafe configuration.

Please note that it is possible to overwrite your configuration or, even worse, your software packages if these commands are not typed properly. Use at your own risk and triple check your typing before hitting enter.

The normal mode configuration will have to be extracted manually like so:

Extract the flash to /tmp:

mkdir /tmp/flash
cd /tmp/flash
tar xf /dev/hda2
ls

You should see 3 directories: etc, root and usr. Your wan.conf will be located in usr/local/sand/wan.conf. To edit your wan.conf:

pico usr/local/sand/wan.conf

To disable sand on startup:

cd etc/rc.d/rc.router; mv S20sand K20sand

Save it back to flash: THESE COMMANDS ARE VERY IMPORTANT. If they are mistyped or throw an error your configuration might be lost if you reboot before fixing it.

cd /tmp/flash
tar cf /dev/hda2 *

Troubleshooting Questions - Getting Technical Support

If you cannot get your router to work, or have difficulty understanding error messages or diagnosing problems with your line, and the wiki documentation does not address the problem you are seeing, send a detailed e-mail describing your problem to support@imagestream.com or contact us by telephone or fax. We will be happy to assist you.
Please have the following information ready before you contact ImageStream:
  1. The serial number of your router
  2. Your contact information
  3. A brief description of your problem
  4. The full text of any error messages you receive
  5. Copies of the relevant configuration files from your router
This manual, accompanying release notes and related documentation are constantly evolving, so please check the Web site periodically for the latest revisions. Your input and suggestions to improve this document are appreciated.

Contact Information

World Wide Web: http://support.imagestream.com

Electronic Mail: support@imagestream.com

Telephone: 574.935.8484

Fax: 574.935.8488

Personal tools
Router software releases