Network Troubleshooting Solution

Preface

This paper listed different inevitable scenarios of network engineers have to face during network troubleshooting, and illustrated the general challenges engineers will inevitably encounter. It also gives summary of the challenges and the keys to solve them.

Challenge

The world is becoming smaller, the distance between people gets shorter and shorter, and communication can no longer be baffled by physical obstacles - thanks to the information technologies, especially the media: network.

The network environment is getting more and more complex due to the rapid increase of using IT system, and different types of industries, enterprises all have IT to support their business. While different business trenches their demands and funds for Information Technology, more and more various technology and applications are involved into the modern network, and unexpected errors may happen during these overlapped technology instances: routing mechanisms, fault tolerant configurations, firewall policies, integrated 3G gateway, Wi-Fi frequency, endpoint health, management software etc. Each technology units could be a fault point and any unit encounters an error will absolutely cause unforeseen effect to other units, and then consequently result in a series of troubles in your network.

And finally you need to face the chaos.

Let's see how this complexity would be firstly.

The features of modern network

In the old days of mainframes, the computing works are done in the mainframe and what the users do is just connect to the mainframe and receive proceed data. The network is pretty simple and easy at that period of time. But the PC aroused then, computing capability is not the only privilege of mainframe. The computing force was distributed in the entire network - each host runs, computes, processes data, send/receive information. When there is an error, the administrators can no longer easily find the root and fix it. This is the problem what brought by the trends of network evolution.

The following tow trends brought the heaviest burden for administrators.

Intensive Application

No matter how small a company is, the basic IT infrastructure is a must: printer, computer, router, switch, or even more. What the virtualization technology brought us is something more than the physical hardware we have. We can have one single PC with VMware work as multiple servers providing DHCP, DNS, DC, etc; and we can have a dynamips to emulate a real router for lab test; and there are coming cloud computing technology and the trend of SaaS to give as services through Internet without hardware.

As what we can deduce from the above saying trends, even the infrastructure may be simple and tatty, or even there is only one server here in the company, we can have hundreds of applications and services. The bad things are:

• Any single error on each application or service can give deadly strike to others and then the entire network.
• What we can see, and the direct symptom, possibly is quite far away from the where the root problem is. It's hard to tell.
• The interactions between applications and/or services are mostly unforeseen. It's hard to determine who, or what application caused an error on another one.

For example

We have a bad experience of using the new CRM system newly developed by the R&D dept. The helpdesk failed to figure out what the problem is and passed it to the network engineer. The network engineer said the application is not responding fast enough so he turned to the R&D and asked them to fix it. The R&D claims the application is all right. There must be something else there.

Finally, it turns to be the internal DNS server had a wrong record and it told the clients to connect to the gateway, and the gateway redirected them back to the real server.

This is a typical example.

Now we can see what the intensive application brought us in troubleshooting.

Note: Complex relationships

• Between network devices
• Between applications
• Between application and network devices
• Between new resource/function and security

Sparse topology

The edge of network is stretching out and the concept of physical perimeter faded away. Instead, the logical perimeter is split into different horizons and scattered in multiple aspects of network. The VPN expanded our network perimeters; the cloud computing fade out the edge; the Internet helps us extend out apperception to other side of the earth.

We now have networks across different locations, with barriers, distance. However, a troubleshooting process inevitably requires go back and forth between different peers.

• Send traffic from one peer and test it from the other one;
• Test both ends to locate which is the error one;
• Go along a specific path to find out any point of failure;
• Etc.

It is ok to have telnet/SSH or other remote login applications to operate from a different location. But what if the problem is losing of network connection? It is not even possible to ping, no mention telnet/SSH.

It is ok to quarter-deck between two departments next to each other. Would it be ok if you need to have round-trips multiple times? Is it ok to go back and forth between two buildings, two cities?

These are the problems bright by the daily expanded network. To minimize the cost, you should accurately locate the failure point before try to go to the sites, thus lessen round-trip times.

Solution

It is easy to infer that in the entire procedure from notice the symptom to finally apply solution and fix the problem, the "find the cause of problems" portion costs the most.

There are lots of solutions in the market for users, but if you have no idea what the source of problem is, and just keep solving the visible symptoms, there's not going to be an end.

Colasoft Capsa is a product which is widely used for troubleshooting. This tool provides statistics support and gives an insight view at frame level. It enables administrators to trace the digital bits running in your network and translate them into human readable information, thus find the source of problems, and then choose the right solution right against the problem rather than merely symptoms. Fix a problem once for all.

Next Steps