Ping a network engineers best friend!

Most network folks are comfortable with a standard Ping command.    Some even know that you can add options to ping to set packet size and repetitions, but at the end of the day, Ping is a level three ICMP command.

Ping Command Syntax

ping [-t] [-a] [-n count] [-l size] [-f] [-i TTL] [-v TOS] [-r count] [-s count] [-w timeout] [-R] [-S srcaddr] [-4] [-6] target [/?] As an example: ping -n 5 -l 1500 www.google.com  the ping command is used to ping the hostname www.google.com. The -n option tells the ping command to send 5 ICMP Echo Requests instead of the default of 4 and the -l option sets the packet size for each request to 1500 bytes instead of the default of 32 bytes. Ping is generally a knee jerk reaction for a network engineer!  It will establish and demonstrate connectivity between network devices.  It can be used to determine latency and jitter and is a very quick, effective and easy to execute network test tool.

ShoreTel Connectivity

ShoreTel administrators learn very quickly that the first place you go when someone complains about the phone system is the ShoreWare Director portal to the "Quick Look" section.   You quickly scan the screen for RED!   (That is what is so simple about alarms:  Green is good, Yellow means something needs attention and Red is bad)!   In this example the site known as Carol can not connect with the the Bob site.   Bob however can connect with the Ted and Alice Sites.  Carol can also connect with the Ted and Alice sites.   How do you visualize this in ShoreTel?   You go to the second screen and by clicking on  Connectivity and you see the famous ShoreTel Christmas Tree! This is very not good!   Sites can not communicate and calls are failing all over the place!  For  example the trunk group that terminates on HQ SGT1K-03 (Line 13), for example will not be able to complete an incoming call to a user on the user switch Central Time Zone (Line2).  The RED box at the intersection of two lines helps you visualize connectivity or lack of connectivity.   Why is this happening?

Lets Ping and find out!

The network engineer will undoubtedly do a Ping test.   You will ping from the source IP address of one site, to the destination ip address of the other sites device.  Now if the test fails, the next step is to figure out where the network connection is broken.   The more challenging problem is more perplexing.   What happens if the Ping is succesful?  You do your Ping test and you get an excellent reply and the network is not broken at the Layer 3 level.  What then?

Enter lsp_ping

ShoreTel has a proprietary protocol named lsp_ping.  This ping makes it possible to test network connectivity at the higher L4, transport level by enabling you to Ping with a port number.    In the example Bob can Ping Carol, so why is the BOX red?   The answer might be to run an lsp-ping  command.   Unlike Ping which you can run from your local computer to a destination ip address, you will need to telenet or Ssh into a ShoreGear switch to run lsp_ping.   To do this you will need to run the ipbxctl  security  command from a ShoreTel server first, then telnet into the device and run your test. It will look something like this: lsp_ping "192.168.10.12", 100 This will setup a ping to port 5440 at the target device of 192.168.10.12 and the quotations are required!  The comma 100 means, send 100 packets in this test.

What is lsp_ping?

Shoregear switches all keep a copy of the current configuration and know where the end points (i.e. handsets ) are and how to get there.   This is one of the strong points of the ShoreTel architecture.   You do not need to check with a server to get the current configuration, and if the server crashes, ShoreTel keeps on trucking!   The proprietary lsp or location service protocol, keeps updating configurations around the deployment reporting any changes.  They do this through ports like 5440 and if they can not update, they assume that switch is off line and thus you can not connect to it. If you study the screen shot above and you knew the IP address of all the devices, you would be puzzled as to why two devices in the same subnet, using the same network path, going through the same network connection should have different connectivity status reports?  Every engineer has asked "what changed in the network" and every client answers the same; "nothing".   If is always frustrating when you get a situation reported like the graphic above.   A working network for months with no problem and then suddenly the Christmas tree appears!  Especially when everyone reports that nothing has changed in the network! ShoreTel TAC will always tell you a failure of lsp_ping means a network issue.  Most times it is, but this time it was not! The take away from this discussion is that Ping can establish network layer connectivity, but ShoreTel can still have a connectivity failure.  You will then need to trouble shoot higher up the stack and look at the transport level and perhaps the application level.  In this scenario, we mirrored all network traffic to a packet capture device.  We ran Wireshark on the packet captures while doing lsp_ping and were surprised to see that lsp_pings on port 5440 were in fact not only sent, but received by the traget device, yet no answer packets were returned.  At first we though that it might be that a UDP packet, being connection-less, would not report, but looking at working sessions we determined we should see a return packet!

ShoreTel V-phone virtual appliance  coma!

The interesting fact in this mess was that the failure was always between the two sites (main location and a data center); and that the pair was always a hardware switch and a virtual switch!   The virtual switch seemed to be in some kind of coma as it did not return lsp_ping packets yet showed green in quick look.  As the issue was causing major ShoreTel call failure and a lot of people had investigated the network with no root cause identified, a decision was made to rebuild all the virtual switches or Vphone and Vtrunk as ShoreTel calls these linux based OVA files.  This cleared the problem with no changes to the network and all is now Green! Summary Recommendation on SG-Vphone and SG-Vtrunk We recommend the use of ShoreTel SG-Vphone virtual appliances only as failover and not as production resources.   Inside those ShoreGear Orange boxes is a rack full of dedicated micro-processors that provide DSP or digital signal processing resouces.   When these are vitualized all that processing most now be done in software.  What a work load!