Category: Voip Service & Solutions

WANtelligence – Software Defined Networks!

Posted on by DrVoIP

WANtelligence?

Recently while working an RFQ project to re-mediate the WAN for a major health care practice management company,  we had the opportunity to listen to the major carriers describe their network solutions.   The client company was seeking a new partner to provide network connectivity options for their several major corporate locations, three data centers and over 600 remote office locations.   The company is growing expotentially and expected to double its WAN requirements and end points with in 12 months.  The network was to provide Internet, data, voice and video communications and as such,  the Service Level Agreements were to be consistent with those required to provide toll quality voice and full motion video.   Historically the company had used ipsec VPN tunnels over Direct Internet Access (DIA) facilities that ranged from bonded T1 to shared media broad band cable and occasionally free space transmission when terrestrial connections were impossible.  Each remote site also used Cradlepoint 4G for fail over and redundancy.

MPLS connectivity to rural America?

Not withstanding San Diego’s recent loss of the Chargers there are some 32 NFL cities in the United States.   Contracting for a high quality, high speed WAN solution in any of these cities is really not a challenge, most any carrier can help you.   The real challenge is getting the same connectivity  to Margaretville, New York or East College Station, Texas.   If you are searching for an MPLS connection to a branch office in a rural American area and you will find that it is rubric cube of challenges!  No one carrier has a foot print that will cover all your requirements and if you are looking for “one neck to choke” when things go black,  you may be giving up network visibility to  facilitate easy order entry and administration.   Work with an aggregator and you will most definitely lose your visibility to the underlying carriers who together knitted your WAN solution.

(Using our super secret online carrier database, www.buildinglit.com , we attempted to check out broadband connectivity to Margretville New York,  just for kicks to prove a point.  Click here for results.)

We listened attentively as Metel,  AT&T, L3, Century Link and Earthlink stepped up to the podium and delivered their very best solutions.   In all but one case, each carrier presented an MPLS solution.   All carriers mentioned SDN, but only one carrier led with that solution and recommended it as the solution for this client opportunity.   Most of the presentations began to sound the same, each carrier was the absolute  best in the marketplace,  each had the widest foot print, with outstanding customer service and the best proven technology!

What is SDN anyway?

Most folks can remember way back in 2007 when Steve Jobs introduced the device that would not only change Apple, but redefine mobile communications forever.  Jobs had given AT&T the exclusive right to provide network connectivity for the new device.  It was something of a “bet your company” play for both Apple and AT&T.   When the iPhone App Store came online, after Jobs opened the iPhone to developers,  the demand for bandwidth exploded and the AT&T network all but collapsed under the weight of all the new devices.  AT&T had to act quickly, faster than building new network capacity would allow.   The solution was to apply software in a way that network capacity would be effectively increased while physical plant was being deployed.   AT&T pioneered software-enabled networking!

For those not familiar with the concept of Software Defined Networks, or what we call WANtelligence, think of two roads connecting a location across a river to a main highway on the other side.   One road might be a more direct path, but not well paved.   The other road might be well paved but not very direct, with more of a circuitous route that added miles to the  distance of the direct path (i.e. Latency).   From time to time, the roads might be backed up with traffic, slowing or stopping your efforts to get to the main highway (i.e. Jitter).  A software defined solution works something like the popular mobile app Waze or Google Maps.   As  you are traveling, the map sends updates and suggest alternative routes.   An SDN solution would not only inform you of conditions, but automatically just take the action of routing you over the best route!

At the Branch Office end point, SDN components require multiple connections to the main broadband highway.  Generally, this might be a broadband connection like MPLS in addition to a DIA link.    The onsite device, under control of a cloud based optimizer, does a packet by packet analysis of the best path solution.   Again, much the way your traffic app knows your originating location and destination,  the SDN knows source and destination and the status of the entire network options between and through central control, optimizes path selection.

There are many vendors of the equipment that might be located at your Branch Office, but they might not provide the total solution.    A complete SDN requires that a centralized intelligence, typically a cloud based portal, be aware of the network and all other end points on that network.   Some devices can also include functions that might otherwise require separate devices.  For example, Firewall, Intrusion Detection, Network Optimization, Compression and Encryption in addition to the flow control and VRF functions that make up the elements of Software Defined Networks.   We found a Gartner Summary that is worth a review and includes a summary of many of the vendors in this exploding market segment.  We note that VeloCloud, a CISCO investment, seems to have it all and is establishing a defensible market share and leadership position.    VersaNetworks seems to be a component play and both companies have been included in proof of concept offerings by major carriers.   You can also download the white paper “Delivering Managed Services” from the DrVoIP site along with other free white papers on related subjects!

WANtelligence Summary

Software defined networks are in fact a reality and are rapidly becoming a cost effective solution that can assure high quality voice and video communications across the WAN.   We continue to work with this technology and have hands on experience with both products and services.   WANtelligence is our agnostic approach to developing Software Defined Networks!  Our Certified Network Architects add expertise and expereince to your enterprise when considering how to include SDN in your network plan!   Give us a call, or write DrVoIP@DrVoIP.com and we will share what we know!

Update note –

Cisco closes on $610M Viptela purchase

|About: Cisco Systems, Inc. (CSCO)|

Cisco Systems (NASDAQ:CSCO) has wrapped its acquisition of privately held Viptela.

Cisco had come to a $610M (and assumed equity) deal to acquire the software-defined wide area networking company in May.

Viptela will join the Enterprise Routing team inside Cisco’s Networking and Security business under senior VP DAvid Goeckeler.

Shares are up 0.8% premarket.

 

ShoreTel Board takes company private?

Posted on by DrVoIP

“ShoreTel  (SHOR) is in final stages of rolling out ShoreTel Connect”. Who cares? Technology issues aside, ShoreTel remains dog meat in the financial markets. Since the company’s public offering in 2007, the stock has never sustained a price above the first day closing of $12.50, itself a modest premium over the offering price of $9.50 a share. Currently SHOR is trading below the rejected MItel acquisition price proposal. Market capitalization continues to hang around the $430M valuation.

Ring Central (RNG) with a market cap of $1.5B and 8X8 (EGHT) with a market cap of $1.3B are pacesetters and models for what ShoreTel should be! Why are these companies valued so highly yet ShoreTel remains the living dead? Maybe Considering divesting the customer premise business? One of the key difference between these three companies, is that only one company is dealing with the challenges of manufacturing hardware. The CPE product line has greater value to a company that wants to focus on that segment. Can you walk both sides of the street, CPE and Cloud? If ShoreTel management wants to keep its head in the cloud, go for it.

Clearly public shareholders are never going to see a return on investment, so why not put them out of their misery and take the company private? Hell the only folks that made any money on this deal are insiders anyway. So take the company private, remove the quarterly pressure and public disclosures, retrench and maybe get it right for a future offering. Divest the CPE product line to a company that wants to develop that segment and commit to making the product a real player in the enterprise market.

Clearly the parts are more valuable than the whole! Customers, employees and Shareholders would all benefit and the Board would finally look like they know what they are doing. IMHO – DrVoIP
From the desk of DrVoIP

Recording ShoreTel phone calls!

Posted on by DrVoIP

Call Center not Required!

Long the standard in boiler room call center applications, recording calls is often a requirement outside the call center.    There are any number of reasons to record calls for including compliance, clarity and certainty and just management of customer quality service.   There are a variety of recording solutions in the market and they all have very different feature sets.  Do you need the ability to annotate calls?   To search on more complex parameters that time and date?     If an employee is being recorded and transfers that call to the HR Director of the company, should that segment of the call be recorded?   Do we want to record inbound calls only?  Both inbound and outbound?  Do you have multiple locations in your deployment?   Do employees have the right to start and stop recordings?  Add notes?  You will also have to consider how recording are to be stored and for how long?  This can have a huge impact on hardware requirements.   Just some of the characteristics you might want to consider as you look for for a recording solution.  The list of features is long and there are many options to chose from.   It is best to get a solid requirements document together and to make sure that you fully understand that recording alone is not all there is!

Recording Server Applications

ShoreTel has an optional recording function that can be very effective!   It can be installed as a single server solution or as a multiple server, multiple site solution.   It can record all calls, or just in one direction only.   You can select who is recorded and you can also select the archive location and time frame.   The solution is deployed on a ShoreTel server, either HQ or DVS depending on the deployment model.    There are actually four modules that can be installed: the recording server; the client side, the web player and the administrative module.   The solution uses the integrated recording functionality of the ShoreTel phone system and most of the usual user group and class of service settings apply.

Generally you will create a route point with a call stack as deep as the number of calls you want to simultaneously record, and put it on the server that will host the recording application.   You will also need a route point for the player and you will need to create a user who will proxy the recording functionality.   The server install is very simple and conforms to the usual point and click install expectations of a Windows server application.  The configuration is very simple, just provide the route point extension number, note the port for recording and click install!

The Administration Application

The Administrative application enables you to configure the specifics of the recording server and  to create profiles.  This application can be installed on the server or on a PC that can reach the server.   The configuration options are very easy to understand and simple to enter.  Basically they deal with where files should be stored and for how long.  You also have the option of either saving the recordings to a file system, or saving them as a voice message.

recorderadmin

You then create profiles that are used to customize different groups of extensions.   For example, you might want persistent recording for some extensions and not others.   This means that the recording continues even if the call is transferred.  Do you want inbound and outbound recordings?   And exactly who should be recorded!  The profile also dictates if recordings are to be made all the time, or sampled as a percentage or by a defined schedule.

Client Side Options

The client side application is installed on a pc and is optional tray icon.   You would use this only if your profile enables users to start and stop recordings and to tag recordings.   If this is not a privileged that you want to extend to those be recorded there is no need to install this optional application at each desk.  The use of the client is profile dependent.

recorderclient

The Player Application

The last application defines the player and is very useful for visually managing recordings.  You can use a phone to play recordings , but most folks find this web application to be more useful.   The application must be installed on a server as it uses IIS, but the recordings are played locally on a windows machine that has a sound card!   Each extension is listed and there is a time and date stamp on the recording file.  You have the option of storing other file information, like ShoreTel call properties to enrich the identification of a recording.

ShoreTel Recording Player

 

All in all, the ShoreTel Recording Application is a sweet suite!  It gets the job done and at a price point that compares more than favorably with the third party Recording applications found in the market place.   We recommend it for both call center and general recording applications when you are on a ShoreTel iPBX!   Give us a call and we can help point you in the right direction or get this installed and configured for you!

 

 

 

 

TEXT-2-AGENT Sending text and pictures to your call center?

Posted on by DrVoIP

Can you send a picture to your ShoreTel or CISCO Call Center

We have been integrating SMS to CISCO UCCX and ShoreTel Call Center deployments for customer service scenarios for some time.  The interface is very simple:  Login, pick a number from anywhere on the planet; assign a “keyword” and match it to an email address or list!   When someone text your “keyword” to your number, the text is converted to email and sent on to that address.   The email recipient can then hit reply and we convert the email back to SMS and forward it back to the original Cell phone.  Optionally, you can build a membership list with and auto response and the ability to send a bulk text to the list!

smsportal

What is new in Version 3.1

We have released Version 3.1 and it now makes it possible to send not only text (SMS), but pictures (MMS)to the “next available agent” in your Call Center.    The application now supports:

  • Inbound SMS to email Address based on “keyword” match with REPLY ability; – this enables you to use one number and many different keywords that point to different email addresses.  The recipient can hit reply like any other email and send a response back to the sender of the SMS, usually a client who has a cell phone!
  • Inbound MMS to default email address – enables a cell phone to send a photo or picture into the customer service organization which will route to the default email address.
  • Inbound SMS to outbound SMS; – Stealth mode, you can forward an incoming SMS to another SMS.
  • SMS to List based on “keyword” match – This enables you to build an SMS marketing list.   Each list is built by “keyword” match and you are able to send auto responds and “bulk” SMS
  • Outbound email to SMS;
  • Email to SMS – enables a customer service representative to send a text from an email client.   The address can be a keyword group list or an individual cell phone number.

Each transaction has a unique “ticket” number, all transactions are searchable, logged and archived for HIPPA and PIC compliance.

We are finding that Insurance, medical, auto and tech support applications in particular are receptive to the concept of enabling clients to send a picture by text to a customer service representative.

sms1

Want A demo?

We can easily set you up with a demo account. A basic single number solution with 500 SMS credits is about $25 a month!  Send the word “DEMO”   (all ONE word, watch that auto correct) to 424-348-4000 and include your email address for an example of how this could work on a ShoreTel ECC or CISCO UCCX; or just create an account!

Note – Your do NOT need a formal call center to implement this technology.   An email distribution list to customerservice@yourcompany.com works just as well!

TOP ShoreTel Connect Installation Gotcha’s !

Posted on by DrVoIP

So you are upgrading to ShoreTel Connect!  Watch out for these blistering hot spots:

Assume you have a call center that works three shifts, with 15 Agents per shift.  In the Pre-Connect world, you would define 45 agents but you would only need licenses for 15.  The Pre-Connect ECC license strategy was “concurrent users”.  The new license strategy requires you to license 45 Agents.  Additionally, all agents have to be named in the iPBX, costing you an additional “per seat” license for the newly named agents.  Before, 45 agents could log into extensions Agent1 through Agent15 using their AgentID to differentiate them for reporting purposes.   Wow!  I hope you don’t find this out the hard way.

Here are some of our favorite ShoreTel Gotcha’s:

1 – Ever have to move route points from one server to another?   You CAN NOT use the batch utility to do this and must remove and then reinstall each route point.   So much fun on a large contact center deployment ( If you know SQL you can actually modify the configuration database to do this).

2 – Want to change IP address of your HQ?   If you are using the new family of 400 phones, you will have to touch each phone individually and manually CLEAR the Configuration for the new Option 156 and DHCP values to take hold.  How would you like to upgrade 3K phones across five times zones to learn that gem!

3 – Gee, would it not be nice to use DNS Name resolution in the Phone Setup?  You could just change the IP address in DNS and update your entire deployment!

4 – Is it no time to get rid of the DB9 Serial IO cable for switch configuration?  It is the 21st century and we sill do not have USB on ShoreTel switches?  When was the last time you had a serial printer cable with an DB9 connector hanging around your tool bag?

5 – Setting up OBDC connectors still use 32-bit connector and do NOT support AD login.   If you are connecting to Microsoft SQL, for example, if you use an AD credential your ECC Script will fail.

6 – Active Directory  integration is a one way sync out of the box.   Even adding the ShoreTel professional Service add on for AD will not solve all your issues!

7 – ECC Shifts will destroy you if you do not fully understand their use and impact.   Create a Shift to close on of your Customer Service Queues at 1PM and end up closing your entire call center!  (See this Blog entry).

8- You can not enter anything in the External number field for transfer a call but  9+1+NPA- NXX-XXXX.  No 9+011-44-204-668-5000 for example.  It will not work! Not for a call handling mode, CTI route point, External Assignment.  Nada. no bueno, NFG!

Useful and Helpful Event Log error entries!  Our Personal favorite:

ShoreTelError

Send us your favorite Installation and Maintenance Gotcha’s and we will update the list and credit you for the contribution!

DrVoIP@DrVoIP.com

 

 

Configuring Compliance Recording with CISCO Workforce Optimzaiton

Posted on by DrVoIP

For compliance will you Record Audio or Screen?

There are a range of products and services that can be used to “record” phone calls on a CISCO CUCM, UCCX and UCCE. These products range in sophistication from services that simply save a wav file of a recording which you can search for by time and date, to very sophisticated recorders with advanced index and search capabilities. Some products even include speech recognition functions that can be used to search files for a particular call or even handle a recording base on the audio content.  CISCO Workforce Optimization and Advanced Quality Management adds desktop screen recording and metrics that can also be used for evaluations and service observing.

All of these products have one characteristic in common; they require you to configure a recording capability in your CUCM to copy the media stream from the target source to the recording server. There are a number of options for doing this including SPAN recording and other options that require you to configure your Ethernet switches to accommodate the recording function.  The simplest method to copy media streams is to use the Built in Bridge or BiB of a CISCO phone.  Clearly this can only work for a specific set of CISCO phones, but most phones support this function.

General Recipe!

The following is a basic general recipe for setting up your CISCO CUCM for recording and identifies some of the decisions you have to make along the way.

  • Gateway or Phone;
  • Notification or not; Notify the caller, or the Agent or both?
  • Route Pattern for Recorder along with a Partition and Calling Search Space.
  • Create a Recording Profile;
  • SIP Trunk Setup between CUCM and Recording server; and options CUCM and Gateway;
  • Identify Users and add them to the proper CTI Recording and Monitoring Group;
    CTIPemissions
  • Enable BiB on the phone;
    BiB
  • Enable Recording on the DN;  Auto always or Selective and reference Recording Profile;

Basically you are creating a conference call between the “source media”, the CUCM and the Recording server.    The actual configuration has a lot of details and you will need to carefully consider each element, but the actual setup is routine for your average deployment engineer.   We summarized them below in video format.

You can also use CISCO XML services to enable SELECTIVE recording in which a phone is recorded only on the demand of the user.  This uses the Calabrio XML api and Phone Service option in CISCO.  This will enable the user to push RECORD, PAUSE, RESUME and STOP.  The second video walks you through the configuration options for that service.  This can also be deployed in Finesse as a desktop button!

Build ShoreTel Connect inside your own private Cloud using AWS!

Posted on by DrVoIP

Placing your ShoreTel HQ in the “cloud”?

Moving the ShoreTel HQ server to a data center to increase system resiliency, reduce or eliminate down time and increasing overall recovery times has always been high on the check list for business continuity and disaster preparedness.    Our preferred “data center” however is Amazon Web Services, or AWS for short!   We have been deploying ShoreTel in AWS as a “private” cloud solution for some time and have several blogs on the subject.

Do you already have an Amazon Account?

If  you have a regular old Amazon book buying account, you already have all you need to log into AWS and get started building out your own virtual private cloud!    Though there is a lot to learn,  in less than 15 minutes you can spin up a Windows 2012 Server in a virtual private network and then link it back to your onsite location with an AWS provided VPN Gateway!

The simplest ShoreTel/AWS deployment model

The simplest of VoIP deployment models is the placement of the ShoreTel Connect Server in an AWS Region and availability zone of your choice.    Typically, we defined a private subnet in three different AWS availability zones and then launched a ShoreTel Connect server.    The availability zones provide additional resiliency  options.  It is even possible to setup an Elastic load balancer than can move from one ShoreTel HQ server to a standby duplicate in another availability zone in the very unlikely situation of a AWS availability zone going off line!

You can interconnect your ShoreTel Connect VPC  with your remote sites over a VPN, ultimately moving to a “direct connect” circuit and only using the VPN for backup.   The remote sites will have ShoreGear resources to support localized carrier access and onsite user phone services.   The distributed nature of the ShoreTel architecture makes this a natural deployment model.   This is  by far the simplest of the deployment options and one that everyone who is considering moving a ShoreTel HQ server to a data center should consider.

Even Ingate in the Cloud?

With ShoreTel Version 14, virtual switch resources make it possible to create the entire deployment in your VPC.  You can even deploy your Ingate as a virtual Session Border Controller, in the AWS cloud and centralize your SIP carrier access.    This is a bit more demanding then spinning up a Windows server but now that AWS enables you to import vmware machines, it is an exciting option.

Importing vmware based ShoreTel machines

The secret to deploying ShoreTel vSwitches in the AWS cloud is to first build the machines as vmware machines in your local environment with an IP that can be duplicated in your private virtual network.   Once your machines are created, you can then import them into AWS.

The options for deploying VOIP in your own “private cloud” have never been more flexible.   Your CFO is going to be impressed when comparing AWS to the cost of building out your own data center or renting space in a collocation facility.   You have all of the benefits and none of the cost associated with a typical infrastructure build out.    Connection options are unlimited and you can access AWS facilities on a global bases!

The Video clip demonstrates a ShoreTel HQ and ECC Server in an AWS VPC, with a VPN back to the main office site.   The office site contains ShoreGear switches for SIP trunk access and 400 series phone support.  There is a synergy when integrating AWS and ShoreTel that every CIO should be seriously considering.    Give us a call and we can help make this happen for your company!

 

Add DNIS routing to your ShoreTel ECC Contact Center!

Posted on by DrVoIP

Why Route by DNIS?

Routing by the number the caller dialed, or DNIS is the preferred routing strategy for any Call Center call flow.  Clearly you can assign a DID phone number to a specific call flow and anyone who knocks on that door is answered by the same group of agents.   It is much more efficient to grab the DNIS information, however, and use it to index a database to retrieve the call routing information.  In this way, we only need one door to the call center!  The DNIS might be used to route a call to the proper product or service group and it may also be used to retrieve client information that the call center Agent needs to see displayed  in order to provide a custom care answer prompt.

Consider the requirements of a Hospital that is providing “centralized scheduling services” for 1000’s of primary care physicians.  When the inbound call is presented to the Agent, the requirement is that the caller be greeted with a customized answer prompt.  For Example:  “Doctor Leary’s office, are you calling to make an appointment?” or “Thank you for calling Doctor Williams”.  This type of dynamic call handling can best be managed by using DNIS information to retrieve the Doctor’s name from a database   We do this regularly in CISCO UCCX and ShoreTel ECC Call Center solutions and the process is essentially the same for both solutions.

ShoreTel ECC Route by DNIS example

First, we need to create a DNIS Map in the ShoreTel PBX; a ‘route point/IRN ‘ combination to pass the call to the ECC;  and an ODBC connector from the ECC server to your favorite SQL database server.   The SQL server would host the database your scripting application needs to access in order to obtain the correct answer prompt.  Lets assume that the database contains a very simple table structure:

DABASE = DNIS_listofDoctorsOffices = (Field1 = DNIS Number, Filed2 = OfficeName, Field3 = QUEUE_IRN)

You would then write a simple script to take the incoming DNIS information and use it to index the database and get the OfficeName and maybe the Customer Service Queue that handles that office (City or State or what have you).  There is no limit to the information you could retrieve and present to the Agent,  For example: Name, Service Class (Platnium, Gold or Silver), Renewal date, last order, shipment date, the list goes on.   In this simple example the script would take the DNIS and use a SQL expression to retrieve the answer prompt data:

Select * from DNISlistof DoctosOffices where DNIS = %DNIS_NAME%Sample ECC Script Screen

Creating a DNIS MAP in ShoreTel iPBX

In the ShoreTel iPBX Trunk Group it is necessary to create a DNIS map for two reasons:  First, the ShoreTel ECC can not read the DNIS directly, it requires the administrator to fill in the “dialed number” column in the DNIS map.  The ECC has a mandatory call profile filed named DNIS-NAME which will be auto filled with the information you provide in the DNIS map “dialed number” column.     Secondly, unlike a DID number that might be directly mapped to an extension, we need a way to get the incoming call connected to the IRN on the ECC that is running the DNIS SQL lookup  Script.   In this example, the Destination field of the DNIS Digit Map in the ShoreTel iPBX Truk Group points to the Route Point/IRN in the ECC that supports the script.

 

DNISMAP

POPing the Agent Display with useful Data

The ShoreTel ECC has two variables data types: Mandatory or System Variables; and User create Variables.  The Mandatory variables are system call parameters like ANI or DNIS and a long list of other system based data.   ANI contains the digits that make up the caller identification and that is also often used to retrieve database information.   If you are using ANI you will need to do some string manipulation to strip off the +1 from the 10 digit number, or format to match your database.   User created variables are the name you create for the fields you will get from your database.  Useful examples would be CustomerName, DateOfService, AccountBalance and RenewalDate.    Any Variable, User created or System,  can be pushed out to the Agent Display within the ShoreTel Communicator.

ScreenPop

What is your Call Center Application Requirement

We have seen it all, so we are always interested in your requirements for custom CRM integration and Call Flow management.  Give us call or drop us an email and play “stump the vendor”.   We would love the challenge of finding yet another new ShoreTel ECC or CISCO UCCX Contact Center application requirement!

[show_related ids=”2828, 2610 ,2447, 1378″]

What Carrier can provide Fiber to my branch office?

Posted on by DrVoIP

What Carrier do I use for this location?

If are responsible for planning out a WAN connectivity solution for your VoIP deployment, you need to know what carrier services your target circuit location. This can lead to the most frustrating experiences an engineer can have! You actually have to rely on someone else to provide information so you can finish your work! Even a simple point to point VPN tunnel requires you to figure out what carrier options are available at your target location. How do you do that? Start calling a list of carriers and asking the first line call center sales folks if they can provide an internet circuit to your branch office in Syracuse, New York? You do a google search and you end up with a list of possible candidates and then you start your outbound calling! Maybe you have a friend who is a sales rep for a circuit aggregator, so you try that option.

The secret Carrier database!

What if you could go to a website, you don’t even need to talk with a sales person, you just plug in an address and Viol! A list of all the Carriers that can service that location magically appears! X marks the spot of every Fiber drop that carrier has in the specified distance from your target address. Not just the carrier your aggregator wants to show you, but all the carriers that can service that target location. You even get a Google map street photo of the location! What if you could just click on that magic X and get a quote! Now that is freaking awesome!

We have been working on a very large WAN deployment to a ShoreTel system that has over 500 branch offices! Now try and knit together that circuit map without a database resource that you can directly tap. We discovered a website that makes the process as simple as entering a location address. Blow out your candle Pilgrim you search has ended, just click here  enter a Street location and you you will get a list of carrier solutions.

buildinglit.com

The good folks at BuildingIT have made finding WAN solutions as simple as locating an Uber Driver!    You don’t have to talk to a sales person, but if you do, they have some of the smartest circuit folks in the industry.  Can’t find fiber for  your Laramie WO location, ask sales to quote a solution through the website and they will come back with any number of alternative solutions, priced and ready for the next phase of your deployment, installation.   They even offer  bundled project management so you don’t have to worry the deploy.  One throat to choke, one website to research and one solution that makes a lot of sense to us!

 

ShoreTel lsp_ping and the SG-Vphone coma!

Posted on by DrVoIP

Ping a network engineers best friend!

Most network folks are comfortable with a standard Ping command.    Some even know that you can add options to ping to set packet size and repetitions, but at the end of the day, Ping is a level three ICMP command.

Ping Command Syntax

ping [-t] [-a] [-n count] [-l size] [-f] [-i TTL] [-v TOS] [-r count] [-s count] [-w timeout] [-R] [-S srcaddr] [-4] [-6] target [/?]

As an example: ping -n 5 -l 1500 www.google.com  the ping command is used to ping the hostname www.google.com. The -n option tells the ping command to send 5 ICMP Echo Requests instead of the default of 4 and the -l option sets the packet size for each request to 1500 bytes instead of the default of 32 bytes.

Ping is generally a knee jerk reaction for a network engineer!  It will establish and demonstrate connectivity between network devices.  It can be used to determine latency and jitter and is a very quick, effective and easy to execute network test tool.

ShoreTel Connectivity

ShoreTel administrators learn very quickly that the first place you go when someone complains about the phone system is the ShoreWare Director portal to the “Quick Look” section.   You quickly scan the screen for RED!   (That is what is so simple about alarms:  Green is good, Yellow means something needs attention and Red is bad)!   In this example the site known as Carol can not connect with the the Bob site.   Bob however can connect with the Ted and Alice Sites.  Carol can also connect with the Ted and Alice sites.   How do you visualize this in ShoreTel?   You go to the second screen and by clicking on  Connectivity and you see the famous ShoreTel Christmas Tree!

xmastree

This is very not good!   Sites can not communicate and calls are failing all over the place!  For  example the trunk group that terminates on HQ SGT1K-03 (Line 13), for example will not be able to complete an incoming call to a user on the user switch Central Time Zone (Line2).  The RED box at the intersection of two lines helps you visualize connectivity or lack of connectivity.   Why is this happening?

Lets Ping and find out!

The network engineer will undoubtedly do a Ping test.   You will ping from the source IP address of one site, to the destination ip address of the other sites device.  Now if the test fails, the next step is to figure out where the network connection is broken.   The more challenging problem is more perplexing.   What happens if the Ping is succesful?  You do your Ping test and you get an excellent reply and the network is not broken at the Layer 3 level.  What then?

Enter lsp_ping

ShoreTel has a proprietary protocol named lsp_ping.  This ping makes it possible to test network connectivity at the higher L4, transport level by enabling you to Ping with a port number.    In the example Bob can Ping Carol, so why is the BOX red?   The answer might be to run an lsp-ping  command.   Unlike Ping which you can run from your local computer to a destination ip address, you will need to telenet or Ssh into a ShoreGear switch to run lsp_ping.   To do this you will need to run the ipbxctl  security  command from a ShoreTel server first, then telnet into the device and run your test. It will look something like this:

lsp_ping “192.168.10.12”, 100

This will setup a ping to port 5440 at the target device of 192.168.10.12 and the quotations are required!  The comma 100 means, send 100 packets in this test.

What is lsp_ping?

Shoregear switches all keep a copy of the current configuration and know where the end points (i.e. handsets ) are and how to get there.   This is one of the strong points of the ShoreTel architecture.   You do not need to check with a server to get the current configuration, and if the server crashes, ShoreTel keeps on trucking!   The proprietary lsp or location service protocol, keeps updating configurations around the deployment reporting any changes.  They do this through ports like 5440 and if they can not update, they assume that switch is off line and thus you can not connect to it.

If you study the screen shot above and you knew the IP address of all the devices, you would be puzzled as to why two devices in the same subnet, using the same network path, going through the same network connection should have different connectivity status reports?  Every engineer has asked “what changed in the network” and every client answers the same; “nothing”.   If is always frustrating when you get a situation reported like the graphic above.   A working network for months with no problem and then suddenly the Christmas tree appears!  Especially when everyone reports that nothing has changed in the network!

ShoreTel TAC will always tell you a failure of lsp_ping means a network issue.  Most times it is, but this time it was not!

The take away from this discussion is that Ping can establish network layer connectivity, but ShoreTel can still have a connectivity failure.  You will then need to trouble shoot higher up the stack and look at the transport level and perhaps the application level.  In this scenario, we mirrored all network traffic to a packet capture device.  We ran Wireshark on the packet captures while doing lsp_ping and were surprised to see that lsp_pings on port 5440 were in fact not only sent, but received by the traget device, yet no answer packets were returned.  At first we though that it might be that a UDP packet, being connection-less, would not report, but looking at working sessions we determined we should see a return packet!

ShoreTel V-phone virtual appliance  coma!

The interesting fact in this mess was that the failure was always between the two sites (main location and a data center); and that the pair was always a hardware switch and a virtual switch!   The virtual switch seemed to be in some kind of coma as it did not return lsp_ping packets yet showed green in quick look.  As the issue was causing major ShoreTel call failure and a lot of people had investigated the network with no root cause identified, a decision was made to rebuild all the virtual switches or Vphone and Vtrunk as ShoreTel calls these linux based OVA files.  This cleared the problem with no changes to the network and all is now Green!

Summary Recommendation on SG-Vphone and SG-Vtrunk

We recommend the use of ShoreTel SG-Vphone virtual appliances only as failover and not as production resources.   Inside those ShoreGear Orange boxes is a rack full of dedicated micro-processors that provide DSP or digital signal processing resouces.   When these are vitualized all that processing most now be done in software.  What a work load!