Troubleshooting ShoreTel Archives - DrVoIP

Configuring a SIP Trunk TIE Line between CISCO CUCM and ShoreTel iPBX

Posted on September 6, 2017 by DrVoIP

Merging Systems is fun and easy?

In a world of swirling mergers and acquisitions, IT organizations are expected to integrate the different systems so that they all work together. In the world of IP PBX systems, that might result in having to make a ShoreTel system play nice with a CISCO system. Actually, both systems support SIP and though it is sometimes more of an art than a science, it can be configured and made to work. In fact we are seeing this request a lot lately. Not sure why, but there are a lot of ShoreTel/CISCO integrations out there in the real world! This video cheat sheet takes a look at a real world configuration from both the ShoreTel side and the CISCO side, so there is something for everybody who has to work with either or both systems.

Quick Topology Overview

In our example we have a CISCO Version 11 deployed in France and a ShoreTel version 14.2 CPE solution deployed in several NY locations. This will be an interesting configuration as both systems have over lapping extension numbers. For example, both systems have the range of 4100-4199 as extension numbers. This along with a requirement for tail end hop off or TEHO at both ends makes for an interesting configuration puzzle. We determined to take it a step at a time and get a basic SIP TIE line working between the two systems. Once that was up an operational, we could then focus on the basic dial plan strategy that would enable us to resolve the extension number conflict.

We are not going to use a session boarder controller at either end. The good news is that it will greatly simplify the configuration as the two SIP proxies will directly exchange SIP messages. In the case of the CISCO, the SIP trunk will originate on the call manager server itself and will terminate on a ShoreTel SG appliance (one of those orange boxes for you CISCO guys) with SIP Trunk resources sufficient for the number of channels you want to set up. The bad news is, that on the CISCO side it makes trouble shooting a bit more time consuming. If you had a CUBE in the mix you could easily open up CCSIP Messages in the debug on the IOS device, but when you do not have IOS you need to download logs from RTMT. On the ShoreTel side, there is the ability to do a remote wire shark capture from the HQ server diagnostic panel using the SG appliance as the end point source. (As a VOIP engineer SIP debugging should be second nature).

To avoid the use of a SBC on either end, we have to configure the solution entirely within a private IP topology. In this case we have a 10.0.0.0 /24 network on the ShoreTel side and a 192.168.53.0/24 network on the CISCO side. The WAN connection is an MPLS network.

ShoreTel and CISCO dial plan strategies

CISCO does not really care about over lapping extension numbers of difference in extension number ranges. When I say, they don’t care, I mean they have a strategy for dealing with it. You might have a multi-site deployment as a result of a recent acquisition and you have four digit extension numbers at one site, and five digit extension numbers at a different site. CISCO can deal with this. ShoreTel not so much. With ShoreTel you would have to convert the four digit site to five digits (you can expand from four to five digits, but not five to four) if you are going to have one dial plan. In CISCO you could actually let both sites coexist with different extension number lengths.

CISCO has a verbose dial plan strategy that expects you to create route “patterns” that when matched by dialed digits, point to a gateway or route list that contains a destination reachable by that route pattern. In our example, we set up a dial pattern of 51.xxxx with a pre-dot discard that points to the SIP trunk TIE Line over to the ShoreTel. The SIP Trunk configuration is relatively simple and is shown in the video below.

ShoreTel requires you to configure a Trunk Group and then put your individual trunks into the group. These individual trunks are basically SIP channels and the video also shows this configuration. The difference in the strategy of both systems is interesting to note. CISCO is routing to this trunk group based on a route patter that matches dialed digits. ShoreTel is going to want you to define both an access code and list out the OPX (off premise extension) numbers that this trunk can reach.

From a digits dialed perspective, ShoreTel is going to match the dialed digits against the list of OPX extensions to route calls over the Trunk Group that contains a matching OPX list. Note that no access code will be required, though it will be required in the Trunk Group configuration.

SIP Trunk Group Profiles

Both systems required a reference to a SIP Profile that defines specifics of the call setup messaging. Sometimes this is a bit more of an art than a science and it may take a bit of tweaking. CISCO has a standard SIP Profile and an associated SIP Security profile both of which will be referenced in the SIP trunk configuration. The only change I typically make to the SIP Profile on CISCO is to activate the “enable PING option” so that I can keep track of my trunk status which would otherwise be “unknown”.

I made all my SIP Profile changes on the ShoreTel Trunk side. This is the list of configuration options we applied and it was the result of trying and retrying, so this list alone should be worth the blog read as it most definitely works in our example:

OptionsPing=1
OptionsPeriod=60
StripVideoCodec=1
DontFwdRefer=1
SendMacIn911CallSetup=1
HistoryInfo=diversion
EnableP-AssertedIdentity=1
AddG729AnnexB_NO=1
Hairpin=1
Register=0
RegisterUser=BTN
RegisterExpiration=3600
CustomRules=0
OverwriteFromUser=0
1CodecAnswer=1

If you have never configured a ShoreTel SIP Profile, the video walks you though this in detail. ShoreTel provides a “standard tie-line” sip profile and suggests that you use it. In fact if you change it, they generate a strong message urging you not to do so, but change it we did to the settings above and we had a more than acceptable result.

Extension conflict resolution

ShoreTel trunks do not out of the box allow abbreviated dialing. The Standard ShoreTel domestic dial plan really wants what amounts to an E164 format. ShoreTel will take all digits dialed and attempt to construct a +1 NPA – NXX – XXXX dial string. So in this example, we are not going to be able to dial 51 as an access code on the ShoreTel side, then output only four digits. Not going to work. As outlined above, ShoreTel will want you to define the OPX extension reachable by the encapsulating trunk group in a detailed list as part. The good news is that this means that you only have to dial the distant extension number and ShoreTel will see it as an OPX and route it over the associated trunk.

Clearly this means these extensions are unique. What are we going to do? We have extensions on both sides of the TIE Line that are not unique, they are the same 41xx. ShoreTel will allow you to create translation table to over come this limitation. We could create a range of 61xx for our OPX list and then convert it to 41xx in the translation table and that would eliminate our problem. Not my favorite solution but it may be the one we have to go with. The down side is that we have no eliminated the 61xx range for use on the ShoreTel side. Additionally, it is not very friendly to directory systems as people have to remember that if you are calling 4304 in France from New York, you need to dial 6304 not 4304 another colleague in NY!

Ideally we want to make use of that access code. It would be nice to dial 51 + XXXX and be done with it, but as previously explained ShoreTel does not easily support that option. It may be possible to write a custom dial plan and apply it to this specific trunk group. For example, when working with paging adapters that have zones, you want to dial the paging amplifier and then dial your zone. The custom dial plan for this would by “;1I” but you would still have to dial an OPX number. For example, you would create the trunk group and maybe put 1234 as the OPX which would route the caller to the paging adapter. The custom dial plan rule would be interpreted by ShoreTel as “do not echo any digits” enabling the paging adapter to play a second dial tone, and then you could touch tone out some digits. Sounds like this would be a good solution but you would end up dialing eight digits, but it would free up a range of extensions that might otherwise be eaten alive by the required translation table.

ShoreTel Update

At the end of the day, the solution was to create a custom dial rule and apply it to the ShoreTel Trunk Group. Users inside the ShoreTel would dial 50+xxxx but as a result of the custom dial rule, the ShoreTel would see 999-555-xxxx and only pass the xxxx. Remember that ShoreTel wants you to list the extension numbers on the other side of the TIE line in the OPX list of the Trunk Group. In a situation in which you have extension conflict you can use a translation table, but often that is not the most friendly solution. Ideally you will want to dial and access code, get connected to the distant phone system and then out dial the extension digits regardless of extension conflict. To do this on a ShoreTel system will require a custom dial plan modification to the trunk group. The document on this subject is very limited so you either have to pay ShoreTel Professional Services or be prepared to play detective. We went the detective route, so hit us up for the solution.

Understanding ShoreTel Connect ECC Shifts!

Posted on August 8, 2016 by DrVoIP

Why use Shifts at all?

Most call centers have “on hours” and “off hours” that route callers to different options based on time of day and day of week. ShoreTel ECC has a concept entitled SHIFTS. Shifts have day types: Weekday, Weekend and Holiday. You go to System Parameters in the Contact Center Director, then Schedules and the create a Shift. Name the Shift something useful, specific the day type and the time the shift action should occur. You then apply it to the IRN you want to alter, open or close based on your shift schedule. If you do not understand shifts, however and how they are utilized by the ECC you are at risk of shutting down your call center unintentionally.

Let’s take a closer look at how Shifts work!

Each IRN in the ECC that uses Shifts, has a default setting that specifies where the call should route, if there is no corresponding shift instruction. The Shift might specify a Script, Service or Device. For example you might have Shift named “M-F” defined for day type Weekdays that is active at 8:00AM and specifies that the call should be routed to a script for handling calls during “on hours”. You might then also add another Shift named “Lunch” for a day type Weekdays and set if to activate at 11:59 AM and point it to script that plays an announcement alerting callers that the call center is closed for lunch and invites them to call back at 1PM (never happen but it makes for a good lesson example). Now we create another Shift also named “M-F” for day type Weekday and set it to activate at 1PM. This action points the calls back to the script for “on hours”. Clearly, you also created a Shift for “M-F” for day types Weekday and set the action time at 5PM. The action is to send the call to the script for After hours. Make sense? Seem easy?

Danger – The trouble Spot with Shifts!

Let’s assume you have an IRN for Sales and an IRN for Technical Support and you apply the shifts we defined above, to both IRN’s, but on the sales IRN they do not close for lunch! For this reason, you do not apply the Shift entitled “Lunch” to the Sales IRN. Let’s assume the default route for the Sales IRN is the Closes script, so that after hours calls defaul if not specified in the Shift list to the after hours call handling. So what do you think will happen to the Sales IRN at 11:59?

The fact that you created a Shift and one now exists for “Lunch” during the week is the important take away. Once you apply Shifts to an IRN, ALL shifts apply even if you did not apply that shift to your IRN. In this simple example, the Sales IRN will follow the Default route because you did not specific an alternative destination in you shift list on the Sales IRN for Lunch time. The fact that the Shift exists in the ECC database, requires that you specify an action for that Shift! This is where most folks get into trouble!

What Version of ECC is your Shift definition?

In versions before Connect and ECC 8, Shifts required you to open and close a shift. After version 8, you are not required to close a shift but it would be a good practice to do so! You must account for all shifts, even if they are to relevant to your IRN. The only way around this is to NOT apply any Shifts to an IRN. If the IRN does not use Shifts at all, it will ignore all Shifts! A bit confusing but once you “get it” it is easy.

When in Doubt press Explain Shifts!

In the IRN definition press “Explain Shift” and then select the day you want to explore. The list will populate with all the Shifts that impact that IRN. Remember it does not matter if it is on your list! What matters is that the Shift exists for the ECC and you must specify an action, or in the absence of a direction, the default destination for that IRN will be followed!

TOP ShoreTel Connect Installation Gotcha’s !

Posted on July 25, 2016 by DrVoIP

So you are upgrading to ShoreTel Connect! Watch out for these blistering hot spots:

Assume you have a call center that works three shifts, with 15 Agents per shift. In the Pre-Connect world, you would define 45 agents but you would only need licenses for 15. The Pre-Connect ECC license strategy was “concurrent users”. The new license strategy requires you to license 45 Agents. Additionally, all agents have to be named in the iPBX, costing you an additional “per seat” license for the newly named agents. Before, 45 agents could log into extensions Agent1 through Agent15 using their AgentID to differentiate them for reporting purposes. Wow! I hope you don’t find this out the hard way.

Here are some of our favorite ShoreTel Gotcha’s:

1 – Ever have to move route points from one server to another? You CAN NOT use the batch utility to do this and must remove and then reinstall each route point. So much fun on a large contact center deployment ( If you know SQL you can actually modify the configuration database to do this).

2 – Want to change IP address of your HQ? If you are using the new family of 400 phones, you will have to touch each phone individually and manually CLEAR the Configuration for the new Option 156 and DHCP values to take hold. How would you like to upgrade 3K phones across five times zones to learn that gem!

3 – Gee, would it not be nice to use DNS Name resolution in the Phone Setup? You could just change the IP address in DNS and update your entire deployment!

4 – Is it no time to get rid of the DB9 Serial IO cable for switch configuration? It is the 21st century and we sill do not have USB on ShoreTel switches? When was the last time you had a serial printer cable with an DB9 connector hanging around your tool bag?

5 – Setting up OBDC connectors still use 32-bit connector and do NOT support AD login. If you are connecting to Microsoft SQL, for example, if you use an AD credential your ECC Script will fail.

6 – Active Directory integration is a one way sync out of the box. Even adding the ShoreTel professional Service add on for AD will not solve all your issues!

7 – ECC Shifts will destroy you if you do not fully understand their use and impact. Create a Shift to close on of your Customer Service Queues at 1PM and end up closing your entire call center! (See this Blog entry).

8- You can not enter anything in the External number field for transfer a call but 9+1+NPA- NXX-XXXX. No 9+011-44-204-668-5000 for example. It will not work! Not for a call handling mode, CTI route point, External Assignment. Nada. no bueno, NFG!

Useful and Helpful Event Log error entries! Our Personal favorite:

Send us your favorite Installation and Maintenance Gotcha’s and we will update the list and credit you for the contribution!

DrVoIP@DrVoIP.com

ShoreTel lsp_ping and the SG-Vphone coma!

Posted on March 31, 2016 by DrVoIP

Ping a network engineers best friend!

Most network folks are comfortable with a standard Ping command. Some even know that you can add options to ping to set packet size and repetitions, but at the end of the day, Ping is a level three ICMP command.

Ping Command Syntax

ping [-t] [-a] [-n count] [-l size] [-f] [-i TTL] [-v TOS] [-r count] [-s count] [-w timeout] [-R] [-S srcaddr] [-4] [-6] target [/?]

As an example: ping -n 5 -l 1500 www.google.com the ping command is used to ping the hostname www.google.com. The -n option tells the ping command to send 5 ICMP Echo Requests instead of the default of 4 and the -l option sets the packet size for each request to 1500 bytes instead of the default of 32 bytes.

Ping is generally a knee jerk reaction for a network engineer! It will establish and demonstrate connectivity between network devices. It can be used to determine latency and jitter and is a very quick, effective and easy to execute network test tool.

ShoreTel Connectivity

ShoreTel administrators learn very quickly that the first place you go when someone complains about the phone system is the ShoreWare Director portal to the “Quick Look” section. You quickly scan the screen for RED! (That is what is so simple about alarms: Green is good, Yellow means something needs attention and Red is bad)! In this example the site known as Carol can not connect with the the Bob site. Bob however can connect with the Ted and Alice Sites. Carol can also connect with the Ted and Alice sites. How do you visualize this in ShoreTel? You go to the second screen and by clicking on Connectivity and you see the famous ShoreTel Christmas Tree!

This is very not good! Sites can not communicate and calls are failing all over the place! For example the trunk group that terminates on HQ SGT1K-03 (Line 13), for example will not be able to complete an incoming call to a user on the user switch Central Time Zone (Line2). The RED box at the intersection of two lines helps you visualize connectivity or lack of connectivity. Why is this happening?

Lets Ping and find out!

The network engineer will undoubtedly do a Ping test. You will ping from the source IP address of one site, to the destination ip address of the other sites device. Now if the test fails, the next step is to figure out where the network connection is broken. The more challenging problem is more perplexing. What happens if the Ping is succesful? You do your Ping test and you get an excellent reply and the network is not broken at the Layer 3 level. What then?

Enter lsp_ping

ShoreTel has a proprietary protocol named lsp_ping. This ping makes it possible to test network connectivity at the higher L4, transport level by enabling you to Ping with a port number. In the example Bob can Ping Carol, so why is the BOX red? The answer might be to run an lsp-ping command. Unlike Ping which you can run from your local computer to a destination ip address, you will need to telenet or Ssh into a ShoreGear switch to run lsp_ping. To do this you will need to run the ipbxctl security command from a ShoreTel server first, then telnet into the device and run your test. It will look something like this:

lsp_ping “192.168.10.12”, 100

This will setup a ping to port 5440 at the target device of 192.168.10.12 and the quotations are required! The comma 100 means, send 100 packets in this test.

What is lsp_ping?

Shoregear switches all keep a copy of the current configuration and know where the end points (i.e. handsets ) are and how to get there. This is one of the strong points of the ShoreTel architecture. You do not need to check with a server to get the current configuration, and if the server crashes, ShoreTel keeps on trucking! The proprietary lsp or location service protocol, keeps updating configurations around the deployment reporting any changes. They do this through ports like 5440 and if they can not update, they assume that switch is off line and thus you can not connect to it.

If you study the screen shot above and you knew the IP address of all the devices, you would be puzzled as to why two devices in the same subnet, using the same network path, going through the same network connection should have different connectivity status reports? Every engineer has asked “what changed in the network” and every client answers the same; “nothing”. If is always frustrating when you get a situation reported like the graphic above. A working network for months with no problem and then suddenly the Christmas tree appears! Especially when everyone reports that nothing has changed in the network!

ShoreTel TAC will always tell you a failure of lsp_ping means a network issue. Most times it is, but this time it was not!

The take away from this discussion is that Ping can establish network layer connectivity, but ShoreTel can still have a connectivity failure. You will then need to trouble shoot higher up the stack and look at the transport level and perhaps the application level. In this scenario, we mirrored all network traffic to a packet capture device. We ran Wireshark on the packet captures while doing lsp_ping and were surprised to see that lsp_pings on port 5440 were in fact not only sent, but received by the traget device, yet no answer packets were returned. At first we though that it might be that a UDP packet, being connection-less, would not report, but looking at working sessions we determined we should see a return packet!

ShoreTel V-phone virtual appliance coma!

The interesting fact in this mess was that the failure was always between the two sites (main location and a data center); and that the pair was always a hardware switch and a virtual switch! The virtual switch seemed to be in some kind of coma as it did not return lsp_ping packets yet showed green in quick look. As the issue was causing major ShoreTel call failure and a lot of people had investigated the network with no root cause identified, a decision was made to rebuild all the virtual switches or Vphone and Vtrunk as ShoreTel calls these linux based OVA files. This cleared the problem with no changes to the network and all is now Green!

Summary Recommendation on SG-Vphone and SG-Vtrunk

We recommend the use of ShoreTel SG-Vphone virtual appliances only as failover and not as production resources. Inside those ShoreGear Orange boxes is a rack full of dedicated micro-processors that provide DSP or digital signal processing resouces. When these are vitualized all that processing most now be done in software. What a work load!

Trouble Shooting ShoreTel ECC Scripts

Posted on January 22, 2016 by DrVoIP

ShoreTel ECC Future?

The ShoreTel ECC or enterprise contact center is a remarkable product in many ways. Though we are depressed to see that it has apparently taken a back seat to ShoreTel Connect as it relates to product enhancements, it remains a formidable player in the contact center space. Clearly, if you are deploying a ShoreTel phone system, then it may be the only viable option. We note that ShoreTel has made a new acquisition for contact center functionality in the cloud, so the potential of supporting ECC product enhancements may even be less hopeful as product development resources shift to the “cloud” and ShoreTel Connect.

Remember EasyRun?

We have worked with the ECC since before it was a ShoreTel product. It actually originated as an OEM product, a re-branding of a software solution brought to market by EasyRun, an Israeli based company founded by Avi Silber, long time VP of Software Development for Telrad. ShoreTel ultimately executed a software source code licnese and the rest is history. Unfortunately, we seem stalled at Version 9 of ECC, which was not a major evolutionary step from ECC Version 8.

At any rate, the product does a super job in small to medium sized Call Centers and meets the minimum daily adult requirement for call center functionality. One particular function enables the system to integrate with popular database solutions like Microsoft SQL. This enables the system to take on some very sophisticated applications that include routing inbound calls based on the return of data in the customer service database. One of them most asked questions of marketing professionals as it relates to Contact Centers: “are all customers created equal”. You might want to route an incoming call based on the callers status in your customer database as a Platinum client or as a deadbeat on credit hold.

SQL database dips are almost essential for any Contact Center offering. ShoreTel enables this functionality through OBDC connectors to the host SQL server or CRM system. One of the challenges for design and implementation engineers is testing the design and results of the SQL data dip. Historically, ShoreTel has not provided a lot of debug tools here and documentation on inner workings of the ECC is not generally available. If you are bold and go poking around in the BIN files, you will note a lot of exe files and if you are curious, brave and inquisitive you might take on an exploration of what these files are used for.

Undocumented ShoreTel Debuggers!

One file in particular seems to launch a very useful test tool. We have never found any documentation on it, and it has become something of a legend among we ECC implementation engineers. If you are fortunate enough to maintain a relationship with other engineers that share information, you can build up a library of useful tools based on shared results from others. Kudu’s to one such brilliant engineer, Bill Fraedrich for sharing his growing list of FC_Thingy functions and the other members of the development community who regularly publish results.

Route Caller by ANI (caller ID)?

Recently we had to create a SQL data dip to pull back a customer record using Caller ID or ANI as the database index. Now anybody who has done any Contact Center CRM integration regardless of vendor, knows that you have to do some string manipulations to strip off the +1 that will be passed by the carrier as part of the ANI information. ShoreTel ECC, CISCO UCCX, it does not matter it is all relatively the same. Once you clean up the ANI you can pass it off as part of a SELECT command and go get your data. Then manipulate the data to find the fields you are looking for to make your routing determination.

The undocumented debugger!

The issue becomes how do debug a script that is not producing the results your design expected? Again, ShoreTel comes up short here as it relates to documented debug tools. The CISCO UCCX, for example, has a step by step debugger built right into the script editor, which is really helpful for those of us who have to design, implement and test these scripts. It turns out that ShoreTel has one as well, you just have to know where to find it. Again it is not documented and is well known only to those that know it well.

In this video clip we take a look at the tool and show an example of how you might use it to unravel a particular SQL data dip problem. We have found a number of these tools and we are always looking for documentation and road maps produced by those who have gone this way before! Next blog we will take a look at a TAPI debugger that is also very useful when troubleshooting ShoreTel phone system Communicator issues.

At any rate, the ShoreTel ECC has great potential and is a wonderful solution when applied in the proper environment. You can create some very sophisticated routing applications based on a variety of CRM integration, custom software solutions and IVR scripts. Despite all our grumbling and complaining, we have not found anything we cant make work on a ShoreTel ECC!

V14 Trouble Shooting ShoreTel SIP

Posted on April 30, 2015 by DrVoIP

ShoreTel SIP is just not that hard to configure! When things go right, and it all works, it is relatively painless and very easy. When things don’t work however, you will need to sort out the problem and figure out how to make it work. The good news is that SIP is a clear text, english like “requests” and “response” message exchange that, once understood can be relatively easy to work with. Using packet capture tools like Wireshark can be a real assist. ShoreTel has built “remote packet capture” into the V14 diagnostic portal and makes debugging sip issues even more easy to digest.

The message exchange for SIP “extensions” is not much different than that for SIP “trunks”. So to learn SIP why not study a successful SIP connection first, before diving into a debug session. What we have tried to do in this tech tip, is to illustrate the process of a SIP extension registering with a ShoreTel proxy server. We picked a configuration that is comparatively complex, but not unlike one that you might find in the real world. SIP extensions can unit remote office workers into a seamless call handling workgroup. As such your remote worker will traverse a couple of firewalls and routers on its way to the SIP proxy.

In this configuration we register a SIP soft phone, remotely from the ShoreTel HQ site, over a site to site VPN tunnel. The VPN tunnel is between a CISCO ASA and a SonicWall TZ250. We note that 90% of the SIP issues we see having nothing to do with the configuration of the ShoreTel equipment and everything to do with the network devices, routers, switches and firewalls that are part of the SIP solution. Small business solutions, for example, tend to bring the SIP trunks into the phone system over the same connection they bring in their internet connection. This means the SIP trunk is passing through the firewall, another most likely candidate for inducing a SIP failure.

This tech tip walks you through a successful SIP registration of a remote soft phone. In our next tech tip, we will walk you through configuring a SIP trunk through a SonicWall and then look at fixing some broken SIP connections!