I just wanted to thank JT 10000000x for his help in isolating my client's FIOS packet loss issue to lcr22/lcr02.lsan.ca.frontiernet.net. The issue wasn't limited to just our circuit and affected others in the area, although maybe they didn't notice. We were losing almost exactly 10% on certain routes. JT is who told me what to ping to isolate the problem, and without his input I'm fairly certain this would have never been solved.
At every twist and turn tech support just blew us off. Even the FrontierDirect forum, which DID eventually get back to me, also connected me to a tech that deflected and wasn't helpful.
In swoops JT and miraculously our problems are solved.
I've been at this since the 7/10/17 and tech support couldn't have been less helpful. Some choice quotes:
Called level 1 support, "Call Level3"... originally thought this might be a peering issue (I did call L3 and they laughed at me)
From my FrontierDirect assigned tech - - Didn't bother looking at the vast amount of WinMTR dumps, and evidence I provided.
"Hello - was able to get ahold of my NT. Our NT do not see any issues on our side they are thinking there is an issue with your VOIP servers. I did give them all the information that you gave me."
From my FrontierDirect liason after I asked they check again, and provided them even more evidence:
"I have forwarded your concerns to *****'s supervisor to see if your service can be reviewed again."
IF????
And now here's all the stuff I provided FrontierDirect with.
-------
Your Name and Address: **********
Your Account number and PIN (required): ***********
A description of your problem: 6-20% packet loss via certain routes. We've ruled out our internal network. Seems to happen with routes involving Level3... for instance routing to Google DNS, 8.8.8.8 works perfectly always. We are rolling out a VOIP system and about 10% of the time traffic to/from our circuit encounters packet loss right around 10%. The problem is intermittent but can be easily duplicated within minutes of each trace.
Any troubleshooting/diagnosis that has already been performed: See below:
outbound.vitelity.net
|------------------------------------------------------------------------------------------|| WinMTR statistics || Host - % | Sent | Recv | Best | Avrg | Wrst | Last ||------------------------------------------------|------|------|------|------|------|------|| xxxx-er5poe-edge-gw.xxxxxxxxxx.local - 0 | 37 | 37 | 0 | 0 | 0 | 0 ||l300.lsanca-vfttp-169.gni.frontiernet.net - 0 | 37 | 37 | 1 | 2 | 32 | 1 || 172.102.102.230 - 0 | 37 | 37 | 3 | 5 | 8 | 4 || ae8---0.scr01.lsan.ca.frontiernet.net - 0 | 37 | 37 | 3 | 3 | 8 | 4 || ae0---0.cbr01.lsan.ca.frontiernet.net - 0 | 37 | 37 | 3 | 3 | 13 | 3 || lag-101.ear2.losangeles1.level3.net - 78 | 9 | 2 | 5 | 5 | 5 | 5 || No response from host - 100 | 7 | 0 | 0 | 0 | 0 | 0 || broadvox-di.ear3.denver1.level3.net - 8 | 27 | 25 | 38 | 38 | 38 | 38 || No response from host - 100 | 7 | 0 | 0 | 0 | 0 | 0 || 66.241.97.205 - 0 | 37 | 37 | 37 | 37 | 42 | 42 || 64.2.142.190 - 12 | 26 | 23 | 0 | 36 | 37 | 36 ||________________________________________________|______|______|______|______|______|______| WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider
outbound.vitelity.net
|------------------------------------------------------------------------------------------|| WinMTR statistics || Host - % | Sent | Recv | Best | Avrg | Wrst | Last ||------------------------------------------------|------|------|------|------|------|------|| xxxx-er5poe-edge-gw.xxxxxxxxxx.local - 0 | 79 | 79 | 0 | 0 | 2 | 0 ||l300.lsanca-vfttp-169.gni.frontiernet.net - 0 | 79 | 79 | 1 | 2 | 39 | 1 || 172.102.102.230 - 0 | 79 | 79 | 3 | 5 | 13 | 5 || ae8---0.scr01.lsan.ca.frontiernet.net - 2 | 75 | 74 | 3 | 3 | 7 | 4 || ae0---0.cbr01.lsan.ca.frontiernet.net - 0 | 79 | 79 | 3 | 4 | 23 | 3 || No response from host - 100 | 15 | 0 | 0 | 0 | 0 | 0 || No response from host - 100 | 15 | 0 | 0 | 0 | 0 | 0 || broadvox-di.ear3.denver1.level3.net - 7 | 64 | 60 | 37 | 37 | 38 | 38 || No response from host - 100 | 15 | 0 | 0 | 0 | 0 | 0 || 66.241.97.205 - 0 | 79 | 79 | 37 | 37 | 41 | 37 || 64.2.142.190 - 9 | 59 | 54 | 36 | 36 | 37 | 36 ||________________________________________________|______|______|______|______|______|______| WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider
sip.flowroute.com
|------------------------------------------------------------------------------------------|| WinMTR statistics || Host - % | Sent | Recv | Best | Avrg | Wrst | Last ||------------------------------------------------|------|------|------|------|------|------|| xxxx-er5poe-edge-gw.xxxxxxxxxx.local - 0 | 45 | 45 | 0 | 0 | 0 | 0 ||l300.lsanca-vfttp-169.gni.frontiernet.net - 0 | 45 | 45 | 1 | 1 | 8 | 1 || 172.102.98.180 - 0 | 45 | 45 | 3 | 5 | 8 | 4 || ae8---0.scr01.lsan.ca.frontiernet.net - 0 | 45 | 45 | 3 | 3 | 5 | 3 || ae0---0.cbr01.lsan.ca.frontiernet.net - 0 | 45 | 45 | 3 | 5 | 37 | 21 || No response from host - 100 | 8 | 0 | 0 | 0 | 0 | 0 || ae-1-4.bar1.lasvegas1.level3.net - 0 | 45 | 45 | 8 | 11 | 43 | 11 || ae-1-4.bar1.lasvegas1.level3.net - 0 | 45 | 45 | 8 | 11 | 43 | 8 || switch-comm.bar1.lasvegas1.level3.net - 0 | 45 | 45 | 9 | 11 | 14 | 11 || be76.las-core7-1.switchnap.com - 0 | 44 | 44 | 10 | 11 | 14 | 10 || be771.las-agg7s3-1.supernap.com - 0 | 45 | 45 | 10 | 10 | 12 | 11 || flowroute.demarc1.switchnap.com - 0 | 45 | 45 | 9 | 9 | 11 | 9 || 216.115.69.190 - 0 | 45 | 45 | 10 | 10 | 14 | 11 || 216.115.69.144 - 10 | 33 | 30 | 10 | 10 | 11 | 10 ||________________________________________________|______|______|______|______|______|______| WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider
CNN.com
|------------------------------------------------------------------------------------------|| WinMTR statistics || Host - % | Sent | Recv | Best | Avrg | Wrst | Last ||------------------------------------------------|------|------|------|------|------|------|| xxxx-er5poe-edge-gw.xxxxxxxxxx.local - 0 | 78 | 78 | 0 | 0 | 3 | 0 ||l300.lsanca-vfttp-169.gni.frontiernet.net - 0 | 78 | 78 | 1 | 11 | 578 | 1 || 172.102.104.58 - 11 | 55 | 49 | 0 | 6 | 9 | 7 || ae8---0.scr02.lsan.ca.frontiernet.net - 18 | 47 | 39 | 0 | 5 | 6 | 5 || ae1---0.cbr01.lsan.ca.frontiernet.net - 11 | 55 | 49 | 0 | 5 | 13 | 5 || No response from host - 100 | 15 | 0 | 0 | 0 | 0 | 0 || 4.69.153.121 - 0 | 78 | 78 | 4 | 4 | 13 | 4 || ae-1.fastly.lsanca20.us.bb.gin.ntt.net - 0 | 78 | 78 | 5 | 5 | 7 | 5 || 151.101.65.67 - 18 | 47 | 39 | 5 | 5 | 10 | 5 ||________________________________________________|______|______|______|______|______|______| WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider
I can provide any other info or traces you'd like. This happens without any firewall as well, we've tested with same results with laptop connected directly to the Ethernet handoff.
Thank you so much for any help you can provide!!
-----
I'd like to add that I contacted JT via email prior to posting previously. He wrote:
----------------------------
I received your email. The level 3 hops are policed to protect control plane, but your transit ICMP stats are telling. I am on vacation for the next week and a bit out of touch. If you can attempt to ping lsanca-lcr-21/22 and have packet loss, it is probably at the PON/OLT. Or, perhaps a circuit throwing errors.
One other test would be connecting via a VPN if possible, as that will change destination/port of the traffic and likely hash it to a different link.
IPs of LCR-21/22 - 172.102.74.9 and 172.102.74.11
IPs of scr01/02.lsan.ca - 74.40.0.104 and 74.40.0.105
If you see packet loss to any of those, please reach out to the Direct forum. I don't work in the NOC, just built the infrastructure and cut over the FiOS edge to our backbone. I have influence in the NOC, but do not directly work on issues.
----------------------------
I went ahead and performed the tests against the IPs specified. All were fine, except 172.102.74.11, which yielded almost the exact packet loss we are encountering.
|------------------------------------------------------------------------------------------|| WinMTR statistics || Host - % | Sent | Recv | Best | Avrg | Wrst | Last ||------------------------------------------------|------|------|------|------|------|------|| xxxx-er5poe-edge-gw.xxxxxxxxxx.local - 0 | 2012 | 2012 | 0 | 0 | 2 | 0 ||l300.lsanca-vfttp-169.gni.frontiernet.net - 0 | 2013 | 2013 | 1 | 2 | 122 | 1 || lcr02.lsan.ca.frontiernet.net - 12 | 1397 | 1236 | 4 | 6 | 52 | 7 ||________________________________________________|______|______|______|______|______|______| WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider
|------------------------------------------------------------------------------------------|| WinMTR statistics || Host - % | Sent | Recv | Best | Avrg | Wrst | Last ||------------------------------------------------|------|------|------|------|------|------|| xxxx-er5poe-edge-gw.xxxxxxxxxx.local - 0 | 30 | 30 | 0 | 0 | 0 | 0 ||l300.lsanca-vfttp-169.gni.frontiernet.net - 0 | 30 | 30 | 1 | 2 | 39 | 2 || lcr02.lsan.ca.frontiernet.net - 10 | 22 | 20 | 5 | 6 | 8 | 7 ||________________________________________________|______|______|______|______|______|______| WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider
|------------------------------------------------------------------------------------------|| WinMTR statistics || Host - % | Sent | Recv | Best | Avrg | Wrst | Last ||------------------------------------------------|------|------|------|------|------|------|| axco-er5poe-edge-gw.axcoadhesive.local - 0 | 98 | 98 | 0 | 0 | 2 | 0 ||l300.lsanca-vfttp-169.gni.frontiernet.net - 0 | 98 | 98 | 1 | 1 | 8 | 1 || lcr02.lsan.ca.frontiernet.net - 9 | 73 | 67 | 4 | 6 | 8 | 5 ||________________________________________________|______|______|______|______|______|______| WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider
|------------------------------------------------------------------------------------------|| WinMTR statistics || Host - % | Sent | Recv | Best | Avrg | Wrst | Last ||------------------------------------------------|------|------|------|------|------|------|| xxxx-er5poe-edge-gw.xxxxxxxxxx.local - 0 | 652 | 652 | 0 | 0 | 2 | 0 ||l300.lsanca-vfttp-169.gni.frontiernet.net - 0 | 652 | 652 | 1 | 2 | 171 | 1 || lcr02.lsan.ca.frontiernet.net - 10 | 473 | 426 | 4 | 6 | 11 | 7 ||________________________________________________|______|______|______|______|______|______| WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider
Here's a long term ping against lcr02, you can see and almost exact 10% loss consistent with what we are seeing on the circuit:
Ping statistics for 172.102.74.11: Packets: Sent = 28876, Received = 25965, Lost = 2911 (10% loss),Approximate round trip times in milli-seconds: Minimum = 4ms, Maximum = 168ms, Average = 6ms
----
Here are further traces from our Asterisk system we are developing which coincide with the packet loss issues. We experience the same packet loss (only on receiving end) regardless of SIP provider. We use both Vitelity and Flowroute servers, and I've included a remote extension behind Spectrum cable to show that regardless of receiving source, the problem persists.
You can see that every X calls, there is almost a perfect 10% loss on the receiving end. The sending side never has any issues.
These calls were placed within about a 10 minute period for each provider/node. I can easily recreate the issue by simply calling over and over again. You will get a lossy call 25% of the time give or take. Take note of the "Lost %" section on the receiving end indicating the packets lost on the call.
Also, if I leave a "lossy" call on for 20 minutes versus the short traces below, the packet loss is identical long term and never recovers.
Incoming calls, via Vitelity - Vitelity -> Internet -> FIOS -> PBX:
Peer Call ID Duration Recv: Pack Lost ( %) Jitter Send: Pack Lost ( %) Jitter140.239.143.5 4c6d2f765be 00:00:11 0000000599 0000000000 ( 0.00%) 0.0000 0000000553 0000000000 ( 0.00%) 0.0005140.239.143.5 67bf9521561 00:00:15 0000000798 0000000000 ( 0.00%) 0.0000 0000000753 0000000000 ( 0.00%) 0.0001140.239.143.5 605daca075e 00:00:16 0000000858 0000000000 ( 0.00%) 0.0000 0000000813 0000000000 ( 0.00%) 0.0015140.239.143.5 38a9f579621 00:00:15 0000000742 0000000083 (10.06%) 0.0000 0000000784 0000000000 ( 0.00%) 0.0001140.239.143.5 0d86aca84c5 00:00:17 0000000930 0000000000 ( 0.00%) 0.0000 0000000886 0000000000 ( 0.00%) 0.0002140.239.143.5 5676a2a4371 00:00:17 0000000927 0000000000 ( 0.00%) 0.0000 0000000882 0000000000 ( 0.00%) 0.0007140.239.143.5 3a04f221599 00:00:20 0000001087 0000000000 ( 0.00%) 0.0000 0000001043 0000000000 ( 0.00%) 0.0007140.239.143.5 385faf96134 00:00:19 0000000996 0000000000 ( 0.00%) 0.0000 0000000952 0000000000 ( 0.00%) 0.0003140.239.143.5 1caf8b7c6af 00:00:25 0000001175 0000000145 (10.98%) 0.0000 0000001283 0000000000 ( 0.00%) 0.0001140.239.143.5 3a138bf7250 00:00:21 0000001126 0000000000 ( 0.00%) 0.0000 0000001079 0000000000 ( 0.00%) 0.0003140.239.143.5 507682c552a 00:00:23 0000001190 0000000000 ( 0.00%) 0.0000 0000001147 0000000000 ( 0.00%) 0.0008140.239.143.5 392644a434c 00:00:22 0000001060 0000000111 ( 9.48%) 0.0000 0000001099 0000000000 ( 0.00%) 0.0000140.239.143.5 1404ecec6be 00:00:22 0000001150 0000000000 ( 0.00%) 0.0000 0000001103 0000000000 ( 0.00%) 0.0001140.239.143.5 055636c513c 00:00:20 0000001077 0000000000 ( 0.00%) 0.0000 0000001032 0000000000 ( 0.00%) 0.0007140.239.143.5 055636c513c 00:00:20 0000001077 0000000000 ( 0.00%) 0.0000 0000001032 0000000000 ( 0.00%) 0.0007140.239.143.5 36ab4e1d5df 00:00:24 0000001078 0000000117 ( 9.79%) 0.0000 0000001220 0000000000 ( 0.00%) 0.0001
Calls from remote extension, behind Spectrum Cable - Spectrum -> Internet -> FIOS -> PBX:
Peer Call ID Duration Recv: Pack Lost ( %) Jitter Send: Pack Lost ( %) Jitter172.112.11.79 13fddf00656 00:00:37 0000001546 0000000159 ( 9.33%) 0.0000 0000001716 0000000000 ( 0.00%) 0.0013172.112.11.79 ea1da0306e3 00:00:19 0000000833 0000000000 ( 0.00%) 0.0000 0000000829 0000000000 ( 0.00%) 0.0020172.112.11.79 539f6dffcc3 00:00:18 0000000714 0000000073 ( 9.28%) 0.0000 0000000792 0000000000 ( 0.00%) 0.0030172.112.11.79 b6b42dd3428 00:00:28 0000001307 0000000000 ( 0.00%) 0.0000 0000001306 0000000000 ( 0.00%) 0.0023172.112.11.79 b6176b8a748 00:00:22 0000001018 0000000000 ( 0.00%) 0.0000 0000001018 0000000000 ( 0.00%) 0.0027172.112.11.79 1bb56246fa7 00:00:26 0000001064 0000000110 ( 9.37%) 0.0000 0000001187 0000000011 ( 0.93%) 0.0022172.112.11.79 6fced586f31 00:00:19 0000000771 0000000087 (10.14%) 0.0000 0000000867 0000000000 ( 0.00%) 0.0015172.112.11.79 341d7dbff65 00:00:07 0000000253 0000000000 ( 0.00%) 0.0000 0000000250 0000000000 ( 0.00%) 0.0017
Outbound calls to Flowroute - PBX -> FIOS -> Internet -> Flowroute:
Peer Call ID Duration Recv: Pack Lost ( %) Jitter Send: Pack Lost ( %) Jitter216.115.69.144 7082aaa71a7 00:00:26 0000001216 0000000000 ( 0.00%) 0.0000 0000001217 0000000000 ( 0.00%) 0.0002216.115.69.144 0bcc9d8d077 00:00:19 0000000890 0000000000 ( 0.00%) 0.0000 0000000809 0000000000 ( 0.00%) 0.0011216.115.69.144 68358f6878f 00:00:23 0000001067 0000000000 ( 0.00%) 0.0000 0000001068 0000000000 ( 0.00%) 0.0007216.115.69.144 5939f42a295 00:00:19 0000000875 0000000000 ( 0.00%) 0.0000 0000000876 0000000000 ( 0.00%) 0.0002216.115.69.144 0602188f103 00:00:12 0000000465 0000000053 (10.23%) 0.0000 0000000525 0000000000 ( 0.00%) 0.0001216.115.69.144 1a1d92da399 00:00:19 0000000859 0000000000 ( 0.00%) 0.0000 0000000862 0000000000 ( 0.00%) 0.0004216.115.69.144 5b517b7e19b 00:00:21 0000000938 0000000000 ( 0.00%) 0.0000 0000000835 0000000000 ( 0.00%) 0.0007216.115.69.144 05d98aba37e 00:00:16 0000000692 0000000000 ( 0.00%) 0.0000 0000000692 0000000000 ( 0.00%) 0.0004216.115.69.144 6628c7bc54d 00:00:23 0000000937 0000000095 ( 9.21%) 0.0000 0000001043 0000000000 ( 0.00%) 0.0001
And some straight ping fails:
outbound.vitelity.net
Ping statistics for 64.2.142.190: Packets: Sent = 55, Received = 47, Lost = 8 (14% loss),Approximate round trip times in milli-seconds: Minimum = 36ms, Maximum = 36ms, Average = 36ms
sip.flowroute.com
Ping statistics for 216.115.69.144: Packets: Sent = 47, Received = 43, Lost = 4 (8% loss),Approximate round trip times in milli-seconds: Minimum = 10ms, Maximum = 18ms, Average = 10ms
↧