| Author |
 Topic  |
|
|
Pesos
Honorable But Hopeless Addict
USA
3506 Posts
Status: offline |
 Posted - 06/29/2012 : 2:52:20 PM
|
We have an office in Central California that a few months ago started seeing one of the strangest issues I've ever encountered.
We started getting user reports of lags/pauses in their Citrix sessions and cutouts/dropped calls in Lync. Nothing out of the ordinary - we just assumed something was wrong with the T1 and latency was spiking.
Latency was indeed the issue, but it wasn't on the T1 - we were seeing it between the workstations and the firewall itself on the LAN. What should be sitting at 1ms consistently was randomly jumping up into the 20s and 30s for short periods of time - with nothing but a single laptop plugged directly into the juniper firewall. Latency was far worse on desktops that were running through a switch to the firewall.
We replaced the switch - no change. We replaced the firewall - no change. I finally got the chance to go onsite (it is not conveniently located) and replicated the issue with my own laptop.
Now the weirdest part... When running a ping test from one of the workstations to a printer on the LAN, I noticed that the latency spikes were cyclical and in small batches were repeating in EXACTLY the same sequence! Like this:
Reply from 192.168.13.75: bytes=32 time<1ms TTL=255
Reply from 192.168.13.75: bytes=32 time<1ms TTL=255
Reply from 192.168.13.75: bytes=32 time<1ms TTL=255
Reply from 192.168.13.75: bytes=32 time=561ms TTL=255
Reply from 192.168.13.75: bytes=32 time=155ms TTL=255
Reply from 192.168.13.75: bytes=32 time=358ms TTL=255
Reply from 192.168.13.75: bytes=32 time=561ms TTL=255
Reply from 192.168.13.75: bytes=32 time=155ms TTL=255
Reply from 192.168.13.75: bytes=32 time=358ms TTL=255
Reply from 192.168.13.75: bytes=32 time=561ms TTL=255
Reply from 192.168.13.75: bytes=32 time=155ms TTL=255
Reply from 192.168.13.75: bytes=32 time=358ms TTL=255
Reply from 192.168.13.75: bytes=32 time=561ms TTL=255
Reply from 192.168.13.75: bytes=32 time=56ms TTL=255
Reply from 192.168.13.75: bytes=32 time<1ms TTL=255
Reply from 192.168.13.75: bytes=32 time<1ms TTL=255
Reply from 192.168.13.75: bytes=32 time<1ms TTL=255
I have never seen anything like that before and I assume it can only mean that some kind of strange interference is being created in order to produce the exact amount of latency over and over again.
According to staff, the suite above them was moved into around the same time - and the army office next door was remodeled at the same time too. Perhaps some piece of equipment was introduced that is causing a weird kind of interference? It would make sense that the workstations suffer more as their wiring runs are longer and go through the walls - vs. a laptop test with a direct and short Ethernet cable directly into the firewall...
Any thoughts? Anyone seen something like this before? I have no idea how to proceed!
thanks!
Wes
|
-Wes |
|
|
don2007
Honorable But Hopeless Addict
1974 Posts
Status: offline |
Posted - 06/29/2012 : 7:31:33 PM
|
I took a chance & searched for "latency spikes" in Google. To my surprise, a bunch of gaming related posts appeared. Did someone decide to run a gaming server there? It's just a guess. I never would have thought of it had I not done that search.
That doesn't rule out the moves that occurred in the building, although I don't see why they would cause that. |
Dyslexic people untie. |
 |
|
|
Pesos
Honorable But Hopeless Addict
USA
3506 Posts
Status: offline |
Posted - 06/29/2012 : 7:35:23 PM
|
Thanks Don... The machines are completely under our control and dont' have anything rogue running.
Also, I disconnected the firewall, carried it across a large room to rule out interference from the equipment in the closet, and hooked up just it and a laptop and still saw latency spikes... |
-Wes |
|
|
|
JamesNT
Moderator
USA
3150 Posts
Status: offline |
Posted - 06/30/2012 : 12:32:36 AM
|
Are there any big antennas in the area spewing RF? I once had a client have poor networking because they were across the street from a bus station that had a big antenna on the roof just spewing Radio Frequency all over the place.
Any network cards going crazy?
JamesNT |
James Summerlin
www.jamessummerlin.com |
|
|
|
wobble_wobble
Honorable But Hopeless Addict
Ireland
4517 Posts
Status: offline |
Posted - 06/30/2012 : 11:03:49 AM
|
A hub or port mirror a switch and Netmon.
You have latency, but what other traffic is there?
What about netstat -s 5 and retransmits?
|
Joe
After everything that has happened during the month of Jan 07, I do believe that pigs fly backwards!
http://whatismyv6.com/ |
|
|
|
Pesos
Honorable But Hopeless Addict
USA
3506 Posts
Status: offline |
Posted - 06/30/2012 : 11:05:47 AM
|
| There is no other traffic... Can repro with just the firewall (no WAN) and a single laptop straight in with Ethernet... We have run tests isolating each host on the network straight to the firewall and still see it - even after switching out the firewall... And the problem gets worse the longer the cable run... |
-Wes |
|
|
|
JamesNT
Moderator
USA
3150 Posts
Status: offline |
Posted - 06/30/2012 : 12:38:49 PM
|
Is the cable running over any electrical lines? I think there still may be an errant RF generator nearby. . .
JamesNT |
James Summerlin
www.jamessummerlin.com |
|
|
|
Pesos
Honorable But Hopeless Addict
USA
3506 Posts
Status: offline |
Posted - 06/30/2012 : 12:40:15 PM
|
That's what I'm saying about some new piece of equipment that may have been introduced either by the new tenants upstairs or the Army office next door during their renovation (all around the same time)...
Going to send a letter to the landlord but was mainly wondering if anyone has seen anything like this so we know what kind of equipment to hopefully track down and get rid of =) |
-Wes |
|
|
|
wobble_wobble
Honorable But Hopeless Addict
Ireland
4517 Posts
Status: offline |
Posted - 07/02/2012 : 12:25:01 PM
|
Lot of suggestions from the lads here, but can you move the switch to a different part of the room and try re-cabling printer and workstation?
Other options;
IOS Bug on the switch.
Microwave oven (possible electro active oven, foundry etc)
Pin/ Nail through cable somewhere.
|
Joe
After everything that has happened during the month of Jan 07, I do believe that pigs fly backwards!
http://whatismyv6.com/ |
|
|
|
Pesos
Honorable But Hopeless Addict
USA
3506 Posts
Status: offline |
Posted - 07/02/2012 : 12:28:19 PM
|
Can't be the switch as the problem persists using just a laptop and the firewall across the room completely isolated...
Can't be a single cable per the same test above...
Seems to me it has to be interference in the air?
Unplugged all microwaves etc in their office - but still suspecting something in one of the newly moved-into spaces or renovated office... very curious about this one =) |
-Wes |
|
|
|
mm_0_mm
Old Timer
USA
562 Posts
Status: offline |
Posted - 07/02/2012 : 12:54:21 PM
|
| I would assume if emf was the cause, an AM radio would pick up the interference? (assuming you can find an AM radio!) |
|
|
|
Isaac
Here To Stay
USA
208 Posts
Status: offline |
Posted - 07/02/2012 : 1:32:04 PM
|
| Call in "Ghost Busters", or take the firewall off site, another building, home, and test it there. I would personally bet its Army's fault. ;) |
|
|
|
Pesos
Honorable But Hopeless Addict
USA
3506 Posts
Status: offline |
Posted - 07/02/2012 : 1:48:16 PM
|
| My money is on the army too lol. We tested the second firewall before sending it down to the office... |
-Wes |
|
|
|
Xenophane
Honorable But Hopeless Addict
Denmark
3070 Posts
Status: online |
Posted - 07/02/2012 : 3:55:52 PM
|
| Have you tried your test using STP cable ? |
Microsoft Powershell MVP
SIG> George Bernard Shaw : The power of accurate observation is commonly called cynicism by those who have not got it. </SIG>
You can read my blog at www.xipher.dk |
|
|
|
Xenophane
Honorable But Hopeless Addict
Denmark
3070 Posts
Status: online |
Posted - 07/02/2012 : 4:02:51 PM
|
| Otherwise you have a great excuse to build a Farraday cage :) |
Microsoft Powershell MVP
SIG> George Bernard Shaw : The power of accurate observation is commonly called cynicism by those who have not got it. </SIG>
You can read my blog at www.xipher.dk |
|
|
|
JSCLMEDAVE
Administrator
USA
6116 Posts
Status: online |
Posted - 07/02/2012 : 4:03:58 PM
|
| Claus getting all Tesla on us. <g> |
Tim-
“This too shall pass" |
|
|
|
wobble_wobble
Honorable But Hopeless Addict
Ireland
4517 Posts
Status: offline |
Posted - 07/02/2012 : 5:04:39 PM
|
Well the switch and firewall should have a ground case around the HF sensitive bits, if not the full case. Whereas the twisted pair will probably be one long antenna.
|
Joe
After everything that has happened during the month of Jan 07, I do believe that pigs fly backwards!
http://whatismyv6.com/ |
Edited by - wobble_wobble on 07/02/2012 5:05:03 PM |
|
|
|
lacrosseboy
Old Timer
550 Posts
Status: offline |
Posted - 07/03/2012 : 07:36:11 AM
|
| What kind of cabling is being used? |
Thomas Deimel
Keeper of the Holy Potato |
|
|
|
aval
Honorable But Hopeless Addict
USA
3276 Posts
Status: offline |
Posted - 07/07/2012 : 10:03:15 PM
|
If you can come up with one, yes, trying a STP might be interesting.
Or try later at night or on the weekend when the cause of the problem is possibly turned off? |
|
|
|
Pesos
Honorable But Hopeless Addict
USA
3506 Posts
Status: offline |
Posted - 07/09/2012 : 12:18:49 PM
|
| When I went in person it was 6pm on a weekday... Going to reach out to the landlord now and see what he may have installed upstairs... |
-Wes |
|
|
| |
Topic |
|
strangest problem i've seen in 15 years
