There are a couple of things that could be causing issues when you LAN is attached. One being spyware - if you decode the packets you can find out what IP address is still broadcasting traffic. There could also be an error in someone's IP address, gateway, or subnet mask making them jump the router. I assume you are running a firewall, so it may be prudent to upgrade to the latest and greatest version. Your ISP should be able to provide you with the address of the communicating device if you don't have a sniffer. They should also be able to tell you what type of traffic (by IP port, anyway) so that you can more easily narrow this down.
If you don't have a sniffer and they can't or won't help you, try downloading one of the demo versions or a free one just to get you started. More than likely, you have a device or two that is being a bandwidth hog or perhaps your server is trying to communicate with your routers because it is routing.
As you don't say you are having any problems accessing websites, I am assuming that DNS is not a problem, but rather a single device or maybe a couple that are driving your latency up. If you try these things and are still having problems, feel free to send me your results and we can move on to another step.
Dig Deeper on SDN and other network strategies
Related Q&A from Carrie Higbie Goetz
Proper SBC configuration requires partnering with providers and security teams to examine circuits and potential traffic. Learn the best practices ... Continue Reading
Organizations need SIP services to connect and disconnect call sessions. Now some SIP services are offered through APIs, which expand the use cases ... Continue Reading
A few factors can indicate it's time to replace a business phone system. Learn when it may be time to switch and how to decide on a course of action. Continue Reading