ARP Storms & Broadcast Issues in Virtual Networks: How to Troubleshoot Fast

[johny899]

Do you ever observe your virtual network slow down and realize there is no apparent reason for the slowdown? It can be frustrating! I have experienced this one time myself, and after the fact, I found out it was caused by an ARP storm. ARP storms are minor issues that can quickly turn into major issues.

What Causes ARP Storms​

An ARP storm occurs when a device repeatedly asks the same question: “Who has this IP address?”

Asking that question one or two times would be considered normal. However, when a device asks that question hundreds of times—that would be an ARP storm.

It’s like a group chat where nobody stops saying “Hello???” over and over again.

An ARP storm can occur because of one of the following:

• A misconfigured VM sending too many ARP requests
• A malfunctioning network card sending packets
• A loop in a virtual switch forwarding the same packets constantly
• A piece of malware generating fake ARP traffic

When an ARP storm occurs, ARP traffic can saturate your network, CPU utilization can become high, and your VMs can slow down.

The Impact of Broadcast Problems within Virtual Networks​

Virtual networks process broadcast traffic using software, making a small problem more severe. Remember, you might notice your VMs are slow to respond at that time? One virtual machine may have sent too much broadcast traffic and everything was congested; therefore, the VMs were slow and possibly disconnected.

For example, there may be

• Slow response
• VM disconnecting
• High CPU utilization on the hypervisor
• Network spammed with broadcast packets

I'd like to mention an instance where I had a virtual machine freeze every few minutes because another virtual machine was constantly sending out ARP packets.

Real Causes of ARP Storms​

Let's discuss the real causes of this problem.

• Virtual switch is configured wrong
• Two devices are configured with the same IP
• Security features are turned off
• Loop in the network
• Too many devices on one large broadcast domain

Any one of them can introduce a storm.

The Best Fixes for ARP Storms​

Here are the options that work.

• Activate ARP suppression in your hypervisor
• Use VLANs to split the network
• Enable storm control in the virtual switch
• Enable anti-spoofing checking (such as ARP Inspection)
• Monitor across your network with something like Zabbix or Prometheus

A good ARP suppression setting should fix many issues quickly.

Final Thoughts​

ARP storms are a nuisance, but you can fix them once you know what causes them. With the right settings, the virtual brings back the speed and stability you purchased.

Next time you find your VMs slowing down for no reason you can check if an ARP storm caused the issue, and you may be able to quickly fix it.