How to Detect SYN Flood Attacks with Capsa Network Protocol Analyzer & Create Automated Notification Alerts

This article explains how to detect a SYN Flood Attack using an advanced protocol analyser like Colasoft Capsa. We’ll show you how to identify and inspect abnormal traffic spikes, drill into captured packets and identify evidence of flood attacks. Furthermore we’ll configure Colasoft Capsa to automatically detect SYN Flood Attacks and send automated alert notifications .

Denial-of-Service (DoS) attacks are one of the most persistent attacks network admins face due to the ease they can be carried out. With a couple of commands, an attacker can create a DoS attack capable of disrupting critical network services within an organization.

There are a number of ways to execute a DoS attack, including ARP poisoning, Ping Flood, UDP Flood, Smurf attack and more but we’re going to focus on one of the most common: the SYN flood (half-open attack). In this method, an attacker exploits the TCP handshake process.

In a regular three-way TCP handshake, the user sends a SYN packet to a server, which replies with a SYN-ACK packet. The user replies with a final ACK packet, completing the process and establishing the TCP connection be established after which data can be transferred between the two hosts:

However, if a server receives a high volume of SYN packets and no replies (ACK) to its SYN-ACK packets, the TCP connections remain half-open, assuming natural network congestion:

By flooding a target with SYN packets and not responding (ACK), an attacker can easily overwhelm the target’s available ports. In this state, the target struggles to handle traffic which in turn will increase CPU usage and memory consumption ultimately leading to the exhaustion of its resources (CPU and RAM). At this point the server will no longer be able to serve legitimate clients requests and ultimately lead to a Denial-of-Service.

Detecting & Investigating Unusual Network Traffic

Fortunately, there are a number of software that can detect SYN Flood attacks. Wireshark is a strong, free solution, but paid versions of Colasoft Capsa make it far easier and quicker to detect and locate network attacks. Graph-oriented displays and clever features make it simple to diagnose issues.

As such, the first point of call for detecting a DoS attack is the dashboard. The overview of your network will make spikes in traffic quickly noticeable. You should be able to notice an uptick in the global utilization graph, as well as the total traffic by bytes:

Click to enlarge

However, spikes in network utilization can happen for many reasons, so it’s worth drilling down into the details. Capsa makes this very easy via its Summary tab, which will show packet size distribution, TCP conversation count, and TCP SYN/SYN-ACK sent.

In this case, there’s an abnormal number of packets in the 128-255 range, but admins should look out for strange distributions under any heading as attackers can specify a packet size to suit their needs. However, a more telling picture emerges when looking at TCP SYN Sent, which is almost 4000 times that of SYN-ACK:

Click to enlarge

Clearly, there’s something wrong here, but it’s important to find the target of the SYN packets and their origin.

There a couple of ways to do this, but the TCP Conversation tab is easiest. If we sort by TCP, we can see that the same 198-byte packet is being sent to our victim PC on port 80:

Click to enlarge

After selecting one of these entries and decoding the packets, you may see the results below. There have been repeated SYN packets and the handshake isn’t performed normally in many cases:

Click to enlarge

The attack becomes most clear when viewing IP Conversation in Capsa’s Matrix view, which reveals thousands of packets sent to our victim PC from random IP addresses. This is due to the use of IP spoofing to conceal their origin. If the attacker isn’t using IP spoofing, Capsa’s Resolve address will be able to resolve the IP address and provide us with its name. If they are, finding the source is likely far more trouble than it’s worth:

Click to enlarge

At this point, we can be certain that an SYN flood attack is taking place, but catching such attacks quickly really pays. Admins can use Capsa’s Alarm Explorer to get an instant notification when unusual traffic is detected:

A simple counter triggers a sound and email when a certain number of SYN packets per second are detected. We set the counter to 100 to test the functionality and Capsa immediately sent us an alert once we reached the configured threshold:

Capsa also lets users set up their own pane in the dashboard, where you can display useful graphs like SYN sent vs SYN-ACK, packet distribution, and global utilization. This should make it possible to check for a SYN flood at a glance when experiencing network slowdowns:

Alternatively, Capsa’s Enterprise Edition lets admins start a security analysis profile, which contains a dedicated DoS attack tab. This will automatically list victims of an SYN flood attack and display useful statistics like TCP SYN received and sent. It also allows for quick access to TCP conversation details, letting admins decode quickly and verify attacks:

Click to enlarge

Together, these techniques should be more than enough to catch SYN floods as they start and prevent lengthy downtime.

Summary

This article explained how SYN Flood Attacks work and showed how to detect SYN Flood attacks using Colasoft Capsa. We saw different ways to identify abnormal traffic spikes within the network, how to drill into packets and find evidence of possible attacks. Finally we showed how Capsa can be configured to automatically detect SYN Flood Attacks and create alert notifications.