DDoS mitigation without cloud protection

One morning at around 6:20 am, I am woken to the notification sound of my developer mailbox. It was an email notification from the webserver’s event monitor about high CPU utilization. I immediately go to my terminal and ssh into the affected box.

The first things I check are the system resources; CPU and memory utilization. I use htop and it’s showing me abnormally high CPU usage coming from the PHP-fpm processes.

Next, I quickly check the incoming and outgoing connections with netstat. There are quite a lot for this early hour. It’s looking like an attack.

Whilst checking the access logs with tail -f it shows me a lot of requests coming through. The visual pattern of the log indicates to me that this is not genuine traffic. There are too many uniform lines scrolling past. Genuine traffic usually has a different visual shape due to users making additional requests for site resources (CSS, JavaScript and images).

Because of a previous risk assessment, I already had a good idea of the attack vectors and which functions cause the most CPU usage – the language and currency switching features and the search features (including the multifacet category navigation). I had already created rate limiters (ngx_http_limit_req_module) for this area, preventing abuse from most attacks, but this attacker had worked out the timing and configuration – send 1 request, do not follow the redirect, and then change IP address.

At this point, the attackers were mostly targeting the language switch function. This feature makes a backend call, gathers the translated URL information and returns a 302 Temporary Redirect.

This attack happened over 2 days. On the first day, the traffic all originated from a single country. This was easy to mitigate as the client whose web server this belongs to does not provide services to this country. I grabbed a list of IP addresses that are specific to the origin country from https://www.ip2location.com/free/visitor-blocker, created a new rule with ipset and then applied the rule to iptables.

Day 2

Day 2 proved to be more of a challenge. This time the attack was truly distributed, with origins spanning across the entire globe. In addition to this, they started attacking another function, the category layered navigation. There was one thing that set these requests apart from other requests, using the same HTTP User-Agent.

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36

In addition to this was the reuse of request tokens containing the following unique identifiers, Y2FubmFiaX and gvcmVzdWx0.

Regarding the category layered navigation, most of these requests had been cached by Varnish and the others fell out of the range provided by the search. The cached requests were ignored but the ones that fell out of the range were filtered separately.

This information enabled me to create a regular expression location match on the edge server, filtering and logging the requests, and preventing them from reaching the backend.

At this point, I decided that the best way to minimize friction for genuine users being false-positive matched was to return a 302 Temporary Redirect to the appropriate location. In the case of the language switcher, I reduced the functionality of the feature instead of disabling it, returning the user to the homepage of the specified language. With the layered navigation, I returned them to the category. A rough example of the configuration is below.

location ~ "Y2FubmFiaX|gvcmVzdWx0" {
	if ($http_user_agent ~ "Chrome/68\.0\.3440\.75"){
	    return 302 $url;
	    access_log /var/log/nginx/ddos.access.log full;
	}
}

location ~ "(filter_[a-z]+=[0-9]+&){5,}" {
	if ($http_user_agent ~ "Chrome/68\.0\.3440\.75"){
	    return 302 $url;
	    access_log /var/log/nginx/ddos.access.log full;
	}
}

(example nginx config showing regex request matching)

They occasionally switched tokens, but that was just the case of finding those requests making it through and updating the filters.

Tactics, Techniques and Procedures

Now that I had the attack under control, it was time to make a decision on how to handle this. I could simply block the responsible IP addresses, but experience says this would just agitate the attacker, giving them a reason to come back harder.

Looking at how the requests were coming in I saw that they were iterating through a loop of IP addresses in the same order. I started to block small chunks, instead of blocking them all. To visualize this, imagine that each hyphen below is an IP Address, and the red coloured ones are being blocked. I intended to create the illusion of an intermittent website failure. This is part of the attackers intention here, to deny service. So I tried to give them this.

--------------------------------------------------------------------------------

This illusion seemed to work. The attacker unleashed an additional few thousand IP addresses, another botnet, with the hopes of finally bringing the site down. These additional requests are already being filtered but it enabled me to gather more intelligence before finally using the banhammer.

David Bianco’s Pyramid of Pain

Utilising my knowledge of the Tactics, Techniques and Procedures of the attack and understanding the goal of the attacker, how they reorganise themselves during the attack, their tools and their next move, I was able to mitigate this attack.

Analysing the log files after the event, I found wget requests before, during and after the attack. Additionally, requests from Protonmail fetching the favicon around the same time as the wget requests make me think that someone, possibly the attacker, was using Protonmail to send the website URL. Maybe an update to their client on the status of the attack?

Important Notes

Sometimes a DDoS is just a cover for a deeper attack. If you are experiencing a DDoS, it is wise to look for signs of a different attack. Searching the logs for signs of SQL injection, cross-site scripting, directory traversal, and remote code execution using search terms such as:

• UNION SELECT
• INSERT INTO
• SLEEP
• OR 1=1
• INFORMATION_SCHEMA.PLUGINS
• DBMS_PIPE.RECEIVE_MESSAGE
• <script>
• alert(1)
• alert(‘xss’)
• /etc/passwd
• \\/..\\/..\\/..
• nslookup
• ping
• perl
• whoami

If you find these, then your priorities need to shift and focus on this instead of the DDoS. The DDoS in this case is there to create log noise, misdirect, and create weaknesses with the application while it tries to process the additional requests. If you are not using a WAF (Web Application Firewall), install one.