Blog

The call: server running slowly, pages taking tens of seconds to load.

The initial analysis: after an age trying to log in, a loadavg of 62, a HTTP access log scrolling faster than the eye can follow. It feels like some sort of DDoS.

One site seems to be the cause of the problems. A site that went live the previous day.

What appeared to be a few thousand Windows XP computers were all making POST requests to the root of one website. Dumping the traffic showed nothing coherent: these appeared to be random data. About 6000 ISP networks were involved, with five figures of IP addresses making the POST requests. All had the same user agent string. None appeared to be legitimate.

Quick fix:

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /

# BEGIN Dirty Fix
RewriteCond %{HTTP_USER_AGENT} "MSIE 6.0; Windows NT 5.1"
RewriteCond %{THE_REQUEST} "POST / HTTP"
RewriteRule .* - [F]
# END Dirty Fix

This made any POST request to / with a user agent string of IE 6 on XP be served a 500 failure page. Those three lines immediately started to bring the load on the server down from 62 to below 1.

We also noticed a large amount of TCP connections being opened to port 25 of the same server, with some junk data sent, before immediately disconnecting. This wasn’t causing much load, as exim just ignored the traffic as not being part of a normal SMTP session. But it was causing data to be stored up in the log files.

But what was causing this? Why was so much junk traffic being sent? But at the same time, it wasn’t so much junk traffic as to be a realistic DDoS attack.

Faelix eventually tracked this down to the PushDo virus. PushDo connects to numerous websites and sends junk to them in a POST as cover traffic to hide its real command and control data. The customer’s domain appeared to be one of the domains used by one of the variants of PushDo, which also generates outbound cover traffic to port 25 of the same server. Presumably this is to make the C&C traffic like a needle in a haystack, thwarting security teams’ attempts to capture it. We toyed with the idea of serving EICAR back to the “attacking” IP address, hoping that this might trigger antivirus signatures within the compromised hosts or their networks, but felt that this might impact the reputation of our customer were any infected end users to complain, “why is your site serving a virus to me?” (when in reality they are already infected with actual malware).

To further lower the load on the customer’s server, Faelix configured front-end Varnish cache servers to absorb the PushDo HTTP requests (and ignore the SMTP connections entirely), serving back an error page with a message:

403 Your computer is infected with the PushDo virus. https://labs.opendns.com/2014/07/31/pushdo-crypter/