What to do when your Ubuntu server is used by botnet to spread email spam

After the Ynterview.com update I decided to perform final user test. No validation email was (properly) delivered after the new user registration. Gmail marked it as a SPAM and moved to the SPAM folder. I saw that something is wrong.

Blacklists check

First thing is such case it to check MxToolBox blacklist tool. Server IP address was included in the industry most serious email spam blacklists: BARRACUDA, CBL, ivmSIP, SORBS SPAM, Spamhaus ZEN.

Server Check

I let it test for open-relay by SMTP Diagnostics and Mailradar. Nothing. Hmm, so… probably the most helpful command when it comes to problem with emails is:

postqueue -p

It shows how many emails generated by the server weren’t delivered and were placed into the “queue” (system tries to deliver them again and again). It returned +30k entries – wow, what a number!

Command

postsuper -d ALL

deletes them all. I used it in order to have a “clean shield”. Number of new deferred emails rose quickly. I disabled Posffix:

service postfix stop

but it didn’t help. It meant that spam emails were generated by some PHP script.

My tip was that “sophisticated” spam bot broke captcha in some registration or contact form.

When you do use postqueue -p command you can see number-letter string like CA4784615B assigned to every message. It’s internal ID of the broken email. Use that string with following command

postcat -q CA4784615B 37C

and you can read that message and see what’s inside. This is what I have found. Most important part in this case was originating script located in the message header. It tells you which PHP script generated it:

X-PHP-Originating-Script: 1000:dirs.php(232)

Next step was to find all the scripts with the name dirs.php:

find /var/www -name "dirs.php"

There was only one. Bingo.

Penetration Method

Backup the script, delete it and think about penetration method: how the file was placed on my Ubuntu server?

It was part of the website directory I own since November 2014. My conclusion was, that malicious script was uploaded together with the rest of the files from previous owner. Good, my passwords were not compromised this time.

Delist Requests

Next step: delist requests. I googled out blacklists one by one and asked them for the removal. CBL gave me also one additional message:

IP Address xxx.xxx.xxx.xxx is listed in the CBL. It appears to be infected with a spam sending trojan, proxy or some other form of botnet.
It was last detected at 2015-06-03 06:00 GMT (+/- 30 minutes), approximately 1 days, 9 hours ago.
This IP is infected (or NATting for a computer that is infected) with the kelihos spambot. In other words, it’s participating in a botnet.

It’s 2 days after the cure and everything looks normal – MxToolBox shows again only the green color 🙂