So it was obviously done for a hidden, longer term purpose, rather than to take the site down, and that worried me too. Could it be rescued?
This is a long post, detailing the steps I took to trace the problem, minimise the damage and rescue the website, so the steps I took are linked in this step-by-step list:
How I noticed something was wrong
Because I usually use Firefox, which hadn’t had any problems with the site, I only realised my site was hacked when I tested a new page in Internet Explorer 8. There was an error message I hadn’t seen before.
The message said, ‘This website wants to run the following add-on: ‘Remote Data Services Data Control’ from ‘Microsoft Corporation’. If you trust the website and the add-on and want to allow it to run, click here’.
Now this website and I have been through stormy times together. If I’ve ever made a lot of sales calls, spent a whole day cold calling at trade shows, or I’m at home with flu and a hyperactive toddler, that’s the day the site will crash. True to form, I had just applied to some new affiliate programs and was about to stop work for the day and start cooking for my son’s 4th birthday party. So did I trust this website? Erm…
At first, I thought maybe there was a script I hadn’t noticed in some content I had just added from Wikipedia, so I checked the source code, but I didn’t see anything unusual there. I browsed to other pages on the same website, and then back again, and didn’t see it again. I wondered if it was something left over from my MSN Internet Explorer home page, so I went back to that and it wasn’t there. Then I went back to the same page, and there it was again.
What is Microsoft ‘Remote Data Services Data Control’?
I tried searching for ‘Remote Data Services Data Control’, assuming it was some kind of Microsoft script, and found a post on a Microsoft Security newsgroup leading to this ‘Can You Spot the Fake?’ post on Hosts News, warning,
‘any time you see that warning “Remote Data Services Data Control” watch out! … this is NOT from Microsoft! This is the generic warning IE7 throws up when an exploit is trying to enter the system.’
Starting to feel alarmed, I following a trackback from Hosts News to the post that saved my day (though ruining my night first ) : the Little Big Tomatoes post The one with the evil jscript on my blog.
HTTP Debugging with Fiddler
The hacked site there appeared to be WordPress based, but the errors looked similar. The post showed a useful free web debugging tool called Fiddler, which logs all HTTP(S) traffic between your computer and the Internet. Fiddler allows you to inspect all HTTP(S) traffic, set breakpoints, and “fiddle” with incoming or outgoing data. There’s an instruction video for Fiddler here.
So I downloaded Fiddler and ran it. I’d never run Fiddler before, but when I loaded the web page again in Internet Explorer, Fiddler picked it up automatically. So I looked down the list of http requests, and in the middle of my normal HTTP requests (site name blocked out) I found these (don’t go to any URLs listed here!):
‘Yourgoogleanalytics’ and ‘Statscounter’???
Looking down the list, I saw ‘statscounter’ and did a double take: I do use Statcounter for website statistics, but here it’s misspelt, and I hadn’t used Google Analytics at all, so that was another red flag. Some Google Adsense HTTP requests appeared around the same time, but it just looked dodgy. Was this really something Google Adsense did?
The next requests, for ‘dos.ms’ were completely unfamiliar, and called PHP scripts called ‘redir.php’ and ‘in.php’, whose names alone sounded worrying. I wondered: shouldn’t hackers have called them something less sinister, perhaps kittens.php and flowers.php? Or had they just stopped pretending? Most of all, I wondered what was going on with my website.
I should say at this point that the website often needs work and doesn’t get much traffic (a good thing, for once). In the words of Courtney Tuttle, it’s a time sucking money pit. But I leave my money pits on my own terms, thanks.
So there weren’t any visitors right then and I thought I had a bit more time to figure out what was going on.
“Have you looked at the code in menus/milonic_src.js ?
What do you suppose is the long line starting with this: document.write(unescape(‘%3C%69%66%72%61%6D%65%20%73%72%63%3D ?
Andrew Martins security blog lists the yourgoogleanalytics.us domain as being used for ‘Money Mule Recruiting’, and gives an example of Money Mule Recruiting on a related domain:
“During the trial period (1 month), you will be paid 2000 USD per month
while working on average 3 hours per day, Monday-Friday, plus 5
commission from every transactions or task received and processed. The
salary will be sent in the form of wire transfer directly to your
account. After the trial period your base pay salary will go up to
3,500USD per month, plus 5 commission.”
I couldn’t see my visitors falling for that, so I hoped if it was hacked it was for money mule recruitment rather than the various trojans and downloaders that were mentioned on other sites as possibilities.
So I downloaded those three files, and checked them one by one in Notepad, and when I got to the top of moscom.jquery.js, there it was:
document.write(unescape('%3C%69%66%72%61%6D%65%20%73 ...and so on.
I didn’t know if the file had always been like that, but that line of code appeared before the header section with the package and file names and looked very suspicious, so that was enough for me: it was clear the website had been hacked.
I felt awful, thinking of people visiting my website, and being at risk from trojans and viruses.
Preventing further infection
I didn’t want to be infecting my visitors with malicious downloads, so I took the site offline via the Global Configuration settings.
I set off a backup of the database and files, in case uninstalling or repairing anything might trigger off some kind of malicious code. Then I noticed that the backup program itself was trying to access the malicious files, so I stopped it and downloaded the files and database separately.
Telling the Web Hosting Provider
Then I emailed my web hosting provider, told them the situation, and apologised for the inconvenience. I told them that I had identified the infected files and taken the website offline to visitors, and that I would be uninstalling the infected components and upgrading everything that needed upgrading.
I was pretty nervous about this, having heard of other hosts deactivating hacked accounts, and I stared at the backups, willing them to go faster…
That didn’t work. My web hosts were really helpful though, and sent me this reply:
“It appears that your site has been subject to an attack via a method known as script injection. Typically, this works by forcing a site to execute code when it was expecting to process another input, fake .txt files are often used for this purpose.
Because script injection attacks the site code itself, it is able to completely avoid webserver security. Unfortunately, some content management systems (especially older versions of Joomla) are extremely susceptible to this form of attack.
Here’s the best way to get your site back up and running:
- Begin by clearing out your public_html directory, ensuring no anomalous files or hidden files are left, this way you know that any backdoors the hacker might have left in are completely eradicated.
- Change any passwords relating to the site, along with the ftp password.
- At this point we can turn scripting back on for you – as the hacker’s entryway is now gone.
- Restore the site from your last known good backup.
If you feel confident removing this infection by hand, please inform us when the site is completely clean and we will turn it back on for you. However, if the infection does recur, we would ask that you completely clear the public_html directory before we reactivate the site again.
I would also suggest contacting the author of the script as it appears there maybe some security issues with the code. “
Script injection? I’d heard of cross site scripting, and SQL injection (see the ‘Bobby Tables’ cartoon about this), but not script injection. A quick look at the Wikipedia page told me that script injection (or code injection) is the more general name for the type of website hacking that adds extra code to a computer program, and can be accomplished using cross site scripting or SQL injection, among other methods.
Checking my Desktop PC
I’d like to say the next thing I did was to run the virus and spyware checker on my local PC. But in fact, I changed all my hosting account’s FTP and database passwords, and then thought, “What if I have keylogger spyware on my local PC…?” Then I ran the virus and spyware checker on my desktop PC and changed my passwords all over again.
When your website has been hacked, your desktop PC’s security is easy to overlook (apparently!), but can be crucial. This post from Brian Teeman’s blog discusses why it’s important to consider keylogging trojans and viruses when dealing with hacked Joomla sites.
Checking on the Joomla security forums (the Joomla 1.0.x security forum and the Joomla 1.5.x security forum), I found advice to check the desktop PC with several different antivirus and spyware tools, as each of them might miss something different.
I was able to run AVG antivirus, PC Tools Anti-virus, and Lavasoft’s Ad-Aware. I also tried Kaspersky, but it wouldn’t install because it found remnants of an old AVG version, which I couldn’t find and couldn’t uninstall.
Some more recommended security tools can be found listed here:
Free virus checkers:
- AVG free anti virus
- House Call
- Bit Defender
- PC Tools free antivirus for Windows XP and Vista
I ran the quick check first, then the thorough ones, but my desktop PC was apparently ok.
I also discovered Secunia’s free check for vulnerable software on desktop computers.
Securing the Web Hosting Account
So I decided that in the long run it might be quicker to take the drastic route. I uninstalled the infected component first, then deleted all the files. While the files were deleting, I used phpMyAdmin to export a text file of the database, and I searched it for anything I could think of that had been out of place so far.
Finding the Vulnerabilities
Obviously, I wanted to avoid this happening again, so it would help to know how the hackers got in.
I’d added extra code to my php.ini and .htaccess files that was recommended on the Joomla Security forums for blocking common exploits, so I’d thought my site would be safer than most, but apparently part of it wasn’t. So where was the problem, and how could I track it down?
I emailed the author anyway, as advised by my web host, and eventually received a reply telling me not to store my files with permissions set to world writable. Ok, that’s valid advice, except that wasn’t what happened and I don’t know why he thought it was.
Another place where I have often seen hacking attempts is the Friendly URLs list of my Search Engine Friendly (SEF) URLs program. When hacking attempts arrive via the browser URL, they are not known URLs, so the SEF program makes FURLs for them. In the past I’ve often checked through these to see which components are being targeted, but this time I’d recently cleared them out and deleted the invalid FURLS. So I didn’t find anything there.
The next thing I did was to download the access and error logs from my web hosting server. All I found from the error logs was that one of my graphics files was missing. The access logs were long text files, but only went back to about 5 days after I installed the files that became infected.
I started looking back through them. I’ve seen SQL injection attempts in my access logs in the past, and there was one almost immediately in the most recent log file. But when I looked up references to SQL injection in the component that was targeted, the attack I saw was known and the vulnerability had been patched many versions previously. I contacted the author and he confirmed this (although not long afterwards they did produce a new security release). I also didn’t see any administrator activity or unusual file accesses in the log entries around it.
I found lots of hacking attempts by the bot libwww-perl, which is often used for automated hacking attempts. Many were targeting Community Builder, which I hadn’t seen targeted before, and many were bringing up my ‘page not found’ page.
I continued trawling through the logs, finding nothing that looked successful. Occasionally, the infected files appeared when I didn’t think they should have done. Eventually, the log files ended, leaving me with that half a week gap.
I changed tack and itemised the components in the website to see what needed upgrading.
Joomla itself was up to date, but Community Builder was known to have an exploit, and I had put off upgrading it because the new version required PHP 5, and I had only had PHP 4.4.7 on my server. In the meantime, however, my web hosts had installed a capability to switch hosting accounts to the upgraded PHP version.
Improving Security in Future
So the first change I made was to upgrade the PHP version to 5. Then I restored the files from a backup taken before installing the infected component, working on the assumption (or hope!) that the site was only hacked once.
I had already made the .htaccess and php.ini changes recommended in the Joomla Security Forum, but I added some new lines to my .htaccess file to block libwww, as recommended in these useful and detailed blog posts: ‘Hackers, Botnets and libwww-PERL’ by WebGeek and ‘Block LIBWWW-PERL and web addresses to protect your site from botnets’ by incrediBILL.
Unfortunately, there are some legitimate uses of Libwww-perl that would also be blocked, but so far the only disadvantage I have found to blocking it is that if I want to run the W3C link validator, that uses Libwww-perl so I have to temporarily allow it again.
I also added lines to block the domains used in the hacking attempts. For more .htaccess protection, the Webmasterworld forum has a useful discussion on ways to block bad bots.
I had a couple of small PHP errors to fix after upgrading to PHP 5, and then I tested the site thoroughly using Internet Explorer and Fiddler, and all seemed ok.
Next I upgraded Community Builder, followed by checking and upgrading any other components that needed it.
My users’ passwords are all stored in encrypted form, and the site does not collect personal information about them, so there wouldn’t be a security risk from that. I checked the user list to see who had logged in recently, and checked their data. Then I changed the passwords to prevent fraudulent logins in future. If they try to log in, the ‘forgot password’ option can be used to reset passwords. It’s not great, but no damage has been done there.
The users’ passwords did give me some scary moments: this is another example of why it’s so important to use different passwords for anything important you do online: if the users’ passwords had been on a website where they weren’t encrypted, and they used the same password for their email, that would be extremely risky. And if I’d used the same passwords for logging in to my website as I did for online banking, email or anything involving credit cards, for example, I could be in deep trouble now. If you’re not convinced, check this entertaining but scary “One Man’s Blog” post on “How I’d hack your weak passwords”.
Looking back, I may be in good company too: I can think of several online services I’ve joined where the password has mysteriously stopped working. None of them told me why, and some of them did have more information of mine than I’d want to let go.
Why was my website a target for hackers?
I can think of a couple of reasons why my site might have been targeted:
- When I first launched it, I posted links in various ‘Site Showcase’ type threads in the forums for the Joomla components I’d used. At the time, I thought this was great, because it brought me visitors from all over the world, as well as a truly improbable sounding Alexa ranking. But then, when any of those components develop a vulnerability, my site is an obvious target. Thankfully, component owners tend to develop patches or eventually take their sites offline, and my links with it. But I wouldn’t post links there again unless it was truly relevant to the website’s focus.
- Social networking sites are increasingly becoming targets for hackers (see Facebook and Twitter becoming Top Targets for Hackers at Brick House Security), which could lead to Community Builder being targeted more than in the past, with hackers assuming that any site using Community Builder would be a social networking / membership site. Plus, the older version of Community Builder had a vulnerability posted. I haven’t been able to find out what it is anywhere, but hackers could have tried attacks in many variations
- What’s most likely, considering that my website isn’t a high value target, is that hackers were using bots such as libwww-perl to run scripted attacks with many variations, and eventually one of them worked. Woohoo, lucky me! Should have kept all my extensions up to date
Online Tests for Malware in your Website
There is an online service called Unmask Parasites which tests web pages for malware. To check your web pages you can either bookmark their service or place their Security Check shortcut links on every page you want to be able to check.
Every time you click the Security Check link the Unmask Parasites service will automatically check the referring web page so you don’t have to type any URLs.
The shortcut links should work for any public web pages, however, I have found since then that when I checked a hacked oscommerce website, the security report service appeared to crash with error messages, so if you see errors they should also be treated as a warning that your website may have been hacked.
Google Malware / Security Warning
If your website has been unlucky enough to be flagged by Google as hosting malware or in some other way risky, this post explains how to get that warning removed from your website’s Google listings: How to remove ‘This site may harm your computer’ from Google search results.
There’s some more information here about Google’s safe browsing diagnostic pages.
General Website Security Links
- Description of a recent attack that infected 40,000 websites
- Sign up for Websense Attack Alerts
- Websense Security Labs Blog
- SANS: Top Cyber Security Risks
- Stop Badware: Tips for Cleaning and Securing Your Website
Joomla Security Links
- Joomla 1.0.x security forum
- Joomla 1.5.x security forum
- Joomla! Security Checklist
- Joomla Advisory: dealing with hacked websites and hacking attempts
- List of Vulnerable Joomla! Extensions
If you’ve also had your Joomla website hacked, you might find these security links useful, and I wish you luck with it! In the meantime, with the files backing up, then deleting, then restoring, and the anti-virus running, the computer ran so slowly that I had lots of time that night to get things ready for my little boy’s birthday party: I may have felt like a zombie but at least the scumbag hackers didn’t ruin his special day. Comments, advice, suggestions etc welcome – if there’s one thing I’ve learned above all from this it’s that I’d rather improve my knowledge of website security at more convenient times