An Open Letter to Rackspace Cloud Hosting

I just received an automated email from Rackspace that made my brain melt. It’s no secret that a lot of websites have been hacked lately.

One thing they seem to have in common is that they’re all running Wordpress, and a lot of them are hosted at the Rackspace Cloud.

Dear Alison,

Since we host hundreds of thousands of applications at The Rackspace Cloud, we have a unique vantage point from which we can identify security trends and patterns. Lately, the industry has seen an elevated level of attempts to take advantage of code vulnerabilities in the software powering websites. Hackers are a common and persistent threat to any website, but there are steps you can take to protect yourself and to make your websites and applications harder to exploit.

Please read over the important tips below. We have dedicated security experts who work to protect our infrastructure, but since we can’t fix or upgrade code on behalf of our customers, it’s important for you to know and regularly implement security best practices in the code you run. We need your help and involvement to ensure your own sites are as protected as possible. If you have any questions about security, please reply to this email and we’ll be happy to help.

HERE’S WHAT OUR SECURITY TEAM HAS RECENTLY IDENTIFIED:

1. The current data that we’ve collected points to application-based vulnerabilities being exploited. Hackers commonly scan sites for insecure applications, plugins, or other pieces of code and then work to take advantage of the software exploits they find.

2. Applications using the popular blogging software WordPress appear to be mostly targeted, but WordPress isn’t the sole target of the malicious groups / persons.

3. Your site does not have to be high-profile to be targeted. Hackers often scan random sites for signs of software known to be vulnerable (older versions of popular software with publicly known security holes, for example).

HERE’S WHAT YOU SHOULD DO NOW TO PROTECT YOUR SITES:

1. This is probably the most important tip: For any application you use, be sure to maintain the most current stable version. Often, an application might be updated to a new minor version solely to address a security hole that’s been discovered. Be sure to subscribe to any news lists and feeds available for your applications to make sure you are aware of updated versions as soon as they are released.

2. Many applications, like WordPress, have optional plugins developed by the community. Since these add-ons are often not as well vetted, it’s extremely important to carefully evaluate and manage third party application plugins, themes, or other functionality that is introduced to a running web application. Most hackers are exploiting these plug-ins

3. It’s imperative to choose strong passwords. Randomly generated strings of letters, numbers, and symbols are best. Avoid words and phrases in your passwords. The unfortunate reality: passwords that are easy to remember are also easy to guess. (Ex: Replacing o by the number 0 is not a recommended tactic.)

4. Change your passwords on a regular basis and change them immediately when you have any hunch that your site may have been attacked.

5. Be as restrictive as possible with users and file permissions. Remove write permissions from files that aren’t likely to change frequently. Some programs have install files that should be deleted after installation. If you’ve installed something or written code for testing purposes or experimentation, it’s best to remove it afterwards. Only keep the files and code on your account that are active and necessary.

As a site owner, you need to take an active role in guaranteeing security of your code and applications. The good news is that our support staff is happy to help you with any questions or concerns you may have. Recovering from a hack or exploit is extremely time-consuming and frustrating. The preventive steps outlined above can make a world of difference in keeping your sites secure.

Finally, if you suspect your site has already been compromised, you should take immediate action. This knowledge base article can help you through the right steps:

http://cloudsites.rackspacecloud.com/index.php/Recovering_from_and_Dealing_with_a_Site_Compromise

Sincerely,
The Rackspace Cloud Security Team

I want to preface this by saying there are a LOT of people that work at Rackspace that are absolutely awesome. The guys I know from Twitter are amazing, and helpful and care about customer happiness more than I can even say. None of this is their fault. This is NOT about them. This is about something fundamentally wrong with priorities at Rackspace, in my opinion.

I replied:

Too little, too late. I could have (and did) tell you all of this already.

And unfortunately, running the most recent version of Wordpress doesn’t help. This week, I have personally had to repair 11 Wordpress websites hosted on the RS Cloud that were hacked, all were running 2.9.1 and had very few plugins in common. The plugins they do have in common, like WP-Supercache, are plugins Rackspace suggests to keep the CPU-cycle raping down to a minimum. And WP-Supercache is a mature plugin that is very well supported so it seems unlikely (although certainly not impossible) that it is the vector.

And thanks to your logfiles not being able to be viewed in real time (as they are owned by root), this leaves web developers that actually have a clue very few options for forensically backtracking the vector.

I would like to know what Rackspace is doing to help developers isolate these issues? Are logfiles being programmatically reviewed for malicious traffic? Without SSH access and the ability to tail apache logs, we cannot do this ourselves within any kind of timeframe that will be useful in preventing or mitigating an attack. If I am going to continue hosting with Rackspace, I want to be assured that Rackspace is actually doing something to help us protect ourselves other than send emails that overstate the obvious.

Your support staff, at least most of the level 1 techs, are completely and utterly incapable of handling anything relating to hacks. They are slow and under-educated, regardless of how well meaning they might be.

You guys are in the position where you can help isolate these vectors. What steps are you taking? You need to up your game, or I’m bailing, and likely taking a lot of people with me. There is a lot of buzz going around about these vulnerabilities being specific to Rackspace Cloud, as it seems the vast, vast majority of the Wordpress hacks have been on RS CS hosted sites.

I have confronted several of your higher-ups in the Cloud, including CTO John Engates, multiple times over the past year, begging for better tools to monitor security, offering to pay extra for them. Simple tools that even terrible, insecure Cpanel servers have. The entire purpose of Mosso, when it was created, was to target web developers – at least that’s how it was pitched to me. Web developers. Professionals. Many of us with over a decade of experience in this business. You deny us SSH and real-time Apache logs, but do nothing to provide us with any tools we would need without access to those basics – and then to add insult to injury, you send us a form letter that tells us to use good passwords and keep Wordpress up to date? If your target is still the web development community, it’s time to nut up or shut up. We’re already doing all of these things, and we’re still getting fucked. It makes us look bad, it costs us time and money, and the trust of our clients.

Your customers are under attack, and I want to know what you plan to do to help us protect ourselves and our clients, or I am taking my business to a company that values my time and reputation.

I would not have published this letter to my blog if this were not something that I have been asking for, over and over and over, for the entire year I’ve been with Rackspace Cloud. I have tried to keep my issues with Rackspace off the grid, because overall I have felt like they’ve been trying to work with me to keep me happy. But this was just too much.

No one is sorrier than I am that it came to this.

Also check out: