Snipe.Net - Geeky Stuff
Twitter
Currently: @jkprime ugh.. Yeah. It's just embarassing. For men and women. in reply to jkprime 25 mins ago

An Open Letter to Rackspace Cloud Hosting

Posted on January 28, 2010 by in Featured, Web Development
closeThis post was published 2 years 10 days ago. There is a chance that some APIs or software versions discussed have changed since this article was written.

I just received an automated email from Rackspace that made my brain melt. It’s no secret that a lot of websites have been hacked lately.

One thing they seem to have in common is that they’re all running WordPress, and a lot of them are hosted at the Rackspace Cloud.

Dear Alison,

Since we host hundreds of thousands of applications at The Rackspace Cloud, we have a unique vantage point from which we can identify security trends and patterns. Lately, the industry has seen an elevated level of attempts to take advantage of code vulnerabilities in the software powering websites. Hackers are a common and persistent threat to any website, but there are steps you can take to protect yourself and to make your websites and applications harder to exploit.

Please read over the important tips below. We have dedicated security experts who work to protect our infrastructure, but since we can’t fix or upgrade code on behalf of our customers, it’s important for you to know and regularly implement security best practices in the code you run. We need your help and involvement to ensure your own sites are as protected as possible. If you have any questions about security, please reply to this email and we’ll be happy to help.

HERE’S WHAT OUR SECURITY TEAM HAS RECENTLY IDENTIFIED:

1. The current data that we’ve collected points to application-based vulnerabilities being exploited. Hackers commonly scan sites for insecure applications, plugins, or other pieces of code and then work to take advantage of the software exploits they find.

2. Applications using the popular blogging software WordPress appear to be mostly targeted, but WordPress isn’t the sole target of the malicious groups / persons.

3. Your site does not have to be high-profile to be targeted. Hackers often scan random sites for signs of software known to be vulnerable (older versions of popular software with publicly known security holes, for example).

HERE’S WHAT YOU SHOULD DO NOW TO PROTECT YOUR SITES:

1. This is probably the most important tip: For any application you use, be sure to maintain the most current stable version. Often, an application might be updated to a new minor version solely to address a security hole that’s been discovered. Be sure to subscribe to any news lists and feeds available for your applications to make sure you are aware of updated versions as soon as they are released.

2. Many applications, like WordPress, have optional plugins developed by the community. Since these add-ons are often not as well vetted, it’s extremely important to carefully evaluate and manage third party application plugins, themes, or other functionality that is introduced to a running web application. Most hackers are exploiting these plug-ins

3. It’s imperative to choose strong passwords. Randomly generated strings of letters, numbers, and symbols are best. Avoid words and phrases in your passwords. The unfortunate reality: passwords that are easy to remember are also easy to guess. (Ex: Replacing o by the number 0 is not a recommended tactic.)

4. Change your passwords on a regular basis and change them immediately when you have any hunch that your site may have been attacked.

5. Be as restrictive as possible with users and file permissions. Remove write permissions from files that aren’t likely to change frequently. Some programs have install files that should be deleted after installation. If you’ve installed something or written code for testing purposes or experimentation, it’s best to remove it afterwards. Only keep the files and code on your account that are active and necessary.

As a site owner, you need to take an active role in guaranteeing security of your code and applications. The good news is that our support staff is happy to help you with any questions or concerns you may have. Recovering from a hack or exploit is extremely time-consuming and frustrating. The preventive steps outlined above can make a world of difference in keeping your sites secure.

Finally, if you suspect your site has already been compromised, you should take immediate action. This knowledge base article can help you through the right steps:

http://cloudsites.rackspacecloud.com/index.php/Recovering_from_and_Dealing_with_a_Site_Compromise

Sincerely,
The Rackspace Cloud Security Team

I want to preface this by saying there are a LOT of people that work at Rackspace that are absolutely awesome. The guys I know from Twitter are amazing, and helpful and care about customer happiness more than I can even say. None of this is their fault. This is NOT about them. This is about something fundamentally wrong with priorities at Rackspace, in my opinion.

I replied:

Too little, too late. I could have (and did) tell you all of this already.

And unfortunately, running the most recent version of WordPress doesn’t help. This week, I have personally had to repair 11 WordPress websites hosted on the RS Cloud that were hacked, all were running 2.9.1 and had very few plugins in common. The plugins they do have in common, like WP-Supercache, are plugins Rackspace suggests to keep the CPU-cycle raping down to a minimum. And WP-Supercache is a mature plugin that is very well supported so it seems unlikely (although certainly not impossible) that it is the vector.

And thanks to your logfiles not being able to be viewed in real time (as they are owned by root), this leaves web developers that actually have a clue very few options for forensically backtracking the vector.

I would like to know what Rackspace is doing to help developers isolate these issues? Are logfiles being programmatically reviewed for malicious traffic? Without SSH access and the ability to tail apache logs, we cannot do this ourselves within any kind of timeframe that will be useful in preventing or mitigating an attack. If I am going to continue hosting with Rackspace, I want to be assured that Rackspace is actually doing something to help us protect ourselves other than send emails that overstate the obvious.

Your support staff, at least most of the level 1 techs, are completely and utterly incapable of handling anything relating to hacks. They are slow and under-educated, regardless of how well meaning they might be.

You guys are in the position where you can help isolate these vectors. What steps are you taking? You need to up your game, or I’m bailing, and likely taking a lot of people with me. There is a lot of buzz going around about these vulnerabilities being specific to Rackspace Cloud, as it seems the vast, vast majority of the WordPress hacks have been on RS CS hosted sites.

I have confronted several of your higher-ups in the Cloud, including CTO John Engates, multiple times over the past year, begging for better tools to monitor security, offering to pay extra for them. Simple tools that even terrible, insecure Cpanel servers have. The entire purpose of Mosso, when it was created, was to target web developers – at least that’s how it was pitched to me. Web developers. Professionals. Many of us with over a decade of experience in this business. You deny us SSH and real-time Apache logs, but do nothing to provide us with any tools we would need without access to those basics – and then to add insult to injury, you send us a form letter that tells us to use good passwords and keep WordPress up to date? If your target is still the web development community, it’s time to nut up or shut up. We’re already doing all of these things, and we’re still getting fucked. It makes us look bad, it costs us time and money, and the trust of our clients.

Your customers are under attack, and I want to know what you plan to do to help us protect ourselves and our clients, or I am taking my business to a company that values my time and reputation.

I would not have published this letter to my blog if this were not something that I have been asking for, over and over and over, for the entire year I’ve been with Rackspace Cloud. I have tried to keep my issues with Rackspace off the grid, because overall I have felt like they’ve been trying to work with me to keep me happy. But this was just too much.

No one is sorrier than I am that it came to this.

Also check out:

If you think this article kicked ass, subscribe to the RSS feed or follow me on Twitter! Share with your friends, or leave a comment below (or better still, do both!) My entire concept of self-worth is in your hands, so that makes you kind of a big deal. Srsly.

This entry was posted on Thursday, January 28th, 2010 at 7:07 pm and is filed under Featured, Web Development. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.
  • http://en-gb.facebook.com/davenatts Dave Nattriss

    I've yet to be affected as only have on WordPress site, but great reply to them.

  • http://scobleizer.com Scobleizer

    I totally agree and have been yelling about this internally too. Unfortunately the infrastructure that Rackspace Cloud on makes it hard to turn on a dime and ship the features we both want, but we are working on these issues. This letter is aimed at getting people to at least use best practices, which as when I got hacked not everyone is.

  • http://www.snipe.net snipe

    Hi Robert – I totally get that. But really, as a professional, this letter only ads insult to injury. When I tried to talk to Engates about this stuff, even casually, at that tweetup you and I met at, he looked at me like I had six heads. They need to decide who their target is and work with that. RS CS was sold to me as being web developer friendly, made specifically for folks like me, and that's simply not true. Given the present situation, when one of my clients gets hacked, this is how the conversation goes:

    Them: Oh shit! My site's been hacked!
    Me: Okay, don't panic. I'll go in and fix it immediately.
    Them: How did this happen???
    Me: I don't know – I might be able to tell you in roughly 24 hours. Maybe.
    Them: What happens if we get hacked again in the meantime?
    Me: Uhm, we're fucked? I go in and fix it again. Oh yeah, on your dime.

    Which wouldn't be the end of the world if *I* wasn't the one that sold them on Rackspace Cloud in the first place.

    We need to be able to respond quickly to these types of things, beyond just replacing the files and praying to the space god Zorkon that it doesn't happen again.

  • http://www.snipe.net snipe

    Hi Robert – I totally get that. But really, as a professional, this letter only ads insult to injury. When I tried to talk to Engates about this stuff, even casually, at that tweetup you and I met at, he looked at me like I had six heads. They need to decide who their target is and work with that. RS CS was sold to me as being web developer friendly, made specifically for folks like me, and that's simply not true. Given the present situation, when one of my clients gets hacked, this is how the conversation goes:

    Them: Oh shit! My site's been hacked!
    Me: Okay, don't panic. I'll go in and fix it immediately.
    Them: How did this happen???
    Me: I don't know – I might be able to tell you in roughly 24 hours. Maybe.
    Them: What happens if we get hacked again in the meantime?
    Me: Uhm, we're fucked? I go in and fix it again. Oh yeah, on your dime.

    Which wouldn't be the end of the world if *I* wasn't the one that sold them on Rackspace Cloud in the first place.

    We need to be able to respond quickly to these types of things, beyond just replacing the files and praying to the space god Zorkon that it doesn't happen again.

  • http://www.bynkii.com/ John C. Welch

    size and infrastructure as an excuse only show that Rackspace has no real procedure in place to deal with this kind of thing.

    WordPress being an attack vector, and a popular one is not new. The ass-raping your own blog took shows that. That was how many months ago? From what anyone can tell, Rackspace did fuck-all nothing to deal with this better, and so now, once again, WordPress installations all over Rackspace are once again getting pillaged like they had given Genghis Khan the finger.

    it is absolutely astonishing that Rackspace is still acting like they have not plan 1 in place to deal with, and i say this not just as some guy with a blog, but as an IT director of a company that uses Mosso or whatever the ass name it has now, to host some of our more important client sites. Mosso is not our only option, and if Rackspace does not get their shit together, and I mean right bloody quick, I will, without qualm, or even raising my pulse, put a bullet in our relationship with your employer, and move to someone who CAN handle life's little issues better.

    as far as my own site, while I would love to move to cheaper hosting, right now, I will HAPPILY pay the $200+ a month bynkii.com costs me, because unlike Rackspace, when shit starts happening at digital.forest, they aren't making lame excuses about being “too big to fix things”, they're working day and night to fix it and tell me why it broke, and why it won't break again.

  • http://twitter.com/djg Dan Grossman

    Don’t be so quick to rule out WP-SuperCache. That seems to be a common element in a lot of recently hacked WordPress sites.

  • http://twitter.com/ChrisDrake Chris Drake

    This is the fundamental flaw with 99.9% of hosting companies. Rackspace is VERY proficient with hardware and networking but they lack (like others) in application protection. Putting the responsibility on the site owners to continually do zero-day updates is irresponsible. Open source with all of the plug-ins will continually be riddled with vulnerabilities. The key is to block the root problem – the hacker's (or bots) traffic patterns.

    Yes… you should always keep your WP install fully up to date and this letter is a good effort to use best practices and protect yourself if the hosting company won't. There are hosting companies who are beefing up their security on the application layer but Rackspace Cloud isn't one of them.

  • http://twitter.com/ChrisDrake Chris Drake

    John – just checked out your copyright line on your website… hilarious.

  • Pingback: An Open Letter to Rackspace Cloud Hosting

  • Pingback: Open Letter to Rackspace Cloud Sites « No Suitable Nodes

  • till

    So, the bottom line is… WP is vulnerable, even in the latest most stable? Or is it RackSpace?

    One my friends got hacked 24/7 for a while, it was pure insanity. All the debugging he did, etc. – could have been so easy with a little more of the big picture but the support was not even remotely helpful.

    I think he ended up writing his own log files (through a prepend directive in PHP) to gather more information. But that'll eat away on resources.

    Speaking of which, what strikes me about Mosso/Cloud Sites – the cycle stuff. And no one being able to tell you which script uses how many. It's totally nutcase, it could be a random number generator in the back and then you are billed. This is pretty unbelievable and I don't see how this targets people who know more than the average John Doe.

    Your blog post just confirmed what I've been telling him for months. Go get something from slicehost, it's probably a lot, lot more cost effective. ;)

  • http://www.snipe.net snipe

    Heya Till,

    From what I can grok, it may be related to multiple instances of WordPress running under the same account (ergo same *nix uid). If one WP install is outdated and vulnerable, it can poison the others, set up backdoors, etc. I am still waiting to confirm whether or not the client I repaired a bunch of sites for had an older version tucked away in that account. I didn't poke around much, since he was paying me by the hour.

    Yeah, the billing cycle thing has been a thorn in my side since the beginning. I run two vbulletin sites (and I'm sure you know how much of a memory pig they are, especially without access to memcache), a very busy but low-query non-profit site and a pile of blogs, and I was going over CPU cycles by 200% every month. They've tried to work with me on this, especially for the non-profit stuff, after Robert C went through heroic measures to make sure RS followed up on what they had told me 6 months prior that made me decide to switch. They had told me overages for non-profit stuff would be comped, and it took 6 months to get that sorted.

    The shit that annoys me about this is that with ALL of these RS customers getting hacked, there's NO WAY they don't see the injections in their logs. Most of the attacks are coming from the malware site centiyo. I asked them to block it at the firewall because after the sites had been cleaned up, we were getting hammered with 700+ simultaneous requests involving that site trying to pull in a CGI script. (I'm preparing a writeup on the hack – just taking some time to sort through everything I documented while fixing.) If I wasn't using Chartbeat, which notifies me if there is an unusual spike in traffic over what I usually see, I'd be *paying* for those requests. My site normally has between 10 and 50 simultaneous users. 700 at once, you can imagine how that would fuck up my CPU cycle billing. If I'm paying by the CPU, and they won't tell me what scripts are using it, they could at the very least have some sort of automated email get fired off saying “Oh hai! If things keep going the way they are, you'll be paying 14x what you paid last month. Have a nice day!”

    Seriously though – even Cpanel can manage to fire off an email letting you know that there's suspicious activity detected, and Cpanel is a giant piece of shit.

  • lomifeh

    I am surprised they don't offer shell access or real time access to the logs. These seem like fundamental things for anyone who does this stuff for a living. On of the reasons I did not go with RS for my personal sites was that.

    SSH access I can see why they'd be leery but the logs? That's a no brainer to setup. Even a shell can be done smartly to minimize risk. My current host has been fairly proactive about stuff. If they see trends or something comes up on your account that they think may be a risk they email you going “hey you should do something about this.”

  • http://www.mosso.com jbryce

    Thanks for the feedback. We've been in touch with you directly, and I'd like to continue that. You make good points, and I'm sorry you haven't been satisfied with our response up to this point. We do take the shared responsibility of application security seriously, and are working on a number of projects to help our customers be able to improve their security posture and to do as much as we can in the overall hosting environment.

  • http://www.jasonn.com jasonn

    Robert, I use you as an example at least twice a week regarding website security. You're connected to the smartest folks in Silicon Valley and yet you've been hacked… multiple times. Even smart people don't use “best practices,” because it's a lot of work.

    This is the downside to running your own stuff. You DIY fanatics are going to get hacked, or spend all your time reading security reports and updating your software. Have a life, or do it yourself. Pick one.

  • http://twitter.com/joseduenas Jose Dueñas

    More than 2 months ago, the Disk Space Used is broken down. It show you are using 0 GB of 8 GB. The response about that: “We are working on that, we don't know when it will be ready again”.
    I was very happy with them, but every single day is getting worse… I hope this kind of details will get better soon. The have very good support technicians.

  • http://www.snipe.net snipe

    Hi Dan – I think that's true simply because WP-Supercache is so popular, and Rackspace even recommends it for high traffic sites, so it's a pretty safe assumption that most of the WP blogs on RS CS are using WP-Supercache. Correlation is not equal to causation.

  • Leo

    Right! John that's awesome man!

  • http://www.snipe.net snipe

    I'm not sure if its a scaling issue or the possibility of log poisoning that prevents them from giving us real time logs

  • http://twitter.com/jmarler Jon Marler

    I'm a Racker, and I just wanted to let you know your post is getting a lot of attention here. And not just from our marketing and cloud departments. Lots of Rackers are talking about this, and when lots of Rackers get involved, good things happen. We don't want to just keep you happy, we want to be the vehicle that makes your business grow. There is a lot of work happening on this issue, and good things will come of it, but not overnight.

    Feel free to reach out to me anytime.

  • http://www.centernetworks.com centernetworks

    Till is talking about me. Snipe – I'd love to have a email or phone chat with you – can you send me an email?

  • lomifeh

    I didn't think of the log poisoning, but if you offer the stats then raw logs as a tarsal that would alleviate that potential issue. But the scalability should not be an issue I'd think.

  • http://www.snipe.net snipe

    Hi Jon – thanks for your reply. I can only *imagine* that its getting a lot of attention – hopefully there aren't too many dart boards with my picture on them posted in the office just yet.

    I hate doing this – I have always felt like Rackspace's intentions, if not their executions, have been good – and I have gotten to know and love so many people there that I generally don't air dirty laundry in public. It's like I said in my There's No Such Thing As a Social Media Marketer post. It becomes harder to give up on a company when you're invested in their employees. But I feel like this is a conversation I've been having for a year, and this is just the first time when there has been enough of an issue to get my panties in a wad.

    I believe part of the issue here is transparency. I spoke with Adrian late last night on the phone and Jonathan Bryce emailed me today. According to them, RS has been working on addressing this problem on a higher level – but because they didn't communicate that to their users, it felt like everyone was getting fucked and Rackspace wasn't doing anything about it.

    For example, one vector would be an older installation of WordPress on the same account. If someone has 8 WordPress accounts and upgraded all but one, that one can be a vector for poisoning the upgraded, solid installations. I don't know when RS realized this, but what should have happened is that the FIRST question out of every level 1 tech's mouth when responding to a customer running WordPress that's been hacked is “Are you running more than one installation of WordPress?” Lots of people use WordPress as a blog, but many use it as a CMS for sites that they don't update very often, s upgrading the WP install can get overlooked. It's not a “real” blog, so it falls off the radar. That is obviously still the site owner's fault and responsibility, but RS could have positioned themselves to be much more proactive and helpful if they nipped that right in the bud. My client had called RS several times when his sites were hacked, and not once did anyone ask him about it. I asked him today, and it turns out he did have a “static” website that runs an older version of WP. Was it his fault for not updating? Absolutely. But RS could have done so much more to mitigate frustration.

    IF we are going to be working in an environment where we don't have the tools we're used to having at our disposal (ssh, svn, log file tails, etc), then we NEED Rackspace to up their game in helping us and being proactive. We HAVE to have one of the two, since we evidently can't have both. On my own server, I had multiple tools set up to keep us safe. Brute force detection, mod_security, scalp, etc. I can't set those up here, so I need to know that you guys are offering me at least the same level of protection that I'd have been able to cobble together myself.

    In the interest of full disclosure, the automated security email you guys sent out wasn't a bad thing. Of course you should tell people to lock their stuff down. I reacted the way I did because I had just finished repairing the 12th hacked WordPress site in 3 days. While I was doing the restore on one, FTP randomly stopped working. 15 minutes and a level 2 tech later, it was back up. 10 minutes later, I had to login to the sites admin to check a ticket my client had opened, and the admin was down for maintenance for 1.5 hours. I reloaded a minute or two later, and it said it would be down for 2.5 hours. So yeah – I was pissed. The email was the last straw, not the cause of my frustration, and I absolutely agree and support you sending it out to your users. I just wish the proactivity hadn't ended – or started – there.

    So barring SSH and real-time logs (and better mysql/uptime), what do I want? I want to be notified when traffic to my site increases from 30 simultaneous users to 700, all attempting to do malicious things. Since I pay for my cpu cycles, if I hadn't been notified by Chartbeat, you can bet I'd be losing my fucking mind next month when my bill was 25x what it normally is. I want to know that at least basic security efforts are in place to block malicious traffic, and that when malicious traffic is detected, it gets blocked from the firewall or at least denied based on the request. If someone is trying to do a RFI or LFI attack, there are lots of ways to detect that at a technology level.

    I had started using Cloud Services partly to evaluate them as a possible candidate for the company I work for. We have several dedicated boxes and spend about $5k a month on hosting. At this point I just don't feel comfortable recommending them to my company. If something goes wrong, and it was my recommendation, it's my ass.

  • http://www.snipe.net snipe

    I don't have your email address – you can find me at snipe@snipe.net

  • http://twitter.com/jmarler Jon Marler

    No dart boards yet … ;) The thing about Rackers is when we hear that our customers aren't happy, we all feel responsible, and we all want to find a way to turn it around. We love our customers, and we want you to be so happy with Rackspace that instead of blogging your frustrations, you can blog about how AWESOME we really are!

    I can certainly understand why you are so upset. I can tell you we are doing a lot of work under the covers to stabilize, and that good things are coming.

  • till

    Yeah, so weird.

    In this case (CN), he didn't have another WP install. At least I couldn't find any. But yeah, maybe something else was hacked and it just infected the WP. It's impossible to “trace” without any logs though.

    All in all, I don't see mosso/cloud sites as a platform for developers. I mean, neither is MT's grid insanity. It's just not designed for people who know a little more. It's perfect for everyone who doesn't like their 1and1 account. And even 1and1 has shell on larger packages.

    Maybe cloud servers is more suited since they are so cheap and you basically get your own root. Then again, developers are not necessarily great system administrators.

    In the end I never manage to understand what makes people want to host with them at that kind of money — especially when you go over you can easily justify a dedicated server from RS with the “fanatical support”, etc.. There are probably a ton of small providers out there which are more suited for your business — people who care.

  • till

    Hehe… btw, did you ever email them for help? Or security advice?

    It's totally nuts. They'll send you a prettyfied grep output which highlights so called malicious functions in your code. It's beyond useful.

  • http://www.snipe.net snipe

    1and1 has shell, but it's a limited shell and none of the log files are readable. At least that was the way for my friend's account. I wouldn't recommend 1and1 to anyone, based on my limited experience with them.

    Yah, I switched to mosso after running my own box for 5+ years. (I still have it, hooked up in a colo in San Diego). While I didn't mind sysadmining the the box most of the time, I was glad to not have to do it anymore. It was very time consuming and everything look me a little longer than it should, since it's not something I had to do ever single day.

    The *individuals* at RS do care. I do believe that. I know enough of them on a personal level that I'm sure of that. Robert James Taylor, Adrian Otto and Robert Collazo have spent extensive amounts of time with me on specific issues, often working on it in their free time. There is no lack of dedication with many of the staff there. Although I can't speak for all of them, the vast majority have bent over backwards to try to help me when I needed it.

    I've been with a lot of hosts over the years – or worked with them because of clients, and they were all crap. The service we get on our dedicated boxed with Rackspace is very good, and very expensive. a dedicated machine with Rackspace isn't an option for me personally or for my non-profit organization. But it does show that their support *is* very good. I think the cloud issue is more a combination of problems.

  • http://www.centernetworks.com centernetworks

    fyi i have sent you an email – thanks

  • Pingback: Did Techcrunch Become a Wordpress VIP? | CenterNetworks

  • Warrock

    How is WordPress vulnerabilities Rackspace's problem in the slightest? I can't fathom how you people are rationalizing that it's the web-host's problem to cover your asses…

  • http://www.snipe.net snipe

    Try reading the post and the comments before chiming in. No one is saying Rackspace should be keeping WordPress updated for them – it's an issue of tools and transparency.

  • Warrock

    Well I did, thoroughly; and what I'm getting – especially in the comments – is that people are expecting their webhost (in this case Rackspace) to baby them and hold their hands in covering their asses from hackers. Why is this expectation so prevalent?

    Where did this entitlement philosophy arise that a web-host should provide for you all of the traffic analysis, attack vector analysis, and proactive policing that you expect?

    Cloud Sites is not a fully managed product. I don't see anywhere in their SLA that they are obligated in any way shape or form to act as your security advisers, I've never had an issue with my sites on their service, except when it was clearly my own fault.

    I'm sure they have their own internal issues, but from the people I've spoken with in their organization, the VAST majority of security issues on their system rest squarely on the site administrators/developers themselves. I have never heard of a site compromised SOLELY because there was a vulnerability within the Cloud infrastructure. Can you name a single instance where it was the Cloud infra-structure and the Cloud infra-structure alone that caused a site to be hacked? ….

    At any rate, I feel like everyone here including yourself are just so pathetically incensed by the fact that you have to work harder to make sure you're safe. I mean, FFS, they are not billed as an Enterprise level solution. If you need so much hand-holding, why don't you people just get a managed server?

    Can you name a single other host that does anything remotely like what you're pining away about, without insulting Rackspace's entire level 1 team in the process (who have always been AWESOME with all of my issues)?

    Pithy whining…that's all I see here.

  • http://www.snipe.net snipe

    Okay, so perhaps you read them, but missed it somehow. My point was that IF we're not going to have the tools available to us (such as ssh, real-time logs, etc), then they SHOULD step up their game for helping us with security, since we're not ABLE to do it ourselves. I moved from a dedicated box (which I still have in a colo in San Diego) specifically because this was sold to me as a solution that would work out well for devs.

    If you're going to cripple me, you at least need to help me up the stairs.

  • http://twitter.com/elazar Matthew Turland

    I don't think it's unreasonable at all for customers of web hosting providers, “cloud” or no, to expect that they include access to web server logs as part of their services. From what I can tell, that's the main thing that Alison is asking for here: not security advice or automated patching of software, but simple log access. She's not claiming that WP itself hasn't or doesn't have security issues and she's not expecting RS to fix them for her, she just wants access to logs so she has some means to investigate the issues herself. She's already mentioned that any provider that gives access to cPanel, horrible as it is, provides this. To say that RS CS doesn't and should isn't whining, it's a perfectly valid point. If RS CS isn't intended to be an enterprise solution, I'd have to ask exactly what it *is* intended to be. Certainly not a service for the web developers at whom its marketing was aimed, and “internal issues” is an all too nice way of describing this shortcoming and the customer response to it.

  • sympathizer

    it's the new Server Beach. it shouldn't have the Rackspace name on it and I see your point, you are correct, it is an issue. when it was still SliceHost, the target audience was people like yourself – informed professionals who were willing to learn from the forums, etc. but who also had a clue – NOT everyone should be in the Cloud.

  • Devon Lambert

    Hey Snipe,

    How come you don't give the Rackspace Servers option a go around? From what I can gather by reading the specs, it appears to provide root access. Did you not want to manage the whole server. I'm partially asking this question because I myself am in the middle of making the decision between Rackspace and Amazon EC2. When I stumbled upon your post it made me completely question Rackspace and has now thrown me in favor of Amazon's Cloud. That being said I was also wondering if they had made some false claim on their site regarding support as well as Cloud Server options?

    Thanks for the great post and please do update and let us know how this all turned out?

  • http://www.snipe.net snipe

    Hi Devon – I actually still have a dedicated server in a colo in San Diego. I had moved everything off that server because I was tired of having to manage my own box. I've been managing my own box for over 10 years, and it's just too time-consuming. I work full-time, commute 4.5 hours a day and run a non-profit org. It was just getting to be too much. I have heard good things about their Servers product, and we have several regular dedicated boxes at Rackspace, so it seems like it's the Cloud product that's still green and struggling. I have been dealing with Rackspace for one reason or another for years, and their support really is good. I would recommend them any day, just not the Cloud product.

  • BJ

    I am glad I am not the only one. I have noticed more and more of our WordPress websites having issues. I just asked via chat and was told to keep the WordPress up-to-date and file permissions set to 775 and 777. I just fixed one where it was the latest and the file permissions were fine. I am wondering, are you still with Rackspace? I am starting to look at other options to see if it would make sense to move hosting companies, but is there one out there that is better or is this just par for the course. Like you, I have yet to find a good solution and up to now, I had thought Rackspace Cloud might be it. I feel just as you, I like mostly everything about the Rackspace cloud, but this issue. Your post and letter to them is spot on!

  • ZanderChance

    Snipe, sorry to hear about this.. You're actually the one that convinced me to make the switch from my dedicated box to Mosso last April. I've also had my fair share of issues with them (including one month where my bill was nearly $400!), but aside from that and a few outages, it's been ok overall. I'm willing to let a few things slide, as long as I don't have to deal with the sysadmin work anymore! :)

    I saw your more recent post “The cloud is a lie”.. If you happen to come across a better solution, please let me know. Thanks!

  • http://www.snipe.net snipe

    Hi Zander – it's definitely better now than it was. Overall, my uptime and stabilitiy is improving, to the point where I probably won't move just yet. After implementing a LOT of caching, my bill is back under control, but the black box that is their billing still irks me. If it wasn't such a colossal pain in the ass to move 100 sites, I'd probably have given up already. Really though, it's the frustration of not having SSH/log files. Since the beginning of the year, I have had to fix at least 20 websites that were hacked, and doing any kind of forensics in that situation is damn near impossible. I've been looking at managed dedicated/managed VPS solutions, just to I know what my options are if I end up having to leave. Hopefully I won't.

  • ZanderChance

    Good to hear.. I also had to get creative with my caching, I'm very particular about our plug-in usage, and utilize a few other techniques to get my compute cycle usage under control. It's a pain sometimes (especially when I'm managing 40+ sites), but I guess that's life in the cloud.

    I come close to the 10K/mo line each month, but I haven't crossed it in probably 6 months now! I've been asking for better tools to monitor cycle usage since I signed up, but that's a losing battle.

    Glad to hear things are better now.. Thanks.

  • http://twitter-fail.com/ Twitter Fail

    Wow. I found this because I was researching the breach last week that caused 7 of my blogs to be hacked. I've been with them for years, and have put up with stuff like this over and over because I thought it was probably my fault and didn't look to see how many other people were affected. It's time to say goodbye to Rackspace. They just don't seem to have what it takes to manage their cloud properly.

  • Imagineer

    Maybe its time to move on to Page.ly. as they are focused on security by hosting their site with Firehost
    • Automatic WordPress Core/Plugin updates
    • Backed by the Bulletproof security of Firehost.com
    o Managed Redundant Firewall Protection
    o Managed Redundant Web Application FireWall Protection
    o Managed Redundant DoS/DDoS Mitigation
    o Multi-level Intrusion Prevention and Detection
    o http://www.firehost.com/why/compare

    They sounds good, but I have yet to use any of their services yet. At least I don’t get any horror stories while searching “Page.ly wordpress hacked” as compared to “Rackspace cloud wordpress hacked”. This is how I found your blog.

    • http://www.snipe.net snipe

      Rackspace has gotten a lot better since that article was written. A lot. You probably get that many search results on “Rackspace wordpress hacked” because Rackspace is a much more popular host. Everyone had heard of Rackspace – the same isn’t true for page.ly, so it’s a sheer numbers game at that point. WordPress gets hacked often, most of the time due to the user not updating their install or plugins. Correlation does not imply causation. Not saying Rackspace is perfect, but I’m also not willing to move 200 websites to a company I’ve never heard of. I’ll keep an eye on them and see how it goes over the next year or two though, so thanks for the heads up ;)

  • http://vitruvianhosting.com Webmaster

    WordPress is shit; quit using it if you want security. You get what you pay for.

    The infrastructure is solid.

    • http://www.snipe.net snipe

      There are ways to secure it – but if you think there were not legitimate factors in the hosting environment that made it harder to track down the vector, you’ve never done any hack forensic work. They have changed a few of those things over the past several month, making it a little easier. (No more load-balancer IP address for example.)

      Funny about your domain name – I own vitruvianarts.com.

      • Webmaster

        How about that…

        It looks prettier than mine also; grats =D

        Yeah the shared environment makes it a pain in the ass to do a number of things; but real hosting breaks the average bank. I just wouldn’t rely on wordpress for much of anything.

        And you’re right; I haven’t done much forensic work but I know the RackSpace Cloud. It’s an amazing infrastructure, and they are constantly seeking to improve it. And they’re the only dudes I can ever get on the phone – so even if you’ve got something to bitch about atleast you can reach them.

      • http://www.snipe.net snipe

        Hah – thanks – that design is 8 years old. I should probably fix that.

        You can reach them, sure – but WordPress is actually pretty stable these days. Hacks are generally a result of an older install that never got updated, or a crappy plugin. It’s nice that I can call them for help, but it would be better if I didn’t have to in the first place. That’s my point.

        Some of the things I’ve been asking for from RS are things I’ve been asking for for two years. And these are reasonable things that other grid/cloud hosting companies offer.

        They do have an amazing infrastructure, and they’ve gotten a LOT better over the past year. MySQL is way more scalable now than I was, and they handle some logging stuff a lot better than they did. But it doesn’t mean I can’t be frustrated that it’s still not where it should be. (Also, this article was written almost a year ago. They fixed a lot since then.)

  • Remembering

    Off subject a bit, but Rackspace centric … I’m just so sad. I too have lauded Rackspace, and have paid more for the support I get. However, a month ago, I noticed our account at a ridiculously high dollar amount. I emailed my account person and said I thought something was wrong. We never carry a balance over $500 … we were more than double that. A reply came, with a copy to someone in accounting, asking them to provide backup so I could audit the fees. I did, and to my surprise, found no errors in Rackspace’s statements. I wrote back and asked for two things to happen, a. that my server be cut once I migrated to a cheaper solution, also at Rackspace, and b. that we begin making payments to pay down this balance, starting with a payment before the end of 2010.

    Yesterday, we were suspended for non payment, and the only way to get this resolved was to pay the entire balance or $900. I am a very small business, with only myself and a part time person. This outage and resulting firestorm of communication has left me bitter and angry.

    I brought the account to the attention of the accounting department, not to be punished, but to protect my relationship with them. And, with my clients. However, some of my sites are still dark and Rackspace has refused to reverse the decision to require the full balance before we go live.

    I’m so angry. Thank you for the post, and for the warning. I operate several WP installations for clients and our ‘cheaper solution’ is a cloud.

    Cole
    Remembering.com (still down)

  • Edward Rendini

    So you’re on top of your shit and others aren’t. Get used to it. Stars are stars. There’s only a few of them in the grand scheme of things. Everyone else is faking.

  • Ttx1

    Sir I am sorry but get out of the cloud a doctor put peoples files in the cloud and they all lost their info with in 3 to 10 days so get out of the cloud it is very bad