MIR is no longer hosted on the Rackspace Cloud

For a good few years we’ve been hosting Mobile Industry Review (“MIR”) on the Rackspace Cloud.

I’ve loved it. The concept is this: You pay $100 a month minimum. That buys you 10,000 ‘compute cycles’ and a ton of diskspace and bandwidth. For a reasonable sized site like MIR with around 250,000 visitors a month, it’s rare to go over the 100 dollar amount. Cost, however, isn’t the key issue for me: It’s availability.

I absolutely loved finally being free of the hardware panic. With the Rackspace Cloud, you upload your code, files, everything to their main server. And your site is then served via high-availability sooper-dooper cluster. Super-shit-hot-wonderful-amazing stuff. That means that should the Rt. Hon. Scoble link to you — as he has done now and again — I don’t feel it, technically. Rackspace’s infrastructure scales dynamically and boom, 10,000 people are served instantly.

Yes you’ll burn through your ‘compute cycles’, bandwidth allotment and so on — but technically, you won’t have a problem.

It’s genius. Utter genius. And I just couldn’t get enough of it. Having hosted sites for years on dedicated servers, I always had to make sure I was near an SSH or Telnet connection (BlackBerry is your friend for this, by the way — the first Blue BlackBerry, the 7230, was it? Or 7320? That had telnet capabilities) so I could login and restart Apache (web server) or MySQL (database server) in case something had screwed up. Anyone who currently manages a dedicated server knows what I mean. Despite putting an inordinate amount of control measures and monitoring, you still have to keep on checking stuff. And then — usually when you’re on your way out to have a nice dinner — you’ll get a note from a client’s client’s mate’s friend’s budgie’s mate, to tell you that the site doesn’t seem to be responding. On closer investigation, you’ll find it’s not the 20-second fix you’d hoped for… and that… arse, yes, you will need to replace the hard disk and… shit, yes, despite doing backups hourly, you have lost 6 hours worth of data. Or you can recover that last 6 hours, just, it’ll take 11 hours. And the client’s screaming… and the site’s down. And you’re now getting complaint emails from end-users.

Arse. Arse and thrice arse.

I think we’ve had about five periods of sustained downtime, each no longer than about 30 minutes, in around a year. Usually to do with a DNS error or something like that. All resolved quickly. All painless. Utterly phenomenal.

But, here’s the bad news.

WordPress.

The content management system that runs MIR and a ton of other sites is susceptible to occasional ‘issues’. And if you haven’t set it up right — or if you’re running fast during setup — there are ample opportunities for hackers to inject malware into your site and boom, it’s a total arse to fix. The malware is more or less automated. Scans discover holes that allow systems to place control files on to your site with write permissions. Before you know it, every page of your site will contain hidden links to an array of porn and spam sites. Google is pretty good at tracking it and advising users not to visit your infected site. Annoying but useful.

So WordPress needs to be installed and configured correctly. And it needs to be regularly updated.

I’ve never had a real problem until a few months ago.

From the brief reading I’ve done, Rackspace suffered some kind of security issue at some point. Michael over at Smackdown has a good summary of his findings. The security issue resulted in a ton of my sites — and I host a lot of them — getting infected.

It’s been a total nightmare. The malware installs itself into a ton of different hard to find and hard to recognise locations across your filespace. So the only way to actually be sure it’s gone is to completely reinstall everything. From scratch. In some instances, the malware even infects your database.

I don’t have a lot of time to arse around with this kind of thing. I spent an afternoon recently looking into the scale of the damage. I did a good few hours looking to see if I could find any competitors to use for hosting.

In the end, I thought ‘screw it’.

And I spun up a Rackspace Cloud Server. (This is separate from a ‘Cloud Site’ — which is how I hosted MIR, Mobile Developer TV, and so on)

I configured the server. I stuck on the firewall, Apache, MySQL etc.

I transferred the sites.

And now we’re back on a dedicated server.

Yes I’m still doing business with Rackspace — I’m just hugely disappointed I’ve had to stop using their Cloud Sites system. I did my best to try and clear out the rubbish malware, but it was absolutely everywhere. And I no longer had the confidence in their ability to deliver decent service. Further, I didn’t want to make a new Cloud Sites account only to find it was hosted on the same compromised-but-not-but-still-dodgy server IP.

As for the rest of the dedicated server industry? Rubbish. I looked around for instant server activations. I can’t be spending days evaluating services and when I need technology infrastructure, I want it now, not at 9am the next morning once some bored customer service agent has finished having his morning coffee.

So I used my Rackspace account to create a dedicated server (instance). The one I chose is $0.12 per hour, plus bandwidth. That’ll do.

As a measure of speed, the server instance was live in about 45 seconds. Apache was installed by about 1 minute 20. Heh. And MIR was moved, in its entirety — lock, stock and DNS barrel — in about 3 hours.

This is the temporary solution. We’re live, we’re free from malware, we’re free from infection from the rest of the Cloud Sites. And I can get on with screaming about the mobile industry’s total lack of innovation, rather than messing around with firewall config scripts.

If you’ve any dedicated/cloud/hosted server suggestions for Mobile Industry Review and our band of sites, they’d be most welcome. I now have the patience to calmly evaluate things.