In the last days of September 2014, Rackspace customers were faced with multiple issues regarding service uptime, security concerns and network outages. The worst hit were the ones who were using Rackspace’s Chicago datacenter.
I happened to be one of those customers. This is my story.
Rackspace: A beacon of hope for all things hosting
Let me begin by saying that I love Rackspace. I salute the kind of service they provide, I appreciate the pricing that they charge. I even admire some of their key people.
I always hold Rackspace above the other mortal hosting firms … why .. well because they are fanatical about their support. So much so that it’s part of the core values at Rackspace.
This singular point is why I hold them above the others.
In India, where anyone with a global ip decides to have their own hosting and start providing this as a service, the quality of hosting is a major issue. A lot of the times, as a techie I have found to my own chagrin that the inhouse team of the hosting provider is completely clueless about hosting, networking and security. Add to it the prohibitive costs that these providers charge … that puts them not only at par with Amazon or Rackspace, but also at times much higher than them.
The outage
As a result, I almost always promote Rackspace as the go to hosting provider for all of the people I work with. What this means is that I am referring their services not because I am an affiliate (no I am not), but because I truly believe in them and their ability to deliver great uptimes. However, this belief stands shattered.
In the last days of September, there was a massive outage (almost lasting 2 days) with all Rackspace servers. Whereas, 13 Llama’s server did not face major issues and it was up and running most of the times, some of our main clients did!
My first reaction was that there might be an issue with the webserver settings. So I promptly logged into the server and lo, I could not even reach the server much less tweak apache settings. Clearly, this was a network issue due to some problem. I tried catching the chat executive on Rackspace’s site. The chappie tried his best, however after 20 minutes it was clear that he had absolutely no idea what the problem was … or that he was not willing to acknowledge that there was a problem with the datacenter’s network.
I raised a ticket, after a couple of hours of waiting, there was an update and the support team acknowledged the network issue. They also suggested a fix, however that fix was not working.
Things get ugly
After trying different approaches, the support executive say that I will have to upgrade my membership and purchase a managed services slab so that the person can take a look at my server. I would have done this had there been any issue with the server. However, I had not made any changes to the server due to which this outage would have been caused.
This is where I start losing patience.
- I am being kept in the dark of the actual problem at hand
- The server changes / outage was due to some security upgrade that they are doing, and due to this I have to pay them more to solve a problem that they have created
- My client’s website is still down
I give up on Rackspace support and try solving this on my own.
Simple Solution
In the end, what came to my rescue is the cloud!
I made a back-up image of the server in question and spun a new server with the same configuration and that image. Within 10 minutes, a new server was up and running. I updated the nameserver records and that’s that.
The problem was solved for me. Rackspace, however had major problems.
The Apology
I was angry at Rackspace’s support. However, this heartfelt apology from Taylor Rhodes drove me to reply to his email … inspite of knowing that most likely this email was sent as a mass email (an excerpt is below).
Those of you who are longtime Rackspace customers know that we have a strong record of open, timely communication with you. We reach out to you whenever there’s an issue. We answer the phone whenever you call. We do everything we can to find a solution. This past weekend, our engineers worked tirelessly with customers and partners to remediate the Xen vulnerability.
Such an email warrants a reply!
The catharsis
When an organization of the caliber of Rackspace (there is no denying their ability and scale) mucks-up, that is when I question things. Here’s an excerpt of the email that I replied to the mass email (I was not expecting a reply).
I don’t want to complain about my individual problems. I am sure your team has had thousands of those. I just want to point out to you that one of the core values with which I look up towards Rackspace was “Fanatical Support”. I believe that you have failed at maintaining this core value.
Within a week, I received a personal reply from Taylor Rhodes and it was addressed by one of the management members in his APAC team. This is great response.
What is broken may not be fixed
However …
What I wanted addressed was not my server issue. I sorted that out myself, thank you.
What I wanted addressed is how does a large organization manage to adhere to its core values? In India, I see a lot of large firms (even the Fortune 500 ones) who tout high values, however, how many of them manage to uphold them when engaging with their customers?
With regards to this, I was looking at Rackspace in high esteem … they had (until now) managed to be awesome in support. What changed then … yes, the crisis they were facing was a lot, and their support team would have been facing several angry customers. I grant them that. There is no elegant solution to this.
However, how do you intrinsify a core value? At what point in time do you do a reality check and say …
Hey, are our core values being upheld?
That’s a difficult question to answer. Personally, if you ask me, I have yet to brainstorm the core values of my even own organization!! However, the real question that I would ask myself or anyone who has put out their core values is … to what extent are you going to go to uphold those?
Would 1 customer complaining out of 200,000 be neglected (as it’s an extremely negligible sample size)? Or would the 200,000 complaining customers be a big enough number to take necessary action.