Thanks for taking the time to comment, and apologies for letting you down on the customer experience side … but like you said … this is already making us a stronger company - managerially and operationally.

Hopefully amongst the lessons learnt are:
+ Know where the trip switches are and what each are for (I’m assuming that that notice posted over-simplifies the technicalities of the incident)
+ Load testing (including surge testing) needs to be improved
+ Customer management of this incident was p**s poor, and needs to be radically overhauled (because you can’t improve on something that doesn’t exist). A major incident notification policy of some form is needed. You clearly don’t have that, or chose to ignore it.

What doesn’t kill you makes you stronger .. so hopefully you’ll grasp the opportunity to improve the areas lacking, as highlighted in the comments above.

.cg

Michael … as Marketing Director it is my responsibility to manage customer communications, and I totally agree that a faster response should have been forthcoming. There’s a few reasons for that not happening - me not being here, me not understanding the issue enough to communicate it, and the rest of the company being focussed on supporting affected customers.

However that is not an good enough excuse or reason, what it is though is a valuable lesson. A Crisis Management Communication Strategy is being formulated where in the event of anything like this happening there will be full communication with customers on the status and progress of what is happening.

Tony: I hope to be ripping us the water on Saturday with my 9m Yarga ;-). I’ve raised this issue with our Technical Operations Director - there’s two things - one is that we are not actually supposed to talk to our resellers customers - they HATE it when we do this, as in the market it often looks like we are competitors … that might not be the case for Digital Crew (especially since you clearly are aware where the hosting is provided) .. but we have a few resellers who keep us under NDA, as it would really hurt their business - and that is totally fair enough I think.

Secondly, you shouldn’t have had a poor experience with hosting365 regardless of anything - as Marketing Director, Customer Experience is one of my primary focusses, as developing positive relationships with existing customers is the best method for growing a successful business.

Eoghan (and Tony): I’m REALLY sorry you have had a poor experience with us. The overwhelming majority of successfully resolved support tickets would suggest that this is not the norm (thank god … we pride ourselves on ‘delighting’ customers … so this would be a real hit were it the everyday occurrence). If you have any specific complaints in the future, PLEASE contact me directly (and if you have the ticket number so I can go through it personally that would be great) and I assure you it will be actioned.

I’ve had similar problems with a girl in 365 support (on a few separate occasions) with a serious attitude problem too. Can’t be sure if she’s the same you’re referring to though.

I’ve got a lot of respect for the 365 guys and haven’t much else bad to say about them, but I’ve complained before about her, and while their response was very understanding, she’s still there, treating customers like shit!

Ed:

You’ve got to do something about her. Sorry. I’d also like to say that Michael has a point with regards to your management of the issue. There’s was just total darkness. No word from you guys for hours. I would have thought an e-mail would be sent immediately (when the systems were back up).

Looks like we are up and running again.

I guess technology will always let us down but managing the customer is the key. Thanks!

PS: You will get kitesurfing this Saturday! F5 plus mate so take a small kite.

slan
Tony
http://www.finsmag.com

http://www.finsmag.com

I understand tech issues can be outside your control but customer service is not. I just called 365 and got a very poor response from the girl that answered the call. My site is down i don’t need attitude on top of that.

Digital Crew may be my developers (and great ones at that) but my hosting issues are with 365 so I need you guys to help me out.

By 2pm today I need to post the the surf forecast for all our readers who want to know where to go surf, windsurf or kitesurf this weekend.

Any help mate?

Slan
Tony

I think you are missing the point here slightly. Hosting365 may have dealt with the technical issue in a speedy fashion, but they did not “manage� the incident. This was a major incident, affecting ALL of your customers, and there was no communication. There was a comment on the boards.ie site at 15:27 stating that there would be an official statement on the hosting365 site in a few minutes. This announcement did not arrive until after 6pm.

There will always be outages, not normally on this level, but is it important that customers are kept informed.

I hope as a result of this outage you will review your communication procedures to ensure that your customers are kept informed in future.

Michael

The link Stephen posted in the comment above is to a brief explanation from our Technical Director - Joe Lavin. We WILL have a more detailed incident report out as soon as detailed analysis is completed.

Any customer down-time is a disaster. We know your businesses need email and web sites live in order to function - furthermore, without you we wouldn’t have a business - so take it from me when I say we consider the outage today to be critically serious and the entire management team will be involved in the analysis and review.

In fairness to everyone in hosting365, the issue will dealt with speedily, multiple tests done over the day afterwards with our generator and UPS and everything works fine. I won’t comment on the specifics of the problem, but it looks now like one of those freak things that find an unbelieveably small glitch in a system design and manages cause a problem.

Yes we are annoyed - but we’ve learnt a valuable lesson - redundant power systems aren’t enough - and wait until you see what we’re building at the moment that our engineers keep calling ‘data centre porn’ !!!

Bear one thing in mind … we have NEVER had an outage in our data centre before, we are the ONLY hosting company in Ireland that owns our own network, and therefore, our Engineers work physically at your box and web services, ensuring the fastest recovery in the case of anything going wrong.

The goal of this blog is to communicate with our customers, so please, post any feedback and suggestions on this here.