Friday the 13th – no one really knows where the superstition originates from, however there are many claims throughout history.
Legend has it that if;
- If 13 people sit down to dinner together, one will die within the year.
- The Turks so disliked the number 13 that it was practically expunged from their vocabulary (Brewer, 1894).
- Many cities do not have a 13th Street or a 13th Avenue, many buildings don’t have a 13th floor.
- If you have 13 letters in your name, you will have the devil’s luck (e.g., Jack the Ripper, Charles Manson, Jeffrey Dahmer, Theodore Bundy, and Albert De Salvo all have 13 letters in their names).
- There are 13 witches in a coven.
In any instance, here at Compare the Cloud we have a craving for superstition, and as today is Friday the 13th we figured we would indulge ourselves! So we’re highlighting some true horror stories with Cloud and the industry.
The following contribution is courtesy of Neil Cattermull.
Replication is the Devil!
So, I recall some years ago a true situation that could have easily been avoided. One major international organisation had dual redundancy for all of their core systems, tested regularly with failover tests documented and publicised at board level. One frosty morning during the post New Year celebrations, a certain power provider happened to be lurking around a particular street corner with a cordoned off manhole cover. By lunchtime there was a creepy occurrence where lights were flickering, and random desktop computers restarting by themselves – creepy. Then, if by some cataclysmic coincidence, the office was immersed in darkness, shortly followed by a plethora of swearing.
Somewhere in the distance, close to the fire exit, fluorescent jackets were wandering around ushering staff to the stairs.
Fast forward to the IT floor – Panic stations had kicked in and the clucking of acronyms were heard like battery hens in warehouses. Bang, the lights come back on and sighs are a welcome relief. However partial power had been restored but not to the core server rooms in the basement. Communications had been established with the outside world but the Big Computing rooms were isolated and many of the core IT services had failed. Enter Headless chicken mode AKA Disaster Recovery, came into play.
Disaster Recovery was invoked (all core services switched to run from the replication partner data centre) and amazingly, as per the test results from 4 months ago, core systems were restored within the next 3 hours, FANTASTIC!
One whole day later, with the root cause fixed (a rogue over zealous electrician taken to hospital), the decision to revert back to primary IT infrastructure onsite was taken. After vigorous testing, hours of through the night working by dedicated IT staff. And then switch gets flipped and normal working was restored! Some time later the help-desk calls came flooding in with similar descriptions… WHERE’S MY DATA!?
Enter Stage Left: THE REPLICATION DEVIL. Within all the testing procedures, scripts, processes and sign off sheets the one thing that had not been tested was… Fail-back replication of data from services from running in disaster recovery mode. When testing in disaster recovery mode it had never crossed anyones mind, and unfortunately assumed working, that failback replication of data worked. What followed on for many weeks was the never ending support help-desk calls asking for missing data.
The moral to this story, be prepared for failure but be prepared to fail back!
Plus, a bonus from our fearless leader, Dan Thomas;
The Cloud Company in question shall remain nameless. An salesperson in a very slick suit from a cloud organisation oversold the benefits of cloud and their company services.
The dream was sold to the end-user organisation of lower costs and manageability of all applications and services. The end-customer based on the salespersons advice moved all critical systems including databases and created a pool of hosted desktops. The customer dream wasn’t realised the day the broadband connection they were using for a 50 person organisation collapsed.
With no back-up line and a highly saturated connection that was next to useless the company went ‘dark’ for 60 working days whilst a more secure leased line connection was invoked.
Moral of the story, cloud is useless without decent connectivity.