Whether it is virtual servers, shared hosting, or anything in between – maintaining 100% reliability for a website and email platform is difficult. If you host a site for a protracted length of time it is inevitable that you will suffer from some type of service disruption. The key to recovery is having data backups and redundancy. This lesson was recently brought to the fore through an unfortunate VPS outage at 123-reg.
Here is a summary of the facts: As first reported by the BBC on April 18, 2016 – 123-reg was conducting a “clean-up” operation on its VPS systems when a coding error in its software “effectively deleted” customer websites. The company subsequently reported that the fault was limited to 67 servers out of 115,000. Further the specifics of the problem were outlined by 123-reg as follows:
”This script is run to show us the number of machines active against the master database. An error on the script showed ‘zero-records’ response from the database for some live VPS. For those customers, this created a ‘failure’ scenario – showing no VMs and effectively deleting what was on the host. As a result of our team’s investigations, we can conclude that the issues faced having resulted in some data loss for some customers. Our teams have been and continue to work to restore.”
Thus began a trying past 2 weeks for those VPS customers who had not either paid additional fees for 123-reg to back their sites up or had not backed up their virtual servers themselves.
As you can imagine – those customers who lost the use of their servers and email became quite upset. Here is a very small sampling of a few of the early Twitter responses to the outage.
@123reg Received email from 123-reg, can't restore my sites, they offer an apology. They are not getting away with that – bloody angry!!
— Art Tutorials Online (@PriestleyonArt) April 18, 2016
— James Tanner (@james_a_tanner) April 18, 2016
— Daniel Naracci (@DanielNaracci) April 18, 2016
As events unfolded it became clear that some number of customer sites had their data deleted completely – as 123-reg had no data to restore the affected sites, they had become effectively unrecoverable. 123-reg did hire data recovery firm Kroll to help restore customer data and as of May 3, 2016 a number of customer sites have been restored. A further controversy ensued when 123-reg offered those customers whose data and websites could not be recovered 6 months VPS hosting for free and a year of free backups by way of compensation. As the following Tweets can attest – the offer did not go over well with some customers.
— InnMaster (@OnlineStocktake) April 20, 2016
— Josh Packham (@photographerXA) April 20, 2016
Another customer brought up a good point – if a website is down for a period of time, it may be penalised in the search listings by Google. Another reason to ensure that your website stays up.
@123reg Hi, our VPS is still not up & our websites no longer rank on google.Will they return to their place on google when site back online?
— Anthony Pearson (@anthonypears09) April 25, 2016
The most important lesson for both hosting providers and their customers is this: always have backups of your critical data. You simply cannot rely on any single source of backups. That is the redundancy needed. If necessary – pay your hosting provider to back up your online data and then create a backup of your own (not stored on the same server). Also be sure to test restoring the data – simply having a backup file isn’t enough. You have to be certain that the file is able to restore your site and data if needed.
One final note on this incident – the parent company of 123-reg is now for sale. This means that 123-reg itself is now up for sale. It seems unlikely that the VPS deletions had anything to do with the decision to sell, but the timing is unfortunate for HEG as any buyer would certainly conduct due diligence on the 123-reg brand and deletion incident.
Details on the proposed sale of HEG can be found here.
For further reading on 123-reg and the VPS outage:
TrustPilot Reviews: https://www.trustpilot.com/reviews/5717ac760000ff000952aaa1
Various articles detailing the specifics of the 123-reg VPS outage: