Going beyond the dial tone

As you are painfully aware, we are currently experiencing significant issues with one of our mail clusters, mailbox.perceiva.com. We have had engineers working since 3:30a Friday morning to resolve the issue and are making significant progress. We are in the process of restoring from out backups and are taking extra steps to ensure there is no mail corruption and once services are restored, that we will no further issues.

We do certainly realize that this has been an incredible inconvenience and I promise to work with you to find some what to attempt to make it up to you. I ask for your continued patience as we work to a resolution.

EDIT 4:00p EDT – Backups appear to be viable, restore is underway.

EDIT 9:15p EDT – Restore continues, we have about 400gb of mail to be restored in total. It is approximately 50% finished at this point from disk space calculations.

EDIT 7:00a EDT – Restore has completed. We are validating the restore, allowing the mail queue to deliver pending messages from Friday/Saturday and we will be allowing client access shortly.

EDIT 8:15a EDT – Services have been restored. A full root cause analysis is being developed and sent to customer contacts.

EDIT – Message from Greg, CEO of Perceiva -

I wanted to share my sincerest and deepest apologies for the outage experienced on mailbox.perceiva.com. There are several things that went wrong, the culmination of the perfect storm if you will. Our engineers are developing a formal root cause analysis. Once completed, we will be sending a notification all all customer contacts (those listed as the “admin” of the domain) with details on what happened, why it happened, what we are going to do to mitigate future risks and of course what we are going to do in an effort to make up for the experience we have all gone through in the last 48 hours.