Now that we have fully restored functionality to our POP3/IMAP email service, we would like to take a few moments to share what exactly caused this outage, our efforts to restore email and the steps we are taking to ensure that this doesn’t happen again. We know that many of our customers were impacted by this outage and we sincerely apologize.

As many of you are already well aware, yesterday we suffered a service disruption on our POP3/IMAP email. This event rides on the coat tails of the service disruption that we had last fall. So, how did this happen again?

Well… The outage occurred because a Raid Controller bit the dust, crippling the entire POP3/IMAP storage system. Our other email services Reflector, SMTP, and Backup MX were not affected.

“I thought you guys weren’t managing email in house anymore? Why haven’t you transferred me to the new service provider?”
The process to transfer accounts over to the new service provider has proven to be a very arduous task. We are now working around the clock to get everyone transferred over. Although users have still been on the old configuration since the outage in the fall, we moved the system to an older, more tested configuration that had proven to be reliable in other situations.

“What was the resolution process like?”
Resolution to the outage was a team effort. Alerts started going off at 4am that the network was down and our engineers were on it immediately. Our senior network engineer was on the first flight out of Reno to the data center to replace the faulty hardware. Our support staff did their best to give whatever information was available to customers that called in and submitted tickets. The problem was that there wasn’t a definitive time frame for the resolution process, because it was all riding on how quickly the installation and data migration to the new Raid Controller would go.

While we waited on the hardware installation, our engineers created a temporary webmail fix. All email functionality was available via the fix.  (Including the emails that had been received since the network disruption occurred.)

Slowly but surely, the Raid Controller rebooted and loaded. Around 7pm last night, email was back up and running with all delayed email delivered.

For our customers that were affected by this outage, we will provided a 5 day pro-rated credit of our email service. Customers will not have to do anything in order to receive this credit, as it will automatically be applied to their account.

Last, but certainly not least, we want to apologize. We know how important email is and we understand our customers frustration with the system. We have amped up our efforts to complete the migration to the new service provider for the remaining customers we support. We are doing everything we can to learn from this situation to drive improvements to not only our mail service, but all of our services.