RESOLVED / Mar 11: Service outage due to a power outage

We're working to restore access to Hushmail after a power outage at our hosting facility took the service offline Saturday morning at 7:02 AM Pacific Time. At this time, all Hushmail services are unavailable.

Update 9:48 AM Pacific Time: After becoming aware of the outage Saturday morning, our network operations team identified that our hosting facility was offline as the result of a power outage. We're looking into why backup power systems didn't prevent the service outage, and preparing to bring Hushmail services back online as soon as power is restored.

Update 10:55 AM Pacific Time: Power has been restored at our hosting facility, and we're now working to bring services back online. At this time, the Hushmail website is online, but all email access is still unavailable (webmail, POP/IMAP, Hushmail for iPhone).

Update 11:34 AM Pacific Time: Email access has been restored for many customers, but inbound email delivery will be delayed for the time being. We're now working to finish restoring access for all remaining customers, and to eliminate email delivery delays.

Update 1:15 PM Pacific Time: All customers can now sign in to webmail, POP/IMAP, and Hushmail for iPhone. Some customers may continue to experience inbound email delivery delays for the next hour.

Summary of events:

At 1:49 AM PST, there was a power outage at our hosting facility and the building’s generator started operating normally. However, at 6:15 AM PST, the building generator shut down unexpectedly, due to a failure in the generator's fuel management system.  With the generator out, the data center’s chillers shut down.  By 6:55 AM PST, the data center reached critical high temperatures, which caused the UPS devices to fail. The data center was left with no power sources and all services went offline. The power at the data center was restored at 8:36 AM PST. Hushmail technicians began a sequenced restart of all our servers in the data center. All Hushmail services were fully operational by 11:58 AM PST.