Downtime
NLSHARED5 Degraded
Started 16 Feb at 02:11pm EST, resolved 21 Feb at 10:53am EST.
Resolved
At this time we have restored all backups.
Updated
The initial restore process has been completed.
We're now going through partial or failed restores, and restarting the restore process from a previous restore point.
Updated
Roughly 76% of the way through the restoration.
Updated
We're still at roughly ~65%. We are tracking the progress, it is not hung - the restore is per-account, and it's been going over some rather large accounts.
Updated
The restore is at 60%.
Updated
Up to 40% at this point.
Updated
20% of accounts have been restored up to this point.
We will post further updates on 18 Feb 02:00 UTC
Updated
10% of accounts have been restored up to this point.
We will post further updates at 21:00 UTC.
Updated
The server is now ready, and restores from yesterday's backup have started.
Updated
Server provisioning is still ongoing. We will post an update as soon as accounts restorations begin.
Updated
We're currently performing the initial server setup. Accounts will start restoring shortly.
We will share more information in one hour again.
Updated
After a catastrophic RAID failure, data on nlshared5 has been corrupted. We're preparing to rebuild the server from scratch and restore from our latest backups.
There's currently no ETA, but it will definitely take a few hours to complete.
We will share more information in one hour again.
Updated
We are taking the server offline to prevent any additional complications. We will update as soon as a new server has been stood up and restored from the most recent nlshared5 backup.
Updated
nlshared5 has been restored. Continuing the instability we are still working on an emergency migration to another host.
Updated
The host instability has become even more severe. We are continuing work on relocating NLSHARED5 to another node.
Updated
We are observing the anomalous behavior on nlshared5's host again.
Updated
We have recovered the host and will continue to monitor the situation.
Created
We began observing disruptions to the hardware nlshared5 is on. We are working to restore services.