Back
Downtime

NLSHARED5 Degraded

Started 16 Feb at 02:11pm EST, resolved 21 Feb at 10:53am EST.

nlshared5
Resolved

At this time we have restored all backups.

Posted 21 Feb at 10:53am EST.
Updated

The initial restore process has been completed.
We're now going through partial or failed restores, and restarting the restore process from a previous restore point.

Posted 18 Feb at 09:52am EST.
Updated

Roughly 76% of the way through the restoration.

Posted 18 Feb at 03:00am EST.
Updated

We're still at roughly ~65%. We are tracking the progress, it is not hung - the restore is per-account, and it's been going over some rather large accounts.

Posted 18 Feb at 02:22am EST.
Updated

The restore is at 60%.

Posted 18 Feb at 12:27am EST.
Updated

Up to 40% at this point.

Posted 17 Feb at 09:55pm EST.
Updated

20% of accounts have been restored up to this point.

We will post further updates on 18 Feb 02:00 UTC

Posted 17 Feb at 06:34pm EST.
Updated

10% of accounts have been restored up to this point.

We will post further updates at 21:00 UTC.

Posted 17 Feb at 02:11pm EST.
Updated

The server is now ready, and restores from yesterday's backup have started.

Posted 17 Feb at 09:06am EST.
Updated

Server provisioning is still ongoing. We will post an update as soon as accounts restorations begin.

Posted 17 Feb at 07:03am EST.
Updated

We're currently performing the initial server setup. Accounts will start restoring shortly.

We will share more information in one hour again.

Posted 17 Feb at 05:31am EST.
Updated

After a catastrophic RAID failure, data on nlshared5 has been corrupted. We're preparing to rebuild the server from scratch and restore from our latest backups.
There's currently no ETA, but it will definitely take a few hours to complete.

We will share more information in one hour again.

Posted 17 Feb at 04:32am EST.
Updated

We are taking the server offline to prevent any additional complications. We will update as soon as a new server has been stood up and restored from the most recent nlshared5 backup.

Posted 17 Feb at 12:11am EST.
Updated

nlshared5 has been restored. Continuing the instability we are still working on an emergency migration to another host.

Posted 16 Feb at 10:41pm EST.
Updated

The host instability has become even more severe. We are continuing work on relocating NLSHARED5 to another node.

Posted 16 Feb at 10:24pm EST.
Updated

We are observing the anomalous behavior on nlshared5's host again.

Posted 16 Feb at 09:05pm EST.
Updated

We have recovered the host and will continue to monitor the situation.

Posted 16 Feb at 04:34pm EST.
Created

We began observing disruptions to the hardware nlshared5 is on. We are working to restore services.

Posted 16 Feb at 02:11pm EST.