Litespeed is now fully tested & operational, server side caching is available. The cluster is fully operational & stable.
An overview of this incident & the resolution will be posted here within 3 days.
The canada300 cluster has been stable today, sysadmins have been performing additional testing throughout the day.
Litespeed will be enabled during a maintenance window of OCT. 24 - 4AM - 4:30AM ET
It possible that short outages will occur during this time, to allow for a restart & testing to ensure no cache corruption.
The results of that testing will be posted here tomorrow morning by 10AM ET.
The issue on canada300 is now resolved, however our sysadmin team will be performing additional testing during a maintenance window between 3-5AM ET. It's possible that short outages will occur during this period. Please note that this will only affect the canada300 cluster.
Full transparency, we're seeing blank front page issues with approx. 10 sites, restoring the public_html is resolving that issue. We have those restores rolling out live now.
We're now going through the server, site by site & will be logging any initial issues we see. Those are will be handed over to a sysadmin to help troubleshoot anything additional that is happening with this cluster. This cluster serves 215 sites & approx. 25% of that number were affected.
Continuing to check for corrupted files.
Sites are coming back online, file checks are underway, checking for corrupted files.
We're seeing sites come back online, awaiting feedback from sysadmins on the file system health & next steps so that we can provide you with more details on what to expect.
The investigation has shown that the issue could be caused by a file system error, as the cluster is only restarting in read only mode.
We're now attempting to perform a file system check & repair, that process can take time. Updates will be made here.
We also have plans in place if necessary to start failover procedures to another data centre failover server.
We have data centre staff actively investigating, we should have an eta very soon.
No ETA at this point, they are still investigating at the data centre.
We're currently experiencing what appears to be a network issue on the canada300 cluster. We're investigating, more details here shortly.