As many know, on Sunday morning we switched to a new database server, which can handle significantly higher traffic, has automatic failover, and includes many additional benefits (including faster load times).
Around 10am PDT, at first indication so far, our incoming email parsing got a significantly higher volume than normal, which caused the database to slow down. This didn’t affect the site, but as we were trying to debug this, the database triggered its failover. The failover itself worked fine, but the database came back in read-only - which overloaded the application servers. Once we found out about the read-only issue (note this is a new database), we were able to quickly restart the application servers, which resolved the issue.
We are still working to determine why the database slowed down due to the higher volume, and we will post a more detailed post-mortem when available.
Posted Aug 07, 2018 - 11:46 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Aug 07, 2018 - 11:37 PDT
Identified
Potential Root cause identified. Further investigation in process. Site is working correctly for most members, but we are continuing to monitor for continued stability.
Posted Aug 07, 2018 - 11:30 PDT
Investigating
Under investigation, potential root cause identified. We are continuing to monitor and will provide updates as they are available.