The increase in traffic this afternoon at around 4PM UTC caused processing load on the front end and database servers to increase to the point where requests for the console and signup form submissions was not able to be handled. The cause of the increase has yet to be fully investigated but initial findings point to a Anti-virus message issue. This caused the monitoring file uploads to increase to a point beyond the servers capacity to handle it.
To fix this problem the Development Operations team:
- Restarted the database on the back end servers
- Restarted Apache on the front end servers
- Removed unprocessed and processed monitoring files
- Optimized the Anti-virus tables involved in the messaging
We will be reviewing the servers we have in the US looking for upgrade opportunities over the next week. We will also implement a new monitoring file handling mechanism for the following release which will increase our ability to cope with this kind of issue.
We sincerely apologize for any inconvenience this may have caused.