5:15pm Pacific / 1:15am GMT — Approximately an hour ago, our engineering team identified a problem with the greylisting service that was impacting some inbound messages to the Amsterdam datacenter for those customers that have greylisting enabled. The issue was a result of the removal and introduction of new database clusters in the Amsterdam facility. The mail handling servers were still querying one of the older database servers for greylisting data, resulting in multiple greylisting deferrals in that datacenter rather than just one. (Greylisting in the California and Germany datacenters was not affected.) We have validated that the issue is now resolved, and it should not recur.
11:45 AM Pacific / 7:45 PM GMT — The Amsterdam datacenter has been operating at normal levels for the last two hours, and appears stable. Our operations team is continuing to monitor, and will take any further actions necessary.
10:00am Pacific / 6:00pm GMT — The engineering team has been in the process of troubleshooting the issues at the Amsterdam datacenter, which seem primarily due to issues at the database tier. The database issues have been substantially mitigated, and we are seeing fewer errors at the control panel. Inbound message processing has also improved, and the team is continuing to work on this. Message processing at the California and Germany datacenters is operating normally. We will post further updates here as more information becomes available.
6:20AM Pacific / 2:20PM GMT — Our engineering team has detected an issue affecting control panel access and is investigating. This issue may impact the ability of some customers to log in and navigate the control panel. Some email may be briefly delayed as well. Thank you for your patience as we work to continue improving the service.
04:45 Pacific / 12.45 GMT – The operations team has resolved the database performance issue, and is continuing to monitor the speed of the service.
1120 Pacific / 1920 GMT – The mail operations team is continuing to work to remediate a database performance issue in the Amsterdam datacenter, and are working on performance improvements. In the meantime, delays in mail delivery may be experienced.
Updates will be posted as they occur.
Thank you for your patience as we work to improve the ControlEmail experience.
7:45am Pacific / 3:45pm GMT — Our operations team is currently investigating high load on some of our database servers in the Amsterdam datacenter. The load is diminishing and there may not be any customer impact, but it is possible that some messages in that datacenter may be processed more slowly than normal. We will continue to post updates here as necessary.