Issue with Greylisting at Amsterdam Datacenter (Now Resolved)

5:15pm Pacific / 1:15am GMT — Approximately an hour ago, our engineering team identified a problem with the greylisting service that was impacting some inbound messages to the Amsterdam datacenter for those customers that have greylisting enabled. The issue was a result of the removal and introduction of new database clusters in the Amsterdam facility. The mail handling servers were still querying one of the older database servers for greylisting data, resulting in multiple greylisting deferrals in that datacenter rather than just one. (Greylisting in the California and Germany datacenters was not affected.) We have validated that the issue is now resolved, and it should not recur.

Update on Issues at Amsterdam Datacenter

10:00am Pacific / 6:00pm GMT —  The engineering team has been in the process of troubleshooting the issues at the Amsterdam datacenter, which seem primarily due to issues at the database tier.   The database issues have been  substantially mitigated, and we are seeing fewer errors at the control panel.   Inbound message processing has also improved, and the team is continuing to work on this.    Message processing at the California and Germany datacenters is operating normally.  We will post further updates here as more information becomes available.

Control panel proxy errors

6:20AM Pacific / 2:20PM GMT — Our engineering team has detected an issue affecting control panel access and is investigating.  This issue may impact the ability of some customers to log in and navigate the control panel.  Some email may be briefly delayed as well.  Thank you for your patience as we work to continue improving the service.

Update: Database Load in Amsterdam Datacenter

1120 Pacific / 1920 GMT – The mail operations team is continuing to work to remediate a database performance issue in the Amsterdam datacenter, and are working on performance improvements. In the meantime, delays in mail delivery may be experienced.

Updates will be posted as they occur.

Thank you for your patience as we work to improve the ControlEmail experience.

Database Load in Amsterdam Datacenter

7:45am Pacific / 3:45pm GMT — Our operations team is currently investigating high load on some of our database servers in the Amsterdam datacenter.  The load is diminishing and there may not be any customer impact, but it is possible that some messages in that datacenter may be processed more slowly than normal.  We will continue to post updates here as necessary.