Resolution of Networking Issue

10:30am Pacific / 6:30pm GMT – The networking issue within the Amsterdam datacenter has been largely resolved, and our engineers have updated the routing within that datacenter to circumvent any remaining issues.  Our monitoring shows that message processing has returned to normal within that facility; we will continue to monitor and will take any further action necessary.  Other datacenters were unaffected by the issue.

Networking Issue at Amsterdam Datacenter

10:00am Pacific / 6:00pm GMT – Our monitoring has detected a networking issue within the Amsterdam datacenter that may be impacting message processing within that facility.  That datacenter is used primarily for backup (lower MX priority) traffic, but some primary traffic is also routed through that datacenter.  We are in contact with the networking team at that datacenter, which is working to resolve the issue.  We will post further updates here.  Other datacenters are unaffected.

Resolution of Message Processing

6:30am Pacific / 2:15pm BST — An issue seen earlier this morning with a third-party partner has been resolved, and all datacenters are now processing mail normally.  We are working with the partner to determine the root cause, and we are also working on implementing better means of mitigating the risk of any future operational issues with this partner.

Update on Message Processing

1:00 PM Pacific / 9:00 PM BST —  General inbound and outbound message processing continues to be stable and operating at normal levels in all datacenters.   We are seeing some unusual behavior with journaling messages specifically from Office 365 to some of our systems for the archive; our engineers are currently investigating this.

Update on Message Processing

7:45pm Pacific /  3:45 AM BST —  Message processing is operating normally in both the Amsterdam and California datacenters, and has been stable for the last several hours.  Our engineers have identified a primary culprit and fix for the issues in Amsterdam, and are in the process of implementing additional systems in California to mitigate the risk of a recurrence.  We will continue to monitor and will post any significant updates here.

Resolution of Message Processing at Amsterdam

3:00pm Pacific /  11:00 PM BST —  Our engineers have identified and resolved a problematic database server that contributed to the message processing issues in Amsterdam.   Mail flow in that facility has improved and is now operating at normal levels.   Additionally, our engineers are working to improve performance at the California datacenter; that work will be progressing over the next several hours.   We will continue to post updates here.

Update on California and Amsterdam Datacenters

1:15pm Pacific / 9:15 PM BTC — Our engineers are continuing to work on the issues affecting customers experiencing delays with inbound messages sent through the California and Amsterdam datacenters. Outbound message processing and other datacenters are not affected.  We will post updates as further information becomes available.