Resolution of Message Processing

6:30am Pacific / 2:15pm BST — An issue seen earlier this morning with a third-party partner has been resolved, and all datacenters are now processing mail normally.  We are working with the partner to determine the root cause, and we are also working on implementing better means of mitigating the risk of any future operational issues with this partner.

Update on Message Processing

1:00 PM Pacific / 9:00 PM BST —  General inbound and outbound message processing continues to be stable and operating at normal levels in all datacenters.   We are seeing some unusual behavior with journaling messages specifically from Office 365 to some of our systems for the archive; our engineers are currently investigating this.

Update on Message Processing

7:45pm Pacific /  3:45 AM BST —  Message processing is operating normally in both the Amsterdam and California datacenters, and has been stable for the last several hours.  Our engineers have identified a primary culprit and fix for the issues in Amsterdam, and are in the process of implementing additional systems in California to mitigate the risk of a recurrence.  We will continue to monitor and will post any significant updates here.

Resolution of Message Processing at Amsterdam

3:00pm Pacific /  11:00 PM BST —  Our engineers have identified and resolved a problematic database server that contributed to the message processing issues in Amsterdam.   Mail flow in that facility has improved and is now operating at normal levels.   Additionally, our engineers are working to improve performance at the California datacenter; that work will be progressing over the next several hours.   We will continue to post updates here.

Update on California and Amsterdam Datacenters

1:15pm Pacific / 9:15 PM BTC — Our engineers are continuing to work on the issues affecting customers experiencing delays with inbound messages sent through the California and Amsterdam datacenters. Outbound message processing and other datacenters are not affected.  We will post updates as further information becomes available.

Message Processing at California and Amsterdam Datacenters

7:00 AM Pacific /  3:00 PM BST —  Our operations team is investigating slower than normal message processing with some of our message handling servers, primarily at the California datacenter and secondarily at the Amsterdam datacenter.   The team is working to identify a culprit and resolve the issue as quickly as possible.   We will post further updates here.

Possible Routing Issues in California

10:30am Pacific / 6:30pm BST – Our engineers are investigating a handful of mail handling servers in the California datacenter that are responding slowly, possibly related to a brief BGP/routing issue at that facility earlier this morning.  We will post further updates here.