0510 PDT / 1310 BST – Our engineers are currently troubleshooting a recurrence of the issue with the disk subsystem on the I-3 mail server, which is used for their email hosting by a small number of MAX Mail customers.
The http://mail.i-3.com service and control panel access will be unavailable from 0800 EDT/1300 BST on 23/05/2016 until 1200 EDT / 1700 BST on 23/05/2016 for essential maintenance.
During this time, I-3 mail customers will not be able to send or receive mail through the I-3 mail server directly, but no messages will be lost, and those customers can use the continuity feature within the MAX Mail control panel (http://mail.maxfocus.com) to both review any incoming messages in queue and to send new messages as necessary. We will update the status here with any updates as they become available.
10:30am Pacific / 6:30pm GMT – The networking issue within the Amsterdam datacenter has been largely resolved, and our engineers have updated the routing within that datacenter to circumvent any remaining issues. Our monitoring shows that message processing has returned to normal within that facility; we will continue to monitor and will take any further action necessary. Other datacenters were unaffected by the issue.
10:00am Pacific / 6:00pm GMT – Our monitoring has detected a networking issue within the Amsterdam datacenter that may be impacting message processing within that facility. That datacenter is used primarily for backup (lower MX priority) traffic, but some primary traffic is also routed through that datacenter. We are in contact with the networking team at that datacenter, which is working to resolve the issue. We will post further updates here. Other datacenters are unaffected.
6:30am Pacific / 2:15pm BST — An issue seen earlier this morning with a third-party partner has been resolved, and all datacenters are now processing mail normally. We are working with the partner to determine the root cause, and we are also working on implementing better means of mitigating the risk of any future operational issues with this partner.
1:00 PM Pacific / 9:00 PM BST — General inbound and outbound message processing continues to be stable and operating at normal levels in all datacenters. We are seeing some unusual behavior with journaling messages specifically from Office 365 to some of our systems for the archive; our engineers are currently investigating this.
7:45pm Pacific / 3:45 AM BST — Message processing is operating normally in both the Amsterdam and California datacenters, and has been stable for the last several hours. Our engineers have identified a primary culprit and fix for the issues in Amsterdam, and are in the process of implementing additional systems in California to mitigate the risk of a recurrence. We will continue to monitor and will post any significant updates here.
3:00pm Pacific / 11:00 PM BST — Our engineers have identified and resolved a problematic database server that contributed to the message processing issues in Amsterdam. Mail flow in that facility has improved and is now operating at normal levels. Additionally, our engineers are working to improve performance at the California datacenter; that work will be progressing over the next several hours. We will continue to post updates here.