8:45am Pacific / 4:45pm GMT — Our monitoring has detected slower than normal message processing in our California and Amsterdam datacenters. This may delay some inbound messages, primarily for some North American customers. Our operations team is investigating now and believes this is related to an increase in load; we will publish updates here as more information becomes available.
17:30 GMT / 09:30 Pacific – Our alerting systems have notified us of an issue with one outbound mail handling server within our North American datacentre. This is currently being worked on by our operations team to restore full functionality for this server. The issue is causing all mail which is stored on this server (quarantined and queued mail) to present an error message within the console if viewed, and this mail will not release until the servers functionality is restored. This will only impact a small number (up to 5%) of outbound messages for non EU customers. We will post further updates to this site when they become available
6:20AM Pacific / 2:20PM GMT — Our engineering team has detected an issue affecting control panel access and is investigating. This issue may impact the ability of some customers to log in and navigate the control panel. Some email may be briefly delayed as well. Thank you for your patience as we work to continue improving the service.
12:15PM Pacific / 8:15PM GMT — Our monitoring picked up some emerging problems with one of our database servers. This may contribute to message delivery times, as well as reducing the responsiveness of certain control panel features such as the Message Delivery Logs and Quarantines.
Our engineering team is investigating further. We will post updates here as soon as more information is available.
11:45AM Pacific / 6:45PM GMT — Microsoft’s engineers have confirmed that the fix for their problem displaying our digests and other similar messages has been deployed, and this issue should be fully resolved. To their knowledge, no future messages should be affected.
Please contact our support team as well as Microsoft support if the issue persists or recurs.
8:00AM Pacific / 3:00PM GMT — We have verified that this spike in load was indeed the result of slow responses from one of our spam/virus detection engines. We have temporarily circumvented that engine and restarted impacted servers at our AMS1 datacenter while we investigate further, and our monitoring confirms that load has returned to normal.
11:45AM Pacific — Microsoft confirmed that the digest display problem is the result of a recent update to their systems which introduced a compatibility issue within the component responsible for processing email attachments. Their engineers have developed a fix and are working on deployment; they could not issue a definitive ETA, but cautioned that the deployment process may take an “extended” period of time. We have pressed for more definitive information and they have committed to an informational update by late afternoon Pacific time. We are also investigating possible changes in the encoding of the digests which could help. In the meantime, Microsoft believes that only a small number of customers are impacted by this, and we can confirm that at least some customers using Office 365 are indeed unaffected.