11:45AM Pacific / 6:45PM GMT — Microsoft’s engineers have confirmed that the fix for their problem displaying our digests and other similar messages has been deployed, and this issue should be fully resolved. To their knowledge, no future messages should be affected.
Please contact our support team as well as Microsoft support if the issue persists or recurs.
8:00AM Pacific / 3:00PM GMT — We have verified that this spike in load was indeed the result of slow responses from one of our spam/virus detection engines. We have temporarily circumvented that engine and restarted impacted servers at our AMS1 datacenter while we investigate further, and our monitoring confirms that load has returned to normal.
7:00am Pacific / 2:00pm GMT — Our monitoring has detected a high load on a number of mail handling servers at the Amsterdam datacenter. Preliminary indications are that the issue is being caused by slow responsiveness of one of the anti-virus engines used in the filtering. Our operations team is investigating and we will post updates here as they become available.
9:45am Pacific / 4:45pm GMT — Per our previous posts, Microsoft has been in the process of deploying at Office 365 a bug fix that is expected to resolve the issue in displaying certain messages including the junk mail digests. Microsoft is now estimating that the deployment will be complete by the end of the day on Friday.
8:30am Pacific / 3:30pm GMT — The removal of the problematic database cluster from production has resulted in stabilization of message processing, which has been operating at normal levels over the last hour. We will continue to monitor the system and will post updates here in the event of any changes.
6:45am Pacific / 1:45pm GMT — Our operations team has removed the problematic database cluster from production, and message processing is returning to normal. We will continue to post updates here.
5:45am Pacific / 12:45pm GMT — Our operations team has detected a database performance issue that has been impacting message processing in the Amsterdam database and resulting in delays for some inbound messages. The team is currently in the process of taking the relevant database cluster out of production to address the issue. We will post updates here as work continues on this.