Console slow in US during agent updates

We are currently experiencing a high amount of load in the US after the release of the 1.18 console. This could make the console slow and even unresponsive for short periods. This is due we believe to the size of this agent release which is stressing our infrastructure as agents update automatically.

We are currently in contact with DevOps, monitoring the situation and working through the issues. We apologize for any inconvenience caused.

Scheduled maintenance in ANZ

Digital Pacific will be performing an urgent upgrade of core networking equipment within their data center during the maintenance window outlined below. During this time, there will be a 10 minute network service interruption for all GFI Cloud customers on the dashboard. This is to upgrade core networking equipment.

Maintenance Start Date & Time:

  • Sydney (Australia – New South Wales) Wednesday, 29 January 2014, 04:30:00 AEDT UTC+11 hours
  • Dundee (United Kingdom – Scotland)   Tuesday, 28 January 2014, 17:30:00   GMT  UTC

Maintenance End Date & Time (AEDT):

  • Sydney (Australia – New South Wales) Wednesday, 29 January 2014, 05:00 AEDT UTC+11 hour
  • Dundee (United Kingdom – Scotland)   Tuesday, 28 January 2014, 18:00:00   GMT  UTC

Status: Scheduled
Maintenance Type: Network Upgrade
Expected Downtime: 10 Minutes

We apologize for any inconvenience this may cause.

US territory back to normal

The increase in traffic this afternoon at around 4PM UTC caused processing load on the front end and database servers to increase to the point where requests for the console and signup form submissions was not able to be handled. The cause of the increase has yet to be fully investigated but initial findings point to a Anti-virus message issue. This caused the monitoring file uploads to increase to a point beyond the servers capacity to handle it.

To fix this problem the Development Operations team:

  • Restarted the database on the back end servers
  • Restarted Apache on the front end servers
  • Removed unprocessed and processed monitoring files
  • Optimized the Anti-virus tables involved in the messaging

We will be reviewing the servers we have in the US looking for upgrade opportunities over the next week. We will also implement a new monitoring file handling mechanism for the following release which will increase our ability to cope with this kind of issue.

We sincerely apologize for any inconvenience this may have caused.

Unusual increased traffic load in the US

We are experiencing unusual increased traffic to the US instance of GFI Cloud, causing increased system and network load. At this point we have no indication that this is malicious, however we are investigating into that direction as well. This is causing the console and signup forms to perform poorly or not at all.

We are still investigating but hope to have the issue resolved soon.