Connectivity issues affecting US North data centre

Incident Report for Global Alerting Platform (GAP)

Postmortem

Microsoft has provided a root cause analysis for the service interruption experienced on 06/11/2017 between 19:33 UTC and 21:07 UTC which resulted in connectivity problems affecting the US North data centre.

The interruption was caused by a problem that arose during a planned maintenance activity to increase capacity within the data centre. Microsoft’s monitoring systems detected the problem however automatic rollback of the configuration change did not restore connectivity. Following further investigations, a manual rollback was performed by Microsoft's engineers and service was then restored.

As a result of this incident, Microsoft will:

Model future routing policy changes using their own internal tools
Update their verification processes for routing policy changes to include checks from other regions

Posted Nov 16, 2017 - 09:23 UTC

Resolved

Microsoft has identified that a network configuration change resulted in connectivity issues affecting the US North data centre. They have taken steps to resolve the problem and service has been restored.

Microsoft has informed us that engineers performed a change to the service configuration to BGP (Border Gateway Protocol) to mitigate the issue. They will continue to investigate to establish the full root cause in approximately 72 hours, at which point a postmortem will be provided.

Posted Nov 06, 2017 - 21:33 UTC

Monitoring

We have observed a restoration of connectivity. We will continue to monitor the situation until we hear from Microsoft that the problem has been fully resolved.

Posted Nov 06, 2017 - 21:10 UTC

Identified

Microsoft has advised that an underlying network infrastructure event is responsible for the connectivity issues affecting the US North data centre. Microsoft engineers are working to resolve the problem as soon as possible.

Posted Nov 06, 2017 - 20:26 UTC

Investigating

We are aware of connectivity issues affecting the US North data centre. We will provide another update as soon as further information becomes available.

Posted Nov 06, 2017 - 20:19 UTC