Phoenix Network Connectivity Issues: August 12

Early this morning, the CubedHost operations team was made aware of network connectivity issues affecting customers with servers in our Phoenix, AZ datacenter. Since then, our team has been in continual communication with the datacenter, working to get the issues resolved.

We have confirmed that only a portion of our Phoenix servers are affected. If you’re located on Phoenix Node 1, 5 – 7, 9 – 13, 15, 21, 22, or 25, your server should not be affected. All other Phoenix Minecraft nodes are likely experiencing lag and/or intermittent connection drops.

We’ll keep this post and our Twitter updated with the latest news regarding these issues.

— Updates —

[9:00 AM CDT] Receiving reports of lag affecting Phoenix Minecraft servers. We’ve begun an investigation and contacted the datacenter for more information. The cause of the lag has been identified as 10%+ packet loss and increased latency to some Phoenix Minecraft nodes.

[10:00 AM CDT] Team is in contact with the datacenter whose network operations team is investigating the packet loss.

[3:00 PM CDT] Network operations team at the datacenter is currently investigating the issue and has identified a routing issue that may have been to blame. After working around the routing issue, the problem still persists.

[6:00 PM CDT] Our team is still working with the datacenter in an effort to isolate the source of the problem and get it resolved.

[11:00 PM CDT] Datacenter networking team has scheduled an emergency maintenance to address the connectivity issues affecting a subset of their customers. This is scheduled for 4 – 6 AM CDT (GMT-5) and should completely resolve the aforementioned issues.

[12:45 PM CDT] The emergency maintenance has been pushed up to 1 AM CDT (GMT-5) / 11 PM MST (GMT-7), to begin approx. 15 minutes from now. More detail will follow as the maintenance progresses.

[2:50 AM CDT] Affected services in Phoenix have gone offline while routing changes are implemented.

[3:00 AM CDT] Service has been restored.  Latency has returned to normal. Zero packet loss. All services appear healthy.