Unscheduled Outage: Node 15 Phoenix

This outage has been resolved and service fully restored.

Around 7 AM CDT (GMT-5), we received alerts that Node 15 Phoenix went offline. Upon investigating, we were unable to determine the cause or nature of the issue as all services appeared to be online and functioning properly at the time. Minecraft servers were accessible and responding. Over the next 5 hours, our team worked to identify the cause of this issue, noting occasional network flukes and other strange occurrences. We began to notice a large number of “connection refused” and “timed out” errors – errors typical of large DDoS attacks. Upon further investigation, we were able to determine that no such attack was taking place.

If you believe this downtime was detrimental to your server/community, please contact our support team to receive compensation immediately.

— Updates —

[12:15 PM] Upon performing a hardware reboot, the server failed to come back online. NOC staff at our datacenter has been dispatched to intervene and bring the system back online.

[2:00 PM] The NOC staff at our Phoenix datacenter have responded with the cause, and provided us with additional details. We’re awaiting IPMI / KVM access to restore service.

[3:00 PM] The NOC has provided us with further information regarding the issue at hand.
The kernel panic was caused by a bad kernel update. We’ve rolled back to an earlier kernel and the server is now online.
Customers on the IPs – 150 may experience packet loss at this time due to routing issues within our Phoenix facility. This will be resolved as soon as possible.

[5:45 PM] IP routing conflicts have been resolved. Service to Node 15 Phoenix has been fully restored at this time.