Category Archives: Maintenance & Outages

Unscheduled Maintenance on Node 29 France

[11:22 PM] All servers are back online. If you have any further issues, please submit a ticket.

[11:11 PM] Node is back online. Servers coming up in a few moments…

[10:48 PM] It’s taking a bit longer than expected. We hope to have the system back up shortly.

 = = = = =

Hello all, sorry to cause a bit of inconvenience tonight. Node 29 France is currently undergoing a bit of emergency maintenance to restore full functionality to our Multicraft daemon. All Minecraft servers will restart multiple times and may be offline for up to 10 – 15 minutes, granted there are no unforeseen issues. We’ll keep you updated.

Node 1 / 15 Arizona Data Loss

It is with great regret that we inform our customers that data from between 12 PM and 12 AM CDT on Oct. 5/6th on nodes 1 and 15 Arizona has been lost. Due to simultaneous hardware failures and other unforeseen circumstances, we were forced to wipe and re-install these systems completely, restoring data from our last available backup: Oct. 5th @ 12 PM CDT.

We understand the implications of this and sincerely apologize for both the overall inconvenience and loss of productivity. If you feel that this has greatly impacted your server, please don’t hesitate to submit a support ticket to receive compensation or answers to any questions you may have.

Scheduled Server Maintenance – October 5th @ 11PM CDT

If you are seeing a “FAILED TO BIND TO PORT” error in your server’s console, please submit a support ticket.

=== Updates ===

[7:22 PM CDT] Node 15 AZ has been brought back online! If you were affected by this extended downtime, please submit a ticket and our team would be glad to compensate you.

[7:00 PM CDT] Node 1 AZ is back online! If you were affected by this extended downtime, please submit a ticket and our team would be glad to compensate you. Node 15 maintenance is still underway with an expected startup within 30 minutes.

[6:50 PM CDT] Node 1 and 15 have undergone OS reinstallation and data will be recovered from our backup system. We expect to have these nodes back online within the next 1 – 2 hours, granted no other issues arise.

[12:00 PM CDT] Node 1 AZ still under goes recovery after a dual hard drive failure. Node 15 AZ was brought back online but seems to be suffering from an unknown hardware failure. We will keep investigating these issues and report back with any further information once available.

[3:45 AM CDT] Node 21 AZ has been brought back online. A hard drive replacement will be scheduled within the next 24 hours with brief downtime. Node 1 AZ has received a new hard drive and is currently rebuilding data; once complete, the second drive will be replaced and all servers started up thereafter. Node 15 AZ is down, awaiting a response from the datacenter.

[3:15 AM CDT] Node 1, 15, 21 Arizona are currently undergoing continued maintenance while we investigate possible hardware failures. All other locations and nodes have been successfully updated at this time – no further downtime is expected.

[2:45 AM CDT] Canada, Virginia, Australia, and London updates have been completed. Node 15, 21 Arizona have encountered issues during the update process and we are working to manually resolve them; minor hardware failures are a possibility. Node 1 AZ is currently awaiting HDD replacements. France updates are still in progress as well.

[1:30 AM CDT] We’ve run into some issues with Node 1 AZ. We’re in the process of completing the updates and other maintenance across our other nodes, while investigating these issues. Node 1 AZ may experience prolonged downtime while we address what may be a hardware failure; other nodes should see no more than brief downtime during the update process.

===

We have scheduled a maintenance for October 5th, 2013 at 11PM CDT. This maintenance will incur a short amount of downtime while we update our system software, and improve upon our early failed-hardware detection.

The following services will be affected:
* Minecraft – Phoenix, Arizona
* Minecraft – Ashburn, Virginia
* Minecraft – London, UK
* Minecraft – Roubaix, France
* Minecraft – Quebec, Canada
* Minecraft – Sydney, Australia

In addition, the Multicraft panel corresponding to your location may become unavailable for up to 2 hours while these updates are taking place.

The downtime is estimated to be between 15 minutes and 120 minutes (2 hours).
As always, unless unforeseen issues appear during the upgrade/update process, we will try our best to remain to this time frame.

Thank you for your patience.

London, UK Network Routing Issues

Update: Latency appears to have returned to normal, we’ll keep an eye on it over the next few hours and update this post if anything changes.

At around 17:00 UTC (12:00 PM CDT) today part of our UK network began experiencing severe routing issues causing very high latency and packet loss. This is caused by an error between a global network carrier and our datacenter; these types of problems must resolve themselves over the next few hours. Once the servers are fully accessible again, we’ll update this post.

The following systems are affected:

  • Node 8 London, UK (node8.ln.uk.mc.cubedhost.org)
  • London, UK SQL (ln.uk.sql.cubedhost.org)

France Multicraft Maintenance

Update: The maintenance has been completed. Your server’s name, jar file, world name, and some other misc. settings may have been reverted/changed. If you have any trouble accessing your Multicraft panel, please submit a support ticket.

We are currently performing a maintenance on our France Multicraft panel and MySQL over the next 24 hours. If your server is inaccessible, simply submit a support ticket and our team can get you squared away ASAP.

USA Maintenance – Sept. 4th/5th

[4:45AM CDT] Update: Node 13 Arizona is now back online. Performance on this system should be significantly better, as the failing motherboard has been replaced. Thank you for your patience.

[1:12AM CDT] Update: The USA software update maintenance has been completed successfully, with less than 5 minutes of downtime per node. If you have trouble connecting to your server, please submit a support ticketNode 13 AZ is still undergoing hardware maintenance. We’ll keep you updated on that situation.

—————

As we noted in our previous post last week, we’ve scheduled a planned maintenance to replace the second CPU in US Node 13 AZ (Phoenix, Arizona). The downtime may be up to 1 – 2 hours, but hopefully it will be quick.

In addition, we have an additional planned, USA network-wide maintenance as noted in an email sent out in the past 24 hours. Downtime should be fairly brief as we’re only doing minimal software updates. These downtimes will be rolling, starting with Node 1 AZ and ending with Node 8 VA.

Both maintenances start at 12AM CDT (Sept. 5th). We’ll keep you updated with the details.
 

Node 8 AZ SYN Flood

Update: It appears the attack has finished. If it begins again, we’ll update you.

We recently discovered a continual SYN flood DDoS attack on Node 8 Phoenix, Arizona. This attack is only affecting the Multicraft daemon itself, and not any Minecraft servers. While this attack goes on, you may be unable to control your server via our control panel; however, your players should be able to connect and play normally. We’ll keep you updated as the situation progresses.

Note: Our node status page will show Node 8 AZ as offline – this is incorrect.

Node 13 Phoenix, Arizona Hardware Replacement

A maintenance to resolve this issue completely has been scheduled for next Thursday, September 5th at 12AM CDT. A new blog post will be made about this event once we get closer to that night.

——-

We recently discovered that Node 13 AZ, a dual E5-2620 system, has a single failed processor; in order to resolve this issue, we’ve taken the node offline and are allowing our datacenter’s NOC to investigate. Once the server is back online and all servers are accessible, we will let you know.

[1:45am CDT] Update: The server was brought back online after around 2h to await further actions from the NOC.

[10:00am CDT] Update: The NOC has taken the server offline to perform the requested maintenance.

[5:25pm CDT] Update: The node is back online and all servers should be accessible again. If you were affected by this downtime, please submit a support ticket to receive compensation. We have scheduled a planned maintenance for September 5th @ 12AM CDT to replace the failed CPU. During this time, your server will be offline for up to 2 hours.

France SQL Offline

(Update) France MySQL services have come back online. It appears the issue was related to network routing and has been resolved.

Around 12:30pm CDT, the France SQL server went offline due to unknown reasons. We’re currently working with the datacenter to get it back online as soon as possible. In the meantime, any servers located in France may experience issues with plugins and server instability. Once we have more information, we’ll update this post.

Node 3 Sydney, Australia Downtime (round 3)

(UPDATED) Please use the IP 27.50.72.60 instead of 27.50.80.218; just add your server’s port number to the end. You do NOT need to set your server to offline mode, as this is your new server IP

As you may know already, Node 3 AU is currently inaccessible again. Unfortunately, as before, we have no control over this and we are at the mercy of our Australian datacenter. We’re actively working with their team to get the node back online and accessible to everyone, but in the meantime, you may not be able to connect or join your Minecraft server(s).

We’re extremely disappointed in the handling of this situation on our datacenter’s side, thus we’re looking into moving over to a different company. We’ll keep everyone updated on the status of this issue, along with any other information that may be of use.

Current StatusOnline (with a new IP)