Category Archives: Maintenance & Outages

Phoenix Network Connectivity Issues: August 12

Early this morning, the CubedHost operations team was made aware of network connectivity issues affecting customers with servers in our Phoenix, AZ datacenter. Since then, our team has been in continual communication with the datacenter, working to get the issues resolved.

We have confirmed that only a portion of our Phoenix servers are affected. If you’re located on Phoenix Node 1, 5 – 7, 9 – 13, 15, 21, 22, or 25, your server should not be affected. All other Phoenix Minecraft nodes are likely experiencing lag and/or intermittent connection drops.

We’ll keep this post and our Twitter updated with the latest news regarding these issues.

— Updates —

[9:00 AM CDT] Receiving reports of lag affecting Phoenix Minecraft servers. We’ve begun an investigation and contacted the datacenter for more information. The cause of the lag has been identified as 10%+ packet loss and increased latency to some Phoenix Minecraft nodes.

[10:00 AM CDT] Team is in contact with the datacenter whose network operations team is investigating the packet loss.

[3:00 PM CDT] Network operations team at the datacenter is currently investigating the issue and has identified a routing issue that may have been to blame. After working around the routing issue, the problem still persists.

[6:00 PM CDT] Our team is still working with the datacenter in an effort to isolate the source of the problem and get it resolved.

[11:00 PM CDT] Datacenter networking team has scheduled an emergency maintenance to address the connectivity issues affecting a subset of their customers. This is scheduled for 4 – 6 AM CDT (GMT-5) and should completely resolve the aforementioned issues.

[12:45 PM CDT] The emergency maintenance has been pushed up to 1 AM CDT (GMT-5) / 11 PM MST (GMT-7), to begin approx. 15 minutes from now. More detail will follow as the maintenance progresses.

[2:50 AM CDT] Affected services in Phoenix have gone offline while routing changes are implemented.

[3:00 AM CDT] Service has been restored.  Latency has returned to normal. Zero packet loss. All services appear healthy.

Node 4 Sydney Hard Drive Swap – June 15th/16th

The hard drive swap has been completed and all servers are back online.

—-

Hello,

Our operations team has identified a failing hard drive in Node 4 Sydney and has coordinated a quick swap with the datacenter to replace the drive.  This is scheduled for 22:00 UTC / 5 PM CDT (GMT-5) June 15th / 8 AM EST (GMT+15). We will be taking a backup immediately prior to the swap in order to maintain complete data consistency and integrity, and to prevent data loss. Downtime is expected to extend about 1 hour.

If you have been experiencing connection, latency, or lag issues on your Sydney server and you’re located on Node 4 Sydney, this should resolve it completely. If you feel that this has negatively impacted your server and would like compensation, please contact our billing team – we would be glad to assist.

If you have any questions, please reach out to our support team.

Regards,

The CubedHost Team

Phoenix Maintenance Updates: June 13th – 16th 2014

All maintenance in Phoenix has been completed at this time.

This post will serve to provide rapid updates regarding any possible downtime for Phoenix-based CubedHost or US-based CraftYourBox customers, from June 13th – June 16th. For more information on this maintenance, please click here.

—————

June 16th – 5:00 MST / 12:00 UTC: Switch upgrades and configuration changes have been completed at this time. Our Phoenix public network has been fully migrated to a new switch. Backend network adjustments will continue throughout the day. No major downtime is expected beyond this point.

June 16th – 3:15 MST / 10:15 UTC: We’re currently performing switch configuration changes which may take around 15 – 30 minutes to fully complete. During this time, service may be inaccessible.

June 16th – 2:45 MST / 9:45 UTC: We are continuing work on swapping affected nodes over to a new switch. Minimal downtime should be expected, for around 2-3 minutes on each node.

June 15th – 12:00 MST / 19:00 UTC: The team will be performing a quick network swap over to a new switch. This will affect a portion of CubedHost Minecraft servers and dedicated servers in Phoenix, AZ. We anticipate downtime of no more than 2-3 minutes. No servers will be restarted during this process.

June 14th – 16:30 MST: All CraftYourBox US nodes online excluding node 30. Upon rebooting the system, the team discovered a hardware failure. We’re working toward a resolution now. This affects only a very small portion of customers.

June 14th – 15:20 MST: CraftYourBox nodes 2 – 9 are back online.

June 14th – 15:00 MST: CraftYourBox US nodes 2 – 25 will be shut down momentarily for a brief hardware swap. We anticipate downtime of no more than 30 minutes.

June 14th – 14:00 MST: All CraftYourBox servers are  back online.

June 14th – 11:00 MST: Routing issues have been resolved and the majority of CraftYourBox US servers should be online again. We’re still investigating routing issues on two nodes.

June 14th – 2:30 MST: We’re continuing to work with our provider to resolve the routing issues affecting some CraftYourBox US servers. This does not affect CubedHost customers.

June 14th – 1:00 MST: CraftYourBox hardware has been migrated. We’re working with our provider to resolve an IP address routing issue preventing servers from coming online.

June 13th – 22:00 MST: CraftYourBox server migrations have begun. CYB nodes will be shut down and brought back online in phases. Our goal is to keep overall downtime to no more than 1 hour.

June 13th – 16:30 MST: Hardware swaps on CraftYourBox nodes 4 and 25 have been completed. Customers on node 25 have been relocated to node 4.

June 13th – 15:30 MST: We have begun swapping out hardware on CraftYourBox nodes 4 and 25.

Phoenix Network Maintenance: June 13th – 16th 2014

All maintenance in Phoenix has been completed at this time.

—-

Hello,

As part of our continuing work to improve and upgrade our Phoenix network, our operations team has scheduled a series of maintenances from June 13th to June 16th 2014. The majority of this time will be spent migrating new hardware to our Phoenix network, and should not affect CubedHost services.

During the outages listed below, the following services may be affected: CubedHost Phoenix-based Minecraft servers, CubedHost Phoenix-based dedicated servers, CubedHost TeamSpeak 3 and Mumble servers, CraftYourBox US-based Minecraft servers. While we do not expect any extended downtime aside from the periods listed below, you may experience brief connection drops lasting no longer than 5 minutes at a time.

You can find the specific details pertaining to any possible downtime or outages below.

— Friday June 13th 2014 —

No major downtime or outages expected for CubedHost customers.

Beginning at 10:00 PM MST (GMT-7), CraftYourBox US-based Minecraft servers will be shut down and nodes will be physically moved to a new cabinet, and onto a new 20 Gb fiber network. We expect downtime to last between 1 – 3 hours. Our team will be making preparations ahead of time in order to prevent extended downtime. All servers should come back online once the migration has been completed.

— Saturday June 14th 2014 —

No major downtime or outages expected for CubedHost customers.

CraftYourBox US-based Minecraft servers will experience brief downtime throughout the day while we perform routine maintenance on the newly migrated hardware.

— Sunday June 15th 2014 —

No major downtime or outages expected.

CraftYourBox US-based Minecraft servers will experience brief downtime throughout the day while we perform routine maintenance on the newly migrated hardware.

— Monday June 16th 2014 —

2:00 – 4:00 AM MST (GMT-7) / 4:00 – 6:00 AM CDT (GMT-5)

Our team will be performing firmware upgrades and configuration adjustments on our primary switch, as well as adding a secondary switch to increase network capacity. We expect a brief outage for some CubedHost Phoenix-based Minecraft and dedicated servers and CraftYourBox US-based Minecraft servers, lasting no longer than 2 hours. This outage will only affect the network; no Minecraft or dedicated servers will be stopped or restarted. During this time, you may notice that your server appears to be offline and that you’re unable to start/stop/restart it from the Multicraft control panel. If this extends past our outlined maintenance period, please contact our support team immediately.

4:00 AM – 2:00 PM MST (GMT-7) / 6:00 AM – 4:00 PM CDT (GMT-5)

Following the switch adjustments, our operations team will keep a close eye on network performance. Further changes to the switch configuration may be needed, which could result in brief outages.

—-

You can find detailed updates for these events here.

If you have any questions regarding this maintenance or others, please feel free to reach out to our support team – we would be more than happy to answer any questions you might have. Furthermore, if you encounter any issues with your server outside of these maintenance periods, please get in touch with our team immediately.

We thank you for your continued patience and cooperation with us while we continually work to improve performance and capacity in our Phoenix location.

Regards,
The CubedHost Team

TS3 / Mumble Control Panel Outage

This issue has been resolved at this time. If you have any trouble connecting to our TeamSpeak and Mumble control panel, please contact support.

——

Hello all,

We’re currently working to resolve a hardware-related outage affecting our TCAdmin control panel. We have worked with datacenter technicians over the past few days to get this resolved with minimal downtime, but these attempts have been unsuccessful.

While this issue is ongoing, you may be unable to make any changes or issue commands to your TS3/Mumble voice servers. We do apologize for the inconvenience and will keep you updated via this post.

If you have any questions or concerns, please feel free to get in touch with our support team.

Regards,

The CubedHost Team

Unscheduled Outage: Node 15 Phoenix

This outage has been resolved and service fully restored.

Around 7 AM CDT (GMT-5), we received alerts that Node 15 Phoenix went offline. Upon investigating, we were unable to determine the cause or nature of the issue as all services appeared to be online and functioning properly at the time. Minecraft servers were accessible and responding. Over the next 5 hours, our team worked to identify the cause of this issue, noting occasional network flukes and other strange occurrences. We began to notice a large number of “connection refused” and “timed out” errors – errors typical of large DDoS attacks. Upon further investigation, we were able to determine that no such attack was taking place.

If you believe this downtime was detrimental to your server/community, please contact our support team to receive compensation immediately.

— Updates —

[12:15 PM] Upon performing a hardware reboot, the server failed to come back online. NOC staff at our datacenter has been dispatched to intervene and bring the system back online.

[2:00 PM] The NOC staff at our Phoenix datacenter have responded with the cause, and provided us with additional details. We’re awaiting IPMI / KVM access to restore service.

[3:00 PM] The NOC has provided us with further information regarding the issue at hand.
The kernel panic was caused by a bad kernel update. We’ve rolled back to an earlier kernel and the server is now online.
Customers on the IPs 66.85.161.146 – 150 may experience packet loss at this time due to routing issues within our Phoenix facility. This will be resolved as soon as possible.

[5:45 PM] IP routing conflicts have been resolved. Service to Node 15 Phoenix has been fully restored at this time.

Phoenix Network Maintenance: April 28th – May 5th

As part of our continuing effort to enhance server and network performance and prepare our infrastructure for future deployments, select Phoenix-based CubedHost Minecraft servers will experience brief downtime while they are relocated to and consolidated on new hardware.

Affected customers will be upgraded from E3-1270 V1/2 systems to E3-1240 V3 systems.

Each maintenance period may last up to 4 hours, however, completion time is expected to last 1 hour or less. During this time, your service may be inaccessible via Minecraft, Multicraft, and FTP.

Below are the specifics for each maintenance period. All times are CDT (GMT-5).

— Monday, April 28th, 2014 @ 10PM
Nodes: 23, 24, 26

— Tuesday, April 29th, 2014 @ 10PM
Nodes: 2, 3, 16

— Wednesday, April 30th, 2014 @ 10PM
Nodes: 4, 5, 6

— Monday, May 5th, 2014 @ 10PM
Nodes 10, 11

To determine which node you’re located on, please visit your Multicraft control panel and see “Node”
Future migrations will include nodes 1, 12, 13, 14, and 15.

As part of this mass migration, new services will be located on our 10 Gbit fiber network. In order to take advantage of this new network, an IP address change is recommended. If you are comfortable with your server’s IP changing, please submit a support ticket and request for your server to be moved to our 10G network.

No IP addresses will be changed without your approval during this maintenance. We strongly disagree with abruptly changing customer IPs and unless an emergency presents itself, we will never do so.

Thank you for your understanding and patience while we work to continually improve and upgrade our service.

If you have any questions, please feel free to contact our support team.

Phoenix Network and Hardware Maintenance – Sunday, Feb. 23rd, 2014 @ 2AM CST

The maintenance has been completed successfully at this time.

We will be performing a network maintenance / hardware swap on Sunday, February 23rd, 2014 at 1AM MST.
The maintenance is expected to last a maximum of 4 hours, with the exception of any unforeseen circumstances.

The following nodes and IP addresses will be affected:
Nodes:
– phx17 – Upgrade from E3-1270v2 to E3-1240v3
– phx18 – Upgrade from E3-1270v2 to E3-1240v3
– phx19 – Upgrade from E3-1270v2 to E3-1240v3
– phx20 – Upgrade from E3-1270v2 to E3-1240v3
– phx25
– phx29
– phx31

If you’re unsure what node you’re on, the following IP ranges will be affected:
– 198.15.97.18-30
– 198.15.96.10-14
– 198.15.73.178-183
– 198.15.74.178-190
– 198.15.77.242-254
– 198.15.77.130-134
– 198.15.86.226-238
– 198.15.87.74-78

The following IP ranges may encounter some brief outages lasting 5 – 15 minutes:
– 69.160.45.0-127
– 69.160.47.0-255

As always, we will try to keep the maintenances down to a minimum timeframe to keep your service online as long as possible.
If you notice extended downtime during/after this period, feel free to drop a request by our support department.

Thank you for your patience while we continue the expansion of our network and upgrading/improving our services.

Regards,
CubedHost, LLC

Node 2 Montreal Emergency Maintenance

Hello all,

Node 2 Montreal is back online at this time.

We’re sorry to report that Node 2 Montreal appears to have a critical hard drive failure resulting in possible data loss. We’ve taken the system offline to prevent further damage and have dispatched a team at the datacenter to replace the affected drive. We will make every attempt to restore the drive array without any data loss. However, data will most likely need to be restored from our nightly backup which was taken this morning at 12am CST (GMT-6).

We do apologize for any inconvenience this causes and fully understand that this is not the level of quality service that you expect from CubedHost. If you believe this is/has significantly impacted you, please reach out to us via support ticket and we would be more than happy to compensate you appropriately.

Thank you