Unscheduled Outage: Node 15 Phoenix

This outage has been resolved and service fully restored.

Around 7 AM CDT (GMT-5), we received alerts that Node 15 Phoenix went offline. Upon investigating, we were unable to determine the cause or nature of the issue as all services appeared to be online and functioning properly at the time. Minecraft servers were accessible and responding. Over the next 5 hours, our team worked to identify the cause of this issue, noting occasional network flukes and other strange occurrences. We began to notice a large number of “connection refused” and “timed out” errors – errors typical of large DDoS attacks. Upon further investigation, we were able to determine that no such attack was taking place.

If you believe this downtime was detrimental to your server/community, please contact our support team to receive compensation immediately.

— Updates —

[12:15 PM] Upon performing a hardware reboot, the server failed to come back online. NOC staff at our datacenter has been dispatched to intervene and bring the system back online.

[2:00 PM] The NOC staff at our Phoenix datacenter have responded with the cause, and provided us with additional details. We’re awaiting IPMI / KVM access to restore service.

[3:00 PM] The NOC has provided us with further information regarding the issue at hand.
The kernel panic was caused by a bad kernel update. We’ve rolled back to an earlier kernel and the server is now online.
Customers on the IPs 66.85.161.146 – 150 may experience packet loss at this time due to routing issues within our Phoenix facility. This will be resolved as soon as possible.

[5:45 PM] IP routing conflicts have been resolved. Service to Node 15 Phoenix has been fully restored at this time.

Phoenix Network Maintenance: April 28th – May 5th

As part of our continuing effort to enhance server and network performance and prepare our infrastructure for future deployments, select Phoenix-based CubedHost Minecraft servers will experience brief downtime while they are relocated to and consolidated on new hardware.

Affected customers will be upgraded from E3-1270 V1/2 systems to E3-1240 V3 systems.

Each maintenance period may last up to 4 hours, however, completion time is expected to last 1 hour or less. During this time, your service may be inaccessible via Minecraft, Multicraft, and FTP.

Below are the specifics for each maintenance period. All times are CDT (GMT-5).

— Monday, April 28th, 2014 @ 10PM
Nodes: 23, 24, 26

— Tuesday, April 29th, 2014 @ 10PM
Nodes: 2, 3, 16

— Wednesday, April 30th, 2014 @ 10PM
Nodes: 4, 5, 6

— Monday, May 5th, 2014 @ 10PM
Nodes 10, 11

To determine which node you’re located on, please visit your Multicraft control panel and see “Node”
Future migrations will include nodes 1, 12, 13, 14, and 15.

As part of this mass migration, new services will be located on our 10 Gbit fiber network. In order to take advantage of this new network, an IP address change is recommended. If you are comfortable with your server’s IP changing, please submit a support ticket and request for your server to be moved to our 10G network.

No IP addresses will be changed without your approval during this maintenance. We strongly disagree with abruptly changing customer IPs and unless an emergency presents itself, we will never do so.

Thank you for your understanding and patience while we work to continually improve and upgrade our service.

If you have any questions, please feel free to contact our support team.

The day that Minecraft Multiplayer broke

Many of you may have noticed that your server – along with many other servers big and small – have gone offline, crashed, or otherwise stopped working. What’s to blame? A tiny bug in Spigot.

At around 3pm CDT (GMT-5), all servers running Spigot (including MCPC+) crashed simultaneously due to a time-specific bug. This caused tens of thousands of servers to cease working, including huge communities such as MCGamer and Hypixel.

As these thousands of players were disconnected, they immediately tried to rejoin their servers, ultimately causing a huge influx of traffic on the Mojang login/session servers and subsequently, a network disruption on their service. This in turn caused widespread outages for all Minecraft servers, but mainly affecting any server running Spigot or MCPC+.

Please see this page for more information.

At this time, we are pushing out continual updates as the Spigot team works hard to completely resolve the bugs. We advise all customers using Spigot or MCPC+ to continue to update their servers regularly in order to stay on the most stable version.

How to update your Minecraft server.

How to update a modpack on your Minecraft server.

If you have any questions or would like our team to update your server for you, please feel free to contact us via live chat or support ticket.

Phoenix Network and Hardware Maintenance – Sunday, Feb. 23rd, 2014 @ 2AM CST

The maintenance has been completed successfully at this time.

We will be performing a network maintenance / hardware swap on Sunday, February 23rd, 2014 at 1AM MST.
The maintenance is expected to last a maximum of 4 hours, with the exception of any unforeseen circumstances.

The following nodes and IP addresses will be affected:
Nodes:
– phx17 – Upgrade from E3-1270v2 to E3-1240v3
– phx18 – Upgrade from E3-1270v2 to E3-1240v3
– phx19 – Upgrade from E3-1270v2 to E3-1240v3
– phx20 – Upgrade from E3-1270v2 to E3-1240v3
– phx25
– phx29
– phx31

If you’re unsure what node you’re on, the following IP ranges will be affected:
– 198.15.97.18-30
– 198.15.96.10-14
– 198.15.73.178-183
– 198.15.74.178-190
– 198.15.77.242-254
– 198.15.77.130-134
– 198.15.86.226-238
– 198.15.87.74-78

The following IP ranges may encounter some brief outages lasting 5 – 15 minutes:
– 69.160.45.0-127
– 69.160.47.0-255

As always, we will try to keep the maintenances down to a minimum timeframe to keep your service online as long as possible.
If you notice extended downtime during/after this period, feel free to drop a request by our support department.

Thank you for your patience while we continue the expansion of our network and upgrading/improving our services.

Regards,
CubedHost, LLC

Node 2 Montreal Emergency Maintenance

Hello all,

Node 2 Montreal is back online at this time.

We’re sorry to report that Node 2 Montreal appears to have a critical hard drive failure resulting in possible data loss. We’ve taken the system offline to prevent further damage and have dispatched a team at the datacenter to replace the affected drive. We will make every attempt to restore the drive array without any data loss. However, data will most likely need to be restored from our nightly backup which was taken this morning at 12am CST (GMT-6).

We do apologize for any inconvenience this causes and fully understand that this is not the level of quality service that you expect from CubedHost. If you believe this is/has significantly impacted you, please reach out to us via support ticket and we would be more than happy to compensate you appropriately.

Thank you

Web Server Migration: Feedback Needed

Since the inception of CubedHost, we have used a single web server based in Phoenix to serve our primary website, billing area, and Phoenix Multicraft control panel. In late December of 2013, we added OVH’s CDN on top of this in order to decrease page load times by caching static content and providing POPs (endpoints) closer to customers. This helped immensely but we wanted to take it a step further.

During the early morning of Feb 11th, our team performed a much needed full migration of our primary web server and database. We moved from a single nginx + PHP web server to a HAProxy-based infrastructure. HAProxy allows us to serve web requests from a single or small number of proxies (and thus IPs) while forwarding these requests to a large number of individual “worker” boxes running nginx + PHP. As of right now, the CubedHost website is operating off of 11 dedicated worker instances. On top of this infrastructure, we have kept our OVH CDN setup for caching. This infrastructure is located in Virginia, while our Phoenix Multicraft panel has been kept in Phoenix.

With this increased distribution of workload, we’re aiming to continually decrease page load times, increase availability, minimize downtime and timeouts from international visitors, and provide redundancy in the event of some form of failure. As such, we’re looking for feedback from our visitors and customers. Please let us know how the new setup works for you – have you noticed an improvement in page load time on our website lately?

Please send all feedback to [email protected], be it positive or negative. Anyone who provides us with feedback will receive 5% off their next month on any Minecraft, voice, or web service.

Thank you!

Web Hosting Service Downtime

This does not affect any Minecraft, dedicated, or voice servers.

Earlier today CubedHost was hit with a 19.3 Gbps DDoS attack aimed at an account on our cPanel web hosting node, specifically on IP 184.164.156.34. This IP has been blocked at the datacenter level in order to prevent network disruption for other services and customers. As such, all websites have been migrated to  184.164.156.35. Service has been restored at this time, but because this involved a DNS record change you may not be able to access your website immediately.

Please try the following steps.

We do apologize for any inconvenience this downtime might cause.

CubedHost + Technic: Fueling the launch of AotBT

Hello all!

Technic‘s latest and heavily anticipated modpack – Attack of the B-Team – was released today. Our team at CubedHost has always been eager to keep our service up-to-date with the latest packs and mods, so we naturally stocked our control panel with v1.0.1. However, when release came around and v1.0.2 was out, Technic’s poor download server had some trouble handling all of these requests. Here’s where we come in. CubedHost picked up the excess traffic via Round Robin DNS — a whole 900 Mbps of it within seconds. We’re still working on adding additional resources to keep speeds steady and fast. In the meantime, here are some awesome graphs from a few hours ago. Enjoy!

Update: It is now Thursday (Feb 6th) and the traffic keeps coming! At one point we were pushing over 3 Gbps. Here are some new bandwidth graphs.

Bandwidth - CubedHost Mirror #1
Bandwidth – CubedHost Mirror #1
Bandwidth - CubedHost Mirror #2
Bandwidth – CubedHost Mirror #2
HAProxy - CubedHost Mirror #1
HAProxy – CubedHost Mirror #1
HAProxy - CubedHost Mirror #2
HAProxy – CubedHost Mirror #2

Cyber Monday Sale – 75% Off!

Hello all!

A certain competitor is having a 75% off sale on Cyber Monday, so we thought, “why not?” 😉

Enjoy 75% off all of our Minecraft servers, including servers in our newest location, Amsterdam! The sales starts Monday, December 2nd and lasts from midnight to midnight (Dec 3rd) CST.

Get ready to grab them quick before the sale is over!

<3 The CubedHost Team

The CubedHost 75% off Cyber Monday promotion applies to all Minecraft servers in all locations. 75% off refers to the first month’s list price before any other promotions. The promotion will be done through the use of a promo/coupon code revealed on the CubedHost website once the promotion starts. Promotions cannot be combined. The Cyber Monday promotion will be valid from 12 AM CST December 2nd to 12 AM CST December 3rd. Deal applies to monthly purchases only, excluding quarterly and semi-annual. 75% off first month only; subscription will recur at the normal list price after the first month. Upgrades are only eligible for 50% off on monthly billing cycles.