Last night after deploying 3.28.0c, some of our servers accidentally had their configuration broken in a way that resulted in approximately twice the number of players being assigned to some servers than what they can normally handle. This was entirely an error on our part and doesn’t have anything to do with previous server disruptions that we have had this launch.
Unfortunately, even though we have an alerting system in place to wake up our server admins in cases where load on our servers is too high, none of the conditions we had in place were triggered by this event. There are some unusual properties of this incident that we are still investigating that caused performance to be bad even though CPU load was not higher than our alerting threshold.
As soon as our server admins were made aware of the problem, the configuration issue was able to be immediately resolved, but this resulted in around 12 hours of degraded performance for some users in some regions.
We have added some new performance metrics to our alerting system which will make sure that this particular failure doesn’t happen again.
We apologise for the impact that this had. This incident was our fault, and we will make sure that it doesn’t happen again.
