Summary for Riot's AMA about EUW Service Issues

As promised, Riot's tech leadership hosted an Ask Me Anything about all the server problems that have been plaguing EUW for the greater part of a few months.
Continue reading for the scoop on the problems and a peek at the technological side of things.

Demorphic started things off by reiterating the purpose of this server problem AMA and introducing the Rioters who participating in the AMA.

"Hi everyone!
I'm Matt "Demorphic" Elliot, and as promised we will be kicking off an AMA today at 6pm UK time with three members of Riot's senior technology leadership:

• Scott "Scott Thru The Heart" Gelb, Vice President of Technology
• Ron "sonicdeathriot" Williams, Vice President of Operations
• David "RiotBanksy" Banks, Technical Director
• Joining us will also be Tomasz "Riot tmx" Ankudowicz, our Live Services Producer for Europe.

We're here to talk with you about the recent service disruptions you've experienced on EUW, and to answer any questions you have about how the League of Legends service operates, and why it sometimes breaks badly. 

I'd humbly ask that you focus your questions on the service/technology side - the Rioters with me today are super busy working on improving the network infrastructure behind League of Legends to avoid issues like this in the future, and they don't often get a chance to hop in and take your questions. 

They will also be fielding questions from multiple European languages, so please bear with us. Realistically we will not be able to answer all questions live. However, we have commited to following up this session with a more comprehensive set of questions and answers before this coming weekend. 

We're going to be doing more deliberate community communication like this moving forward - we understand that not getting this kind of face time with the team at Riot HQ is also a big point of pain for EU players, and we hope this is a valuable step in the right direction.Please kick things off and we will be jumping in at 6pm UK time. You then have our undivided attention for 2 hours. So fire away!"

Now on to the questions! As usual I've tried to clump like minded questions together and format things so the questions flow. It should be noted that the questions in the brackets [ ] are my interruptions of the questions, not the word for word questions asked.


The Problem, EUW, and Communciation

[ What is the extremely tech-lingo filled reason for the problems? ]

"I wanted to share a big wall of text for anybody who'd like to go deeper into the tech today. :) This write-up describes some of the big problems that were impacting all the regions in Europe (and other platforms outside of Europe) until about mid-April (and more recently for Korea): 

EU Core Switch Summary 

Since late October there have been several major outages in Europe that were related to scaling problem with our core switches in Frankfurt. Our Korea players also suffered from this issue recently along with some other painful problems. Last week we finally figured out the root cause of this painful issue. The root cause around the malfunction of the core network switches was that our game servers were redundantly connected with two network interface cards (NIC) that were setup in an active/active load balancing mechanism we use in all of our environments around the world.* The network load balancing was accomplished through the use of a “smart” algorithm that dynamically moved traffic from one NIC to the other.* Since each server was connected to the network twice, every time the server flipped its active NIC, the entire Europe network would have to change its network packet forwarding tables to account for the change.* Specifically the MAC address table inside the switches needed to be updated constantly.* Typically updating a MAC address table isn’t as issue with just a few servers, but it becomes an issue as the server count increases when those severs are using “smart” load balancing. * When the game servers change their active NIC this has to be propagated across the entire network.* It starts from the top of rack switch that the server is directly connected to, sends the change to the core network switches and then is distributed out to all devices connected to the network. As network traffic increases, the servers will swap the active NIC more often to distribute the load, which causes the MAC address tables to be updated at a rate that cannot be handled by the core network switches.* In the case of our Eruope (and Korea) game servers this means the MAC address table was updating millions of times per second during peak playtimes. North America was spared this pain because we have fewer game servers on the core switches than we do in Europe (or Korea).

This is where the network flooding starts that would eventually crash the core switches that support the entire Eruope service. *Inside the core network switch, the MAC address table stores a MAC address mapped to a physical network port that is uses for forwarding and filtering network traffic. **As the rate of updates exceeds the capability of the switch, the switch falls back into a fail-safe mode where it sends the packets to all ports on the switch.* Traffic that is normally only sent to the destination that it addressed for, is now being broadcasted across the entire environment.* As a result all of our servers started getting lots of unneeded network traffic.** This had the net effect of essentially creating a denial of service attack across the entire network.* Welcome to meltdown city. *And when the core switches crashed they did not keep any diagnostic information we could access to help troubleshoot the problem.
** 

To mitigate the above issue, we have done a few things: 

1.*******Short Term Fix: *Disable secondary NICs on the game servers. The idea behind this is that if there is 1 NIC and 1 related MAC address per game server (instead of 2), this completely removes the possibility of traffic jumping around from one NIC to the other.* The network will no longer have to update itself as a result of any server side load balancing. 

2.*******Long Term Fix: Change the environment to use active/passive failover on the NICs.* The server will have a backup connection but it will only use it in the case of the primary path failing.* This will stabilize the environment greatly but still allow us to provide high availability in the case of a localized failure.
* 

3.*******Upgrade the core network infrastructure with hardware that can handle a huge amount of additional load.* Why not, I mean, more power is better, right?! 

a. Upgrade the core switches so if we get surprised again with extra traffic for whatever reason, we are ready for it. *Different designs can keep up better with MAC address table updates which would allow us to better handle the load should something similar happen in the future.* 

b. An improved design will also reduce the chance of one switch failure crashing everything in the data center. 

Current Actions: 

We have been working on a daily basis with our network switch vendor (the same one used by the leading Internet companies and other leading websites). During the outages we’ve been passing the switch vendor logs we gather as we make changes to the Europe (and Korea) network infrastructure, and have been working alongside of them to diagnose the root cause of why the core switch isn’t able to handle the rate of change to the MAC address table. * We have also been working with the vendor to expedite the new network gear to upgrade Europe's network, gear we placed orders for several weeks ago, but due to the very big capacity of this new network gear, it is built to order by the manufacture. Some of this gear is already in installed and has helped stablize EUW and EU Nordic's network since mid-April 2013.

We are sorry of this inconvenience, it has been a very difficult problem to track down and fix. We have learned from this issue and are building a better network to protect the quality of the service in Eruope and in our other service areas." - sonicdeathriot

[ Can the PBE used to avoid these problems in the future? ]

"We do use PBE for testing our deploys, patches, hotfixes, etc. It's in constant use for client-based issues and new releases are always first deployed to the test server and many bugs are caught this way. It does not cover all the re-configurations (this word is unfortunate, but I'm talking about all platform changes, firewall policies, drives, front and backend applications, database modifications, Coherence tweaks, etc) we perform on our network and system infrastructure. On the other hand, PBE lives in our datacenter and is hosted from some of our standard game servers, so in fact it is a part of our bug investigation." - Riot tmx

[ Can you explain why EUW has so many problems compared to other servers? ]

"Service quality across the many regions League of Legends runs is actually about the same when looked across a full year of data, however sometimes a certain region will have a bad run of issues that can last several days. Also, when issues happen our biggest regions do tend to have extra player impact when issues occur. This is due mainly to: 

1. When a big region has an issue that kicks a lot of players out of the game or off the platform it puts a lot of stress on the systems that let players back into the system. This creates complications that lead to the need for us to limit the number of players per second we can let back into the service which means in large outages it can take several hours to let everyone back into the service. Also when this kind of load is placed on the system it can sometimes create secondary issues that will extend the unplanned downtime. There is a lot of work going on in our Operations and Development teams to greatly increase how many players per second we can let into the service and also improve the stability of the service when it is under stress when huge numbers of players are trying to log back in after an outage at peak times. 

2. Our biggest regions like EUW are also our most complicated infrastructures to maintain. This means that new release can take longer to deploy, troubleshooting problems can take longer, and there are just more things that can break, be misconfigured, or have other issues. EUW and EU Nordic are both currently being rebuilt in Amsterdam on brand new servers, network, and secuirty gear (note that game servers will continue to be availble in Frankfurt with even more being added in Amsterdam). This will give players the latest architecture we have been testing for the last several months to ensure EUW and EU Nordic can continue to grow larger and have even better service stability. It is a huge undertaking but coming soon." - sonicdeathriot

[ How big are the EUW servers? ]

"Our EUW data center in Frankfurt is pretty astounding. We rack thousands of game servers globally and house many of them there. In EUW, we are handling many millions of game data packets per second. We are constantly making improvements to network infrastructure, hardware and optimizing our software to handle the ever growing enthusiasm for League." - Riot Banksy

[ What databases and what sort of teams manage such huge systems? ]

"We're on various DBs, and beside Microsoft SQL and some other premium brands we also operate on fancy versions of well-known MySQL. 

All services are hosted from different servers. Actually, clusters of servers divided by functionalities. Let's take chat: once creating the connection to the platform you're first reaching the load balancer, which creates a connection to given chat server. It handles the traffic in the most efficient way, so if one of the chat servers suddenly drops, it does not affect all player base on the platform. 

We're working 24/7. We have our Network Operations Centers all over the world and some of them work in 3 overlapping shifts. Problem might be handled by the team in Santa Monica, or Dublin, or Seoul, or Istanbul or Rio. Additionally, we have our on-call engineers on various continents focusing on every single incident happening anywhere. It doesn't really matter if we're hit in Singapore, Brazil or Europe. The team is always there and people are back to their workstations instantly." - Riot tmx

[ Why not just get more servers? ]

"We are constantly making investments in new testing tools, new monitoring, and higher capacity equipment to handle the growing Europe player community. Currently we actually have enough server infrastructure and constantly installing more to stay ahead of the growth. Our recent issues are not related to server capacity. When we do run into capacity issues they usually involve either network gear tuning, network security adjustments, or tuning the the software/databases that support the service. It can be very difficult to simulate the load our large Europe player base can place on the network and servers for each region, so sometimes we run into unexpected issues during peak service times that require further troubleshooting and tuning." - sonicdeathriot

[ If it is too big to handle, are there any plans to split EUW? ]

"We currently do not have plans to split the EU-West platform. We are constantly upgrading the infrastructure and adding additional capacity to support the growth of EU-West, one of the largest League of Legends regions in the world. We recently addressed one of the technical challenges we've faced with the scale of our core network. Ron Williams, our VP of Operations, is going into more detail on the specific issue and how we’ve solved it in a separate post (http://euw.leagueoflegends.com/board...23700#11923700). We have engineers working around the clock to fix the stability issues and dedicated teams that are focused on scaling our systems proactively to support the live service." - ScottThruDaHeart

[ Are you sure you don't want to split EUW ? ]

"We don't want to split EUW, I think this has been messaged out already. What we do instead is to improve our infrastructure even more. We already have top quality devices in place, but life brings us new solutions every month and we're constantly upgrading our systems. We recently performed large maintenance in the Frankfurt data center where we completely re-designed our network infrastructure or replaced all the database drives with the high-end solutions. It all gives us some additional power and if you take a look a year or two behind, we're now able to provide service for over 100% more players comparing to the past. Also, not sure if you noticed, getting back on EUW platform is now quicker than ever, as we're able to throttle players in much faster in the peak times or during the incidents." - Riot tmx

[ How will you tackle similar problems in the future? ]

"There are always going to be new challenges as we keep growing League of Legends. However we continue to make investments in new data centers, new equipment, and hire smart engineers to provide the best player experience we can." - sonicdeathriot

[ What are your plans to improve communicating server problems with players?]

"Beside keeping the platform in good health, communication is our main goal. It always was, but we know we didn't really do great job here. So, about plans: 

1) We're working on standardizing Live Incident communication in all languages. These will be high quality, automated and localized announcements posted regularly during the incident and we know it's going to work better, than our current system. We're still finalizing the scripts. 

2) We introduced new program called Post-Incident Messaging. Every time our platforms come through the issues, we're going to tell you about it within next 24-48 hours. We will summarize our efforts, tell you what we've done, what kind of problems we noticed, what was the impact and how did we mitigate the incident. We will be actively answering all your questions and we really want to put our best effort into this. 

3) We're also working on improving the communication within the client, as we know that some players don't even check forums for service status. More info coming soon." - Riot tmx

Compensation