Login queues: your first line of defense on launch day

Launch day is both wonderfully exciting and terribly scary. Every developer hopes the players will show up, but if they do en masse, you may be headed straight into trouble. The login queue is the best defense against a flood of players all trying to jump on at once, and should provide both rate limiting and a peak concurrent user cap. Let's explore the anatomy of a login queue, discuss several reasons they frequently fail, and how you can develop a robust solution that protects the player experience during the launch window.

The login queue can be thought of as a bouncer at a nightclub. The bouncer is responsible for choosing who is allowed in, how quickly he allows them in, and stops allowing people in when the club reaches capacity. These are exactly the traits we need for an effective login queue.

Rate limiting

If tens or hundreds of thousands of players show up all at once, there's a risk that your platform may not survive contact, even if the overall capacity of the platform is capable of supporting that many players. For example if your platform can support 100,000 concurrent players, it still may crash if all of those players slam your account service all at once.

Rate limiting, as the name suggests, slows down the rate of traffic to a level of concurrent requests that the platform is capable of handling. This allows players to stream in at a regulated rate, ensuring that the platform remains stable and all players can play and enjoy the game.

Common mistakes

When authoring a login queue, developers frequently are met with the challenge of what to do about potential exploit cases and attack vectors. A common discussion is whether there should be any criteria for entering the login queue in the first place. Frequently the choice is made to perform player authentication before allowing players to enter the queue. This is the equivalent of having the bouncer check IDs before allowing club goers permission to wait in the line outside the door.

This is a surprisingly common, and often fatal, mistake. The account service is the single most common backend service that fails during launch, as it is the first service in the flow, and tends to rely on database transactions, lookups, and cryptographic operations. By slamming the account service at an unregulated rate, the login queue is doomed before it even has a chance.

Best practices

It's important, therefore, to allow unauthenticated users into the queue. While this does technically allow a bad actor to take up slots in the queue, the reality is that there are many more direct approaches for attacking the backend and ultimately it's not a consequential risk. Of course the player must be authenticated, and this check happens after getting to the front of the queue, at the specified login rate.

Max concurrent users

Depending on the game type, genre, and studio capabilities, it's very possible that there is an upper bound to the number of players that can be logged in at once while keeping the platform stable. It's common for developers to worry about the negative player experience caused by a player cap, but it's important to consider that if the alternative is a server crash that results in all players being unable to enjoy the game, a player cap is the better alternative.

Due to the requirement that we do not authenticate the players before joining the queue, and common distributed system architectural patterns, there are consequences of these attributes that make enforcing a max player count a bit more tricky than it sounds.

Common mistakes

Some naive implementations of a login queue might not account for the difference between approving a player login, vs when the player does in fact login and establish a session. For example, if your queue is configured to allow 100 logins per second, and you approve a batch of 100 players, you don't yet know whether all of those players are still around waiting to be let in. Perhaps the player logged out because they were tired of waiting, or they restarted the client in hopes that it would somehow allow them a higher place in the queue. For these reasons, there is no guarantee that approving a batch of 100 logins will result in 100 new player sessions.

As the queue nears a max player limit, you can see that this time delay presents a problem. Do you slow down the login rate early in anticipation of approved clients? Do you keep approving at the configured limit right up to the limit, only to have a large batch of already approved players show up and either blow past the limit, or get rejected during authentication after seemingly being approved?

Best practices

There are several approaches that can help address this matter. At its simplest, if the absolute scale of the platform doesn't actually require horizontal scale, it's possible that a single service instance could handle all of the login queue and login requests. This simplifies the problem as all the state is held within the same service, and coordination can be done via standard concurrency and multi-threaded patterns. While it might seem blasphemous to consider a solution that doesn't horizontally scale, modern hardware is capable of astonishing levels of capacity and throughput when running well engineered software, even to the level of supporting hundreds of thousands or millions of players for some use cases.

Another approach is a heuristic based one, where the login queue tracks its approval rate, and receives reports of actual login rates. With even a small number of samples (say a few minutes worth), in practice the actual login rate is likely to be quite consistent. For example if the login queue is approving 100 logins per second, and the login service reports a rolling average of 80 logins per second, the login queue can use this information plus some measurement of login delay (on average how long after being approved does a client actually arrive) to calculate when to enforce the player cap. This may mean a best effort solution that doesn't enforce the boundary exactly, but in many cases this isn't a problem in practice because it's unlikely that a small number of extra players would overwhelm the system.

Other considerations

Third-party limits

Some studios have chased extremely high volume login rates (thousands or tens of thousands per second), only to discover that the effort spent optimizing to this level was all for nought when a platform auth provider enforces its own rate limits significantly lower. In games, it's not uncommon for the big first party platforms to start timing out requests around ~200 requests per second (this is just a broad figure pulled from past experience, obviously it depends from platform to platform and these limits change based on updates all the time).

Load testing is a valuable exercise for assessing your platform's scale and capacity limits before it's live to players. However, you must frequently mock out these third party endpoints because platforms often don't take kindly to you performing unplanned stress tests against their live services. It's important to understand the limits of the insights one can get from a load testing effort, and be mindful of the diminishing returns of pushing your capabilities beyond what you'll be able to utilize in production.

Capacity limits

Similar to third party limits, if you are running dedicated game servers operated by a hosting provider, the provider may enforce capacity limits on how many instances they can provide. In this case you may need to enforce a player cap not based on what the backend is able to support, but instead the number of active games you are able to host at once.

Protecting all the things

Lastly, while the accounts service is the most common point of failure given it's the first backend service to get hit, and because it performs several complex tasks, it's always possible that other backend services cannot handle the same load of incoming players. In these cases the login queue rate limit must be configured to a level that ensures stable operation for the least capable service within the backend services cluster.

This point may sound obvious, but it can be easy to overlook depending on how much time is spent assessing the platform's launch readiness before launch. Launch readiness is typically evaluated via load testing, and how this load testing is performed can impact how much insight the studio has towards their production needs.

Load testing

Load testing is a topic in its own right and will receive a dedicated article in this series, but we'll touch on a few points related to the login queue.

One expedient approach is to develop an isolated load test fixture that stress tests a single service. This can be built quickly and is relatively easy to configure, orchestrate, and run. The benefit is that you can get a sense for what each individual service is capable of in isolation. The catch, however, is that this kind of testing may not reflect the real-world traffic patterns that the service will operate under in production.

Blended load testing is the process of developing a shared scenario that is modeled after real player behavior. This provides a more realistic load and traffic pattern across all backend services, and can help the team discover secondary issues caused by the various kinds of player and service-to-service traffic that is all happening at the same time. This kind of load testing can meaningfully improve confidence in being production ready, but can be more expensive to build, operate, and maintain over time.

Summary

The goal with this series is not to provide a comprehensive guide to every facet of launching games at scale, but rather to establish key areas of interest or risk and provide insights into common mistakes to avoid and best practices to pursue. I hope this information is informative and useful. We live and breathe this stuff at Pragma, so please reach out and connect with us if you want to talk more. We operate a Discord community of game developers interested in backend development, see you there!

Pragma Engine is a backend game engine, a source licensed tech stack that is designed to run locally, be extended endlessly, and capable of scaling to power the world's largest online games. If you are planning to launch an ambitious online project, Pragma is the fastest and most reliable way to develop your game backend. We're looking forward to building with you!