Tuesday, July 14, 2009

Designing a tournament (Part 1)

This article is part of a series on designing a tournament:
Part 1 - Designing a Tournament (this article)

Spent a bit of last week involved in a few discussions about ranking systems used in competitive computer gaming. This reminded me of an application I had virtually completed to calculate the bias of different types of tournament systems, but I hadn't written up the results. Hopefully this series of articles will address that.

Ever since the second season of QGL back in 1998 I've been designing and analysing tournament systems for use with computer games. Back then there was a great focus on player satisfaction as a goal for the type of tournament to run as we wanted to maximize the enjoyment and participation of players throughout the season. We developed a matching system akin to swiss to be run across several LANs that enabled players to compete against people of their rank and be resilient to players/teams dropping or joining the tournament.

As AusGamers grew to a national body for organising tournaments, we dabbled with finals formats consisting of mainly double elimination. At that time the format was relatively new on the scene, but it held up remarkably well to a number of concerns we had for finding a winner from disparate states in an efficient manner.

Through involvement running the Australian leg of CPL and WCG, a number of other systems were reviewed and the shift to a more spectator oriented position could be seen emerging through the formats chosen. As a keen advocate of double elimination at the time, I was a little disappointed in WCG's insistence on pools of players competing against each other with the top 2 advancing to a single elimination tournament. I could see the reasons why that type of structure was chosen, but believed that it was an inefficient method. I spent another 5 years involved with WCG (partly as Australian tournament director designing my own tournaments, and later at international level as head referee) and had continued to seek ways of eliminating bias.

There's got to be others in the world that have gone through a similar upbringing and who share a passion for tournament design. Hopefully these articles can stir up some debate so that we can develop a body of knowledge about how best to design a tournament for future competitions.

What makes a Tournament?
Wikipedia currently does an Ok job of defining what a tournament is, but not why it exists. What are the goals of a tournament? For me, the primary goal of a tournament is to provide an objective method for finding the competitor with the highest true skill. Who's the best gathered here today? As an adjunct to that, a tournament should also do its best to rank all competitors by their true skill. This becomes especially important when there are rewards given to lesser places.

There are also 2 other goals of varying degrees of desirability that depend on the context of the tournament: player satisfaction and spectator satisfaction. Players wish to derive satisfaction by playing as many games as possible, by being treated fairly, by being exposed to learning opportunities, and by demonstrating their skill to others, whereas spectators want to see entertaining play, close matches and a high degree of skill being displayed. Ideally these goals should be maximized and can shape the type of tournament selected, but not at the expense of its primary goal otherwize it's not a tournement (EG: WWF as a spectacle, handicapping for social play). These will hopefully be expanded upon in future sections.

True skill in a competitive sense is defined as the competitor's own abilities in the game being tested. Ideally everyone should be able to demonstrate their true skill each time they play, however a number of factors can cloud a player's true skill to produce a bias in the final result of each match in the tournament. Another topic for later, but it's mentioned here as acknowledgement that they exist. Initially when analysing tournament structures we will assume that players are able to play flawlessly to their true skill so that they will always win a matchup with someone of lower true skill.

The perfect tournament for me would be one where every competitor is able to play every other competitor simultaneously (but individually) and complete the match instantaneously, while playing at their true skill. Obviously this could never happen, but that type of tournament would be able to state with certainty who was the best competitor at that specific time and place. In reality there are biases all over the place, having to wait for your rounds, having to play people coming off a losing streak or winning streak, not playing everyone, etc. All are biases inherent with the tournament system that you choose. Understanding these biases and choosing a specific type of tournament structure that minimizes bias is at the heart of tournament design.

The final part of the puzzle is resource management. You are almost always going to be constrained to complete the tournament in a specified amount of time. If you can complete the tournament in a shorter amount of time then there are many ways to enhance the tournament to address bias issues or cater to secondary goals. The other main resource is the amount of locations available for matches to be played simultaneously. Not having enough locations for each round will impact on the total time taken for the tournament, however increasing the locations may be impossible or too costly.

Decisions, Decisions:
So, we want to deliver an objective ranking of all competitors while maximizing player and spectator satisfaction, minimizing bias inherent to the tournament structure or to individual matchups, and maximizing location use and time. No worries! Where do we start?

First port of call is to look at the resources you have and the anticipated players to see what types of tournaments are possible. Try single elimination initially as there will be no tournament structure with less resource usage for a competitor vs competitor style tournament. If you still don't have enough resources you'll need to cull the amount of players eligible or beg for more resources and use single elimination.

Next try a league format. If you can fit in a complete league then your tournament biases will be largely minimized, however the resources and time needed make these a rarity in most LAN and computer game competitions. You are most likely going to come out somewhere in the middle; more than enough resources for single elimination, but less than a full league. In this space there are a bevvy of different tournament structures to choose from, but little to no information to gauge how efficient each type of system is.

So how do you choose which is the best system for you? In the coming articles I hope we can come up with a grading system to compare tournament structures against each other in terms of bias and desirability and then explore what each type of tournament structure brings to the table.

Part 2 - A grading system