EPTA Closer Look at How we Measure Balance
Starcraft players love discussing balance. And when they do, there's a variety of statistics they like to cite to support their claims. Some will cite win rates from Liquipedia or Aligulac, others will cite racial representation in Grandmaster League or some tournament, and others still will use metrics like total GSL champions for each race. Whatever the metric, the assumption is that with a sufficiently large player base, differences in skill between players will average out. What is left should be nothing more than a reflection of the absolute winningness of each race, or in a word, balance.
The underlying assumptions here are not always accurate. For example, the last metric listed is obviously quite poor. Even if it is assumed that racial balance is the only factor in determining GSL champions, a tally of GSL champions for each race still wouldn't reflect current SC2 balance; it would only reflect the accumulation of SC2 balance over its history since WoL release. For purposes of assessing the current state of the game, that is obviously useless, and I think most people on Team Liquid know that such a metric isn't good for much more than painting a broad, fuzzy picture of what WoL balance looked like.
But what about win rates or racial representation at the highest level as metrics? Most prefer to cite win rates because they are more statistically rigorous and have a larger sample size; some people prefer to cite racial representation at upper levels because it involves fewer arbitrary decisions about which games, regions, tournaments, etc. should be included in win rate data. But almost everyone will acknowledge that one or the other is pretty much an accurate representation of racial balance, aside from some statistical noise.
But there could be more at play here than statistical noise, specifically because of essential issues with the fact the Starcraft 2 is not one game, but six. Any given player does not just play Starcraft 2; they specifically play three different match-ups, each of which is radically different. Win rate analyses will generally respond to this issue by partitioning the data into particular matchups, and then analyzing balance in each matchup separately. This is probably the best way to salvage data taken from tournament games, but an essential sampling error is introduced when such a technique is employed. And this same error can interfere with racial representation in the higher levels of tournaments, poisoning that, too, as a metric for balance.
I'll explain this sampling error by a hypothetical. I do this partly because it's simpler and easier to understand in the abstract. But I also do this because balance is a notoriously heated issue, and it isn't my intention to argue that any particular match-up is or isn't imbalanced.
The Hypothetical
Imagine a hypothetical 2-race competitive game, with races A and B. The AvB match-up is highly skill-based, and quite balanced. The AvA and BvB match-ups are obviously balanced; but the AvA matchup is highly skill-based, while BvB wins are much more random, with highly skilled B players often losing to much worse B players simply because the matchup is much more chance-based. We'll also assume, for simplicity's sake, that a player's skill is the same in the AvB match-up as in their mirror match-up, even though in reality some players are obviously more skilled at one match-up than another.
A tournament is organized, with a large player pool of both A and B players, and the skill distribution appears to be quite even between the two races. In the first round, players are randomly paired and the winner advances. It doesn't much matter whether the format is single elimination, double elimination, or any other tournament format; all that matters is that each player is either playing the AvB match-up or their respective mirror match, and losers are eventually eliminated.
As stated previously, when the tournament begins the player pool had roughly equal skill of A players and B players. But by the end of the first round, a highly skilled B player is more likely to have been eliminated in a mirror matchup than a highly skilled A player, since the BvB matchup is so chaotic. Each round this pattern continues, until by the later stages of the tournament, the skill level of the B player pool is much lower than that of the A player pool. Obviously this quickly results in lower B representation in the upper levels of the tournament, since the now-weakened B player pool is more likely to fall in AvB matches.
How does this play out in the post-tournament balance statistics? Obviously anyone who looks at racial representation in the top 4, top 8, top 16, etc. will find see the A race dominating. More specifically, the racial representation will start out even, and then steadily drop for the B race as the tournament proceeds. Win rates in the AvB match-up will display a similar pattern; in the first round they should be perfectly balanced, but by the end of the tournament they will have shifted toward the A race. The effect is that an overall win rate of the AvB match-up will favor the A race, even though in a data set of games in which players are randomly paired and keep on playing win or lose, the match-up might appear to be balanced.
Conclusions
So what does that mean for Starcraft 2? Inevitably some balance warrior will use this argument to claim that Terran is actually more under-powered than balance statistics suggest, since the TvT match-up is often thought to be more skill-based than PvP or ZvZ. This is not the intention of this blog, and I believe that line of reasoning to be highly suspect. Starcraft 2 is not a hypothetical 2-race game; it is a real, complex 3-race game, and as such there are more considerations than just the randomness of the mirror match-up. For instance, if TvZ, TvP, and TvT all require quite different skills so very few Terrans are exceptionally good at all three, while all the Protoss match-ups require similar skills so being an all-around player is much more common, then both tournament representation and TvZ win rates would shift in favor of the Zerg race because talented Zergs are less likely to hit an unlucky match-up and get knocked out. There are so many confounding factors, I don't think much can be achieved by attempting to solve for and compensate for whatever error is introduced by this effect.
Ultimately, the only conclusion I wanted to draw is this: our current techniques for measuring balance have essential sampling errors in them. Because the winners in a tournament play more games than the losers, the win rate of any particular match-up is inevitably affected by the state of a race's other match-ups. Even if the other match-ups appear to be balanced, they could still throw off statistics by a disparity in the randomness with which wins are awarded.
Thanks for writing it!
