Statistical Analysis of StarCraft 2 Balance - Page 5

On May 07 2011 02:01 ffdestiny wrote:
Quoting Day9: "You can't really talk about balance before you take a hell of a lot of time analyzing the data." Unfortunately, your article has 5 references, uses a small sample size and jumps to conclusions based on your model. Obviously you're industrious and want to prove a point, but without just statistically gathering the entire database of games played (on this patch) there is no room for an argument on balance. Also, balance is so tied to maps that it almost nearly becomes a moot point to measure racial imbalances rather than map imbalances. There are just so many factors.

How do you measure balance in terms of the whole game? If we measure balance using data by pro players that doesn't indicate the whole game, but the subset.

How do you measure balance in terms of a race? If we measure by race then how do we correlate that data to maps.

How do you measure balance in terms of games? If we measure imbalance by games then how do we account for strategies that are intended to kill the opponent before he or she has expansions--cheesing or all-ins.

How do you measure balance by all of the above? If we measure imbalance by the whole game, the race and the games then how do those relate to one and another, because if we analyze all of the data and it comes up with a statistical win ratio favoring the zerg but then our subsets of data prove that zerg is weaker on certain maps, against certain strategies, etc. this totally negates our first assumption.

You see it's almost pointless to try and argue imbalance?

On May 06 2011 22:25 Elean wrote:
It looks like this model assumes that protoss players are extremely skilled (6 Protoss in the top 10 skilled player), and get to the conclusion that Protoss is underpowered.
Basicaly, it has exactly the same value as Idra saying "I'm the best player, I don't win, thus there is an imbalance".
(actually, this model can converge to different solutions, the particular solution the author got was "protoss players are skilled and protoss are underpowered...", it could very well have converged to "protoss players have no skill and protoss are overpowered")

All the people reading this should understand that this is not a scientific peer reviewed paper.

There is no way, this would be accepted as it is now.
If I were to review this paper I would ask for several modifications, and I would actually reject the paper unless the author answer this question:
How can you tell there is no offset on the "skill parameter" of all the players of 1 race ?

I would also ask a plot of the "skill pararemeter" distribution for each race.

On May 07 2011 01:49 Mactator wrote:
The imbalance issue is not necessarily related to probability of a player winning. The usual notion of "imbalance" refers to specific issues rather than xvy being imbalanced.

If we consider the notorious example of the protoss death ball then many people complains that zerg players can't win against it. Let's assume that is correct. Then the obvious thing to do if you are a zerg player is to avoid getting into late game against a protoss. This may be a very effective strategy and you may even measure a high probability for zerg players to win. The game would still be imbalanced though!

Therefore the question of imbalance is a matter of strategies. To quantify it you need to consider a specific case of imbalance. If for example you can prove statistically that zvp never goes in to late game and if it does then protoss has an extreme win-loss ratio then you can conclude either that
1) there is an imbalance issue
or
2) zerg players are bad at playing late game.

On May 07 2011 02:37 d_ijk_stra wrote:


First of all, I think you read it very carefully. Thank you very much for your interest.
I'll talk in technical sense, since it seems like you have good background in statistics.

The problem you're worried of can happen in "unidentifiable" cases,
that is, there are multiple parameters that can represent the same model.
This is not the case for this problem, since I used either

1) Use LASSO as a L_1 regularier,
2) Use non-informative gamers as baseline

Therefore, things like what you described cannot happen.
The existence of regularizer tries to not have the presence of extraordinary gamers
as much as it can, unless he wins too many games.

It is very important to check identifiability of the model before conducting an analysis,
and it is good for you to check out this issue. I understand for you to miss this point
since 1) I agree that the document is poorly written. It should be rejected in every journal/conference 2) you should've not read it as a professional reviewer

And it is also good to point out that THIS IS NOT A SCIENTIFIC PEER-REVIEWED PAPER.
I DID IT FOR FUN, and the fact that I am a Statistics major does not guarantee that the
analysis is correct. I didn't worry much at this point at the time posting it, but people without
proper background could've misled. Thanks.

On May 05 2011 10:21 professorjoak wrote:
Data set had only about ~620 nonmirror games in it. It would be interesting to use this methodology on the Brood War TSL Season 1 and 2 full ladder replay packs, which have several times more data in them.

I looked into trying a statistical analysis for TSL Season 1 at one point to see if the distribution of build orders on a map had any correlation with win percent. A first glance at the data showed all matchups on any map where I had 100+ games in that specific map and matchup balanced within 52-48. (Which is different than the Korean results in the TLPD which usually split 60-40 or 55-45, though those are based on far fewer games). However, I then realized the data set had many duplicate games from a game between two top ladder players being counted in each player's replay pack and decided it would be too much trouble to properly sort them out so I quit there and didn't take the analysis much further.

On May 07 2011 02:42 d_ijk_stra wrote:


To make a change in a game as a patch, you are definitely correct.
However, there ARE imbalances sometimes.
If you watched BW for a long time, do you remember the infamous map Mercury?
What was the P score? Did P win more than 2 games in that map?

From game to game, yes there are differences.
Even July was defeated in Mercury in OSL final.
However, everyone who has been playing SC1/SC2 for long time KNOWS that
certain maps REQUIRE PLAYERS of certain race to do things x, y, z,....
and thus it leads to imbalance issues.

On May 07 2011 03:02 Elean wrote:

Your model is:

logit(P)=beta_player1-beta_player2+beta_matchup

You use the LASSO method to fit the value of beta_playerx, and beta_matchupx

You get ONE fit, but there are other degenerated solutions, here is the proof:
Take the values of your solution, then decrease by 10000 all the beta_player of protoss players, then increase by 10000 the beta_PvZ and the beta_PvT.
If you do that, you get another fit that is just as good as the one you first had (i.e. all the logit(P) are unchanged). However, now the beta_PvZ and the beta_PvT are extremely high, and protoss become clearly overpowered.


Your model is probably good to estimate how likely a player is to win a match, but it is 100% blind to balance.

The problem is that all the players only play 1 race, and you will never be able to make the difference between "all the protoss players are way better than the others, but protoss is underpowered" and "all the protoss players are noobs, but it's ok since protoss is way overpowered".
There is absolutely nothing you can do about it. Not with this sample of data.

On May 07 2011 05:54 d_ijk_stra wrote:


By LASSO, you mean the existence of (L_1) regularizer.
When you add 10,000 to your parameter, you are being penalized a lot.
I suspect you understand the concept of regularization, sorry.

On May 07 2011 15:31 Elean wrote:

As far as I can tell, LASSO is a least square method that set up a constraint on the L1 norm. Constraint that has no justification in this case.

You have to understand that if 2 models give the exact same results for any match, there is no way to tell which one is better. I explained to you, that there is an infinite number of models that will give the same resuts with different "balance between 2 races". This means you can not tell if there is an unbalance.

I will explain to you on an example, why the L1 constraint does not have any justification.

For simplicity sake, let's consider only 2 races, T and Z, and let's consider that all the players of 1 race have the same skill.
Suppose TvZ is unbalance, and the actuall value of beta_TvZ is 500.
Since all the players made it in the tournaments, they are likely to have roughly the same strengh (skill + balance). This means the Z players have likely a beta_player that is 500 above the beta_player of the T players.

Now, run your mode, with a sample size extremely large. You get the solution beta_TvZ=0, beta_playerZ=0 and beta_playerT=0. This solution clearly minimize the L1 norm, and is also give the exact result for the probability of each match. However, it is completly wrong and does not manage to see the unbalance.

Your method fails to catch any unbalance for the exact same reason everyone can't tell the balance: we don't know if the zerg players are more or less skilled than the terran players.

On May 07 2011 15:31 Elean wrote:

As far as I can tell, LASSO is a least square method that set up a constraint on the L1 norm. Constraint that has no justification in this case.

You have to understand that if 2 models give the exact same results for any match, there is no way to tell which one is better. I explained to you, that there is an infinite number of models that will give the same resuts with different "balance between 2 races". This means you can not tell if there is an unbalance.

I will explain to you on an example, why the L1 constraint does not have any justification.

For simplicity sake, let's consider only 2 races, T and Z, and let's consider that all the players of 1 race have the same skill.
Suppose TvZ is unbalance, and the actuall value of beta_TvZ is 500.
Since all the players made it in the tournaments, they are likely to have roughly the same strengh (skill + balance). This means the Z players have likely a beta_player that is 500 above the beta_player of the T players.

Now, run your mode, with a sample size extremely large. You get the solution beta_TvZ=0, beta_playerZ=0 and beta_playerT=0. This solution clearly minimize the L1 norm, and is also give the exact result for the probability of each match. However, it is completly wrong and does not manage to see the unbalance.

Your method fails to catch any unbalance for the exact same reason everyone can't tell the balance: we don't know if the zerg players are more or less skilled than the terran players.

On May 07 2011 23:47 d_ijk_stra wrote:


That part is already (implicitly) mentioned by other users. If there is NO MIRROR MATCH, your point is right. The existence of mirror match enables you to do such an analysis.

Of course, (as another user already pointed out) you can question it. Every gamer may have different levels of skill depending on the race of his/her opponent. But I think this assumption itself is not too strong to make everything nonsense: we know that most top level players are also good at mirror matches.

On May 07 2011 23:56 d_ijk_stra wrote:


Oh, and it seems like you missed this point: every beta_player of each user is ALSO penalized by LASSO. This is a very important point, but I thought when I say LASSO everyone would imagine every variable is being penalized. Isn't it the usual case? I think not penalizing certain variables is an exceptional case when using LASSO.