|
Since there is a never-ending discussion on balance in Starcraft II, an objective measure would be very useful. This is where the WCS comes in handy. I analysed the points per race to see if there are any significant differences. Because many players are included and points can be earned over several seasons, not just in a single tournament, I think the WCS gives a good picture of pro-level play. I do not mean to generalise these results to other levels of play, and of course in-depth discussions are necessary too, as this analysis says nothing about certain match-ups, builds, units, etc.
Because the points were not distributed normally (very skewed towards the low-end), I ran Kruskal-Wallis tests in PASW 18. With this kind of test, the absolute points are irrelevant, only the relative position is used. The median is therefore a better representative of average race performance than the mean.
 Protoss
43 players
Mean: 1400
SEM: 200
Median: 875
 Terran
38 players
Mean: 1555
SEM: 274
Median: 800
 Zerg
42 players
Mean: 1314
SEM: 200
Median: 700
Race differences
The Kruskal-Wallis test using the Monte Carlo method did not yield a significant result (H(2) = 0.038, p = 0.98). This is almost as close to absence of a difference as possible. I got similar results with an ANOVA, which would be invalid anyway because the assumptions are not met.
Region differences
I also did the analyses per region, and again found no race differences. America: H(2) = 1.38, p = 0.51. Europe: H(2) = 0.68, p = 0.72. Korea: H(2) = 0.15, p = 0.93.The number of players per race per region is about 14, so the sample size is a lot smaller and thus less reliable. However, given the high p-values I think we can be fairly certain there are no race differences, regardless of region.
Finally, I looked at whether certain regions performed better and found no difference (H(2) = 0.96, p = 0.63). I didn't test for differences between nationalities, because Koreans are over-represented.
Thus, all races performed strikingly similar in the WCS.
Note: I posed this on other forums before, but couldn't do so here earlier because of the 3 day waiting period.
|
Your stats say nothing about "deserved" wins. 
EDIT: Great effort though. Always nice to get some numbers out.
|
Yay statistics! <3
What a great idea! Wish I had come up with that! I might just repeat your analysis just for fun 
Yeah, those data are skewed a lot! Which of course isn't really a surprise.
Two things: Why is your n=123 when there are 125 player datas available on wcs.battle.net. Also that list seems to cut players off who made less than 250 points. Dunno if that is a noticable sample at all but it's of course highly unlike to change the outcome of your analysis at all, if we are even at all interesting in that level of play of course =)
Very nice analysis in general! Thanks for the effort!
Also, why is this in the tournament section? This should probably just be SC2 general.
|
fantastic analysis! the game really does seem balanced (at the moment...)
|
On November 14 2013 20:24 Cereb wrote:Yay statistics! <3 What a great idea! Wish I had come up with that! I might just repeat your analysis just for fun Yeah, those data are skewed a lot! Which of course isn't really a surprise. Two things: Why is your n=123 when there are 125 player datas available on wcs.battle.net. Also that list seems to cut players off who made less than 250 points. Dunno if that is a noticable sample at all but it's of course highly unlike to change the outcome of your analysis at all, if we are even at all interesting in that level of play of course =) Very nice analysis in general! Thanks for the effort! Also, why is this in the tournament section? This should probably just be SC2 general.
I'm glad you liked it! Please feel free to do it as well, you know the importance of replication!
When I collected the data a few days ago, the number 125 was actually number 123 (124 and 125 were missing). It looks like it's fixed on the website now, but they just cut off the list at 125 anyway. Is there a longer list then?
I put it in tournaments because it is about a tournament, otherwise people start to complain again it's not representative of "SC2 general", but I think it would fit in there too.
|
Could even justify this by doing simple t-tests, 40ish players is probably enough to make the distribution of the mean relatively normal. Anyway just looking at the numbers it's pretty obvious this wouldn;t give any significant result no matter what test you do.
It's an interesting idea for a test though because it elegantly uses the data of a whole year of HotS and it's actually independent datapoints. The general problem I frown about with lot of these statistical approaches to balance is that they are often coming from a tournament here and there where the data is not independent at all; the good players play more matches and highly influence the data.
I still wish they would just post winrates from grandmaster corrected for skill, that is still the best single way to look at the balance I think.
|
On November 14 2013 20:33 Astro_Peasant wrote:Show nested quote +On November 14 2013 20:24 Cereb wrote:Yay statistics! <3 What a great idea! Wish I had come up with that! I might just repeat your analysis just for fun Yeah, those data are skewed a lot! Which of course isn't really a surprise. Two things: Why is your n=123 when there are 125 player datas available on wcs.battle.net. Also that list seems to cut players off who made less than 250 points. Dunno if that is a noticable sample at all but it's of course highly unlike to change the outcome of your analysis at all, if we are even at all interesting in that level of play of course =) Very nice analysis in general! Thanks for the effort! Also, why is this in the tournament section? This should probably just be SC2 general. I'm glad you liked it! Please feel free to do it as well, you know the importance of replication! When I collected the data a few days ago, the number 125 was actually number 123 (124 and 125 were missing). It looks like it's fixed on the website now, but they just cut off the list at 125 anyway. Is there a longer list then? I put it in tournaments because it is about a tournament, otherwise people start to complain again it's not representative of "SC2 general", but I think it would fit in there too.
Ah, ok! Makes sense! Don't know where to find a longer list, but honestly, I feel it's very much reasonable to just use the top 125, especially since it's not just some arbitrary cut-off, but because that is what Blizzard listed, and it's a pretty decent sample size for the chosen analysis regardless
Hehe, yeah it's not easy selecting the "right" forum! It seems though that "SC2 Tournaments" is mostly for the actual tournaments and their matches, whereas SC2 general is more for discussion overall results of those and this really encompasses the entirety of last season =)
On November 14 2013 20:47 Markwerf wrote:
Could even justify this by doing simple t-tests, 40ish players is probably enough to make the distribution of the mean relatively normal. Anyway just looking at the numbers it's pretty obvious this wouldn;t give any significant result no matter what test you do.
It's an interesting idea for a test though because it elegantly uses the data of a whole year of HotS and it's actually independent datapoints. The general problem I frown about with lot of these statistical approaches to balance is that they are often coming from a tournament here and there where the data is not independent at all; the good players play more matches and highly influence the data.
I still wish they would just post winrates from grandmaster corrected for skill, that is still the best single way to look at the balance I think.
Indeed! But how would you actually adjust for skill? As it's a very ambiguous term I feel. I can't think of anything other than adjusting for MMR which I think they claim to be doing already.
|
On November 14 2013 20:47 Markwerf wrote:
Could even justify this by doing simple t-tests, 40ish players is probably enough to make the distribution of the mean relatively normal. Anyway just looking at the numbers it's pretty obvious this wouldn;t give any significant result no matter what test you do.
It's an interesting idea for a test though because it elegantly uses the data of a whole year of HotS and it's actually independent datapoints. The general problem I frown about with lot of these statistical approaches to balance is that they are often coming from a tournament here and there where the data is not independent at all; the good players play more matches and highly influence the data.
I still wish they would just post winrates from grandmaster corrected for skill, that is still the best single way to look at the balance I think.
So the data was far removed from normality (K-S test), and other assumptions like homogeneity of variance was not met either. ANOVA is basically a t-test for multiple groups, which as I said, had similar results to the nonparametric test.
The good players play more matches because they are better, and thus have more points. And since there are about the same number of players for each race, this is not an issue.
How would one correct for skill in grandmaster? There are many interesting analyses that could be done with ladder data, especially if you include multiple levels like match-ups and match duration, which would give a more detailed picture. But that would take a lot of work to collect unless someone already has a well-ordered dataset.
|
|
|
|
|
|
EPT
