|
On August 17 2010 07:35 neobowman wrote:
Isn't this math and not science?
XD great!!! Is math actually science? Ask Nobel :D
On August 17 2010 07:38 Muirhead wrote:
All this stuff is invalidated because of blizzard's matchmaking service, which will make all but the very best and worst players on the entire ladder converge to a 50% win-rate.
this
When you loose to much, you´ll be matched against weeker opponents. Even against those ranked in 1 - 2 Divisions under yours
great post anyway
|
On August 17 2010 07:35 neobowman wrote:
Isn't this math and not science?
nope.
mathematics is a form a logical deductions based on #s.
this is science because they are logical (or i'd say illogical for this case) inductions based on facts.
|
On August 17 2010 07:18 StarcraftGuy4U wrote:
None of these stats are worthwhile because the matchmaking system does not assign people like they would in a blind study, instead it is actively adjusting the matches so that every player reaches 50%. The numbers you are pulling are worthless for this reason.
QFT. You shouldn't be a scientist. The match making system is designed so that you get players as close to a 50% win rate as possible. So in a hypothetical situation of one race being overpowered, the win %'s of the race will not change, merely the distribution of players. You should look at the distribution per rating by race.
|
given enough games and time winrates for players on any ladder should be approaching 50% so i guess if we assume that blizzard's matchmaking system is working correctly, we can look to racial distribution at top levels for an indication of balance right?
|
I would think the best way Blizzard could test balance at all levels is to have separate hidden ELOs for each MU. Then they could see that the typical Diamond Z is 600 in ZvZ and 550 in ZvT, for example.
|
The problem the way you're calculating the imbalances is that you're assuming that races should have different win percentages. A 1000 point terran may only be as skilled as an 800 point zerg, but both may have 50% win records at their perspective levels. It's just the terran has a racial imbalance allowing him to play competitively at 1000 points.
This graph: http://www.sc2ranks.com/stats/race/us/1
shows that as points goes up, terran starts to dominate the ranks more and more, and zerg gets worse and worse.
|
On August 17 2010 07:41 GagnarTheUnruly wrote:
I wouldn't try to extrapolate my 'results' to the pro level, just because the level of play is so great and the game itself is played so differently. Also, I would guess that performance of top-teir pros on the ladder wouldn't closely correlate with tournament performance, due to differences in their play habits. I think the only way to know if SC2 is balanced at the pro level would be to compile results of tournaments, and I suspect that there haven't been enough of those to give a difinitive answer (BW is pretty streak-y for certain races, for example).
Regarding the other criticisms, I was under the impression that players in the same league get roughly the same player draws as one another. Is this true, or do higher ranked players draw from a different pool of players than lower ranked players in the same league?
Well they have to draw from other leagues to all have >50% win rate.
With Diamond only statistics, there will be some that are <50% and over 50%.
|
On August 17 2010 07:48 ejac wrote:The problem the way you're calculating the imbalances is that you're assuming that races should have different win percentages. A 1000 point terran may only be as skilled as an 800 point zerg, but both may have 50% win records at their perspective levels. It's just the terran has a racial imbalance allowing him to play competitively at 1000 points. This graph: http://www.sc2ranks.com/stats/race/us/1shows that as points goes up, terran starts to dominate the ranks more and more, and zerg gets worse and worse.
Yeah, this is much more scientific "proof" that Terran is in fact OP, and Z is the worst race.
I mean cmon... look at it, you would have to be blind to not see a relationship: http://www.sc2ranks.com/stats/race/us/1
|
On August 17 2010 07:51 Wr3k wrote:Show nested quote +On August 17 2010 07:48 ejac wrote:The problem the way you're calculating the imbalances is that you're assuming that races should have different win percentages. A 1000 point terran may only be as skilled as an 800 point zerg, but both may have 50% win records at their perspective levels. It's just the terran has a racial imbalance allowing him to play competitively at 1000 points. This graph: http://www.sc2ranks.com/stats/race/us/1shows that as points goes up, terran starts to dominate the ranks more and more, and zerg gets worse and worse. Yeah, this is much more scientific "proof" that Terran is in fact OP, and Z is the worst race. I mean cmon... look at it, you would have to be blind to not see a relationship: http://www.sc2ranks.com/stats/race/us/1
that's ignoring the fact that as skill level gets higher sample size becomes smaller and you have to compensate for a margin of error (calculate significance or something?). 70:30 with a sample size of 1000000 is vastly different from 70:30 with a sample size of 10.
|
This doesn't show that the game is balanced at all. All this shows is that matchmaking is pretty good.
|
Re-work your hypothesis based on your new understanding of how the matchmaking system actively creates 50% win rates and run the numbers again!
+ Show Spoiler +
Isn't science fun?
|
Kudos to the OP. This is actually a very well-done statistics test.
Unfortunately, people have said that the matchmaking system will pair players in a way to guarantee balanced win rates. In which case... 
Nevertheless, I think it's pretty interesting to compare the win rate results at the lowest leagues. I would never have expected such a large difference going from bronze to silver. For the longest time, I thought both were pretty much the same animal (as in, a Silver Leaguer only sucks a hair bit less than a Bronze Leaguer.) Either that, or Bronze leaguers suck so bad that the matchmaking system has a hard time giving them "easy wins" to even out the win rates.
|
On August 17 2010 07:45 mahnini wrote:
given enough games and time winrates for players on any ladder should be approaching 50% so i guess if we assume that blizzard's matchmaking system is working correctly, we can look to racial distribution at top levels for an indication of balance right?
I agree. Here's a graph of the racial distributions. The y-axis is proportion of that race in the games played pool for a certain league.
![[image loading]](http://i667.photobucket.com/albums/vv32/GagnarTheUnruly/meanpropgraph.jpg)
This indicates that racial imbalances aren't causing weak races to get held back in lower leagues. If some of the earlier criticisms were true, that matchmaking obviates differences in racial performance, we should see some races gaining prominence and others losing it as you move through the leagues. In particular, the races that indicated as slightly weak in my analysis should fall out of diamond. Comparing silver through diamond you can see that this isn't the case. For example, zerg gets more common.
I think it's reasonable to conclude that the races are pretty balanced, but I acknowledge that some of the criticisms I'm getting are valid.
The analysis was a chi-square analysis comparing observed distributions vs. homogenous distributions (assumed under random sorting).
Also, this is definitely science, because it uses a hypothesis-based testing approach. Math is just a tool to accomplish the science. Whether it's good science seems to be stimulating a rigorous debate LOL.
The 'real' mathematical way to test for imbalance would probably require treating players individually, and using regression-based approaches to predict performance based on race, league placement, etc. That way one could parse out the influence race has on win rate. I don't have access to that kind of data, though. The best would be to control for player as a variable, to see if players consistently perform better with certain races than others.
Edit: these^^ are games played not players active, so take the graph with the appropriate grain of salt.
|
the results are around a statistical mistake.
Also you dont take into account different types of build that player could play / undiscovered ways of play etc.
E.g a nice proof will be to compute a build and check at which point player A could make heavy push on player B, how long is that timing window etc. Checking does equally macroing players will end with a draw or one of them will loose and so on.
|
On August 17 2010 07:53 mahnini wrote:Show nested quote +On August 17 2010 07:51 Wr3k wrote:On August 17 2010 07:48 ejac wrote:The problem the way you're calculating the imbalances is that you're assuming that races should have different win percentages. A 1000 point terran may only be as skilled as an 800 point zerg, but both may have 50% win records at their perspective levels. It's just the terran has a racial imbalance allowing him to play competitively at 1000 points. This graph: http://www.sc2ranks.com/stats/race/us/1shows that as points goes up, terran starts to dominate the ranks more and more, and zerg gets worse and worse. Yeah, this is much more scientific "proof" that Terran is in fact OP, and Z is the worst race. I mean cmon... look at it, you would have to be blind to not see a relationship: http://www.sc2ranks.com/stats/race/us/1 that's ignoring the fact that as skill level gets higher sample size becomes smaller and you have to compensate for a margin of error (calculate significance or something?). 70:30 with a sample size of 1000000 is vastly different from 70:30 with a sample size of 10.
Yes, the sample size is very small, but there are only so many 800+ diamond players to get data from. SC2 needs to be balanced at the highest level. It's the only sample size we have. A small sample size with a analysis that actually makes sense is still infinitely better than one with a large sample size that is completely and utterly flawed. I know the sc2ranks numbers are unreliable due to their size, but there is still enough players above 600/700 to show that the difference in racial distribution is significant. The real question is whether or not players perform better with one race than another, or if more people at the top are just choosing terran. All the OP has shown with his numbers is that the match making system is working properly.
|
OP, I really like what you’ve attempted here (as a probability and statistics major!).
Perhaps a more meaningful test would be a Chi-Square test of independence on the number of diamond level players of each race:
Find percentage of total players of each race (at ALL levels). Using this, find the expected number of diamond ranked players of each race under the assumption that league placement is independent of race. Use the appropriate Chi-Square test statistic to determine the likelihood that the observed proportion of diamond players of each race fits what would be expected under independence.
I think this will help alleviate bias caused by the matchmaking service, as all we care about is that it places people into diamond correctly.
Also, we will have accounted for the fact that more terran players on the whole means more terran players in diamond, if the skill level of players across races is equal.
A problem is that the skill level of players across races may not be equal. But I have to think that players of greater skill tend to choose the “stronger” race, so the relative strength of that race would be reflected in our results.
|
I'm a bit confused about your method, why (and what) are you assuming are chi-square distributed? The win ratio? Wouldn't it be more appropriate to assume a normal distribution assuming that each game is binary distributed, either you will win - or you will lose - and there is a certain probability for one of the events to happen (hm I get a feeling that it is more complicated than that). If that is true, the central limit theorem states that you could approximately assume a normal distribution when the amount of games increases (if my memory serves me right).
I would also suggest that you in your method state your assumptions, and in this case a definition of imbalance would be in place. One factor that such a definition most likely has to include is the skill of the player. We assume that a matchup is imbalanced if players of equal skill lose due to their choice of race rather than to their lack of skill. For this reason I would advise against trying to use numbers in order to balance the game.
|
On August 17 2010 08:04 stochastic wrote:
OP, I really like what you’ve attempted here (as a probability and statistics major!).
Perhaps a more meaningful test would be a Chi-Square test of independence on the number of diamond level players of each race:
Find percentage of total players of each race (at ALL levels). Using this, find the expected number of diamond ranked players of each race under the assumption that league placement is independent of race. Use the appropriate Chi-Square test statistic to determine the likelihood that the observed proportion of diamond players of each race fits what would be expected under independence.
I think this will help alleviate bias caused by the matchmaking service, as all we care about is that it places people into diamond correctly.
Also, we will have accounted for the fact that more terran players on the whole means more terran players in diamond, if the skill level of players across races is equal.
A problem is that the skill level of players across races may not be equal. But I have to think that players of greater skill tend to choose the “stronger” race, so the relative strength of that race would be reflected in our results.
Yes OP, please do, because I played starcraft in my stats classes and crammed to get a B.
|
United Kingdom16710 Posts
statistics on each matchups would be more pertinent and as someone already said, the way matchmaking works means you really cant get a good sense from your results. good effort though.
|
On August 17 2010 08:04 Wr3k wrote:Show nested quote +On August 17 2010 07:53 mahnini wrote:On August 17 2010 07:51 Wr3k wrote:On August 17 2010 07:48 ejac wrote:The problem the way you're calculating the imbalances is that you're assuming that races should have different win percentages. A 1000 point terran may only be as skilled as an 800 point zerg, but both may have 50% win records at their perspective levels. It's just the terran has a racial imbalance allowing him to play competitively at 1000 points. This graph: http://www.sc2ranks.com/stats/race/us/1shows that as points goes up, terran starts to dominate the ranks more and more, and zerg gets worse and worse. Yeah, this is much more scientific "proof" that Terran is in fact OP, and Z is the worst race. I mean cmon... look at it, you would have to be blind to not see a relationship: http://www.sc2ranks.com/stats/race/us/1 that's ignoring the fact that as skill level gets higher sample size becomes smaller and you have to compensate for a margin of error (calculate significance or something?). 70:30 with a sample size of 1000000 is vastly different from 70:30 with a sample size of 10. Yes, the sample size is very small, but there are only so many 800+ diamond players to get data from. SC2 needs to be balanced at the highest level. It's the only sample size we have. A small sample size with a analysis that actually makes sense is still infinitely better than one with a large sample size that is completely and utterly flawed. I know the sc2ranks numbers are unreliable due to their size, but there is still enough players above 600/700 to show that the difference in racial distribution is significant. All the OP has shown with his numbers is that the match making system is working properly.
saying there are enough player to make the imbalance significant doesn't make it so there are calculations for this but i'm terrible and don't know how to do them. though the OP itself doesn't show us anything revelating it does give us a concrete basis off which we can make assumptions such as: blizzards matchmaking works properly therefore we can look towards racial distribution at certain levels to help gauge imbalance.
if there are a proportional amount of zerg in at the top levels as in the general population it means the ladder perceives the skill levels of those zergs to be high which would not be the case if a certain matchup were extremely imbalanced.
|
|
|
|
|
|
EPT![[image loading]](http://img641.imageshack.us/img641/6781/science2020you27re20doi.jpg)
