EPT![[image loading]](http://s10.postimage.org/gyvjiumdx/winrates.jpg)
Ladder-Balance-Data - Page 24
| Forum Index > SC2 General |
|
Not_That
287 Posts
| ||
|
ZjiublingZ
United Arab Emirates439 Posts
On July 15 2012 20:53 skeldark wrote: Update the result with a lot of stats: Result Source Main Data + Show Spoiler + - The data is biased towards EU/US and towards higher skill-rate. Gamescount: 125976 Sc2-Accounts: 45203 -worst to best player: 3200 MMR -one average win/loose on Ladder: +16 / -16 MMR TIME Filter: only between 1 Jan 1970 00:00:00 GMT - 12 Jul 2012 16:52:47 GMT Average MMR per Race + Show Spoiler + Race account count: 15814 Data average MMR: 1539.46 Difference in average MMR per Matchup: T-P: -62.14 T-Z: -117.03 P-Z: -54.89 Average Win-ratio per Race + Show Spoiler + TvP 50.43 Games: 6700 TvZ 46.7 Games: 8118 PvZ 51.61 Games 9189 Win-ratio per Race over Game-Time + Show Spoiler + TvP gamelength,%race1 win,%race2win, %of games 0,44.9,55.1,3.66 5,40.71,59.29,13.9 10,58.32,41.68,24.21 15,59.7,40.3,24.78 20,45.72,54.28,18.31 25,37.79,62.21,9.16 30,35.04,64.96,3.49 35,46.71,53.29,2.49 TvZ gamelength,%race1 win,%race2win, %of games 0,37.13,62.87,3.78 5,33.78,66.22,9.15 10,46.91,53.09,15.96 15,52.51,47.49,22.12 20,47.88,52.12,22.9 25,44.36,55.64,14.3 30,50.0,50.0,6.65 35,48.08,51.92,5.12 PvZ gamelength,%race1 win,%race2win, %of games 0,47.38,52.62,4.57 5,38.3,61.7,11.39 10,59.72,40.28,25.07 15,50.17,49.83,25.36 20,49.97,50.03,17.34 25,53.21,46.79,9.14 30,51.0,49.0,4.37 35,58.89,41.11,2.75 This is really cool. Especially awesome to see TvZ 50/50 at 30 minutes, and PvZ 51/49 at 30 minutes. | ||
|
Atrimex
193 Posts
| ||
|
Account252508
3454 Posts
| ||
|
Rick Deckard
90 Posts
Here's why, imagine there is a race exactly like zerg but is only be played by blind people, call it blind-zerg. This race would have a very low average MMR as blind people obviously can't play starcraft as well as sighted people. But because the blind-zerg race is the same as zerg it's no weaker than zerg. Thus a low average MMR per race doesn't necessarily imply that race is weaker. In conclusion the statistics gathered in the study provided can't be used to make conclusions about how well starcraft 2 is balanced. In practice the data is confounded by the fact that the average skill of of players of different races is not necessarily the same. Dividing race MMR by number of players per race to determine average race MMR doesn't change this. | ||
|
Zacsafus
England255 Posts
On July 18 2012 22:47 Rick Deckard wrote: I believe varying average MMR between races is not an indicator of imbalance. Here's why, imagine there is a race exactly like zerg but is only be played by blind people, call it blind-zerg. This race would have a very low average MMR as blind people obviously can't play starcraft as well as sighted people. But because the blind-zerg race is the same as zerg it's no weaker than zerg. Thus a low average MMR per race doesn't necessarily imply that race is weaker. In conclusion the statistics gathered in the study provided can't be used to make conclusions about how well starcraft 2 is balanced. In practice the data is confounded by the fact that the average skill of of players of different races is not necessarily the same. Dividing race MMR by number of players per race to determine average race MMR doesn't change this. But blind people aren't more likely to pick one race over another, its not possible to imply that one race has worse players on it, at the highest levels everyone is competent and displays good skill so your point isnt valid at high levels, or really at any level because there is no bias between the races of which a handicapped/less-skilled player would pick | ||
|
aintthatfunny
193 Posts
| ||
|
Rick Deckard
90 Posts
On July 18 2012 23:11 Zacsafus wrote: But blind people aren't more likely to pick one race over another, its not possible to imply that one race has worse players on it, at the highest levels everyone is competent and displays good skill so your point isnt valid at high levels, or really at any level because there is no bias between the races of which a handicapped/less-skilled player would pick It's not necessary for blind people pick one race over another for my point to be valid. I don't expect blind people to play starcraft. What I've shown is that differences in average MMR per race doesn't always imply imbalance, even significantly different average MMRs. This is proof by contradiction, it only requires a single counter example to disprove a rule. Given that (my proof is valid that) differences in average MMR per race doesn't imply imbalance in general, it also doesn't per se imply imbalance in starcraft 2. I'm not making a statement about the balance of the game at either high or low skill level. Just pointing out a logical flaw in the argument that different average race MMRs indicate imbalance. As best I can tell the author of the study has concluded because average MMR per race is different therefore the game is imbalance. I believe that logic to be flawed. | ||
|
Account252508
3454 Posts
| ||
|
Lysenko
Iceland2128 Posts
* Not_that's work makes a lot of sense, but it's quite clear (and stated very explicitly in that post) that you can only use that technique to find a master league player's MMR with respect to the 0 point of master league. How have you backed out MMRs for the other leagues, particularly considering how many different offsets there are per league and that you can't see the offsets? More to the point, if that were the case it would also cause the ladder-point-space MMR number not to follow a normal distribution, since it would be a normal distribution with a nonlinear mapping applied to it. Edit: I forgot when I wrote this that lolcanoe's analysis seems to confirm that there is close to a normal distribution to these ladder-point-scale MMR numbers, so this is maybe a moot point. * Finally, MMR is known not to be Elo, so if there are any points in any of these analyses that assume MMR to be Elo, those points are not valid. I only saw one place in Not_that's post where Elo came up, and it was along the lines of "oh, I saw blah blah blah in the data and that roughly reminds me of blah blah blah in Elo, so it's probably correct." That kind of thing is fine, though maybe not as good a reinforcement as he thinks. * Edit: REALLY finally -- to the extent that skeldark is emphatic that there is no visible MMR cap in his data, that calls into question whatever process happened to the data before it wound up in ladder-point-scale MMR space, because we KNOW that there is such a cap. If that process is broken, all bets are off. Bottom line is that I'm not sure that any of these issues are fatal to lolcanoe's analysis of the data set, assuming there's some answer to the first point, though skeldark's approach of generating tons of random games using some black-box code he wrote is a highly dubious way to interpret the data, and the question of why no MMR cap is visible is problematic. (Edit: It may simply be that the MMR cap does not affect enough people in the data set to be clearly evident.) However, even with lolcanoe's confirmation that there is a modest amount of variation between races in this data set, there's still no way to tell why that is -- whether it comes from game design, player-originated biases in race choice, or simply players not having caught up to the current state of the game in their understanding of how, optimally, to play the races against each other. In that light, this whole discussion is a lot of heat and very little illumination. | ||
|
Rick Deckard
90 Posts
On July 18 2012 23:52 monkybone wrote: Yes, this assumes a similar skill distribution for each race. Author has basically just defined balance as average MMR, which doesn't give any information. But taking the skill distribution in consideration, then the MMR distribution gives evidence of balance. OK. That makes sense then. Thank you for the clarification. Personally I'll follow top tournament results for significant imbalance indications. I hope that Blizzard continues to improve the balance of WOL. The win rates over time are interesting in the study. Thank you to whoever took the time to put that together. | ||
|
Not_That
287 Posts
On July 18 2012 23:58 Lysenko wrote: OK, so I've gone back over the "find your MMR in one game" post by not_that and I am confused about a few things that represent a large leap from there to this post. * Not_that's work makes a lot of sense, but it's quite clear (and stated very explicitly in that post) that you can only use that technique to find a master league player's MMR with respect to the 0 point of master league. How have you backed out MMRs for the other leagues, particularly considering how many different offsets there are per league and that you can't see the offsets? More to the point, if that were the case it would also cause the ladder-point-space MMR number not to follow a normal distribution, since it would be a normal distribution with a nonlinear mapping applied to it. Edit: I forgot when I wrote this that lolcanoe's analysis seems to confirm that there is close to a normal distribution to these ladder-point-scale MMR numbers, so this is maybe a moot point. * Finally, MMR is known not to be Elo, so if there are any points in any of these analyses that assume MMR to be Elo, those points are not valid. I only saw one place in Not_that's post where Elo came up, and it was along the lines of "oh, I saw blah blah blah in the data and that roughly reminds me of blah blah blah in Elo, so it's probably correct." That kind of thing is fine, though maybe not as good a reinforcement as he thinks. * Edit: REALLY finally -- to the extent that skeldark is emphatic that there is no visible MMR cap in his data, that calls into question whatever process happened to the data before it wound up in ladder-point-scale MMR space, because we KNOW that there is such a cap. If that process is broken, all bets are off. Bottom line is that I'm not sure that any of these issues are fatal to lolcanoe's analysis of the data set, assuming there's some answer to the first point, though skeldark's approach of generating tons of random games using some black-box code he wrote is a highly dubious way to interpret the data, and the question of why no MMR cap is visible is problematic. (Edit: It may simply be that the MMR cap does not affect enough people in the data set to be clearly evident.) However, even with lolcanoe's confirmation that there is a modest amount of variation between races in this data set, there's still no way to tell why that is -- whether it comes from game design, player-originated biases in race choice, or simply players not having caught up to the current state of the game in their understanding of how, optimally, to play the races against each other. In that light, this whole discussion is a lot of heat and very little illumination. I'll be happy to address the points in your post as well as future ones you may have. The output of F gives what we refer to as dmmr, which is the difference between the player's MMR and the opponent's league and tier offset. For example if player A plays opponent B who is from an unknown diamond division tier and has 300 adjusted points before the game and loses 12 points, we can tell that A's dmmr is 300+-14 (the deviation of F for 12 points change matches) in relation to B's diamond tier offset. If B was in master we would know A's MMR right there and then, however for all leagues with multiple tiers it is more complicated than that. This is why the MMR calculator requires more than a single match for players who play opponents below master before it becomes accurate. Once we have multiple consecutive matches of A against opponents below master, we can infer more about A's MMR by looking at the series as a whole. We know that A's MMR increases after every win and decreases after every loss. From this we can start making predictions of A's opponent's tiers in their leagues. It is a fairly difficult problem and it has taken Skeletor quite a while to get it right, but by the current version of the calculator the predictions are very accurate at high game count. You may dispute his methods, but for the current topic of discussion it's enough to say that the individual races of A or his opponents do not play a role whatsoever. So any potential problem which may exist in the calculation method is unbiased towards producing results that show one race as having a different MMR than the others. Skeletor described his methods in several posts. You call the calculator a black box, but he has released a source code (which afaik got no attention). On to next point: MMR is not ELO (adjusted ladder points are very similar to ELO however). We never claimed that it is, and it's irrelevant. We do not calculate MMR changes, we simply read MMR values and let the ladder system worry about how MMR behaves. We discussed the surprising behavior of MMR several times and how the "uncertainty value" stored for each player is surprisingly the same for all players who have played some matches since buying the game, even for players who experience DRAMATIC MMR fluctuations. You can literally go from Bronze to Master with 90+% winratio and your uncertainty value will be the same as that of a player who is steady. However once again I emphasize it has nothing to do with current topic as we don't care about the uncertainty value or what system governs MMRs, we simply read MMR values. Regarding MMR cap, I haven't dug too deep into it. I looked at #1 ranked GM earlier in the season and his adjusted points (around 820 at the time. It's 981 for #1 on Europe atm) were close to his MMR. That to me indicated that if that player is MMR capped, he's probably very close to the cap as adjusted points and dMMR tend to get roughly close to each other after having played enough games and not experience MMR shifts. For a player who is MMR capped we expect his adjusted points to go quite a bit higher than his MMR and towards his uncapped dMMR, the reason for which I explain in #1 in the next paragraph. The reasons I haven't spent much time on this are: 1) If player A is MMR capped, that doesn't affect A whatsoever. It affects A's OPPONENTS. They receive less points for beating A and lose more points for losing to him. Assuming that only few players at the top are capped, this would spread out across a large multitude of opponents each suffering small impacts from playing against capped players. The capped player's ladder points would otherwise behave as though they are uncapped. They wouldn't notice. 2) We aren't trying to model MMR behavior. We're simply reading it. If MMR is capped - great, we don't care. Those who are capped are GMs anyway (and potentially few high Masters who very much belong in GM but didn't get in due to Blizzard's strange qualification rules for it, i.e. Whitera who didn't get GM until near the end of the season last season). GM league has 1 tier - we easily read MMRs of people who play GMs, we don't care about cap. | ||
|
Lysenko
Iceland2128 Posts
On July 19 2012 04:02 Not_That wrote: Skeletor described his methods in several posts. You call the calculator a black box, but he has released a source code (which afaik got no attention). Are those posts in this thread? If not, can you link to them? (If they are, I'll go back and look again.) I wasn't referring to anything he did to calculate league offsets as a black box, because I don't recall seeing the documentation of that. What I was referring to was the code he talked about using to simulate large numbers of imaginary games in an attempt to guess what the likelihood of a given distribution would be. | ||
|
Not_That
287 Posts
We're always glad when people show interest in the theoretical part, and this is certainly something that would benefit from more minds thinking about, particularly when it comes to the as of yet unexplored territories. Regarding the MMR numbers and the distribution, if you consider a few wins worth of MMR difference between the races as something that falls within the ladder's MMR distribution without being statistically significant, I don't think the part about simulating large numbers of imaginary games adds something to the table that will make you change your mind. As I understand it, it assumes a distribution of MMRs. Whether or not that assumption makes sense considering the data, I propose some of the statisticians who put this thread to so much scrutiny will prove or disprove. | ||
|
lolcanoe
United States57 Posts
On July 18 2012 23:26 Rick Deckard wrote: It's not necessary for blind people pick one race over another for my point to be valid. Actually, it is. On July 18 2012 23:26 Rick Deckard wrote: This is proof by contradiction, it only requires a single counter example to disprove a rule. What an awful, awful way to rehash the causation-correlation concerns. On July 18 2012 23:26 Rick Deckard wrote: Given that (my proof is valid that) differences in average MMR per race doesn't imply imbalance in general, it also doesn't per se imply imbalance in starcraft 2. If your "proof" had been valid, you would've been led to no conclusion, and no conclusion alone. In this particular situation, the opposite of your statement is true - differences in average MMR per race would GENERALLY suggest imbalance, however there are specific instances where causation is a concern. Keep in mind without a time-based model, most statistical tests leave causation-correlation debates to reasonable examination. Likewise, there are concerns about sampling, MMR measurement, normality, and applicability to the highest tier of play. Op has already noted that most players sampled have high average MMR to lessen concerns that the MMR differences was centered around race choices of players who were new to the game, which was the predominant causation-correlation concern. Besides this concern, there seems to be no reasonable account to support the idea that there is a strong race selection-bias at hand here. | ||
|
Lysenko
Iceland2128 Posts
On July 20 2012 14:39 lolcanoe wrote: Op has already noted that most players sampled have high average MMR to lessen concerns that the MMR differences was centered around race choices of players who were new to the game, which was the predominant causation-correlation concern. Besides this concern, there seems to be no reasonable account to support the idea that there is a strong race selection-bias at hand here. Thing is, I don't see any particular reason to expect that the impact of new-to-the-game players favoring Terran wouldn't trail off measurably all the way to master league. After all, a small but nonzero number of players get quite good at the game very fast. Edit: Also, remember that that mechanism was just as much a factor in 2010 as today, so players who went through that process then have had plenty of time to improve their play. An excess of Terrans as one goes farther down in the distribution may simply be an echo of choices made two years ago and the fact that race choices tend to be somewhat "sticky" since people tend to like playing what they know. | ||
|
gillon
Sweden1578 Posts
On July 18 2012 22:47 Rick Deckard wrote: I believe varying average MMR between races is not an indicator of imbalance. Here's why, imagine there is a race exactly like zerg but is only be played by blind people, call it blind-zerg. This race would have a very low average MMR as blind people obviously can't play starcraft as well as sighted people. But because the blind-zerg race is the same as zerg it's no weaker than zerg. Thus a low average MMR per race doesn't necessarily imply that race is weaker. In conclusion the statistics gathered in the study provided can't be used to make conclusions about how well starcraft 2 is balanced. In practice the data is confounded by the fact that the average skill of of players of different races is not necessarily the same. Dividing race MMR by number of players per race to determine average race MMR doesn't change this. You're analogy is simply implying that every Z player is better than every T player. While, statistically speaking, this isn't impossible it's very arrogant to even assume it a possibility. EDIT: Basically, what lolcanoe stated. | ||
|
skeldark
Germany2223 Posts
MMR Filter: Above Master TIME Filter: 1 Jul 2012 00:00:00 GMT - 31 Jul 2012 23:59:59 GMT T: -15.77 Z: -0.77 P: 12.23 MMR Filter: Above Master TIME Filter: 1 Aug 2012 00:00:00 GMT - 11 Aug 2012 11:47:54 GMT T: -15.24 Z: -7.24 P: 17.76 MMR Filter: No TIME Filter: 1 Jul 2012 00:00:00 GMT - 31 Jul 2012 23:59:59 GMT T: -45.24 Z: 28.76 P: 6.76 MMR Filter: No TIME Filter: 1 Aug 2012 00:00:00 GMT - 11 Aug 2012 11:47:54 GMT T: -46.82 Z: 23.18 P: 14.18 | ||
|
MockHamill
Sweden1798 Posts
On August 11 2012 21:23 skeldark wrote: Updated the data: MMR Filter: Above Master TIME Filter: 1 Jul 2012 00:00:00 GMT - 31 Jul 2012 23:59:59 GMT T: -15.77 Z: -0.77 P: 12.23 MMR Filter: Above Master TIME Filter: 1 Aug 2012 00:00:00 GMT - 11 Aug 2012 11:47:54 GMT T: -15.24 Z: -7.24 P: 17.76 MMR Filter: No TIME Filter: 1 Jul 2012 00:00:00 GMT - 31 Jul 2012 23:59:59 GMT T: -45.24 Z: 28.76 P: 6.76 MMR Filter: No TIME Filter: 1 Aug 2012 00:00:00 GMT - 11 Aug 2012 11:47:54 GMT T: -46.82 Z: 23.18 P: 14.18 Interesting. Terran is UP according to the data. But now when they get faster Ravens Terran will dominate for sure. | ||
|
Thrombozyt
Germany1269 Posts
On August 11 2012 21:31 MockHamill wrote: Interesting. Terran is UP according to the data. But now when they get faster Ravens Terran will dominate for sure. I know you're trolling, I'm not sure which group you target... damn.. time to upgrade my sarcasm detector. | ||
| ||
