|
5003 Posts
Edit: looks like i need to read more carefully
|
On August 17 2010 11:36 Biochemist wrote:
I love how 80% of the posts in this thread are pointing out the exact same flaw in the study.
lol, that's true. You did a good job OP. I appreciate your effort.
|
OP, the work you put into this is impressive and I appreciate the effort you put into all of it. But basically, your conclusion was that Starcraft 2 is not perfectly balanced. Honestly, I don't think any matchup other than a mirror matchup can be considered perfectly balanced. I can't say that a study on these numbers was entirely necessary, or even valid for that matter. Continue the pursuit though! Nice presentation.
|
On August 17 2010 11:20 GagnarTheUnruly wrote:Show nested quote +On August 17 2010 11:07 koswinner wrote:actually.. value of investment is the easy thing you could get. I believe if we do not consider those top pros who does not play ladders seriously, the rating point could just be an valid proxy for it.
Omitting more than one significant variable and relying on a single factored model will probably cause your R-square to go to some pathetic value with a huge error term. In practice, we throw this model like this to rubish bin directly instead of trying to interpret its pathetic 'preditictive power'. If you are relying on such thing to support your claim, then don't call it 'scientific', because in no way it is. Your point is just not any better than anyone who argue it's imbalanced based on one of the many potentially significant variable.
This kind of stats argument might be best for PM's because it's drifting from the OT, but I want to make it clear that I never claimed (or should have claimed) that my data had predictive power. That's not really sensible with a simple chi-square analysis. Regarding regressions, excluding important effect variables can cause your r2 to go down, but adding unimportant effect variables artificially inflates r2 and reduces the accuracy of other parameters due to autocorrelations and spurious effects of the excess predictors. Adding predictors will always increase r2 but it's not always a good idea to add predictors. Often it's not the r2 that's important, but the parameter estimates. It's really unimportant because I don't have the data to do a regression and I probably never will.
Are you copying/pasting from some sources? As I cannot see your logic in this statement, which looks abit irrelevant.
What I have been talking is omitting a SIGNIFICANT VARIABLE, which I stressed throughout my arguement, how does it linked to your logic that adding insignificant variable is a bad thing?
Parameter estimates? Oh yea you know about parameter estimates, but don't you even know that the parameter estimates CARRIES ALMOST NO CREDIBILITY if you omitted some important significant variables? For example, variable A has an actual parameter value of -5, variable B has an actual parameter value of +10, variable C has an actual parameter value of +5, then if you run a regression on A alone you may be well gettting some estimate that A has a parameter value of some positive number, depending on the degree of correlation between ABC, which is obviously false and completely bullshit.
R-square is just a side evidence.. not the main one.
I'm looking like an idiot by trying to educate you what is a scientific piece while urs is far from it.. Not wasting more time on it. Enjoy your 'Scientific Study'. :D
User was warned for this post
|
@OP, It's a decent study, but ladder data maybe not good as many have pointed out.
It's better to collect your own data from tournaments where the top pros play, that is where game balance matters most.
|
On August 17 2010 12:28 koswinner wrote:Show nested quote +On August 17 2010 11:20 GagnarTheUnruly wrote:On August 17 2010 11:07 koswinner wrote:actually.. value of investment is the easy thing you could get. I believe if we do not consider those top pros who does not play ladders seriously, the rating point could just be an valid proxy for it.
Omitting more than one significant variable and relying on a single factored model will probably cause your R-square to go to some pathetic value with a huge error term. In practice, we throw this model like this to rubish bin directly instead of trying to interpret its pathetic 'preditictive power'. If you are relying on such thing to support your claim, then don't call it 'scientific', because in no way it is. Your point is just not any better than anyone who argue it's imbalanced based on one of the many potentially significant variable.
This kind of stats argument might be best for PM's because it's drifting from the OT, but I want to make it clear that I never claimed (or should have claimed) that my data had predictive power. That's not really sensible with a simple chi-square analysis. Regarding regressions, excluding important effect variables can cause your r2 to go down, but adding unimportant effect variables artificially inflates r2 and reduces the accuracy of other parameters due to autocorrelations and spurious effects of the excess predictors. Adding predictors will always increase r2 but it's not always a good idea to add predictors. Often it's not the r2 that's important, but the parameter estimates. It's really unimportant because I don't have the data to do a regression and I probably never will. Are you copying/pasting from some sources? As I cannot see your logic in this statement, which looks abit irrelevant. What I have been talking is omitting a SIGNIFICANT VARIABLE, which I stressed throughout my arguement, how does it linked to your logic that adding insignificant variable is a bad thing? Parameter estimates? Oh yea you know about parameter estimates, but don't you even know that the parameter estimates CARRIES ALMOST NO CREDIBILITY if you omitted some important significant variables? For example, variable A has an actual parameter value of -5, variable B has an actual parameter value of +10, variable C has an actual parameter value of +5, then if you run a regression on A alone you may be well gettting some estimate that A has a parameter value of some positive number, depending on the degree of correlation between ABC, which is obviously false and completely bullshit. R-square is just a side evidence.. not the main one. I'm looking like an idiot by trying to educate you what is a scientific piece while urs is far from it.. Not wasting more time on it. Enjoy your 'Scientific Study'. :D
I think that you are immature and have no idea what you are talking about.
|
The game has been out like a month. And the metagame is constantly changing.
Let new strategies come and go. Let the general skill level of players rise. Let the sc2 evolve a bit before looking at statistics. (that sounded cheesy =P)
And btw, the kind of statistics i'd like to see is wins by matchup. Does zerg win more against protoss than terran?
Or game lenght by matchup, which matchup is statistically the longest/shortest.
Which race has the shortest games on average.
Or a graph with race popularity by number of games played on the ladder, over time.
|
OP--
Are you sure that the algorithm for assigning league (and division) placement is blind to race?
Blizzard may want ~1/3 of each division to be comprised of each race and could accomplish this by placing the top x% of each race in each division or by promoting players above a certain fixed threshold, but capping the amount of Terran at ~35% for instance and then creating a new division when additional Terrans join up.
In either of those cases, the question would be begged from the start. Blizzard would have designed for this distribution to occur by rules, not by game balance.
Also, I think it is important to note that race choice is at least partially a function of skill level with a given race OR of perceived balance advantages. Just on that point alone, a good indicator of potential imbalance would be an underrepresented race across the entire population.
It's also worth generalizing from these two points. It is possible that Blizzard has any number of mechanics behind the scene designed to help players maintain a 50% win rate. It may present you with more ZvZs if you win more at ZvZ, for example.
It may give you relatively weak terrans to play against compared to similarly skilled zerg. Because we know with certainty that Blizzard wants all players to have a 50% win ratio and to be well-distributed across leagues (at least until you get to the very top), it is reasonable to assume that other significant factors apart from balance may in part be driving these statistics.
If some of what I've said here is true, we would expect at least some of the following:
*There are fewer Zerg players overall
*Zerg players have poor winrates against Terrans in general
*Zerg players play against Terrans less often
*Zerg players play against Terrans of comparable skill less often
It's the third point that would be most subtle to detect. If Blizzard has a matchmaking system that always lets Zerg play against a Terran opponent who is actually a tick less skilled but represents the match as even, then things would seem statistically balanced all the way up to the very top of the ladder.
That, however, is where it would break down. When you get to the very top as Terran, there would be no Zergs who are a tick more skilled than you to be matched up with. This would mean that a Zerg player could cruise through the ranks and then hit a brick wall when they near the top of diamond.
Consistent with that hypothesis, one would expect the percentage of Zerg at the highest meaningful tier of skill (maybe top 5 diamonds across all divisions? maybe top 50 players on each server? or maybe that's not quite the top meaningful tier, it may need to be top 10 per server, the extreme right on the bell curve) to be underrepresented relative to the ratios found elsewhere on the ladder.
While a much smaller possible sample to examine, I think I've made a good case that it's possible that an imbalance could be cheated against, hidden and swept under the rug through the matchmaking algorithm all the way to the very top where it then could no longer be hidden.
I recommend that someone more statistics minded than I am looks there next for further analysis.
|
Well there too many problems with your study :/ It is a good effort but your data does not reflect what you are trying to see...
Your data are win% of players / race and /league. But what you really want to show is that one race beats the other one and this is not the same thing because people battles are not randomly made because of the matchmaking system.
See it like this : What would happen if Terran would be far more powerful than the 2 others : Every diamond would be Terrans and their battles TvT, which would give you an exact 50% in wins for Terran Diamond. Your method would qualify this as balanced.
So for a better approach you have to take data about population in leagues and compare them to general population (or a part of it like how many X is there in the top 500 of the top 100 000 players compared to how many of X there are in the 100 000).
If you have clear data of like the top 5000 and can order them you can make non-parametric studies (like Wilcoxon rank tests to see a difference in average rank).
You can also work with data involving number of game Xplayer won against X,Y,Z races I think...
But even with more accurate data it's not that easy to consider it as proof or not, because of the simple fact that what we want is the game to be balanced for pro level and not overall levels and there are too few pro games data to tell anything. On top of that take into account that these kind of thing involves the races are played at their best, which is far from the case right now... watch top players discover different ways to play the game for the average player to follow.
|
On August 17 2010 07:18 StarcraftGuy4U wrote:
None of these stats are worthwhile because the matchmaking system does not assign people like they would in a blind study, instead it is actively adjusting the matches so that every player reaches 50%. The numbers you are pulling are worthless for this reason.
QFT
User was warned for this post
|
On August 17 2010 12:02 koswinner wrote:Show nested quote +On August 17 2010 11:18 mahnini wrote:On August 17 2010 10:35 koswinner wrote:On August 17 2010 10:13 GagnarTheUnruly wrote:On August 17 2010 09:26 petered wrote:
Distribution of race amongst leagues is sadly not valid as an indicator of racial balance. It makes the key assumption that players of different skill level are picking the different races at the same distribution. Graphing race distribution against league level isn't a statistical test and therefore it doesn't make any assumptions. What that graph shows is that roughly equivalent numbers of games are being played by a particular race at each league level. What this suggests is that there is no sorting effect, whereupon a weak race is held back into lower leagues because players that favor that race are having trouble advancing because they are losing games with that race. It is an indirect way of testing that hypothesis. Viewed in the context of the other data, it suggests (but doesn't prove) that AMM is not the only, or even an important, factor in keeping race performance even within leagues. I totally agree that it would be great to analyze the data using player as an explicit factor, but I don't have access to that data. This doesn't prove that there aren't matchup imbalances.
Terran could beat Protoss 60% of the time, Protoss could beat Zerg 60% of the time, and Zerg could beat Terran 60% of the time.
At the end of the day each race would have an approximately ~50% win ratio, as supported by your graphs and charts.
However, TvP, PvZ and ZvT would all be imbalanced. The imbalances would just cancel one another out in terms of overall win ratios.
I agree totally. It would be fun to do that but again I lack the data. If someone can get it for me, I'll do that analysis. This is just bs. You are omitting various factors in your analysis. At lower levels, when players get crushed with a certain race, they tend to change race easily. i.e. a significant variable you have omitted from that diagram is attachness to a certain race, which is obviously positively correlated with skill level. This is just because the amount of 'investment' in a certain race increases with skill level, and the players' utility is usually a function of 'value of investment', which is something like max{Value of investment in T, value of investment in P, value of investment in Z}. With the ratio of (value of investment/time or effort invested) an effective indicator of ratio balance, assuming an representative agent who is trying to maximise his utility. To avoid/minimize this problem you should either gather some reliable information about the parameter of this variable or picking some sample which will exclude this, i.e. pick the 'most attached' bracket, i.e. diamond, or even high end diamond, pro leagues and tournament.
Picking some result and trying to interpret it as solely caused by one factor when obviously there are other factors at work is an indication that either you are very biased, i.e. have a strong incentive to distort the result towards a certain direction, or your level of skill in utilizing 'scientific method' is just horrible.
So, this is not science, just some kid trying to prove his view in the name of science with the help of pseudo/naive/broken scientific method.Last edit: 2010-08-17 09:38:21 This post is not very constructive. What you're suggesting is an absurdly complex model. And please don't disparage my abilities as a scientist. I'm actually a really good scientist and I have some skill at dealing with difficult data. I would like to be able to use a regression model to see how race, placement, and matchup affect the performance of individual players, but as I've noted repeatedly I don't have access to that data. In science when you can't get certain data you need to take indirect approaches that often involve making important assumptions. Often, there are ways to test those assumptions either directly or indirectly, but in this case the data set is extremely complicated, particularly due to match placement. Also, I really need to emphasize that very few assumptions are required to do a chi square test. There are no distributional assumptions to the test. It simply tells us very clearly that within each of the leagues, if a match is picked at random the outcome is totally independent of the one race entering that match. The test doesn't assume that the players are distributed randomly among the races or anything like that. It just tests the hypothesis that states are nonrandomly distributed among the categories being analyzed. The data show that within a league the races have quantifiably different but functionally equal chances of winning randomly selected games. This is a point of fact. There are three non-mutually exclusive possible causes for this that I can think of: 1) the balance is good 2) the matchmaking system is accomodating for poor balance 3) the matchup balance or map balance is poor but it evens out when you ignore the confounding factors There is no way to test the third cause, so we need to suspend it for now, and refer to better judement that it is probably happening but may not be extremely important. It's certainly a hypothesis that bears testing, however. The second cause can be tested indirectly by graphing race use frequency with league status. Since there appears to be no pattern, it suggests that the second cause is also not important. This leaves the first cause. Given consideration of the possible causes of this pattern, it is a reasonable conclusion that good balance is probably largely responsible. It also means it's ignorant to make statements like 'Terran is unbalanced,' because there is no evidence to support such a statement, and becasue the evidence that does exist suggests the opposite. This is not to say that high level players like IdrA, who play in a rarefied realm with tight builds and well rehearsed timing, might not sense conditions that give certain races advantages at certain times. Certainly in BW we've witnessed major shifts of the 'metagame' that resulted in periods of dominance for the various races. 1. What I said was actually not a proposition of that model to test the balance issue, that model was just backing my point that that particular variable (attacheness to race) is very likely to be signficant factor in the overly simplified model you were proposing. And I have already enlightened you how to bypass problem like that. i.e. for that particular problem, pick samples within the same group, and I already pointed out that datas for high end diamond is readily available. If you know how to run a regression then I assume you should know the devastating effect it will be in omitting one significant variable, don't you? Not to mention what you omitted is not only one significant variable.. So basically what I was proposing was just a multi-factor model, which is soooo common in practice, your single factor model is just 'absurdly oversimplified'. With such a skyrocketing error term and a tiny R-square caused by omitting significant variables, as a objective scientist I have no idea how could you claim that your overly simplified single factor model could explain anything at all. So the logic is simple, if that model is way toooo simplified to get the result, don't claim you got the result with some scientific method. 2. 'It also means it's ignorant to make statements like 'Terran is unbalanced,' because there is no evidence to support such a statement, and becasue the evidence that does exist suggests the opposite.' ROFL, 'EVIDENCE', you call the result of your overwhelmingly over simplified model .... EVIDENCE?? And you are ignoring all other much more reliable indicator, like proportion of Z at top level, or some opionion pool around the world about 'the weakest race' and 'is ZvT imba'. Nice scientist :D testing for attachedness to race would bring about even more headache inducing factors such as style of play, mechanical requirements, and depth of understanding. we can go on and on about missing factors but we are able to make certain conclusions with the data we already have i think. anyway like i stated before the proportion of top 100 zerg players matches that of the proportion of zerg players in the general population (if someone wants to check my math and do statistical magic on it, that'd be great). a lot of what's going is we have SOME concrete data that weighs in the favor of the races being balanced it's not a 100% thorough scientific study but that doesn't mean you sure turn a blind eye towards it. afterall, the opposing argument is simply referencing anecdotal evidence of zvt being hard and pointing out that A, B, and C top zerg players say it's imbalanced. Attachness actually isn't that difficult if you use some indirect way of testing it. For example you could just test the correlation between change in proportion of each race and the opionion pools about which race is considered as most powerful/imba. Even if for SC2 there isn't enough sample space yet, but we could certainly use other similar type of games such as WC3 and SC1, which probably could be some valid proxy. The datas were available but just nobody really bothered to record it. Some simpler indicator could be some poll asking about whether players would consider/ is considering changing race if their race is having problem. Of course these does not distinguish between different attachness between different skilled players. If you want, just do the same survey for different groups. Your argument is valid ONLY IF proportion of players in each level represents the balance, i.e. ONLY IF other factors are not affecting player's pick of race and change of race, and assuming each race's population has homogenous characteristics, i.e. they have similar ability, some of them does not struggle harder to get their status as opposed to other races. Then with a more detailed breakdown of bracket such as to top 20 or 10 and tournament oriented top pros it will probably be some valid test. But obviously some of the assumptions are just tooo strong/unrealistic, as players do change away from weak races to stronger ones.. Just look at WC3. Though this effect is much less signficant among top players, who already invested significantly in their particular race. So if you see quite a number of, or even a significant proportion of the very top players of one race (which is considered as weakest) is changing to other races (mainly the commonly considered imba one), while no top players from other races changed their race, is this just an accident? Does it say anything? How about win rate of these top players with their respective races in major tournaments? All of them, I believe, are much better / more reliable indicator compared to yours, as they require much simpler/realistic assumptions and are obvious enough to overwhelm the rest.
i suppose a random poll would work but i would seriously question the reliability of such a poll to determine attachedness. i'm not sure i understand why my argument would only be valid with homogeneous races. my argument is that if you use a controlled group to test all races and use that to gauge attachedness you introduce inherit racial biases as all races in starcraft are not played the same. viable play styles, necessary mechanics, and the ability of a player to understand the depth of the race will all factor in.
i think we may have different ideas of imbalance. are we talking about imbalance as in an imbalance of wins:losses or as in one race is inherently inferior to another? even now at the height of TvZ imbalance discussion we see proportional amounts of Zerg players in the top 100. even when every post in this forum makes it seem like the matchup is utterly impossible there are STILL proportional amounts of zergs in the top 100. with PvZ generally considered balance, how come zerg's aren't underrepresented?
if at the tip-top level, let's say the top 10, zerg are underrepresented and there is an imbalance in wins and losses, is that enough to claim inherent racial imbalance? it is much easier to say, "look zerg's are proportionally represented and therefore the game is inherently balance at the moment." than it is to say, "zerg's are currently underrepresented and therefore at the zerg is an inherently inferior race." such an extraordinary claim requires much more evidence don't you think? much more than the opinions of top gamers over a 3 week timespan at least?
|
On August 17 2010 13:03 mahnini wrote:
if at the tip-top level, let's say the top 10, zerg are underrepresented and there is an imbalance in wins and losses, is that enough to claim inherent racial imbalance? it is much easier to say, "look zerg's are proportionally represented and therefore the game is inherently balance at the moment." than it is to say, "zerg's are currently underrepresented and therefore at the zerg is an inherently inferior race." such an extraordinary claim requires much more evidence don't you think? much more than the opinions of top gamers over a 3 week timespan at least?
See the post I just made for my thoughts on this.
In short, I'm proposing that a disproportionate representation of Zerg at the highest possible level (open for debate what that means) may well be the only reliable indicator we'd have of an imbalance given the known sophistication of Blizzard's matchmaking system as well as the potential unknowns in that system.
|
...And I'll add on that note, that the current global top 20 according to sc2ranks.com shows:
10 Terrans
7 Protoss
2 Zerg
1 Random
So that's 50% Terran, 35% Protoss, 10% Zerg, and 5% random.
|
On August 17 2010 13:12 retro-noob wrote:Show nested quote +On August 17 2010 13:03 mahnini wrote:
if at the tip-top level, let's say the top 10, zerg are underrepresented and there is an imbalance in wins and losses, is that enough to claim inherent racial imbalance? it is much easier to say, "look zerg's are proportionally represented and therefore the game is inherently balance at the moment." than it is to say, "zerg's are currently underrepresented and therefore at the zerg is an inherently inferior race." such an extraordinary claim requires much more evidence don't you think? much more than the opinions of top gamers over a 3 week timespan at least? See the post I just made for my thoughts on this. In short, I'm proposing that a disproportionate representation of Zerg at the highest possible level (open for debate what that means) may well be the only reliable indicator we'd have of an imbalance given the known sophistication of Blizzard's matchmaking system as well as the potential unknowns in that system.
it may eventually be but is it even close to being reliable right now and is that alone sufficient to claim inherent racial imbalance? iloveoov once went on a 80% or so tear through the highest leagues in the world, did that mean the other races were unequipped to deal with terran? if we want to use only the best players in the world as data points wouldn't it require MUCH more time to eliminate many factors that would increase variance in results?
|
we already know it's imba, why all the topics about it?
User was warned for this post
|
On August 17 2010 13:22 mahnini wrote:Show nested quote +On August 17 2010 13:12 retro-noob wrote:On August 17 2010 13:03 mahnini wrote:
if at the tip-top level, let's say the top 10, zerg are underrepresented and there is an imbalance in wins and losses, is that enough to claim inherent racial imbalance? it is much easier to say, "look zerg's are proportionally represented and therefore the game is inherently balance at the moment." than it is to say, "zerg's are currently underrepresented and therefore at the zerg is an inherently inferior race." such an extraordinary claim requires much more evidence don't you think? much more than the opinions of top gamers over a 3 week timespan at least? See the post I just made for my thoughts on this. In short, I'm proposing that a disproportionate representation of Zerg at the highest possible level (open for debate what that means) may well be the only reliable indicator we'd have of an imbalance given the known sophistication of Blizzard's matchmaking system as well as the potential unknowns in that system. it may eventually be but is it even close to being reliable right now and is that alone sufficient to claim inherent racial imbalance? iloveoov once went on a 80% or so tear through the highest leagues in the world, did that mean the other races were unequipped to deal with terran? if we want to use only the best players in the world as data points wouldn't it require MUCH more time to eliminate many factors that would increase variance in results?
No one is claiming racial imbalance merely because of the stats at the top of the leagues. People are claiming the imbalance because of the nature of the games being played at the top of those leagues as well. While this thread explores a statistical approach, others have taken an analytical approach to the mechanics of the game itself, and more than a few have been cogent.
Regardless, the scenario I propose, if in fact that is a desired or undesired consequences of Blizzard's matchmaking system, would render statistical analysis of the distribution in leagues entirely moot. The same would go for analyzing the TvZ win-rate across the entire game--it would probably look a lot like 50/50 with a teeny rounding error known as the top of the ladder.
Because in effect, Terrans on the entire ladder would be playing their TvZs with a slight handicap in how the matchup was decided--except for at the very, very top. Where we'd expect to see lopsided racial representation just like what we actually do see. Same for tournaments.
|
On August 17 2010 13:03 mahnini wrote:Show nested quote +On August 17 2010 12:02 koswinner wrote:On August 17 2010 11:18 mahnini wrote:On August 17 2010 10:35 koswinner wrote:On August 17 2010 10:13 GagnarTheUnruly wrote:On August 17 2010 09:26 petered wrote:
Distribution of race amongst leagues is sadly not valid as an indicator of racial balance. It makes the key assumption that players of different skill level are picking the different races at the same distribution. Graphing race distribution against league level isn't a statistical test and therefore it doesn't make any assumptions. What that graph shows is that roughly equivalent numbers of games are being played by a particular race at each league level. What this suggests is that there is no sorting effect, whereupon a weak race is held back into lower leagues because players that favor that race are having trouble advancing because they are losing games with that race. It is an indirect way of testing that hypothesis. Viewed in the context of the other data, it suggests (but doesn't prove) that AMM is not the only, or even an important, factor in keeping race performance even within leagues. I totally agree that it would be great to analyze the data using player as an explicit factor, but I don't have access to that data. This doesn't prove that there aren't matchup imbalances.
Terran could beat Protoss 60% of the time, Protoss could beat Zerg 60% of the time, and Zerg could beat Terran 60% of the time.
At the end of the day each race would have an approximately ~50% win ratio, as supported by your graphs and charts.
However, TvP, PvZ and ZvT would all be imbalanced. The imbalances would just cancel one another out in terms of overall win ratios.
I agree totally. It would be fun to do that but again I lack the data. If someone can get it for me, I'll do that analysis. This is just bs. You are omitting various factors in your analysis. At lower levels, when players get crushed with a certain race, they tend to change race easily. i.e. a significant variable you have omitted from that diagram is attachness to a certain race, which is obviously positively correlated with skill level. This is just because the amount of 'investment' in a certain race increases with skill level, and the players' utility is usually a function of 'value of investment', which is something like max{Value of investment in T, value of investment in P, value of investment in Z}. With the ratio of (value of investment/time or effort invested) an effective indicator of ratio balance, assuming an representative agent who is trying to maximise his utility. To avoid/minimize this problem you should either gather some reliable information about the parameter of this variable or picking some sample which will exclude this, i.e. pick the 'most attached' bracket, i.e. diamond, or even high end diamond, pro leagues and tournament.
Picking some result and trying to interpret it as solely caused by one factor when obviously there are other factors at work is an indication that either you are very biased, i.e. have a strong incentive to distort the result towards a certain direction, or your level of skill in utilizing 'scientific method' is just horrible.
So, this is not science, just some kid trying to prove his view in the name of science with the help of pseudo/naive/broken scientific method.Last edit: 2010-08-17 09:38:21 This post is not very constructive. What you're suggesting is an absurdly complex model. And please don't disparage my abilities as a scientist. I'm actually a really good scientist and I have some skill at dealing with difficult data. I would like to be able to use a regression model to see how race, placement, and matchup affect the performance of individual players, but as I've noted repeatedly I don't have access to that data. In science when you can't get certain data you need to take indirect approaches that often involve making important assumptions. Often, there are ways to test those assumptions either directly or indirectly, but in this case the data set is extremely complicated, particularly due to match placement. Also, I really need to emphasize that very few assumptions are required to do a chi square test. There are no distributional assumptions to the test. It simply tells us very clearly that within each of the leagues, if a match is picked at random the outcome is totally independent of the one race entering that match. The test doesn't assume that the players are distributed randomly among the races or anything like that. It just tests the hypothesis that states are nonrandomly distributed among the categories being analyzed. The data show that within a league the races have quantifiably different but functionally equal chances of winning randomly selected games. This is a point of fact. There are three non-mutually exclusive possible causes for this that I can think of: 1) the balance is good 2) the matchmaking system is accomodating for poor balance 3) the matchup balance or map balance is poor but it evens out when you ignore the confounding factors There is no way to test the third cause, so we need to suspend it for now, and refer to better judement that it is probably happening but may not be extremely important. It's certainly a hypothesis that bears testing, however. The second cause can be tested indirectly by graphing race use frequency with league status. Since there appears to be no pattern, it suggests that the second cause is also not important. This leaves the first cause. Given consideration of the possible causes of this pattern, it is a reasonable conclusion that good balance is probably largely responsible. It also means it's ignorant to make statements like 'Terran is unbalanced,' because there is no evidence to support such a statement, and becasue the evidence that does exist suggests the opposite. This is not to say that high level players like IdrA, who play in a rarefied realm with tight builds and well rehearsed timing, might not sense conditions that give certain races advantages at certain times. Certainly in BW we've witnessed major shifts of the 'metagame' that resulted in periods of dominance for the various races. 1. What I said was actually not a proposition of that model to test the balance issue, that model was just backing my point that that particular variable (attacheness to race) is very likely to be signficant factor in the overly simplified model you were proposing. And I have already enlightened you how to bypass problem like that. i.e. for that particular problem, pick samples within the same group, and I already pointed out that datas for high end diamond is readily available. If you know how to run a regression then I assume you should know the devastating effect it will be in omitting one significant variable, don't you? Not to mention what you omitted is not only one significant variable.. So basically what I was proposing was just a multi-factor model, which is soooo common in practice, your single factor model is just 'absurdly oversimplified'. With such a skyrocketing error term and a tiny R-square caused by omitting significant variables, as a objective scientist I have no idea how could you claim that your overly simplified single factor model could explain anything at all. So the logic is simple, if that model is way toooo simplified to get the result, don't claim you got the result with some scientific method. 2. 'It also means it's ignorant to make statements like 'Terran is unbalanced,' because there is no evidence to support such a statement, and becasue the evidence that does exist suggests the opposite.' ROFL, 'EVIDENCE', you call the result of your overwhelmingly over simplified model .... EVIDENCE?? And you are ignoring all other much more reliable indicator, like proportion of Z at top level, or some opionion pool around the world about 'the weakest race' and 'is ZvT imba'. Nice scientist :D testing for attachedness to race would bring about even more headache inducing factors such as style of play, mechanical requirements, and depth of understanding. we can go on and on about missing factors but we are able to make certain conclusions with the data we already have i think. anyway like i stated before the proportion of top 100 zerg players matches that of the proportion of zerg players in the general population (if someone wants to check my math and do statistical magic on it, that'd be great). a lot of what's going is we have SOME concrete data that weighs in the favor of the races being balanced it's not a 100% thorough scientific study but that doesn't mean you sure turn a blind eye towards it. afterall, the opposing argument is simply referencing anecdotal evidence of zvt being hard and pointing out that A, B, and C top zerg players say it's imbalanced. Attachness actually isn't that difficult if you use some indirect way of testing it. For example you could just test the correlation between change in proportion of each race and the opionion pools about which race is considered as most powerful/imba. Even if for SC2 there isn't enough sample space yet, but we could certainly use other similar type of games such as WC3 and SC1, which probably could be some valid proxy. The datas were available but just nobody really bothered to record it. Some simpler indicator could be some poll asking about whether players would consider/ is considering changing race if their race is having problem. Of course these does not distinguish between different attachness between different skilled players. If you want, just do the same survey for different groups. Your argument is valid ONLY IF proportion of players in each level represents the balance, i.e. ONLY IF other factors are not affecting player's pick of race and change of race, and assuming each race's population has homogenous characteristics, i.e. they have similar ability, some of them does not struggle harder to get their status as opposed to other races. Then with a more detailed breakdown of bracket such as to top 20 or 10 and tournament oriented top pros it will probably be some valid test. But obviously some of the assumptions are just tooo strong/unrealistic, as players do change away from weak races to stronger ones.. Just look at WC3. Though this effect is much less signficant among top players, who already invested significantly in their particular race. So if you see quite a number of, or even a significant proportion of the very top players of one race (which is considered as weakest) is changing to other races (mainly the commonly considered imba one), while no top players from other races changed their race, is this just an accident? Does it say anything? How about win rate of these top players with their respective races in major tournaments? All of them, I believe, are much better / more reliable indicator compared to yours, as they require much simpler/realistic assumptions and are obvious enough to overwhelm the rest. i suppose a random poll would work but i would seriously question the reliability of such a poll to determine attachedness. i'm not sure i understand why my argument would only be valid with homogeneous races. my argument is that if you use a controlled group to test all races and use that to gauge attachedness you introduce inherit racial biases as all races in starcraft are not played the same. viable play styles, necessary mechanics, and the ability of a player to understand the depth of the race will all factor in. i think we may have different ideas of imbalance. are we talking about imbalance as in an imbalance of wins:losses or as in one race is inherently inferior to another? even now at the height of TvZ imbalance discussion we see proportional amounts of Zerg players in the top 100. even when every post in this forum makes it seem like the matchup is utterly impossible there are STILL proportional amounts of zergs in the top 100. with PvZ generally considered balance, how come zerg's aren't underrepresented? if at the tip-top level, let's say the top 10, zerg are underrepresented and there is an imbalance in wins and losses, is that enough to claim inherent racial imbalance? it is much easier to say, "look zerg's are proportionally represented and therefore the game is inherently balance at the moment." than it is to say, "zerg's are currently underrepresented and therefore at the zerg is an inherently inferior race." such an extraordinary claim requires much more evidence don't you think? much more than the opinions of top gamers over a 3 week timespan at least?
The problem with your arguement about those factors are that, if you are familiar with statistical models, you know that it is only reasonable to add more variables if you know that this variable is statistically significant.. adding more insignificant variable will have a detrimental effect about your estimation. So the problem is not that whether you added have impact or not, or are they significant? Very likely not.
As I already pointed out long ago, that if other factors (like change of race) contribute significantly to your proposed result, proportional representation, then it just doesn't carry any credibility. Maybe in some statistical terminology you didn't get it, I'll give you a easy-to-understand example: We could very likely to have that if zerg players never change, the it will get significantly under represented in top 100, but with a significant amount of zerg (especially lower level) quitting their race, zerg looks like proportionally represented in top 100, or even overly represented. So this arguement is just not valid. Due to the current mm system, the validity of racial representation increases exponentially the higher the level is. The problem is that, with top 10 or something, there is not enough sample. But together with the fact that in every region's ladder, whether top 10 or top 20 Z is very under-represented and T is overly represented.
You are just comparing two very naive arguement in your last paragraph..
I have listed out a lot more much better and valid arguement/indicators, why don't you compare "look zerg's are proportionally represented and therefore the game is inherently balance at the moment." to them?
BTW: win% between best players of each races pretty much represent the inherent imbalance, they are almost the same thing with large sample.
PS: your previous arguement is only valid if the players of each races are homogenous rather than the race itself. Because it could be that players of certain races are more hardworking/smarter than others etc., though unlikely.
Well, I'm done with this topic, spent too much time on this..
|
On August 17 2010 13:32 koswinner wrote:Show nested quote +On August 17 2010 13:03 mahnini wrote:On August 17 2010 12:02 koswinner wrote:On August 17 2010 11:18 mahnini wrote:On August 17 2010 10:35 koswinner wrote:On August 17 2010 10:13 GagnarTheUnruly wrote:On August 17 2010 09:26 petered wrote:
Distribution of race amongst leagues is sadly not valid as an indicator of racial balance. It makes the key assumption that players of different skill level are picking the different races at the same distribution. Graphing race distribution against league level isn't a statistical test and therefore it doesn't make any assumptions. What that graph shows is that roughly equivalent numbers of games are being played by a particular race at each league level. What this suggests is that there is no sorting effect, whereupon a weak race is held back into lower leagues because players that favor that race are having trouble advancing because they are losing games with that race. It is an indirect way of testing that hypothesis. Viewed in the context of the other data, it suggests (but doesn't prove) that AMM is not the only, or even an important, factor in keeping race performance even within leagues. I totally agree that it would be great to analyze the data using player as an explicit factor, but I don't have access to that data. This doesn't prove that there aren't matchup imbalances.
Terran could beat Protoss 60% of the time, Protoss could beat Zerg 60% of the time, and Zerg could beat Terran 60% of the time.
At the end of the day each race would have an approximately ~50% win ratio, as supported by your graphs and charts.
However, TvP, PvZ and ZvT would all be imbalanced. The imbalances would just cancel one another out in terms of overall win ratios.
I agree totally. It would be fun to do that but again I lack the data. If someone can get it for me, I'll do that analysis. This is just bs. You are omitting various factors in your analysis. At lower levels, when players get crushed with a certain race, they tend to change race easily. i.e. a significant variable you have omitted from that diagram is attachness to a certain race, which is obviously positively correlated with skill level. This is just because the amount of 'investment' in a certain race increases with skill level, and the players' utility is usually a function of 'value of investment', which is something like max{Value of investment in T, value of investment in P, value of investment in Z}. With the ratio of (value of investment/time or effort invested) an effective indicator of ratio balance, assuming an representative agent who is trying to maximise his utility. To avoid/minimize this problem you should either gather some reliable information about the parameter of this variable or picking some sample which will exclude this, i.e. pick the 'most attached' bracket, i.e. diamond, or even high end diamond, pro leagues and tournament.
Picking some result and trying to interpret it as solely caused by one factor when obviously there are other factors at work is an indication that either you are very biased, i.e. have a strong incentive to distort the result towards a certain direction, or your level of skill in utilizing 'scientific method' is just horrible.
So, this is not science, just some kid trying to prove his view in the name of science with the help of pseudo/naive/broken scientific method.Last edit: 2010-08-17 09:38:21 This post is not very constructive. What you're suggesting is an absurdly complex model. And please don't disparage my abilities as a scientist. I'm actually a really good scientist and I have some skill at dealing with difficult data. I would like to be able to use a regression model to see how race, placement, and matchup affect the performance of individual players, but as I've noted repeatedly I don't have access to that data. In science when you can't get certain data you need to take indirect approaches that often involve making important assumptions. Often, there are ways to test those assumptions either directly or indirectly, but in this case the data set is extremely complicated, particularly due to match placement. Also, I really need to emphasize that very few assumptions are required to do a chi square test. There are no distributional assumptions to the test. It simply tells us very clearly that within each of the leagues, if a match is picked at random the outcome is totally independent of the one race entering that match. The test doesn't assume that the players are distributed randomly among the races or anything like that. It just tests the hypothesis that states are nonrandomly distributed among the categories being analyzed. The data show that within a league the races have quantifiably different but functionally equal chances of winning randomly selected games. This is a point of fact. There are three non-mutually exclusive possible causes for this that I can think of: 1) the balance is good 2) the matchmaking system is accomodating for poor balance 3) the matchup balance or map balance is poor but it evens out when you ignore the confounding factors There is no way to test the third cause, so we need to suspend it for now, and refer to better judement that it is probably happening but may not be extremely important. It's certainly a hypothesis that bears testing, however. The second cause can be tested indirectly by graphing race use frequency with league status. Since there appears to be no pattern, it suggests that the second cause is also not important. This leaves the first cause. Given consideration of the possible causes of this pattern, it is a reasonable conclusion that good balance is probably largely responsible. It also means it's ignorant to make statements like 'Terran is unbalanced,' because there is no evidence to support such a statement, and becasue the evidence that does exist suggests the opposite. This is not to say that high level players like IdrA, who play in a rarefied realm with tight builds and well rehearsed timing, might not sense conditions that give certain races advantages at certain times. Certainly in BW we've witnessed major shifts of the 'metagame' that resulted in periods of dominance for the various races. 1. What I said was actually not a proposition of that model to test the balance issue, that model was just backing my point that that particular variable (attacheness to race) is very likely to be signficant factor in the overly simplified model you were proposing. And I have already enlightened you how to bypass problem like that. i.e. for that particular problem, pick samples within the same group, and I already pointed out that datas for high end diamond is readily available. If you know how to run a regression then I assume you should know the devastating effect it will be in omitting one significant variable, don't you? Not to mention what you omitted is not only one significant variable.. So basically what I was proposing was just a multi-factor model, which is soooo common in practice, your single factor model is just 'absurdly oversimplified'. With such a skyrocketing error term and a tiny R-square caused by omitting significant variables, as a objective scientist I have no idea how could you claim that your overly simplified single factor model could explain anything at all. So the logic is simple, if that model is way toooo simplified to get the result, don't claim you got the result with some scientific method. 2. 'It also means it's ignorant to make statements like 'Terran is unbalanced,' because there is no evidence to support such a statement, and becasue the evidence that does exist suggests the opposite.' ROFL, 'EVIDENCE', you call the result of your overwhelmingly over simplified model .... EVIDENCE?? And you are ignoring all other much more reliable indicator, like proportion of Z at top level, or some opionion pool around the world about 'the weakest race' and 'is ZvT imba'. Nice scientist :D testing for attachedness to race would bring about even more headache inducing factors such as style of play, mechanical requirements, and depth of understanding. we can go on and on about missing factors but we are able to make certain conclusions with the data we already have i think. anyway like i stated before the proportion of top 100 zerg players matches that of the proportion of zerg players in the general population (if someone wants to check my math and do statistical magic on it, that'd be great). a lot of what's going is we have SOME concrete data that weighs in the favor of the races being balanced it's not a 100% thorough scientific study but that doesn't mean you sure turn a blind eye towards it. afterall, the opposing argument is simply referencing anecdotal evidence of zvt being hard and pointing out that A, B, and C top zerg players say it's imbalanced. Attachness actually isn't that difficult if you use some indirect way of testing it. For example you could just test the correlation between change in proportion of each race and the opionion pools about which race is considered as most powerful/imba. Even if for SC2 there isn't enough sample space yet, but we could certainly use other similar type of games such as WC3 and SC1, which probably could be some valid proxy. The datas were available but just nobody really bothered to record it. Some simpler indicator could be some poll asking about whether players would consider/ is considering changing race if their race is having problem. Of course these does not distinguish between different attachness between different skilled players. If you want, just do the same survey for different groups. Your argument is valid ONLY IF proportion of players in each level represents the balance, i.e. ONLY IF other factors are not affecting player's pick of race and change of race, and assuming each race's population has homogenous characteristics, i.e. they have similar ability, some of them does not struggle harder to get their status as opposed to other races. Then with a more detailed breakdown of bracket such as to top 20 or 10 and tournament oriented top pros it will probably be some valid test. But obviously some of the assumptions are just tooo strong/unrealistic, as players do change away from weak races to stronger ones.. Just look at WC3. Though this effect is much less signficant among top players, who already invested significantly in their particular race. So if you see quite a number of, or even a significant proportion of the very top players of one race (which is considered as weakest) is changing to other races (mainly the commonly considered imba one), while no top players from other races changed their race, is this just an accident? Does it say anything? How about win rate of these top players with their respective races in major tournaments? All of them, I believe, are much better / more reliable indicator compared to yours, as they require much simpler/realistic assumptions and are obvious enough to overwhelm the rest. i suppose a random poll would work but i would seriously question the reliability of such a poll to determine attachedness. i'm not sure i understand why my argument would only be valid with homogeneous races. my argument is that if you use a controlled group to test all races and use that to gauge attachedness you introduce inherit racial biases as all races in starcraft are not played the same. viable play styles, necessary mechanics, and the ability of a player to understand the depth of the race will all factor in. i think we may have different ideas of imbalance. are we talking about imbalance as in an imbalance of wins:losses or as in one race is inherently inferior to another? even now at the height of TvZ imbalance discussion we see proportional amounts of Zerg players in the top 100. even when every post in this forum makes it seem like the matchup is utterly impossible there are STILL proportional amounts of zergs in the top 100. with PvZ generally considered balance, how come zerg's aren't underrepresented? if at the tip-top level, let's say the top 10, zerg are underrepresented and there is an imbalance in wins and losses, is that enough to claim inherent racial imbalance? it is much easier to say, "look zerg's are proportionally represented and therefore the game is inherently balance at the moment." than it is to say, "zerg's are currently underrepresented and therefore at the zerg is an inherently inferior race." such an extraordinary claim requires much more evidence don't you think? much more than the opinions of top gamers over a 3 week timespan at least? The problem with your arguement about those factors are that, if you are familiar with statistical models, you know that it is only reasonable to add more variables if you know that this variable is statistically significant.. adding more insignificant variable will have a detrimental effect about your estimation. So the problem is not that whether you added have impact or not, or are they significant? Very likely not. As I already pointed out long ago, that if other factors (like change of race) contribute significantly to your proposed result, proportional representation, then it just doesn't carry any credibility. Maybe in some statistical terminology you didn't get it, I'll give you a easy-to-understand example: We could very likely to have that if zerg players never change, the it will get significantly under represented in top 100, but with a significant amount of zerg (especially lower level) quitting their race, zerg looks like proportionally represented in top 100, or even overly represented. So this arguement is just not valid. Due to the current mm system, the validity of racial representation increases exponentially the higher the level is. The problem is that, with top 10 or something, there is not enough sample. But together with the fact that in every region's ladder, whether top 10 or top 20 Z is very under-represented and T is overly represented. You are just comparing two very naive arguement in your last paragraph.. I have listed out a lot more much better and valid arguement/indicators, why don't you compare "look zerg's are proportionally represented and therefore the game is inherently balance at the moment." to them? BTW: win% between best players of each races pretty much represent the inherent imbalance, they are almost the same thing with large sample. PS: your previous arguement is only valid if the players of each races are homogenous rather than the race itself. Because it could be that players of certain races are more hardworking/smarter than others etc., though unlikely.
ok that makes sense to me but isn't that in itself very simplified? as far as i know battlenet keeps your race singular until you played more games with another race but your stats remain the same so i see what you are saying about lower levels switching races. you're right, there's definitely a possibility there to skew results.
what i was trying to say with the naive arguments is that at the moment proving anything statistically is a one way street in favor of balance, that is you can only prove balance and not imbalance (inherent racial imbalance) because not enough time has passed to do so, at least directly.
though, i'm still not sure what you mean by attachedness to a race. if you mean that simply to be willing or unwillingness to switch race in the face of adversity or perceived imbalance that seems like a very complex thing to do as there many factors that would affect the outcome of that value even though it would not affect the model you are applying attachedness to. does that make sense?
|
On August 17 2010 12:10 Milkis wrote:
taking differences of percents is not a scientific study, nor is it statistically sound.
please don't just look at percents and try to frame it as statistics. this is far from how you would be approaching it. you don't even have a theory of how this actually works out nor do you consider an actual model, you just subtract differences and hope it works out
holy crap what people try to pass as statistics on the internet is absolutely appalling
This is the truth - There's no measure of certainty or variance in these statistics, and you'd have to assume the game is actually balanced to set it all to a bell curve.
What about considerations of unknown variables? What if players new to SC (and RTS games) are more attracted to Terran because of familiarity through the campaign, leading to Terrans having a much lower win% in Bronze league?
|
5003 Posts
The problem with your arguement about those factors are that, if you are familiar with statistical models, you know that it is only reasonable to add more variables if you know that this variable is statistically significant.. adding more insignificant variable will have a detrimental effect about your estimation. So the problem is not that whether you added have impact or not, or are they significant? Very likely not.
It's only reasonable to add more variables if you want to see if the variable has an effect. Statistics in the end is based on theory -- you shouldn't concentrate on the actual model itself but on the theory first.After you figure out what you think may affect the variables, you make a model with the variables.
Secondly, no. If a variable is insignificant, then all it does at best is increase the variance a bit (meaning you'll need more samples). If the sample is large enough then this isn't even a problem. You usually figure out a variable is insignificant after fitting it to a statistical model. You don't know if it's significant or not until you have done so. Missing a significant variable is a much bigger problem than adding in an insignificant one, and you can easily filter out insignificant variables. But even then, that's never for certain.
I can't even comprehend the rest of your post.
This is the truth - There's no measure of certainty or variance in these statistics, and you'd have to assume the game is actually balanced to set it all to a bell curve.
It doesn't have to do with certainty or variance, but the model has no statistical basis at all. You have to start with how you think the data is distributed, and in order to that you need better theory.
Also being balanced has nothing to do with it being normal ("set it all to a bell curve"). Balance is a pesky, if not, impossible think to measure and it only paints a relative picture given the strategies, maps, and the players' innate skillsets at hand. Arguing balance based on statistics will be a lot more complicated than these simplistic approaches.
|
|
|
|
|
|
EPT
