EPTScientific proof that SC2 is imbalanced (sorta) - Page 4
| Forum Index > SC2 General |
|
hadoken5
Canada519 Posts
| ||
|
stochastic
United States16 Posts
| ||
|
mahnini
United States6862 Posts
On August 17 2010 09:09 hadoken5 wrote: I think I posted something like this, but I didn't do it with all the graphs like you did. Good job though! don't kid yourself | ||
|
GagnarTheUnruly
United States655 Posts
On August 17 2010 09:05 darmousseh wrote: You do realize that all statistics relating win percentage are irrelevant because of the matchmaking scheme? Matchmaking ensures that all players have roughly a 50% win percentage. The only way to test imbalance is to have a large number of games not using the matchmaking system. Again, I'm not testing players, I'm testing games. The data I'm using show not that players have a 50% win ratio, but that races have a 50% win ratio on a per-game basis. I'm not saying you're totally wrong, but it's important to recognize the difference and consider the relationship between per-game win rate and per-player win rate as indirectly related. | ||
|
st3roids
Greece538 Posts
Also some builds are extremely powerfull but this doesnt mean that everyone knows and performs them perfect or do the very same builds all the time . | ||
|
petered
United States1817 Posts
More specifically, it assumes that a diamond player is just as likely to choose terran as a bronze player. So the fact that there are more zergs in diamond proportion wise as there are in bronze could suggest that zerg is not weak, or it could suggest that bronze newbies are just less likely to pick zerg. The key to making comparisons is always to hold one factor constant. AMM makes this very very difficult to do. The best way to compare racial imbalances, in my mind, is to collect info on random players, and their win ratios with different races. In that case you are holding the skill of the player constant, and hence you can make some progress. You still have to make some pretty big assumptions, but it will get you farther than pure win/loss ratios. | ||
|
Gnial
Canada907 Posts
On August 17 2010 08:54 GagnarTheUnruly wrote: Another worthwhile conclusion, in my opinion, is that for an individual person who feels they are the victim of race imbalances, what they are actually experiencing is just trouble with a particular matchup. These data offer what in my opinion is pretty strong evidence that race alone is not a strong determinant of success at a given player level. You can't randomly pick a game played by a platinum player and predict the outcome based on the race of that player. This doesn't prove that there aren't matchup imbalances. Terran could beat Protoss 60% of the time, Protoss could beat Zerg 60% of the time, and Zerg could beat Terran 60% of the time. At the end of the day each race would have an approximately ~50% win ratio, as supported by your graphs and charts. However, TvP, PvZ and ZvT would all be imbalanced. The imbalances would just cancel one another out in terms of overall win ratios. | ||
|
nihlon
Sweden5581 Posts
On August 17 2010 08:54 GagnarTheUnruly wrote: Another worthwhile conclusion, in my opinion, is that for an individual person who feels they are the victim of race imbalances, what they are actually experiencing is just trouble with a particular matchup. These data offer what in my opinion is pretty strong evidence that race alone is not a strong determinant of success at a given player level. You can't randomly pick a game played by a platinum player and predict the outcome based on the race of that player. So if I compile a list of all TvZ games in Platinum, will I find a perfectly even winning percentage or not? You can't say the bolded part and then claim the game is balanced... | ||
|
koswinner
United Kingdom27 Posts
On August 17 2010 08:01 GagnarTheUnruly wrote: I agree. Here's a graph of the racial distributions. The y-axis is proportion of that race in the games played pool for a certain league. ![[image loading]](http://i667.photobucket.com/albums/vv32/GagnarTheUnruly/meanpropgraph.jpg) This indicates that racial imbalances aren't causing weak races to get held back in lower leagues. If some of the earlier criticisms were true, that matchmaking obviates differences in racial performance, we should see some races gaining prominence and others losing it as you move through the leagues. In particular, the races that indicated as slightly weak in my analysis should fall out of diamond. Comparing silver through diamond you can see that this isn't the case. For example, zerg gets more common. I think it's reasonable to conclude that the races are pretty balanced, but I acknowledge that some of the criticisms I'm getting are valid. The analysis was a chi-square analysis comparing observed distributions vs. homogenous distributions (assumed under random sorting). Also, this is definitely science, because it uses a hypothesis-based testing approach. Math is just a tool to accomplish the science. Whether it's good science seems to be stimulating a rigorous debate LOL. The 'real' mathematical way to test for imbalance would probably require treating players individually, and using regression-based approaches to predict performance based on race, league placement, etc. That way one could parse out the influence race has on win rate. I don't have access to that kind of data, though. The best would be to control for player as a variable, to see if players consistently perform better with certain races than others. Edit: these^^ are games played not players active, so take the graph with the appropriate grain of salt. This is just bs. You are omitting various factors in your analysis. For example, at lower levels, when players get crushed with a certain race, they tend to change race easily. i.e. a significant variable you have omitted from that diagram is attachness to a certain race, which is obviously positively correlated with skill level. This is just because the amount of 'investment' in a certain race increases with skill level, and the players' utility is usually a function of 'value of investment', which is something like max{Value of investment in T, value of investment in P, value of investment in Z}. With the ratio of (value of investment/time or effort invested) an effective indicator of ratio balance, assuming an representative agent who is trying to maximise his utility. To avoid/minimize this problem you should either gather some reliable information about the parameter of this variable or picking some sample which will exclude this, i.e. pick the 'most attached' bracket, i.e. diamond, or even high end diamond, pro leagues and tournament. Picking some result and trying to interpret it as solely caused by one factor when obviously there are other factors at work is an indication that either you are very biased, i.e. have a strong incentive to distort the result towards a certain direction, or your level of skill in utilizing 'scientific method' is just horrible. So, this is not science, just some kid trying to prove his view in the name of science with the help of pseudo/naive/broken scientific method. | ||
|
rextyrann
Germany41 Posts
On August 17 2010 08:01 GagnarTheUnruly wrote: The analysis was a chi-square analysis comparing observed distributions vs. homogenous distributions (assumed under random sorting). in not one of the post you answer to the problem of the matchmaking system. but this sentence of you should be explanation enough. it is NOT a random sorting. thats why it is a matchmaking system. thats why the test you used doesnt apply on those stats. but kudos to your work. very well presented and appart of the wrong starting point of a random sorting it would be significant. there is just no way of analysing balancing issues just by stats unless we do have all the data about matchup stats and a sample of games outside of the matchmaking system... | ||
|
st3roids
Greece538 Posts
how can u gather that data accurately without working for blizzard. | ||
|
Chronald
United States619 Posts
I think your theory about map-making is the right approach. To really do away with the zerg early game weakness, maps need to be bigger. But these issues will be sorted out soon. Don't forget to check out the iCCup maps, they are way more balanced then the Blizz official ones. | ||
|
holy_war
United States3590 Posts
On August 17 2010 09:53 st3roids wrote: Id like to ask , there literally millions of games each day in bnet. how can u gather that data accurately without working for blizzard. Data crawling of Battle.net profiles from people's accounts. | ||
|
Chimpalimp
United States1135 Posts
| ||
|
GagnarTheUnruly
United States655 Posts
On August 17 2010 09:26 petered wrote: Distribution of race amongst leagues is sadly not valid as an indicator of racial balance. It makes the key assumption that players of different skill level are picking the different races at the same distribution. Graphing race distribution against league level isn't a statistical test and therefore it doesn't make any assumptions. What that graph shows is that roughly equivalent numbers of games are being played by a particular race at each league level. What this suggests is that there is no sorting effect, whereupon a weak race is held back into lower leagues because players that favor that race are having trouble advancing because they are losing games with that race. It is an indirect way of testing that hypothesis. Viewed in the context of the other data, it suggests (but doesn't prove) that AMM is not the only, or even an important, factor in keeping race performance even within leagues. I totally agree that it would be great to analyze the data using player as an explicit factor, but I don't have access to that data. This doesn't prove that there aren't matchup imbalances. Terran could beat Protoss 60% of the time, Protoss could beat Zerg 60% of the time, and Zerg could beat Terran 60% of the time. At the end of the day each race would have an approximately ~50% win ratio, as supported by your graphs and charts. However, TvP, PvZ and ZvT would all be imbalanced. The imbalances would just cancel one another out in terms of overall win ratios. I agree totally. It would be fun to do that but again I lack the data. If someone can get it for me, I'll do that analysis. This is just bs. You are omitting various factors in your analysis. At lower levels, when players get crushed with a certain race, they tend to change race easily. i.e. a significant variable you have omitted from that diagram is attachness to a certain race, which is obviously positively correlated with skill level. This is just because the amount of 'investment' in a certain race increases with skill level, and the players' utility is usually a function of 'value of investment', which is something like max{Value of investment in T, value of investment in P, value of investment in Z}. With the ratio of (value of investment/time or effort invested) an effective indicator of ratio balance, assuming an representative agent who is trying to maximise his utility. To avoid/minimize this problem you should either gather some reliable information about the parameter of this variable or picking some sample which will exclude this, i.e. pick the 'most attached' bracket, i.e. diamond, or even high end diamond, pro leagues and tournament. Picking some result and trying to interpret it as solely caused by one factor when obviously there are other factors at work is an indication that either you are very biased, i.e. have a strong incentive to distort the result towards a certain direction, or your level of skill in utilizing 'scientific method' is just horrible. So, this is not science, just some kid trying to prove his view in the name of science with the help of pseudo/naive/broken scientific method.Last edit: 2010-08-17 09:38:21 This post is not very constructive. What you're suggesting is an absurdly complex model. And please don't disparage my abilities as a scientist. I'm actually a really good scientist and I have some skill at dealing with difficult data. I would like to be able to use a regression model to see how race, placement, and matchup affect the performance of individual players, but as I've noted repeatedly I don't have access to that data. In science when you can't get certain data you need to take indirect approaches that often involve making important assumptions. Often, there are ways to test those assumptions either directly or indirectly, but in this case the data set is extremely complicated, particularly due to match placement. Also, I really need to emphasize that very few assumptions are required to do a chi square test. There are no distributional assumptions to the test. It simply tells us very clearly that within each of the leagues, if a match is picked at random the outcome is totally independent of the one race entering that match. The test doesn't assume that the players are distributed randomly among the races or anything like that. It just tests the hypothesis that states are nonrandomly distributed among the categories being analyzed. The data show that within a league the races have quantifiably different but functionally equal chances of winning randomly selected games. This is a point of fact. There are three non-mutually exclusive possible causes for this that I can think of: 1) the balance is good 2) the matchmaking system is accomodating for poor balance 3) the matchup balance or map balance is poor but it evens out when you ignore the confounding factors There is no way to test the third cause, so we need to suspend it for now, and refer to better judement that it is probably happening but may not be extremely important. It's certainly a hypothesis that bears testing, however. The second cause can be tested indirectly by graphing race use frequency with league status. Since there appears to be no pattern, it suggests that the second cause is also not important. This leaves the first cause. Given consideration of the possible causes of this pattern, it is a reasonable conclusion that good balance is probably largely responsible. It also means it's ignorant to make statements like 'Terran is unbalanced,' because there is no evidence to support such a statement, and becasue the evidence that does exist suggests the opposite. This is not to say that high level players like IdrA, who play in a rarefied realm with tight builds and well rehearsed timing, might not sense conditions that give certain races advantages at certain times. Certainly in BW we've witnessed major shifts of the 'metagame' that resulted in periods of dominance for the various races. | ||
|
MamiyaOtaru
United States1687 Posts
On August 17 2010 10:18 Oddysay wrote: finaly someone who show zerg are not imbalanced and dont need buff This shows no such thing. Matchmaking ruins this sort of analysis. if *if* zerg is really underpowered, zerg players will get placed down until they are playing worse terrans, toss, or other zerg of a similar level until their win rate normalizes. The win rate looks normal, but says nothing about how the terrans and toss they are playing would be ranked a lot lower if they were zerg, or how the zerg would be ranked higher if he was a terran or toss. | ||
|
Oddysay
Canada597 Posts
pratice and get good , stop hope blizzard will fix the game so you can win | ||
|
GagnarTheUnruly
United States655 Posts
On August 17 2010 10:17 MamiyaOtaru wrote: matchmaking kills this. if *if* zerg is really underpowered, zerg players will get placed down until they are playing worse terrans, toss, or other zerg of a similar level until their win rate normalizes. The win rate looks normal, but says nothing about how the terrans and toss they are playing would be ranked a lot lower if they were zerg, or how the zerg would be ranked higher if he was a terran or toss. But we would see that on a graph of race use vs. placement. In that case we would see more zerg in lower leagues and fewer in higher leagues. We don't see such a pattern, suggesting that if matchmaking is having an effect, it probably isn't a strong one. Of course there's no way to know for sure without testing it directly. | ||
|
blueblimp
Canada297 Posts
On August 17 2010 09:26 petered wrote: The best way to compare racial imbalances, in my mind, is to collect info on random players, and their win ratios with different races. In that case you are holding the skill of the player constant, and hence you can make some progress. You still have to make some pretty big assumptions, but it will get you farther than pure win/loss ratios. Yes. In fact, not only is it the best way to evaluate racial imbalance, I claim it's the only way short of getting players to actively participate in a study. To see why this is, consider an arbitrary race, say Protoss. Looking at statistics alone, if you don't use Random, how are you going to tell the difference between "Protoss players are 2x as skilled as non-Protoss players" and "Protoss is 2x as good as other races"? It's completely impossible. I know that's a contrived example, but it's exactly the problem with evaluating Zerg balance, given that the race is pretty unfriendly to newbies. So to sum up: please look at games random players play and evaluate their per-race stats. Edit: I'm not saying this way is flawless either. You'll still have doubts about "well maybe random players tend to be higher-skilled at race X than race Y", but at least that's not likely to make a big difference. | ||
|
AmishNukes
United States98 Posts
| ||
| ||
