Disclaimer: I’m not concerned about game balance at all. I’m hoping to have a discussion on the math & statistics behind Blizzard's adjusted-win-percentage that they rely on heavily.
Late last year, Blizzard released a bunch of ladder statistics on “skill-adjusted-win-percentages” for the different matchups. The reason I have it in quotes, is because they never really explained how they did the skill-adjustments. I’ve always been skeptical about whether such a “skill-adjustment” is really possible.
Well recently, I found the following video where Blizzard partially explains how they calculate the “skill-adjusted-win-percentages.” Watch the first 5 minutes of the following video:
Gist of what they said: Raw league matchup numbers aren’t very meaningful because of matchmaking’s system ability to matchup players with equally challenging opponents. The math guy mentions specifically: Not only does the system put players in 50-50 matches, it also tries to keep the race matchups at 50-50 as well. Because of this, we have to adjust for player skill to calculate the true matchup win rates. Example: a ZvP match is about to be played. The Zerg player’s rating (odds of winning) relative to the Protoss player is 55-45. The Zerg race’s rating relative to the Protoss race is 53-47. If the Protoss player ends up winning, the player ratings will then converge to 51-49. The race ratings will also converge to 52-48.
Their explanation just didn’t click with me. Rating systems such as ELO are great when you’re dealing with a single unknown (relative player strength). But can they really work if you’re trying to differentiate between 2 unknowns? Both relative player skill & race balance? I constructed the following scenario which seems to suggest that this is impossible.
It’s important to first establish the following: Any good rating system, including ELO & the point system, relies on the following principle: • Give each agent (could be a player, or a race) a certain rating as an estimate for how strong the agent is • If 2 agents play and one wins at a higher percentage, the more successful agent should eventually end up with a higher rating • If a higher rated agent & a lower rated agent play against each other, and each wins with an equal percentage, the 2 ratings should eventually converge
The ELO system that Blizzard uses for MMR is an optimized algorithm that allows ratings to stabilize much quicker, but other rating systems that utilize the above principle (including the point-system), can achieve the same results in the long run.
Now going back to the scenario, consider the case where Blizzard releases a new patch which nerfs Zerg and makes it UP relative to both Protoss & Terran (eg, drones now cost 60 min). Consider what will happen to the average Zerg player. He will start losing more than 50% of his games, and his MMR will start dropping. Because of his lower MMR, he’ll start playing against weaker opponents. Eventually, his MMR will stabilize at a level where he starts winning 50% of his future games.
Now let’s say Blizzard had assigned each race a rating as well, to track how “strong they think it is.” Suppose that before the patch, all the races were balanced & had equal rating. Immediately after the patch, because the Zerg population goes through a losing streak, the Zerg rating will drop.
But eventually, the Zerg players will have stabilized their MMR and start winning 50% of their games. At this point, because of the last bullet point in the rating system’s principles (ratings will converge at 50% win rates), the Zerg rating will start increasing again. Remember also that the stabilized Zerg players are playing against opponents of the same MMR, so there’s no way to “account for player skill.” Eventually, the zerg rating will once again converge with the other races, even though Zerg is now UP.
Based on this scenario, it seems impossible to determine whether a race is truly UP, using Blizzard’s rating system. Thoughts? Any ideas on how Blizzard could possibly be “accounting for player skill” in calculating race balance?
agreed. this formula blizzard is using is dumb, however i think (HOPEFULLY) they are using this just to see when one race has a clear advantage (which results could show) and then when they look at the advantages they must look at the races units and high level games to try to decide where the strength of one race is coming from to allow them to win more
The third and arguably most important factor that they exclude from that though is map balance. They will never get a perfect rating unless the system takes the maps in account.
On August 03 2011 13:34 KiLL_ORdeR wrote: The third and arguably most important factor that they exclude from that though is map balance. They will never get a perfect rating unless the system takes the maps in account.
Even in terms of position. ZvT close positions on Shattered versus far positions, for example, has got to be pretty skewed.
Measuring balance using any single measurement tool is going to be faulty. That's why Blizzard uses a large number of different tools and gathers data from all of them individually in order to make balance decisions.
Because they use multiple different tools, an argument like "measuring balance based on this tool is faulty" is silly. If you're trying to say that Blizzard does a bad job at balancing the game, then attack the entirety of their balance system altogether (pro feedback, community feedback, balance design team, matchmaker statistics, tournament statistics, etc).
Don't pick out one part of a system and then act all surprised when it doesn't work when separated from the rest of the system.
Your example assumes that Blizzard would simply ignore the downward trend in both players and the race ratings post patch and wait for the statistics to stabilise before analysing them. While you may be correct that they inevitably will stabilise, history has shown Blizzard won't just wait for that to happen and then announce that everything is hunky dory.
There are a lot of variables (such as maps, new strategies, patches, etc.) which will cause shifts in these trends, and it no doubt takes quite a while to do a thorough analysis on the root cause of such shifts, but I think there's a lot of evidence that this sort of thing is being taken into account.
There's a lot of points to go through this topic so I'll try to answer them as I see them.
But can they really work if you’re trying to differentiate between 2 unknowns? Both relative player skill & race balance? I constructed the following scenario which seems to suggest that this is impossible.
There's a TON of math out there that deal with 2+ independent or unknown variables. You can basically use probability and statistics to predict with a degree of certainty where each unknown lies.
The ELO system that Blizzard uses for MMR is an optimized algorithm that allows ratings to stabilize much quicker, but other rating systems that utilize the above principle (including the point-system), can achieve the same results in the long run.
"Optimized algorithm" is almost ubiquitous with professionally created ladder systems. The general consensus about MMR, however, isn't that it "stabilizes" much quicker, but that it gets to the general skill area much quicker. Evidence seems to indicate that MMR actually isn't very stable at all. It varies widely after each match and gives more of a window of skill range for each player and less of an absolute position among players. Even with perfectly differentiated skill levels, you're likely to see statistical "noise" in every player's MMR.
What you can obtain from MMR, however, is relative win percentage expectations, just like a normal Elo system. However, the uncertainty of this prediction is probably much greater (statistically) than a traditional Elo. Where a fully matured Elo scale may have an expanded uncertainty of ±0.5%, Blizzard's may be closer to an expanded uncertainty of ±1.2%. A greater number of matches and predictions will lower each of these numbers. ***These numbers are completely made up and used as an example.***
• Give each agent (could be a player, or a race) a certain rating as an estimate for how strong the agent is • If 2 agents play and one wins at a higher percentage, the more successful agent should eventually end up with a higher rating • If a higher rated agent & a lower rated agent play against each other, and each wins with an equal percentage, the 2 ratings should eventually converge
For these points, the stability and gradual increase in "points" actually come from ladder points. Since MMR is littered with too much noise, ladder points act as a sort of anchor to place people where they "belong." Your ladder points will drift much slower to the general area of your MMR, while your MMR can properly reflect who to place you against on a good and bad day. This way, you don't have to lose 300 ladder points to start playing people a division lower if you're just doing awful today. You'll maintain a ~50% winrate, but your point gains and losses will become unbalanced and you'll slowly slide downward.
Now going back to the scenario, consider the case where Blizzard releases a new patch which nerfs Zerg and makes it UP relative to both Protoss & Terran (eg, drones now cost 60 min). Consider what will happen to the average Zerg player. He will start losing more than 50% of his games, and his MMR will start dropping. Because of his lower MMR, he’ll start playing against weaker opponents. Eventually, his MMR will stabilize at a level where he starts winning 50% of his future games.
(...etc. until end of post)
This is more of a philosophical question applied to real world statistics. Since the criteria for skill is based on results and not quantitative characteristics, a bit of guesswork and intuition have to come into play. This is where they break up the statistics between leagues, look at tournament results, ask players about balance, et al. If they began to see all Zerg drop down the ladder, they would know that they swung the nerf bat too hard. Even then, however, balance is relatively subjective. Depending on how you look at data, all 3 races could look OP. If you throw psychology into the mix and say things like "people won't try to get better if they're winning more than X% of the time," it gets even harder to balance. As long as they keep a vigilant eye on results across the spectrum without seeing any glaring imbalances (like Zerg get knocked out first round every tournament ever), we can be assured that balance isn't broken.
On August 03 2011 13:34 KiLL_ORdeR wrote: The third and arguably most important factor that they exclude from that though is map balance. They will never get a perfect rating unless the system takes the maps in account.
Even in terms of position. ZvT close positions on Shattered versus far positions, for example, has got to be pretty skewed.
You'd think something like this, and then you learn that metal (with all positions enabled), Z has something like a 60% win ratio.
That's factoring in an "auto-loss" 1/3 of the time.
Personally, I don't feel the game is as imbalanced as a lot of people tend to think
Simplistic example P1 is Zerg, he has 10 wins vs Protoss and 9 losses vs Protoss. Each of these Protoss players had a lower MMR and P1 was favored
Since all 19 games played had P1 as being favoured to win, it does not necessarily translate to a 52.63% win ratio for the Zerg player P1. If overall, Zerg can only hold an aggregate 52% win ratio against substantially less skilled players, then there is potentially something wrong with the match up.
These are all simplistic numbers, but expand it for the entire population and you can see why you need to adjust win percentages for relative skill level.
A lot of people have responded with some variant of "If Blizzard sees all the Zerg players have significantly lower MMR, they'll know there's something wrong." If Blizzard is doing this, then what they're basically doing is comparing the average zerg player's MMR with the average Terran player's MMR. This approach can break for so many reasons, which I'm not going to get into now. But if this is indeed what they're doing... why even put up the smoke screen of complicated formulas leading to "adjusted-win-rates"?
As a lot of people have mentioned, much of Blizzard's balancing act revolves around subjective judgments, which is certainly a valid approach. However, I'm uncomfortable with Blizzard constantly bringing up their "adjusted-win-rates" to support their claim that everything is balanced. If they're going to present this statistic as proof, it only makes sense that we question whether this statistic makes sense.
My central question is this: Does Blizzard's "adjusted-win-rate" hold any value at all? The example I brought up was to show that no matter how UP a race is, the win-rate statistics will still show perfect balance. If so, why shouldn't we throw these statistics in the trash?
Omnipresent, thanks for the awesome link. I was gonna make a very similar post actually, and am very amused/disappointed to see that someone has taken the words out of my mouth
aks, thanks for the detailed reply. When I was talking about stabilization, I was assuming consistent skill levels, to give Blizzard's statistics further benefit. Regarding the math tools that you mentioned exist... if you could give an example of how one of them can be used in this situation, that would be great. In my mind, the problem does seem unsolvable... kinda like a single equation with 2 unknowns.
On August 03 2011 14:09 whacks wrote: Thanks for the responses all.
A lot of people have responded with some variant of "If Blizzard sees all the Zerg players have significantly lower MMR, they'll know there's something wrong." If Blizzard is doing this, then what they're basically doing is comparing the average zerg player's MMR with the average Terran player's MMR. This approach can break for so many reasons, which I'm not going to get into now. But if this is indeed what they're doing... why even put up the smoke screen of complicated formulas leading to "adjusted-win-rates"?
As a lot of people have mentioned, much of Blizzard's balancing act revolves around subjective judgments, which is certainly a valid approach. However, I'm uncomfortable with Blizzard constantly bringing up their "adjusted-win-rates" to support their claim that everything is balanced. If they're going to present this statistic as proof, it only makes sense that we question whether this statistic makes sense.
My central question is this: Does Blizzard's "adjusted-win-rate" hold any value at all? The example I brought up was to show that no matter how UP a race is, the win-rate statistics will still show perfect balance. If so, why shouldn't we throw these statistics in the trash?
Because they won't show perfect balance... You act as if skill and metagame never change. They use these win-rate statistics to track trends and follow the "migration" of large groups of players. In your example, Zerg won't drop down over night. Even after a devastating patch in your example, they have a window of probably a month or 2 to watch this happen and make adjustments accordingly. If you always judge the merits of a system by its ends, you're going to have a very depressing and disappointing life in general...
On August 03 2011 13:34 KiLL_ORdeR wrote: The third and arguably most important factor that they exclude from that though is map balance. They will never get a perfect rating unless the system takes the maps in account.
Even in terms of position. ZvT close positions on Shattered versus far positions, for example, has got to be pretty skewed.
You'd think something like this, and then you learn that metal (with all positions enabled), Z has something like a 60% win ratio.
That's factoring in an "auto-loss" 1/3 of the time.
Personally, I don't feel the game is as imbalanced as a lot of people tend to think
Every time I get delta quadrant I leave, and same with other similar maps, so when i do get meta I am playing somebody I am 99% supposed to win vs.
On August 03 2011 14:13 whacks wrote: aks, thanks for the detailed reply. When I was talking about stabilization, I was assuming consistent skill levels, to give Blizzard's statistics further benefit. Regarding the math tools that you mentioned exist... if you could give an example of how one of them can be used in this situation, that would be great. In my mind, the problem does seem unsolvable... kinda like a single equation with 2 unknowns.
In general, you use systems of equations/linear algebra to solve these problems. You've probably seen stuff like this:
3x+5y=5 5x+2y=7
As long as the results are linearly independent (where basically one equation isn't a multiple of another), you can get answers for x and y. If you then throw in another equation where the answer creates a "contradiction":
3x+5y=7
This is where you get uncertainty (more or less). You can quantify this uncertainty to give you an idea of where the values of x and y lie. The way Blizzard's equations work look, at the very core, like this:
Ax+By+Cz=D
Where x, y, and z each represent race, rating, and map variables (probably winrates ), and A, B, C, and D represent some constant values, like actual the race they're facing, rating of their opponent (or difference), map selection, and the projected winrate. Different operators are used (+, -, *, /, ^, etc), but that's essentially how it works.
On August 03 2011 14:09 whacks wrote: Thanks for the responses all.
A lot of people have responded with some variant of "If Blizzard sees all the Zerg players have significantly lower MMR, they'll know there's something wrong." If Blizzard is doing this, then what they're basically doing is comparing the average zerg player's MMR with the average Terran player's MMR. This approach can break for so many reasons, which I'm not going to get into now. But if this is indeed what they're doing... why even put up the smoke screen of complicated formulas leading to "adjusted-win-rates"?
As a lot of people have mentioned, much of Blizzard's balancing act revolves around subjective judgments, which is certainly a valid approach. However, I'm uncomfortable with Blizzard constantly bringing up their "adjusted-win-rates" to support their claim that everything is balanced. If they're going to present this statistic as proof, it only makes sense that we question whether this statistic makes sense.
My central question is this: Does Blizzard's "adjusted-win-rate" hold any value at all? The example I brought up was to show that no matter how UP a race is, the win-rate statistics will still show perfect balance. If so, why shouldn't we throw these statistics in the trash?
Because they won't show perfect balance... You act as if skill and metagame never change. They use these win-rate statistics to track trends and follow the "migration" of large groups of players. In your example, Zerg won't drop down over night. Even after a devastating patch in your example, they have a window of probably a month or 2 to watch this happen and make adjustments accordingly. If you always judge the merits of a system by its ends, you're going to have a very depressing and disappointing life in general...
LOL, don't take it so personally. I'm not here seeking life coaching The example I gave is an extreme one, but you can easily think up more subtle cases. What if the imbalance has existed ever since game launch? In this case, there is no adjustment at all for Blizzard to track. In fact, any fixes that Blizzard puts in to fix the imbalance, will only show an upward shift for the previously-UP race & a downward-shift for the previously-OP race.
The point I'm trying to get across: If Blizzard's statistics don't work even in the ideal case where player skill & metagame is static and one race is very obviously UP, then of what possible value are they?
did anyone else watch the question and answer section of the video. That was the most tragic thing I have ever seen. Obviously when the majority of the ppl who play ur games have that mindset your going to be in a position where u cant make a well balanced game.
On August 03 2011 14:09 whacks wrote: Thanks for the responses all.
A lot of people have responded with some variant of "If Blizzard sees all the Zerg players have significantly lower MMR, they'll know there's something wrong." If Blizzard is doing this, then what they're basically doing is comparing the average zerg player's MMR with the average Terran player's MMR. This approach can break for so many reasons, which I'm not going to get into now. But if this is indeed what they're doing... why even put up the smoke screen of complicated formulas leading to "adjusted-win-rates"?
As a lot of people have mentioned, much of Blizzard's balancing act revolves around subjective judgments, which is certainly a valid approach. However, I'm uncomfortable with Blizzard constantly bringing up their "adjusted-win-rates" to support their claim that everything is balanced. If they're going to present this statistic as proof, it only makes sense that we question whether this statistic makes sense.
My central question is this: Does Blizzard's "adjusted-win-rate" hold any value at all? The example I brought up was to show that no matter how UP a race is, the win-rate statistics will still show perfect balance. If so, why shouldn't we throw these statistics in the trash?
Yes, Blizzard has equations to match people by their skill levels and then uses another equation to convert that skill level across races (like converting the US dollar to the Euro).
Let's pretend there's a Terran player with a skill level of 2100 ELO and a Zerg player of 1900 ELO. The difference of 200 ELO (in most systems) is probably enough to be sure that the Terran player will almost always win against the Zerg player if TvZ was perfectly balanced. Now let's assume that that TvZ is favorable to the Zerg player, such that, a Zerg player with an intrinsic skill level of 1900 is on even footing with a Terran player of 2100, and is expected to win 50% of the time and lose 50% of the time.
So Blizzard has a formula to convert Terran ELO to Zerg ELO to Protoss ELO, etc. I figure the way they probably do this is look at the statistical curve for mean of each race's ELO. And figure out the disparity. And then they do this at each league.
It's not a perfect system, and there is some uncertainty, but if there were severe issues with balance, they would become very obvious, very quickly.
You bring up a good point, which is that things get tricky in the case where ZvP is balanced, but not ZvT. Check out the link by Omni earlier in the thread. The post is regarding this very topic & analyzes it very well.
However, your analogy breaks down if Zerg is equally overpowered against both Terran and Protoss. Suppose the Zerg player deserves to be at 1900, but because of the racial imbalance, has a 50-50 chance of beating 2100 Terran & Protoss players. His MMR will then rise to 2100 because of the way the ELO system works (in fact, it would never have stabilized at 1900 to begin with). Because of this, when Blizzard looks at ladder results, all they see is a bunch of 2100 Zergs, 2100 Terrans, and 2100 Protoss with 50-50 win rates.
This is exactly the point I'm trying to get across. If ZvT is unbalanced but ZvP is balanced (or unbalanced the other way), then you will see the results reflected in the statistics. But if one race is equally UP or OP against both other races, then all the ladder statistics would become absolutely meaningless.
On August 03 2011 14:09 whacks wrote: Thanks for the responses all.
A lot of people have responded with some variant of "If Blizzard sees all the Zerg players have significantly lower MMR, they'll know there's something wrong." If Blizzard is doing this, then what they're basically doing is comparing the average zerg player's MMR with the average Terran player's MMR. This approach can break for so many reasons, which I'm not going to get into now. But if this is indeed what they're doing... why even put up the smoke screen of complicated formulas leading to "adjusted-win-rates"?
As a lot of people have mentioned, much of Blizzard's balancing act revolves around subjective judgments, which is certainly a valid approach. However, I'm uncomfortable with Blizzard constantly bringing up their "adjusted-win-rates" to support their claim that everything is balanced. If they're going to present this statistic as proof, it only makes sense that we question whether this statistic makes sense.
My central question is this: Does Blizzard's "adjusted-win-rate" hold any value at all? The example I brought up was to show that no matter how UP a race is, the win-rate statistics will still show perfect balance. If so, why shouldn't we throw these statistics in the trash?
Because they won't show perfect balance... You act as if skill and metagame never change. They use these win-rate statistics to track trends and follow the "migration" of large groups of players. In your example, Zerg won't drop down over night. Even after a devastating patch in your example, they have a window of probably a month or 2 to watch this happen and make adjustments accordingly. If you always judge the merits of a system by its ends, you're going to have a very depressing and disappointing life in general...
LOL, don't take it so personally. I'm not here seeking life coaching The example I gave is an extreme one, but you can easily think up more subtle cases. What if the imbalance has existed ever since game launch? In this case, there is no adjustment at all for Blizzard to track. In fact, any fixes that Blizzard puts in to fix the imbalance, will only show an upward shift for the previously-UP race & a downward-shift for the previously-OP race.
The point I'm trying to get across: If Blizzard's statistics don't work even in the ideal case where player skill & metagame is static and one race is very obviously UP, then of what possible value are they?
Edit: I see 2 new posts since I last hit refresh. Reading through those now.
That's where the philosophy and subjectivity comes in. In this case, you take the faith and credit that the game was, for the most part, balanced upon release. This is the ONLY reason phrases like, "That's because Zerg players are better than ____, but it only shows when they play on a fair map!" and "So many crappy Terran players relied on all-ins, and now since the game takes REAL skill, they're losing all the time!" hold any merit what-so-ever. In reality, there wasn't a race at release that so many people flocked to and stayed with that it was overwhelming for the other races.
Essentially, these win-rates are one of the tools in the shed we can use, but we should never rely on them too much. We have to take input from the community, theorycrafting, and our own play experiences to put those win-rates into perspective, and that's exactly what Blizzard does.
I also want to point out that I was just commenting on the very nature of using the "ends" of these systems as a way to rate them. In reality, the reason why Elo systems SUCK for a majority of people isn't because you start at some median and can fall below it, but because it takes so long to start getting matched up against people you can expect to win (or lose) against. The end system is perfect and beautiful, but in a constantly flowing system, it's not so much. Blizzard took a different approach with MMR and allowed people to get 50% win-rates fairly quickly, then let the point system fill in the rest. The result is a developing system that is more pretty, but the end result not so much.
Aks, we both agree that Blizzard uses a lot of other tools in their quest for balance. However, I disagree that the adjusted-win-rate is a tool that's worth keeping around at all. Every tool in your box should have a purpose. A tool that can't fulfill its purpose should be discarded. Even in the most ideal case, even if a race is severely OP or UP compared to the other 2 races, Blizzard's adjusted-win-rate will give no indication of this. I'm assuming you agree with me on this, since you haven't disputed my scenario or offered an alternative. A tool that bad should be junked immediately.
I feel like a large issue is that Blizzard is measuring the skill of a player as the success of a player.
Consider a hypothetical situation with 3 races, X,Y, and Z, and 3 players, x, y, and z of these respective races.
x is highly skilled, y is moderately skilled, and z has low skill. X is vastly underpowered, y is relatively balanced, and Z is vastly overpowered.
Suppose this balances in such a way that x, y, and z each have a 50% win ratio against each other.
According to Blizzard, all 3 players are of equal skill due to their win rates (equal win rate against players of equal MMR means their MMR will be equal and therefore their skill will be equal), and since players of equal skill have equal win rates, the game must be balanced. This of course contradicts our hypothesis regarding balance.
Regardless of the underlying math, these statistics appear to be flawed based on this (they make assumptions that are likely untrue).
I don't think they take into account when (gametime) the cert ant race wins, although this is only important in masters or diamond league because of all the cheese in the lower league, but something could be found like that zergs win 70% in the late game or something like that, that you just can't feel by yourself, obviously the 70% win rate in the late game is not ture but something like that could arise.
This makes no sense. So blizzard is matching up based on both player win percentage and race win percentage? That should mean that the win percentage of all the races should also eventually equal 50% given an infinite number of games played, not because of any map/balance success but instead because of a bad matchmaking algorithm.
On August 04 2011 09:13 McFortran wrote: I feel like a large issue is that Blizzard is measuring the skill of a player as the success of a player.
Consider a hypothetical situation with 3 races, X,Y, and Z, and 3 players, x, y, and z of these respective races.
x is highly skilled, y is moderately skilled, and z has low skill. X is vastly underpowered, y is relatively balanced, and Z is vastly overpowered.
Suppose this balances in such a way that x, y, and z each have a 50% win ratio against each other.
According to Blizzard, all 3 players are of equal skill due to their win rates (equal win rate against players of equal MMR means their MMR will be equal and therefore their skill will be equal), and since players of equal skill have equal win rates, the game must be balanced. This of course contradicts our hypothesis regarding balance.
Regardless of the underlying math, these statistics appear to be flawed based on this (they make assumptions that are likely untrue).
I think one underlying assumption is that there will be some highly skilled players who plays the overpowered race, which is not that big an assumption to make. If that race is overpowered and the player is highly skilled there is no way to get that high an MMR otherwise, and would show up in the tool.
On August 04 2011 09:32 h0oTiS wrote: I don't think they take into account when (gametime) the cert ant race wins, although this is only important in masters or diamond league because of all the cheese in the lower league, but something could be found like that zergs win 70% in the late game or something like that, that you just can't feel by yourself, obviously the 70% win rate in the late game is not ture but something like that could arise.
Define late game. I have seen TvZ matches where both players are barely mining and it has gone on for ages. Is nestea vs sc late game? I certainly saw no hive tech. Was Destiny holding off a push with infestoRs and waiting 20 minutes to break a Terran turtle considered late game. How many bases does Zerg have? How many does Terran have? Did DRG vs MVP reach late game? The metalopolis game was 5 base vs 4 base, late game? It is too abstract a term to be defined with game time. Some would say late game is brood lord/corruptor/infestor vs marine tank medivac Viking ghost. But does rushing to brood lords count as late game then? Nestea had 15 minute brood lords vs Ensnare.
On August 03 2011 14:09 whacks wrote: Thanks for the responses all.
A lot of people have responded with some variant of "If Blizzard sees all the Zerg players have significantly lower MMR, they'll know there's something wrong." If Blizzard is doing this, then what they're basically doing is comparing the average zerg player's MMR with the average Terran player's MMR. This approach can break for so many reasons, which I'm not going to get into now. But if this is indeed what they're doing... why even put up the smoke screen of complicated formulas leading to "adjusted-win-rates"?
As a lot of people have mentioned, much of Blizzard's balancing act revolves around subjective judgments, which is certainly a valid approach. However, I'm uncomfortable with Blizzard constantly bringing up their "adjusted-win-rates" to support their claim that everything is balanced. If they're going to present this statistic as proof, it only makes sense that we question whether this statistic makes sense.
My central question is this: Does Blizzard's "adjusted-win-rate" hold any value at all? The example I brought up was to show that no matter how UP a race is, the win-rate statistics will still show perfect balance. If so, why shouldn't we throw these statistics in the trash?
Yes, Blizzard has equations to match people by their skill levels and then uses another equation to convert that skill level across races (like converting the US dollar to the Euro).
Let's pretend there's a Terran player with a skill level of 2100 ELO and a Zerg player of 1900 ELO. The difference of 200 ELO (in most systems) is probably enough to be sure that the Terran player will almost always win against the Zerg player if TvZ was perfectly balanced. Now let's assume that that TvZ is favorable to the Zerg player, such that, a Zerg player with an intrinsic skill level of 1900 is on even footing with a Terran player of 2100, and is expected to win 50% of the time and lose 50% of the time.
So Blizzard has a formula to convert Terran ELO to Zerg ELO to Protoss ELO, etc. I figure the way they probably do this is look at the statistical curve for mean of each race's ELO. And figure out the disparity. And then they do this at each league.
It's not a perfect system, and there is some uncertainty, but if there were severe issues with balance, they would become very obvious, very quickly.
You bring up a good point, which is that things get tricky in the case where ZvP is balanced, but not ZvT. Check out the link by Omni earlier in the thread. The post is regarding this very topic & analyzes it very well.
However, your analogy breaks down if Zerg is equally overpowered against both Terran and Protoss. Suppose the Zerg player deserves to be at 1900, but because of the racial imbalance, has a 50-50 chance of beating 2100 Terran & Protoss players. His MMR will then rise to 2100 because of the way the ELO system works (in fact, it would never have stabilized at 1900 to begin with). Because of this, when Blizzard looks at ladder results, all they see is a bunch of 2100 Zergs, 2100 Terrans, and 2100 Protoss with 50-50 win rates.
This is exactly the point I'm trying to get across. If ZvT is unbalanced but ZvP is balanced (or unbalanced the other way), then you will see the results reflected in the statistics. But if one race is equally UP or OP against both other races, then all the ladder statistics would become absolutely meaningless.
This is not true because if the Zerg is at 1900 and the Terran at 2100, and Zerg is racially imbalanced towards Terran and the other way against Zerg versus protoss, theres no way the Zerg will stabilize at 2100.
So in your example 1900 Zerg vs 2100 Terran will be 50/50 1900 Zerg vs 1700 Protoss will be 50/50 1900 Zerg vs 1900 Zerg will be 50/50
So Zerg remains relatively at 1900, even though the Terran is of higher skill.
I've studied how TrueSkill works on the Xbox 360 and a single game out of a ton of games played has a very small impact on the change in skill, to the point that the zerg might go up to 1905 and the Terran down to 2095 (if Zerg wins), and theres a smaller pool of games played, if more games are played the change will be way less.
I like that part where they're satisfied with 5% differences in W/L when that percent is based off millions of matches. Even a 1% difference with that many matches means that one race very, very significantly favors the other. Wtf are they talking about when a 55% win rate for a specific race matchup is just "borderline?"
On August 03 2011 13:07 whacks wrote: But eventually, the Zerg players will have stabilized their MMR and start winning 50% of their games. At this point, because of the last bullet point in the rating system’s principles (ratings will converge at 50% win rates), the Zerg rating will start increasing again.
This is the fallacy in your argument. If Zerg players start losing, their ratings fall, and they stabilize at 50/50 win/loss ratios against other-race players with lower MMRs, their rating will be stabilized at a lower MMR. It won't start increasing again without the players getting better or another change to the game.
The way you adjust for skill is to look at overall MMR distribution among each race's population. If one race, let's say Zerg, has a population distribution that's weighted toward lower MMRs, chances are it's the race that's doing it unless there's some external indication that better players systematically favor the other races for some reason.
On August 03 2011 14:09 whacks wrote: If Blizzard is doing this, then what they're basically doing is comparing the average zerg player's MMR with the average Terran player's MMR. This approach can break for so many reasons, which I'm not going to get into now.
Why don't you get into it? I argue that it's an absolutely valid approach, particularly if one evaluates differences in the entire distribution and not just, say, an average.
On August 03 2011 14:37 whacks wrote: What if the imbalance has existed ever since game launch? In this case, there is no adjustment at all for Blizzard to track. In fact, any fixes that Blizzard puts in to fix the imbalance, will only show an upward shift for the previously-UP race & a downward-shift for the previously-OP race.
The point I'm trying to get across: If Blizzard's statistics don't work even in the ideal case where player skill & metagame is static and one race is very obviously UP, then of what possible value are they?
Well, we don't really know what they're doing exactly, so it's difficult to evaluate (and equally difficult to discount.)
However, I don't see why looking at the MMR distributions of race populations wouldn't show a static imbalance just as easily as one that were the result of a new adjustment to the game. If one race's MMR distribution is shifted lower on the MMR axis, has a thinner tail at the low or high end, or has a shape other than a normal distribution, that all points pretty clearly to some type of imbalance, without regard to how or when it was introduced to the game.
On August 04 2011 09:13 McFortran wrote: Regardless of the underlying math, these statistics appear to be flawed based on this (they make assumptions that are likely untrue).
Your argument is true if your sample size is one of each race. However, when you look at tens of thousands of players, why would you argue it's not a good assumption that each race attracts a similar (and normal) distribution of player talent?
On August 04 2011 09:36 dhe95 wrote: So blizzard is matching up based on both player win percentage and race win percentage? That should mean that the win percentage of all the races should also eventually equal 50% given an infinite number of games played, not because of any map/balance success but instead because of a bad matchmaking algorithm.
You've misunderstood. The video is discussing not how their matchmaking works, but how they evaluate whether the races are balanced against each other post-matchmaking.
The matchmaking system assigns a score that numerically predicts the likelihood of that player beating another prospective player. Then, it strives to find other players who are likely to have a 50/50 win percentage vs. the player who hit the button, without regard to race.
On August 04 2011 10:32 bamman1108 wrote: I like that part where they're satisfied with 5% differences in W/L when that percent is based off millions of matches. Even a 1% difference with that many matches means that one race very, very significantly favors the other. Wtf are they talking about when a 55% win rate for a specific race matchup is just "borderline?"
It may be significant in a statistical sense, but remember that the basic problem they're trying to solve is not achieving statistically-insignificant balance but balance that's sufficient for the game to remain entertaining. So, 55% being tolerable is a judgement about how much imbalance will take away from the entertainment value of the game, not a judgement about whether the game is the optimal measurement of the player.
That said, those numbers in their presentation have shifted around a bunch since then, and I think the fact that all races have had success in major tournaments is a pretty good sign that any imbalances that exist are not having a huge impact on the competitive scene.
On August 04 2011 09:13 McFortran wrote: Regardless of the underlying math, these statistics appear to be flawed based on this (they make assumptions that are likely untrue).
Your argument is true if your sample size is one of each race. However, when you look at tens of thousands of players, why would you argue it's not a good assumption that each race attracts a similar distribution of player talent?
If you look at Korea, it would appear that there are significantly more terran players at the highest level than other races. So I don't think the assumption is particularly good in theory or in practice.
The whole purpose of differential equations is to measure things that are constantly changing...
Here is the differential equation from the video:
If you can't make sense of that (I can't!) then I don't know why you're trying to criticize them. The percentages are just simplified representations to present the data to people who aren't math majors, you can't really use them to support random theories.
Here is the problem and appears will continue to be as far as SC2 goes balance wise in regards to ladder...I say in regards to ladder because no matter how blizzard balances the game sooner or later tournaments will make maps that will balance the game regardless of blizzard balance (see broodwar).
Anyway back on topic, the way they balance the ladder is not with the highest level in mind and probably never will be. Things they do or balance for that are counter intuitive to actual balance.
- Constantly add maps that are unique and fun....reality is they force you to play in X way assuming your race is capable of that.
- They balance for the lowest level as well as the highest level....balancing for the fact say zerg bronze players are terrible at X isn't a good way to balance a game even if it means bronze zergs lose a lot and Master / GM zergs don't.
- They balance based on raw statistics with 'player skill' factored in, but due to potentially bad maps by sheer random luck, poor positions, or possibly unknown balance factors...there is frankly no humanly possible way to use a formulaic approach to determine balance on the ladder.
Perfect example of the above...if say Zerg was OP, but only slightly, if zerg players were not truly better then say T or P players....the game would still appear balanced simply because the stats say so. Same goes for any factor such as maps making maybe P struggle but maybe the race is OP so despite the bad maps they win despite it...maps change and all the sudden P rolls everyone. Essentially unless the game was perfectly balanced already there would be no way to account for players actual skill in a formula.
On August 04 2011 11:20 McFortran wrote: If you look at Korea, it would appear that there are significantly more terran players at the highest level than other races. So I don't think the assumption is particularly good in theory or in practice.
The thing is, that simple fact doesn't say much. In the absence of any other information it could mean:
* That Terran is imbalanced for game design reasons for players at the top level of skill.
* That players at the top level of skill favor Terran because it plays in a way that rewards that skill for reasons other than a competitive advantage (for example a playstyle that makes them feel like they're making use of their skills.)
* That the skill distributions of all three races are identical but for some reason Korean players favor Terran, thus all levels of the game are more populated with Terrans.
Edit, sorry, I left an important one out:
* That the most skilled players in Korea have decided among themselves that Terran is favored by imbalances, when in fact it's well-balanced.
The video the OP linked explains that when the numbers suggest something, they go looking through other sources of evidence (pro player feedback, community feedback, their own play experiences, tournament replay analysis, results from testing tools) to try to distinguish between possible causes. Sounds to me like a very reasonable approach.
As for how they create their "skill-adjusted win percentages," it's hard to criticize without a specific understanding of what they're doing. However, I do know that the Battle.net 2.0 team employs at least a few people with a more rigorous statistical background than most of the posters on Team Liquid's site.
My point, of course, is to say that it's completely invalid to criticize any statistical analysis without understanding the details of what they're doing.
Blizzard just has an a priori for racial skill based on the normal distribution where they assume something about what player skill is supposed to be for each race (probably the same mean). Then if this distribution is off, something is wrong.
Of course that something is more likely to be correlation with player skill or other factors such as race potential, rather than raw racial strength. Furthermore, accounting for meta game this will shift through time as races become more well understood than they were before. Because of this, there is no way for Blizzard to know actual balance through this means.
On August 04 2011 11:22 seaofsaturn wrote: The whole purpose of differential equations is to measure things that are constantly changing...
Here is the differential equation from the video:
If you can't make sense of that (I can't!) then I don't know why you're trying to criticize them. The percentages are just simplified representations to present the data to people who aren't math majors, you can't really use them to support random theories.
When I saw the presentation in person, I presumed that that was a joke, because that equation looks a hell of a lot like an orbital wave function in 3D spherical space, from quantum mechanics. However, I'm no expert in multivariate statistics, and it's quite possible that the math looks similar.
Edit: It actually looks like the equation for calculating the probability of a particular quantum state from the wave function. For any real physicists or chemists keeping score.
Edit 2: I'd be very surprised if Blizzard's statisticians worked in spherical coordinates for just about anything.
On August 04 2011 11:34 NATO wrote: Blizzard just has an a priori for racial skill based on the normal distribution where they assume something about what player skill is supposed to be for each race (probably the same mean). Then if this distribution is off, something is wrong.
Of course that something is more likely to be correlation with player skill or other factors such as race potential, rather than raw racial strength. Furthermore, accounting for meta game this will shift through time as races become more well understood than they were before. Because of this, there is no way for Blizzard to know actual balance through this means.
What you say is true, but I think they know this, which is why they check their plans against player feedback and their own testing tools and David Kim's play experiences.
Blizz has people a lot smarter than us working on this all day every day... do any of you really think you can sit here knowing .000001% of what they're doing and criticize them like you know what your're talking about?
On August 04 2011 11:41 Neo.NEt wrote: Blizz has people a lot smarter than us working on this all day every day... do any of you really think you can sit here knowing .000001% of what they're doing and criticize them like you know what your're talking about?
There are certainly people in this forum with deep technical backgrounds. I think it's reasonable for such people to ask tough questions about what they're doing based on that presentation, and some of this thread is just that. That said, people with deep technical backgrounds often jump to conclusions especially when speaking casually, and that's represented in this thread as well.
On August 04 2011 11:22 seaofsaturn wrote: The whole purpose of differential equations is to measure things that are constantly changing...
Here is the differential equation from the video:
If you can't make sense of that (I can't!) then I don't know why you're trying to criticize them. The percentages are just simplified representations to present the data to people who aren't math majors, you can't really use them to support random theories.
When I saw the presentation in person, I presumed that that was a joke, because that equation looks a hell of a lot like an orbital wave function in 3D spherical space, from quantum mechanics. However, I'm no expert in multivariate statistics, and it's quite possible that the math looks similar.
Edit: It actually looks like the equation for calculating the probability of a particular quantum state from the wave function. For any real physicists or chemists keeping score.
Edit 2: I'd be very surprised if Blizzard's statisticians worked in spherical coordinates for just about anything.
Hmm... I have never considered Blizzard the sort of people that would make esoteric jokes at a press conference with how much they always try to be kind to noobs and such, but I guess it is possible...
On August 04 2011 11:46 seaofsaturn wrote: Hmm... I have never considered Blizzard the sort of people that would make esoteric jokes at a press conference with how much they always try to be kind of noobs and such, but I guess it is possible...
It wasn't at a press conference -- it was a presentation at Blizzcon, and yeah they absolutely make incredibly oblique and geeky jokes. (There was one a few years ago in a Diablo 3 presentation about a "Horadric Companion Cube" that stuck with me as particularly funny on a number of levels.)
Besides, it's not that esoteric a joke, if you imagine the speaker getting on Google with the goal of finding the most hilariously complicated-looking equation he could find.
On August 04 2011 11:20 McFortran wrote: If you look at Korea, it would appear that there are significantly more terran players at the highest level than other races. So I don't think the assumption is particularly good in theory or in practice.
The thing is, that simple fact doesn't say much. In the absence of any other information it could mean:
* That Terran is imbalanced for game design reasons for players at the top level of skill.
* That players at the top level of skill favor Terran because it plays in a way that rewards that skill for reasons other than a competitive advantage (for example a playstyle that makes them feel like they're making use of their skills.)
* That the skill distributions of all three races are identical but for some reason Korean players favor Terran, thus all levels of the game are more populated with Terrans.
Edit, sorry, I left an important one out:
* That the most skilled players in Korea have decided among themselves that Terran is favored by imbalances, when in fact it's well-balanced.
The video the OP linked explains that when the numbers suggest something, they go looking through other sources of evidence (pro player feedback, community feedback, their own play experiences, tournament replay analysis, results from testing tools) to try to distinguish between possible causes. Sounds to me like a very reasonable approach.
As for how they create their "skill-adjusted win percentages," it's hard to criticize without a specific understanding of what they're doing. However, I do know that the Battle.net 2.0 team employs at least a few people with a more rigorous statistical background than most of the posters on Team Liquid's site.
My point, of course, is to say that it's completely invalid to criticize any statistical analysis without understanding the details of what they're doing.
I agree. I guess the problem I have is that some people simply assume that these statistics are all-powerful because they don't understand the underlying math (it's just Bayesian inference). In reality they use a lot of methods to determine balance, none of them are particularly good by themselves. It's more of an art than a science in the end; it's not like Blizzard has the answer to everything.
On August 03 2011 14:37 puppykiller wrote: did anyone else watch the question and answer section of the video. That was the most tragic thing I have ever seen. Obviously when the majority of the ppl who play ur games have that mindset your going to be in a position where u cant make a well balanced game.
I went back to the video and watched that part... Ahahahahahahahahhahaaha.
Especially the second guy. I bet he's using IE 6 as his browser.
To everyone saying that you have blind faith in Blizzard: I respect that, but stay out of this thread! I honestly don't care that much about race balance. This entire thread is about math-buffs trying the understand how the stuff works. If you're just going to pop in here & say that we should blindly trust Blizzard, you aren't really contributing anything to the thread.
Maddog: I was giving an example where Zerg is OP compared to both Terran & Protoss. I agree that in the situation you described (where Zerg is OP vs T, UP vs P), the win-rates will look weird & raise flags. But for the case where Zerg is OP relative to both T & P, the Zerg player will indeed show up as 2100, and the win-rates won't flag anything weird. I think this is the most fatal flaw behind Blizzard's adjusted-win-rates.
Lysenko: Thanks for taking the time to post intelligent, detailed replies. I can tell we have a common way of thinking, even if we disagree.
The way you adjust for skill is to look at overall MMR distribution among each race's population. If one race, let's say Zerg, has a population distribution that's weighted toward lower MMRs, chances are it's the race that's doing it
I agree. Looking at each race's MMR distribution is a very valid tool for judging balance. If Blizzard came right out & said this is what they were doing, I might even support it. It's so much simpler & more straight forward than their "adjusted-win-rate" and requires almost no number crunching at all.
However, I don't think this is what Blizzard is doing. Why? Because their adjusted-win-ratios make the game seem so much more balanced than the MMR distribution. Check out the Zerg distribution in comparison to the Zerg player base. They're skewed so much more heavily towards Plat, Diamond & Master's, compared to the 27% player base.
That said, I don't think this is that great a tool for objectively determining balance. Why? Because some races might appeal a lot more to hardcore gamers while others appeal a lot more to casuals. Sounds hard to believe, right? But ask anyone who played a lot of battlegrounds in WoW-classic. Even though the 2 factions were 99% identical, the Horde would dominate the battlegrounds the vast majority of the time over Alliance. This is so true, it's not even a joke. A random Horde team would consistently walk over a random Alliance team, despite there being no balance issues. Why? Because the Horde faction simply appealed so much more to Hardcore gamers, compared to Alliance with their huge population of casuals. Looking at the whopping 40% of Bronze players being Terran, I suspect the same thing is happening in SC2 as well. Hence why looking at MMR distribution, though being handy, is not all that useful.
I'm actually planning on making a different thread regarding this topic, focusing on other statistics that could be used in determining balance. But for now, I wanted to have a discussion about whether the "adjusted-win-ratios" that Blizzard posts make any sense at all. And I'm starting to think the answer is no.
On August 04 2011 11:41 Neo.NEt wrote: Blizz has people a lot smarter than us working on this all day every day... do any of you really think you can sit here knowing .000001% of what they're doing and criticize them like you know what your're talking about?
There are certainly people in this forum with deep technical backgrounds. I think it's reasonable for such people to ask tough questions about what they're doing based on that presentation, and some of this thread is just that. That said, people with deep technical backgrounds often jump to conclusions especially when speaking casually, and that's represented in this thread as well.
Well, then those people should present their credentials.
On August 04 2011 11:22 seaofsaturn wrote: The whole purpose of differential equations is to measure things that are constantly changing...
Here is the differential equation from the video:
If you can't make sense of that (I can't!) then I don't know why you're trying to criticize them. The percentages are just simplified representations to present the data to people who aren't math majors, you can't really use them to support random theories.
When I saw the presentation in person, I presumed that that was a joke, because that equation looks a hell of a lot like an orbital wave function in 3D spherical space, from quantum mechanics. However, I'm no expert in multivariate statistics, and it's quite possible that the math looks similar.
Edit: It actually looks like the equation for calculating the probability of a particular quantum state from the wave function. For any real physicists or chemists keeping score.
Edit 2: I'd be very surprised if Blizzard's statisticians worked in spherical coordinates for just about anything.
Well, why would you be surprised? The wave equation for QM is mainly a probabilistic one, and really even outside of the realm of Physics spherical coordinates are not uncommon and are actually pretty useful.
Errrr, about it being a joke.. well since none of us is really working with Blizzard we kinda have to either believe in that they are using that model or not ...
And besides a problem such as abstract as balance and skill can´t really be tackled solely by statistics, which I have always found it weird why people focus so much on that. I mean, anyone who has worked in statistics can and will know that they can be terribly misleading. But at one point one has to land their work into real life. And Blizzard has acknowledged that just using ladder statistics by itself is meaningless, that is why supposedly they see tourneys and ask pro players|
On August 04 2011 10:32 bamman1108 wrote: I like that part where they're satisfied with 5% differences in W/L when that percent is based off millions of matches. Even a 1% difference with that many matches means that one race very, very significantly favors the other. Wtf are they talking about when a 55% win rate for a specific race matchup is just "borderline?"
Absolutely.
Before looking at win-rate statistics in chess (white vs. black), if you read up on the first-move advantage in chess you get the impression that the discrepancy between white and black pieces must be huge. You read quotes like this: "Teimour Radjabov was the youngest person to beat Gary Kasparov -- and he did so as black!" Yet the statistics lead to "only" a ~55% win-rate for white across high level matches.
There are so many factors that determine the outcome of a game of Starcraft -- skill, randomness (due to imperfect information), psychological factors, other sources of variability -- that whatever inherent imbalance exists will provide a relatively weak signal in win rates.
For example, no one would deny that lag gives you an inherent -- and unfair -- disadvantage. Yet if you have two NesTea clones duking out 1000 matches, where one NesTea is playing with lag and the other isn't, then the win-rate might only be around 55%. Again, this is because a milieu of other factors determine the outcome. Statistically, it's hard to tease apart that one factor in question (whether racial imbalance, or lag).
Also factor in that our brains suck at understanding statistics intuitively. For example, see the birthday problem or that infamous cancer problem that Bayesians are always talking about.
It's easy to get excited about "only" a 55/45% imbalance because those numbers "look" close. Don't trust your gut interpretation of raw statistics.
On August 04 2011 14:00 ero wrote: It's easy to get excited about "only" a 55/45% imbalance because those numbers "look" close. Don't trust your gut interpretation of raw statistics.
Except that from a game design perspective, "does it feel wrong" is a perfectly reasonable question to ask, and maybe even the overriding concern.
In any case, in chess you can make the case that who moves first is the SOLE difference between the two colors, and thus the source of that 5% deviation (or whatever) from even win percentages.
In Starcraft, a 5% difference is likely the result of a large number of different factors each contributing independently -- and correcting such a difference without introducing other issues elsewhere may be a nearly unsolvable problem.
What's "close enough" is always going to be a heuristic, there's no way around it. Also, I'd be very surprised if Blizzard's data has ever suggested a consistent 5% advantage for either side in any matchup.
(Note that that presentation was made three months after the game was released. It's now been about twelve months since the game was released. That's a lot of time for things to change.)
On August 04 2011 13:19 whacks wrote: I agree. Looking at each race's MMR distribution is a very valid tool for judging balance. If Blizzard came right out & said this is what they were doing, I might even support it. It's so much simpler & more straight forward than their "adjusted-win-rate" and requires almost no number crunching at all.
However, I don't think this is what Blizzard is doing. Why? Because their adjusted-win-ratios make the game seem so much more balanced than the MMR distribution. Check out the Zerg distribution in comparison to the Zerg player base. They're skewed so much more heavily towards Plat, Diamond & Master's, compared to the 27% player base.
This is the key problem, as I see it. There's no way (I can think of) to internalise the fact that the distribution of players in each race is not independent of the other factors.
Also, not sure how much to trust that sc2rank data; it show's that all races have a win % above 90 when sorted by region
With regard to that equation, it looks like maximum likelihood estimation or something similar. Almost definitely is a genuine statistical equation and not some out of place particle physics lol.
On August 03 2011 13:07 whacks wrote: Now going back to the scenario, consider the case where Blizzard releases a new patch which nerfs Zerg and makes it UP relative to both Protoss & Terran (eg, drones now cost 60 min). Consider what will happen to the average Zerg player. He will start losing more than 50% of his games, and his MMR will start dropping. Because of his lower MMR, he’ll start playing against weaker opponents. Eventually, his MMR will stabilize at a level where he starts winning 50% of his future games.
Now let’s say Blizzard had assigned each race a rating as well, to track how “strong they think it is.” Suppose that before the patch, all the races were balanced & had equal rating. Immediately after the patch, because the Zerg population goes through a losing streak, the Zerg rating will drop.
But eventually, the Zerg players will have stabilized their MMR and start winning 50% of their games. At this point, because of the last bullet point in the rating system’s principles (ratings will converge at 50% win rates), the Zerg rating will start increasing again. Remember also that the stabilized Zerg players are playing against opponents of the same MMR, so there’s no way to “account for player skill.” Eventually, the zerg rating will once again converge with the other races, even though Zerg is now UP.
No, Zerg players will switch their race, and the imbalance will show up eventually. Blizzard works with that assumption for race adaptability by the players. It's imperfect, but probably optimal for now. In general, there's no problem with ELO over multiple variables. I think I've posted before, for example, that TL's ELO system needs to include a general race and map balance coefficient too at any given moment. I'm glad Blizzard thinks about these details. Thanks a lot for the video!
I've read through the thread, watched the whole video, etc. but I want to highlight some points that have been highlighted already, just to see if some straighter, simpler discussion could come of it.
- A system that ensures a 50% win rating not only in general, but race to race will hide imbalance by virtue of actively seeking that 50% regardless of skill level. This means two players of identical skill with two different races will both be at 50%, but will have very different MMRs if their respective races are imbalanced against one another.
- The law of large numbers means that in such a system, EVERY matchup should be EXACTLY 50%. ANY margin of error lends itself to either race, map or system imbalance. Remember, as the sample size becomes larger, the acceptable margin of error becomes smaller., and to call 2-5% acceptable is silly.
- The point has been made that because of the flaws in the ladder system, we should probably look to tournament results for balance results. Given a large enough sample, and randomly positioned opponents, the percentage of race wins will give an actual measure of balance.
The last point might not be well accepted, but it goes back to the law of large numbers. If you have enough tournament matches, every player will face every other player multiple times, essentially leveling out the skill factor. LZGamer, for example, may balance out Nestea (Losses to Wins), leading to an even 50% Win/Loss across the races. Because matchups are chosen (relatively) randomly, this non-fixed system can actually give us a decent look at balanced percentages of wins.
All of that said, I think it can be fairly well agreed the ladder, with it's system of insuring 50% wins regardless of imbalance vs skill should be abandoned as a method of seeking balance (especially if after millions of games across all races and the given matchmaking system in place, the rates STILL don't balance to 50%).
I've read through the thread, watched the whole video, etc. but I want to highlight some points that have been highlighted already, just to see if some straighter, simpler discussion could come of it.
- A system that ensures a 50% win rating not only in general, but race to race will hide imbalance by virtue of actively seeking that 50% regardless of skill level. This means two players of identical skill with two different races will both be at 50%, but will have very different MMRs if their respective races are imbalanced against one another.
- The law of large numbers means that in such a system, EVERY matchup should be EXACTLY 50%. ANY margin of error lends itself to either race, map or system imbalance. Remember, as the sample size becomes larger, the acceptable margin of error becomes smaller., and to call 2-5% acceptable is silly.
- The point has been made that because of the flaws in the ladder system, we should probably look to tournament results for balance results. Given a large enough sample, and randomly positioned opponents, the percentage of race wins will give an actual measure of balance.
The last point might not be well accepted, but it goes back to the law of large numbers. If you have enough tournament matches, every player will face every other player multiple times, essentially leveling out the skill factor. LZGamer, for example, may balance out Nestea (Losses to Wins), leading to an even 50% Win/Loss across the races. Because matchups are chosen (relatively) randomly, this non-fixed system can actually give us a decent look at balanced percentages of wins.
All of that said, I think it can be fairly well agreed the ladder, with it's system of insuring 50% wins regardless of imbalance vs skill should be abandoned as a method of seeking balance (especially if after millions of games across all races and the given matchmaking system in place, the rates STILL don't balance to 50%).
Holy hell, this post above me is spectacular. I was going to spend at least 20 minutes trying to write up my analysis of it, but instead will only pile on to these excellent points.
~Ladder stats are essentially meaningless. Blizzard's correction may undo the effect of the matchmaking system gradually, but cannot fix the fact that the matchmaking system generated the data in the first place. You cannot analyze the flaw out of a flawed data set. ~~A potential fix would be to turn off the MMR-based matchmaking system for a period of time, and match players on a simple statistic (maybe points, no bonus pool). The resulting data would be strange, but untainted.
~For the design team, balance changes meaning from 'broken' to 'unfun' once the game is developed. As Browder mentioned in the video, strategies become more important than the raw units once the game matures, and designers switch focus ('T wins all games <12 minutes, loses all longer than' would be a balanced, but unfun scenario).
Thus, to Blizzard, balance really is a subjective thing. The stats they throw up will always be "close enough" in their estimation to call sc2 balanced, while player and developer gameplay opinion will drive changes.
greatest post of all time concerning balance in sc2. I knew this 100% a while ago but couldnt put it into words as well as you. I explained this to my friends but if they dont want to believe then they wont. basically nomatter what the balance battle.net statistics will always show close to 50% because the goal and intention of bnet is to make everyone at 50%.
On August 04 2011 11:41 Neo.NEt wrote: Blizz has people a lot smarter than us working on this all day every day... do any of you really think you can sit here knowing .000001% of what they're doing and criticize them like you know what your're talking about?
There are certainly people in this forum with deep technical backgrounds. I think it's reasonable for such people to ask tough questions about what they're doing based on that presentation, and some of this thread is just that. That said, people with deep technical backgrounds often jump to conclusions especially when speaking casually, and that's represented in this thread as well.
Well, then those people should present their credentials.
Or, you can just use your brain and see if their points make sense.
I dislike this "well they probably know better than us" approach, it leads people to follow them blindly. Their answer during this conference was half joke, half nonsense, yet people seem to believe they have a way to calculate win percentages that completely negates the MMR effect and are a great indicator of balance.
That's just not the case, and tournament results should be taken more seriously than the numbers they give us. Of course since you then have a more limited number of players, you can then have a Nestea/MC Protoss/Korean Terran effect that you have to take into consideration. Ultimately, you have to analyse the game because statistical analysis will only get you so far.
Disclaimer: quantitative finance major (that does include a lot of stats) who actually thinks Blizzard does a very good job with balance, I don't have a problem with that. But I do think the numbers they give us is a way to tell us "shut up and play" and have little to no value.
As long as the results are linearly independent (where basically one equation isn't a multiple of another), you can get answers for x and y. If you then throw in another equation where the answer creates a "contradiction":
On August 03 2011 13:07 whacks wrote: Disclaimer: I’m not concerned about game balance at all. I’m hoping to have a discussion on the math & statistics behind Blizzard's adjusted-win-percentage that they rely on heavily.
Late last year, Blizzard released a bunch of ladder statistics on “skill-adjusted-win-percentages” for the different matchups. The reason I have it in quotes, is because they never really explained how they did the skill-adjustments. I’ve always been skeptical about whether such a “skill-adjustment” is really possible.
Gist of what they said: Raw league matchup numbers aren’t very meaningful because of matchmaking’s system ability to matchup players with equally challenging opponents. The math guy mentions specifically: Not only does the system put players in 50-50 matches, it also tries to keep the race matchups at 50-50 as well. Because of this, we have to adjust for player skill to calculate the true matchup win rates. Example: a ZvP match is about to be played. The Zerg player’s rating (odds of winning) relative to the Protoss player is 55-45. The Zerg race’s rating relative to the Protoss race is 53-47. If the Protoss player ends up winning, the player ratings will then converge to 51-49. The race ratings will also converge to 52-48.
Their explanation just didn’t click with me. Rating systems such as ELO are great when you’re dealing with a single unknown (relative player strength). But can they really work if you’re trying to differentiate between 2 unknowns? Both relative player skill & race balance? I constructed the following scenario which seems to suggest that this is impossible.
It’s important to first establish the following: Any good rating system, including ELO & the point system, relies on the following principle: • Give each agent (could be a player, or a race) a certain rating as an estimate for how strong the agent is • If 2 agents play and one wins at a higher percentage, the more successful agent should eventually end up with a higher rating • If a higher rated agent & a lower rated agent play against each other, and each wins with an equal percentage, the 2 ratings should eventually converge
The ELO system that Blizzard uses for MMR is an optimized algorithm that allows ratings to stabilize much quicker, but other rating systems that utilize the above principle (including the point-system), can achieve the same results in the long run.
Now going back to the scenario, consider the case where Blizzard releases a new patch which nerfs Zerg and makes it UP relative to both Protoss & Terran (eg, drones now cost 60 min). Consider what will happen to the average Zerg player. He will start losing more than 50% of his games, and his MMR will start dropping. Because of his lower MMR, he’ll start playing against weaker opponents. Eventually, his MMR will stabilize at a level where he starts winning 50% of his future games.
Now let’s say Blizzard had assigned each race a rating as well, to track how “strong they think it is.” Suppose that before the patch, all the races were balanced & had equal rating. Immediately after the patch, because the Zerg population goes through a losing streak, the Zerg rating will drop.
But eventually, the Zerg players will have stabilized their MMR and start winning 50% of their games. At this point, because of the last bullet point in the rating system’s principles (ratings will converge at 50% win rates), the Zerg rating will start increasing again. Remember also that the stabilized Zerg players are playing against opponents of the same MMR, so there’s no way to “account for player skill.” Eventually, the zerg rating will once again converge with the other races, even though Zerg is now UP.
Based on this scenario, it seems impossible to determine whether a race is truly UP, using Blizzard’s rating system. Thoughts? Any ideas on how Blizzard could possibly be “accounting for player skill” in calculating race balance?
Indeed it is quite hard to judge balance through Blizzard's system. However, when a player starts winning so much that there isn't an opponent fit for him the system can't do anything but match that player with other people that are lower in skill. Now if you were to use this as a way to judge which race is better/easier/overpowered/whatever you want to call it:
The race with the most players excelling in terms of skill should be the easiest race (not neccessarily, but a good assumption if seen on most realms)
This shows the top people in the world in terms of ladder points in descending order
Now I don't have time to do a huge analysis, so let's just take the top 25 from that list and break it down to how many of each race we find (server = Americas):
Zerg: 4 Protoss: 6 Random: 1 Terran: 14
It's clear that there are more terran players dominating the scene (i.e. terran players that are winning more often than losing, therefore consistently facing opponents worse than them)
My take on this... The way Blizzard has defined their balance equation leads to circular logic.
Chess is a good analogy to how badly Blizzard messed up. White has an obvious advantage and it is OP'ed...so say you had players who played exclusively either white or black. Then say Blizzard comes along to determine whether white is OP'ed. They say, we'll going to compare the win/loss records of white players of equal skill level who have a 50/50 win percentage against black players who have a 50/50 win percentage against white players.
So Blizzard fixes their match making system so white players are winning 50/50 against black players of 'equal skill level'. If a white player just keeps on winning...? Well then he was 'just that good' so we won't count him in the stats because we only like matches where the players are equal. That blizzard could run a fancy formula and come to this conclusion that players of equal skill had a 50/50 win % against the opposite color, thus determines that the colors are even is preposterously circular. They are using the adjusted win percentage to determine skill and also to determine matchups of even skill and you can't double dip like that mathematically.
The one plus of the interview was Dustin referring to early/late game balance. THIS is what blizzard should be focused on instead of the pseudo mathematical formulas. If in PvZ, protoss say wins 70% of the time when they rush zerg under 10 minutes, but lose 30% of the time after 10 minutes when zerg macro kicks in, then this is an example of a serious 'fun balance' issues that is far more important than an win/loss balance issue. The game first and foremost has to be fun and interesting and it it just degenerates into 'construct the perfect timing attack' or die because your race can't macro that well, then this makes all the minutes outside of that timing attack very boring. Blizzard messed up with larvae inject...it has made zerg too boom or bust unlike SCBW. Things like micro/positioning/skirmishes just don't happen that often because everything revolves around stupid timing attacks and whether they succeed or not because zerg is too exponential and all or nothing. Would love to see larvae inject nerfed and the zerg units buffed to compensate to make their growth more linear and thus making the other races easier to balance as well.
It may be instructive to consider how, in chess, white pieces tend to have an advantage over black pieces. Among weaker players, there's not much advantage. At higher levels, it starts to become a factor. Among top rated computers, white scores about 55%. However, in chess, players take turns playing white and black. It seems to me that the best way to do it for Starcraft would be to look at top-rated random race players. If there is a tendency for them to win with certain races, then I believe this would reveal something meaningful. To put it briefly, the best random race players' should be immune to MMR's affect on their races winnability. An obvious problem is that the random matchups are not totally analogous to the standard matchups since the non-random player doesn't know his opponent's race off hand.
I think the formula is too complicated. Im in gold but i pwn plats and diamonds(the occasional masters) like nothing. One time a diamond literally moved his army into mine w/o attacking then proceeded to BM me. No one is in leagues they belong, and its f*cking impossible to rank up...
On August 05 2011 11:16 BushidoSnipr wrote: I think the formula is too complicated. Im in gold but i pwn plats and diamonds(the occasional masters) like nothing. One time a diamond literally moved his army into mine w/o attacking then proceeded to BM me. No one is in leagues they belong, and its f*cking impossible to rank up...
I don't think this a thread to whine about "why I am not in the league I deserve". There are too many out there. There are a lot cheesy players in the dia and plat. Winning against them means nothing.
I feel that the current map pool in the ladder does not give us a good balance view. It may make Zerg look underpowered compared to Terran. Blizzard should learn to solve one problem at a time. They have huge complains about the current map pool, and yet they refuse to budge. If they continue to give us such maps, we will never obtain a statisically accurate result.
On August 05 2011 11:16 BushidoSnipr wrote: I think the formula is too complicated. Im in gold but i pwn plats and diamonds(the occasional masters) like nothing. One time a diamond literally moved his army into mine w/o attacking then proceeded to BM me. No one is in leagues they belong, and its f*cking impossible to rank up...
I don't think this a thread to whine about "why I am not in the league I deserve". There are too many out there. There are a lot cheesy players in the dia and plat. Winning against them means nothing.
I feel that the current map pool in the ladder does not give us a good balance view. It may make Zerg look underpowered compared to Terran. Blizzard should learn to solve one problem at a time. They have huge complains about the current map pool, and yet they refuse to budge. If they continue to give us such maps, we will never obtain a statisically accurate result.
the problem is blizzard is ok with their map pool being imbalanced. a quote i used in a blog
Our goal for these two formats is for players to be able to enjoy variety in the gameplay, rather than trying to provide an eSports level of game balance.
just shows that at the end of the day balance is a relative term.
blizzard are ok with their map pools being bad, and tourneys using different maps for competitive play. at the same time blizzard say they use 'ladder data' as a point of reference for balance decisions. they cant have it both ways
On August 05 2011 04:09 Ihpares wrote: - A system that ensures a 50% win rating not only in general, but race to race will hide imbalance by virtue of actively seeking that 50% regardless of skill level. This means two players of identical skill with two different races will both be at 50%, but will have very different MMRs if their respective races are imbalanced against one another.
But the distributions will be different over a large number of players in the imbalanced case. If it's harder to win as Zerg, Zerg players will have systematically lower MMRs, and that'll be visible in the distributions, unless you can make a convincing case that worse players choose Zerg.
- The law of large numbers means that in such a system, EVERY matchup should be EXACTLY 50%. ANY margin of error lends itself to either race, map or system imbalance. Remember, as the sample size becomes larger, the acceptable margin of error becomes smaller., and to call 2-5% acceptable is silly.
I said it before: If you're looking for a quantitative measure of player skill, you're right, but that's not what they're trying to do. They're trying to get close enough to 50% that the game feels fun while you're playing it. That's an enormously looser constraint.
On August 05 2011 07:05 Wren wrote: ~Ladder stats are essentially meaningless. Blizzard's correction may undo the effect of the matchmaking system gradually, but cannot fix the fact that the matchmaking system generated the data in the first place. You cannot analyze the flaw out of a flawed data set.
The problem is that everyone who's been arguing that the data set is "flawed" somehow have been saying so without any reasoning or explanation behind it, other than to completely misunderstand or misrepresent the impact of the matchmaking system on the data set.
Nobody in this thread knows what their data set is or exactly how they're analyzing it, so all the criticism of it is fantasy based on imagined details to fill in the blanks.
On August 05 2011 10:05 Fungal Growth wrote: So Blizzard fixes their match making system so white players are winning 50/50 against black players of 'equal skill level'. If a white player just keeps on winning...? Well then he was 'just that good' so we won't count him in the stats because we only like matches where the players are equal. That blizzard could run a fancy formula and come to this conclusion that players of equal skill had a 50/50 win % against the opposite color, thus determines that the colors are even is preposterously circular. They are using the adjusted win percentage to determine skill and also to determine matchups of even skill and you can't double dip like that mathematically.
If you assume that the distributions of actual player skill for each race are roughly similar, you can derive a mapping from one race's MMRs to another based on the MMR distributions of each race. Doing this requires the matchmaking system they're using. It depends on it completely.
Whether that's a reasonable assumption is something you'd have to look elsewhere to decide, but they do listen to anecdotal evidence from pros and the community, which is what you'd have to do to decide that. So, what's the problem?
On August 05 2011 11:16 BushidoSnipr wrote: I think the formula is too complicated. Im in gold but i pwn plats and diamonds(the occasional masters) like nothing. One time a diamond literally moved his army into mine w/o attacking then proceeded to BM me. No one is in leagues they belong, and its f*cking impossible to rank up...
Indirectly you bring up another point. Blizzard's bonus points reward you for playing...so weak players that play more out rank strong player's that don't play as much. So when the 'matchmaker' as David Kim called it tries to match players of equal ability, how can it ignore points gained from bonus wins? Yet another reason why Blizzard's fancy balance equations can't be trusted.
On August 05 2011 11:16 BushidoSnipr wrote: I think the formula is too complicated. Im in gold but i pwn plats and diamonds(the occasional masters) like nothing. One time a diamond literally moved his army into mine w/o attacking then proceeded to BM me. No one is in leagues they belong, and its f*cking impossible to rank up...
Indirectly you bring up another point. Blizzard's bonus points reward you for playing...so weak players that play more out rank strong player's that don't play as much. So when the 'matchmaker' as David Kim called it tries to match players of equal ability, how can it ignore points gained from bonus wins? Yet another reason why Blizzard's fancy balance equations can't be trusted.
On August 05 2011 11:12 carwashguy wrote: At higher levels, it starts to become a factor. Among top rated computers, white scores about 55%. .
Not to detract the thread too much, but that seems kind of low for white. In the last World chess championships of Anand/Topalov black only won once out of 12 matches...just goes to show the better the skill level...the better able the player (at any game) can magnify his advantage.
On August 05 2011 12:19 KevinIX wrote: Bonus points have no effect on MMR.
Curious...so when the matchmaker determines players of equal skill level, it uses a completely separate point system than the public one (which is largely comprised of bonus points)? If that were the case this shadow rank and the public rank could have quite the disparity. If this shadow rank was so wonderful in determining the skill level of an opponent then why wouldn't Blizzard use it instead of the public ranking system?
This just crosses my mind. After looking at the statistical equation, nature of data set and what Ihpares said, I feel like that the balance and players' skill follow the Uncertainty Principle .
It feels like you cannot really measure two quantities (player skills [MMR] and racial balance [win/lose ration of players]) at the same time so what you can do is to estimate the data as close to the actual data as possible. Like we only can calculate probability density of electron(s), we might only be able to calculate probability of the data of racial balance by using skill adjusted system (similar to using a basic set).
Or maybe I just review too much Physical Chemistry
On August 05 2011 12:19 KevinIX wrote: Bonus points have no effect on MMR.
Curious...so when the matchmaker determines players of equal skill level, it uses a completely separate point system than the public one (which is largely comprised of bonus points)? If that were the case this shadow rank and the public rank could have quite the disparity. If this shadow rank was so wonderful in determining the skill level of an opponent then why wouldn't Blizzard use it instead of the public ranking system?
Because the MMR does not really increase over time when a player reach certain point. So it does not give a positive reinforcement (not showing progress), if we would speak psychologically, to players and would lead to many players stop playing the ladder. However, because the ladder point always increase and can change abruptly, it give the sense of progression.
My question is why do they think statistics from lower leagues matter as much as higher ones? To but it bluntly, everyone is so bad in bronze that race imbalances aren't what's going to decide the winner, plain and simple (contrary to plat and above, where people don't get supply blocked at 11).
On August 05 2011 12:22 Fungal Growth wrote:Not to detract the thread too much, but that seems kind of low for white. In the last World chess championships of Anand/Topalov black only won once out of 12 matches...just goes to show the better the skill level...the better able the player (at any game) can magnify his advantage.
You have to consider what is meant by "score." You have to add half of the draw percentage to white's win percent (and the other half to black's). White won four, black won one, and the other seven were drawn. That means white won 33.33%, 58.33% were drawn, and black won 8.33%. That means white scored 62.5% ((33.33+(58.33/2)). In any case, it's best to take the computer engines' score of 55%, since they're way better than grandmasters.
On August 03 2011 13:07 whacks wrote: Disclaimer: I’m not concerned about game balance at all. I’m hoping to have a discussion on the math & statistics behind Blizzard's adjusted-win-percentage that they rely on heavily.
Late last year, Blizzard released a bunch of ladder statistics on “skill-adjusted-win-percentages” for the different matchups. The reason I have it in quotes, is because they never really explained how they did the skill-adjustments. I’ve always been skeptical about whether such a “skill-adjustment” is really possible.
Gist of what they said: Raw league matchup numbers aren’t very meaningful because of matchmaking’s system ability to matchup players with equally challenging opponents. The math guy mentions specifically: Not only does the system put players in 50-50 matches, it also tries to keep the race matchups at 50-50 as well. Because of this, we have to adjust for player skill to calculate the true matchup win rates. Example: a ZvP match is about to be played. The Zerg player’s rating (odds of winning) relative to the Protoss player is 55-45. The Zerg race’s rating relative to the Protoss race is 53-47. If the Protoss player ends up winning, the player ratings will then converge to 51-49. The race ratings will also converge to 52-48.
Their explanation just didn’t click with me. Rating systems such as ELO are great when you’re dealing with a single unknown (relative player strength). But can they really work if you’re trying to differentiate between 2 unknowns? Both relative player skill & race balance? I constructed the following scenario which seems to suggest that this is impossible.
It’s important to first establish the following: Any good rating system, including ELO & the point system, relies on the following principle: • Give each agent (could be a player, or a race) a certain rating as an estimate for how strong the agent is • If 2 agents play and one wins at a higher percentage, the more successful agent should eventually end up with a higher rating • If a higher rated agent & a lower rated agent play against each other, and each wins with an equal percentage, the 2 ratings should eventually converge
The ELO system that Blizzard uses for MMR is an optimized algorithm that allows ratings to stabilize much quicker, but other rating systems that utilize the above principle (including the point-system), can achieve the same results in the long run.
Now going back to the scenario, consider the case where Blizzard releases a new patch which nerfs Zerg and makes it UP relative to both Protoss & Terran (eg, drones now cost 60 min). Consider what will happen to the average Zerg player. He will start losing more than 50% of his games, and his MMR will start dropping. Because of his lower MMR, he’ll start playing against weaker opponents. Eventually, his MMR will stabilize at a level where he starts winning 50% of his future games.
Now let’s say Blizzard had assigned each race a rating as well, to track how “strong they think it is.” Suppose that before the patch, all the races were balanced & had equal rating. Immediately after the patch, because the Zerg population goes through a losing streak, the Zerg rating will drop.
But eventually, the Zerg players will have stabilized their MMR and start winning 50% of their games. At this point, because of the last bullet point in the rating system’s principles (ratings will converge at 50% win rates), the Zerg rating will start increasing again. Remember also that the stabilized Zerg players are playing against opponents of the same MMR, so there’s no way to “account for player skill.” Eventually, the zerg rating will once again converge with the other races, even though Zerg is now UP.
Based on this scenario, it seems impossible to determine whether a race is truly UP, using Blizzard’s rating system. Thoughts? Any ideas on how Blizzard could possibly be “accounting for player skill” in calculating race balance?
Indeed it is quite hard to judge balance through Blizzard's system. However, when a player starts winning so much that there isn't an opponent fit for him the system can't do anything but match that player with other people that are lower in skill. Now if you were to use this as a way to judge which race is better/easier/overpowered/whatever you want to call it:
The race with the most players excelling in terms of skill should be the easiest race (not neccessarily, but a good assumption if seen on most realms)
This shows the top people in the world in terms of ladder points in descending order
Now I don't have time to do a huge analysis, so let's just take the top 25 from that list and break it down to how many of each race we find (server = Americas):
Zerg: 4 Protoss: 6 Random: 1 Terran: 14
It's clear that there are more terran players dominating the scene (i.e. terran players that are winning more often than losing, therefore consistently facing opponents worse than them)
Now if you look at this data and also look at winning percentage of these top players:
Highest of protoss: 79% Highest of zerg: 75% Highest of terran: 90%
All in all (through this small amount of data, obviously more would be more accurate) it seems:
-Zerg is hardest to reach high level of play with -Terran is most abundant, and also highest winning ratio
In my opinion, this (with more data) is a very good way to show imbalance within the game.
Bias aside, feedback?
Basing any conclusions on statistical data without any sort of null hypothesis testing is horrible. There will always be patterns in data, even if it is entirely random, so you cannot draw conclusions this way.
First, there seem to be a lot of people in this thread who are emotionally invested in saying that Blizzard doesn't know what they're doing despite not having access to what they are doing. Maybe this is because they nerfed your favorite unit. Maybe this is because you don't like their map designs. Maybe it's because Dustin Browder (who isn't deeply involved in game balance) made some comment about the metagame that you think sounded dumb. In any case, having that kind of emotional investment in arguing they're wrong regardless of what they're doing just isn't rational.
Second: While we know from their statements that various in-game statistics and build order information aren't used by the matchmaking system directly, it's quite possible that such information might be a way to adjust for player skill, at least in gross ways. Without a statement from them on whether they're using that data in that way, it's impossible to tell. If they were, of course, it could open up all kinds of arguments about whether their skill-adjusted data were valid, but it's possible that it could be used in some ways that might be insightful.
On August 05 2011 12:13 Fungal Growth wrote: So when the 'matchmaker' as David Kim called it tries to match players of equal ability, how can it ignore points gained from bonus wins?
On August 05 2011 12:19 KevinIX wrote: Bonus points have no effect on MMR.
Curious...so when the matchmaker determines players of equal skill level, it uses a completely separate point system than the public one (which is largely comprised of bonus points)? If that were the case this shadow rank and the public rank could have quite the disparity. If this shadow rank was so wonderful in determining the skill level of an opponent then why wouldn't Blizzard use it instead of the public ranking system?
Forgive me for saying that you have no business posting in this discussion at all if you have not read and understood this post, which, incidentally, is a sticky in this forum:
That post's information is not speculation. Excalibur_Z has direct contact with the developers on these matters. They don't share all the details but they do confirm what he's written there. (That post, btw, is also a sticky on the Battle.net forums, which attests to its accuracy.)
On August 04 2011 11:22 seaofsaturn wrote: The whole purpose of differential equations is to measure things that are constantly changing...
Here is the differential equation from the video:
If you can't make sense of that (I can't!) then I don't know why you're trying to criticize them. The percentages are just simplified representations to present the data to people who aren't math majors, you can't really use them to support random theories.
It's funny they put up some supposedly insane math equation (it's not, you need one year of calculus to understand it), but they don't tell you what anything represents; theta, beta, gamma? Equations are meaningless if that information is omitted. Lol at Blizzard trying to appear transparent, patronization at its best, imo.
"We'll spare you the details, but these are the percentages", sketchy.
On August 05 2011 13:51 kckkryptonite wrote: It's funny they put up some supposedly insane math equation (it's not, you need one year of calculus to understand it), but they don't tell you what anything represents; theta, beta, gamma? Equations are meaningless if that information is omitted.
It takes a year of calculus to read it, understanding it either requires some statistical knowledge, or they were joking.
In any case, it clearly wasn't presented to convey information, and if it is, as someone else above speculated, something related to a maximum likelihood calculation involved in the matchmaking system, it's quite likely that none of the people speaking on that stage were fully up to speed on the statistics involved.
On August 05 2011 13:51 kckkryptonite wrote: It's funny they put up some supposedly insane math equation (it's not, you need one year of calculus to understand it), but they don't tell you what anything represents; theta, beta, gamma? Equations are meaningless if that information is omitted.
It takes a year of calculus to read it, understanding it either requires some statistical knowledge, or they were joking.
In any case, it clearly wasn't presented to convey information, and if it is, as someone else above speculated, something related to a maximum likelihood calculation involved in the matchmaking system, it's quite likely that none of the people speaking on that stage were fully up to speed on the statistics involved.
If you can read it, you can at the very least interpret it. It's like you're saying you might be able to read Korean, but not understand it. Not divulging more than some arbitrary equation clearly tells us that Blizzard doesn't want to tell us anything other than imply they have a big magical formula done by 5 PhDs.
On August 04 2011 11:22 seaofsaturn wrote: The whole purpose of differential equations is to measure things that are constantly changing...
Here is the differential equation from the video:
If you can't make sense of that (I can't!) then I don't know why you're trying to criticize them. The percentages are just simplified representations to present the data to people who aren't math majors, you can't really use them to support random theories.
It's funny they put up some supposedly insane math equation (it's not, you need one year of calculus to understand it), but they don't tell you what anything represents; theta, beta, gamma? Equations are meaningless if that information is omitted. Lol at Blizzard trying to appear transparent, patronization at its best, imo.
"We'll spare you the details, but these are the percentages", sketchy.
you need more than one year of calculus to understand this type of equation. Cal I/II don't even sniff DEs. All the posts in this thread are total crap except those which point out that <1% of the population comprehends the math involved. The details aren't important, because it works. I am amazed at the number of numbskulls who piss and moan about this or that policy from blizzard without doing any anything to contribute to the solution side of things.
Math is extremely hard, the kind of programming talent Blizzard can hire, while not Google/MSFT/AAPL level, is incredibly high. Just be glad that the smartest guys in the gaming industry are working on the IP you love.
On August 04 2011 11:22 seaofsaturn wrote: The whole purpose of differential equations is to measure things that are constantly changing...
Here is the differential equation from the video:
If you can't make sense of that (I can't!) then I don't know why you're trying to criticize them. The percentages are just simplified representations to present the data to people who aren't math majors, you can't really use them to support random theories.
It's funny they put up some supposedly insane math equation (it's not, you need one year of calculus to understand it), but they don't tell you what anything represents; theta, beta, gamma? Equations are meaningless if that information is omitted. Lol at Blizzard trying to appear transparent, patronization at its best, imo.
"We'll spare you the details, but these are the percentages", sketchy.
Greek letters have defined meanings in statistical analysis, they are not unpredefined variables like for example the common latin/roman letter "x".
On August 04 2011 11:22 seaofsaturn wrote: The whole purpose of differential equations is to measure things that are constantly changing...
Here is the differential equation from the video:
If you can't make sense of that (I can't!) then I don't know why you're trying to criticize them. The percentages are just simplified representations to present the data to people who aren't math majors, you can't really use them to support random theories.
It's funny they put up some supposedly insane math equation (it's not, you need one year of calculus to understand it), but they don't tell you what anything represents; theta, beta, gamma? Equations are meaningless if that information is omitted. Lol at Blizzard trying to appear transparent, patronization at its best, imo.
"We'll spare you the details, but these are the percentages", sketchy.
you need more than one year of calculus to understand this type of equation. Cal I/II don't even sniff DEs. All the posts in this thread are total crap except those which point out that <1% of the population comprehends the math involved. The details aren't important, because it works. I am amazed at the number of numbskulls who piss and moan about this or that policy from blizzard without doing any anything to contribute to the solution side of things.
Math is extremely hard, the kind of programming talent Blizzard can hire, while not Google/MSFT/AAPL level, is incredibly high. Just be glad that the smartest guys in the gaming industry are working on the IP you love.
I agree he is wrong. even without having the variables defined you can still tell what the thing does if you can understand it. Everyone can recognize the formula for a parabola without knowing how x or y is applied.
On August 05 2011 14:34 Disquiet wrote: I agree he is wrong. even without having the variables defined you can still tell what the thing does if you can understand it. Everyone can recognize the formula for a parabola without knowing how x or y is applied.
Actually, when it comes to statistics, that's not true. Here's why: every technique that exists for analyzing data sets statistically has implicit assumptions that must be true before the technique tells you what it purports to.
Here's an example: The mean and standard deviation have different meanings if you're looking at a normal vs. a Poisson distribution. When is it appropriate to presume each type of distribution? The answer isn't found in the mathematical equations that describe the distributions' shapes!
Rewatched the vid: The powerful terran early game is still a problem which DB recognizes. Yet it is not something easily fixable without nerfing mid/late game. By design the race is like that and I think we will have to wait til HotS for the problem to truly be "fixed".
On August 04 2011 11:22 seaofsaturn wrote: The whole purpose of differential equations is to measure things that are constantly changing...
Here is the differential equation from the video:
If you can't make sense of that (I can't!) then I don't know why you're trying to criticize them. The percentages are just simplified representations to present the data to people who aren't math majors, you can't really use them to support random theories.
It's funny they put up some supposedly insane math equation (it's not, you need one year of calculus to understand it), but they don't tell you what anything represents; theta, beta, gamma? Equations are meaningless if that information is omitted. Lol at Blizzard trying to appear transparent, patronization at its best, imo.
"We'll spare you the details, but these are the percentages", sketchy.
you need more than one year of calculus to understand this type of equation. Cal I/II don't even sniff DEs. All the posts in this thread are total crap except those which point out that <1% of the population comprehends the math involved. The details aren't important, because it works. I am amazed at the number of numbskulls who piss and moan about this or that policy from blizzard without doing any anything to contribute to the solution side of things.
Math is extremely hard, the kind of programming talent Blizzard can hire, while not Google/MSFT/AAPL level, is incredibly high. Just be glad that the smartest guys in the gaming industry are working on the IP you love.
No, you don't. The level of ignorance in your post is astonishing, after the first year of Calc, you are be able to solve basic differential equations (in my curriculum at least). Pouring money into something must mean it's the best right (US HEALTHCARE/EDUCATION)? To top it off you cite wikipedia.
On August 05 2011 14:15 Alyoshka wrote: All the posts in this thread are total crap except those which point out that <1% of the population comprehends the math involved.
Really? WTF. There are derivatives and integrals, fractions and exponents. Seriously? 1%? Where did you get this number? How are you coming to your various conclusions?
On August 05 2011 14:34 Disquiet wrote: I agree he is wrong. even without having the variables defined you can still tell what the thing does if you can understand it. Everyone can recognize the formula for a parabola without knowing how x or y is applied.
Actually, the variables are assumed to be defined as the set of all real numbers.
W/e guys, I'm not gonna get into a math debate with people who know how to use google and wikipedia.
it seems there main concern is making a player satisfied and happy by giving them the 50% win ratio, but aren't all that concerned with how it misconstrues balance data, so their risking some more reliable numbers for a better experience for the consumer, probably a good move by them
On August 05 2011 15:03 Slago wrote: it seems there main concern is making a player satisfied and happy by giving them the 50% win ratio, but aren't all that concerned with how it misconstrues balance data, so their risking some more reliable numbers for a better experience for the consumer, probably a good move by them
This idea that a 50% individual win ratio is not compatible with a statistical data set that can be used to detect race-wide imbalances is just not true, no matter how much people say it. That individuals are being matched against other individuals with whom they have a 50/50 chance to win says nothing about the information that the overall distributions can communicate.
On August 04 2011 11:22 seaofsaturn wrote: The whole purpose of differential equations is to measure things that are constantly changing...
Here is the differential equation from the video:
If you can't make sense of that (I can't!) then I don't know why you're trying to criticize them. The percentages are just simplified representations to present the data to people who aren't math majors, you can't really use them to support random theories.
It's funny they put up some supposedly insane math equation (it's not, you need one year of calculus to understand it), but they don't tell you what anything represents; theta, beta, gamma? Equations are meaningless if that information is omitted. Lol at Blizzard trying to appear transparent, patronization at its best, imo.
"We'll spare you the details, but these are the percentages", sketchy.
This is a cleaned up version:
At the very least, you can infer their equation takes into account standard deviation. Beyond that, there is more there than something "one year of calculus" can help you understand. You might know the symbols and can figure out the "answer" with the numbers, but one year of calculus will never prepare you for what is actually achieved by the equation itself. I would break some of it down for you, but I'm tired of battling morons in this topic who are here for no other purpose than blindly bashing Blizzard for having foresight far beyond anything they can imagine themselves.
In fact, I'm somewhat surprised that this topic is still alive since the TC all but said outright that he doesn't understand how the system can be used and if it strives to match people on a 50-50 basis, then data from those matches is absolutely worthless. The ONLY person I have seen in this topic with any sort of clarity and education is Lysenko. Because of this, I find it worth the time and effort to address a point he is making:
On August 05 2011 14:44 Lysenko wrote: Actually, when it comes to statistics, that's not true. Here's why: every technique that exists for analyzing data sets statistically has implicit assumptions that must be true before the technique tells you what it purports to.
Here's an example: The mean and standard deviation have different meanings if you're looking at a normal vs. a Poisson distribution. When is it appropriate to presume each type of distribution? The answer isn't found in the mathematical equations that describe the distributions' shapes!
There is indication that Blizzard has made an attempt to correct some of their statistical analysis back at the end of season 1 and beginning of season 2. We went through this shift and scaling of MMR as well as a "fixing" of certain league thresholds because the actual MMR distribution didn't show a Gaussian curve. As far as active population goes, it's assumed that the system actually produced a log-normal pdf, where players who played more often were ending up with a better MMR on average than players who didn't play as much. In essence, casual players would be "feeding" MMR to people who played more, and since there are more casual than hardcore players, a smaller population was ending up on the high end of the scale compared to the low end.
Because of this corrective action in league assignments, we can at least assume that they are constantly watching and correcting equations to measure data obtained through ladder.
On August 03 2011 13:34 KiLL_ORdeR wrote: The third and arguably most important factor that they exclude from that though is map balance. They will never get a perfect rating unless the system takes the maps in account.
Even in terms of position. ZvT close positions on Shattered versus far positions, for example, has got to be pretty skewed.
The saddest part about Close Position maps is that even at a High Masters level i have a higher win % on close pos (about 90%) then on cross positions.
This is mainly because terrans do all'ins or cheesy plays which are easy to stop with proper mechanics.
On August 05 2011 15:13 aksfjh wrote: Because of this corrective action in league assignments, we can at least assume that they are constantly watching and correcting equations to measure data obtained through ladder.
That's an interesting change that I'd failed to understand the importance of at the time. Thanks for pointing that out.
Edit: The reason I'd thought that was a quantum-mechanical equation back at Blizzcon is that during the short time they flashed it up on the screen I thought I saw imaginary exponents in there. On looking again right now, I notice that that's not the case.
On August 05 2011 13:51 kckkryptonite wrote: It's funny they put up some supposedly insane math equation (it's not, you need one year of calculus to understand it), but they don't tell you what anything represents; theta, beta, gamma? Equations are meaningless if that information is omitted.
It takes a year of calculus to read it, understanding it either requires some statistical knowledge, or they were joking.
In any case, it clearly wasn't presented to convey information, and if it is, as someone else above speculated, something related to a maximum likelihood calculation involved in the matchmaking system, it's quite likely that none of the people speaking on that stage were fully up to speed on the statistics involved.
They were clearly joking...
I really don't think they have any way of calculating a "clean" win percentage, as I said before it's just a way for them to send the right message to the players (stop caring about balance, just play).
On August 05 2011 15:03 Slago wrote: it seems there main concern is making a player satisfied and happy by giving them the 50% win ratio, but aren't all that concerned with how it misconstrues balance data, so their risking some more reliable numbers for a better experience for the consumer, probably a good move by them
This idea that a 50% individual win ratio is not compatible with a statistical data set that can be used to detect race-wide imbalances is just not true, no matter how much people say it. That individuals are being matched against other individuals with whom they have a 50/50 chance to win says nothing about the information that the overall distributions can communicate.
Maybe, but that's not at all what Blizzard was trying to say, and has nothing to do with the numbers they gave us which are obviously meaningless.
On August 05 2011 04:09 Ihpares wrote: - A system that ensures a 50% win rating not only in general, but race to race will hide imbalance by virtue of actively seeking that 50% regardless of skill level. This means two players of identical skill with two different races will both be at 50%, but will have very different MMRs if their respective races are imbalanced against one another.
But the distributions will be different over a large number of players in the imbalanced case. If it's harder to win as Zerg, Zerg players will have systematically lower MMRs, and that'll be visible in the distributions, unless you can make a convincing case that worse players choose Zerg.
I don't get why you keep harping on this -- do you have any convincing case that we should believe the race choice to be skill-independent?
There is absolutely nothing symmetric that would lead to any obvious justification here.
Terran is the only race you really play in the campaign. Why would that not have any effect on how many people, and how many new people (who did not play any other race in SC/BW) choose that race? That will be the only race they have serious experience with from single-player, and it's clear that transitioning from T to P/Z is far from easy if you never played P/Z.
Why do you think that all three races are equally frustrating for low-level players? Is it so? I don't know, but there's no symmetry that would justify this assumption a priori. Two races can easily wall off, one cannot. Two races can easily start playing with certain builds which require little scouting or other adaptation (e.g. 4gate and 3rax), that is _very_ helpful for beginners to get a handle of the race; as Zerg you need to play differently to learn how to play the race.
As Protoss you may lose to many early cheeses if you don't 1-base forge (which won't get you far) and if you aren't very careful about how you spend your chronoboosts (to get sentries if needed) early on. As Terran it's a lot easier to survive if you have a wall, since you can build a bunker regardless of whatever else you were doing, and you already have a rax anyway to fill the bunker.
As Terran your scans cannot be stopped, and they don't require any scouting by a unit, which may or may not make it a lot easier and/or more palatable for a disproportionate (more than 1/3rd) number of people. As Zerg you can try to 6pool every game, which may or may not work at low levels, but this is a significantly different strategy from 2raxing or 2gating in terms of how it needs to be stopped.
All these things are examples of how the races play differently in ways which may very well affect which players (categorized by skill level) choose to play those races. What is your evidence that none of these actually matter? It is not reasonable to assume an a priori distribution of players that is equal across all races, *especially* given that early on (beta) certain strategies were often clearly dominant and abusive (roaches vs p at 1 supply, reapers before rax and speed nerfs) -- which may also make players switch races.
[To people other than Lysenko: This is not a balance whine, and don't bother replying if you have nothing to add to the actual point.]
Finally, actual quantitative data: http://sc2ranks.com/stats/all/1/all (This is just the first link I came across, there may be much better numbers. I don't know of any, feel free to reply with them if you do.)
Race distribution by league in bronze: 8.1% random, 32.7% protoss, 39.2% terran, 20.0% zerg.
For every bronze Zerg, there are two bronze Terrans.
If this isn't strong enough evidence to suggest that assuming a priori that all races have roughly equal MMR distributions is not justified, I don't know what is.
On August 05 2011 19:24 bmn wrote: If this isn't strong enough evidence to suggest that assuming a priori that all races have roughly equal MMR distributions is not justified, I don't know what is.
I don't think that you said what you meant to say. Clearly I'm not assuming a priori that all races have roughly equal MMR distributions -- in fact I'm saying that differences in MMR distribution can yield useful balance information. I am suggesting that it's useful to assume that those differences in MMR distribution come from something else than, say, players of X race being systematically worse than players of Y race.
I'm not even saying that this is the analysis Blizzard is doing -- what I'm saying is that the fact that one can extract useful information from those distributions demonstrates the falsehood of saying that a matching system that tries to make every individual game a 50/50 matchup cannot provide information about differences between races, a falsehood that keeps getting repeated here.
Incidentally, while it's interesting what percentage of players in bronze play each race, that's not that useful by itself either -- you'd at least have to normalize the distributions to account for racial preferences across all skill levels.
And, after doing ALL that, maybe you see there's a bulge in the Terran distribution around bronze, and you match that up to people who have shared their experience of playing the game a little bit coming out of the campaign and then quitting the game. Still, there are ways to test that with the data Blizzard have available, to weed out people who don't stick with the game, for example, or look at the subset who have switched races at least once, or whatever.
Nobody said, least of all me, that interpreting this data would be clean, or could be done without applying qualitative judgement. That said, the idea that the aggregate data is useless for this purpose because the matchmaking system automatically seeks out 50/50 matches for players is simply stupid. There's information there. Sure, interpreting it requires checking it against other sources of information, such as player feedback and the developers' own gut feelings. They said exactly that in the video that's been posted in this thread. It doesn't mean the data's useless -- it's just one input to a much bigger process.
Incidentally: the idea that the matching system exists to provide useful balance data makes me shake my head. That is obviously not the case (since the matching system exists for the sole purpose of ensuring that as few players as possible lose more than half their games).
Final thought: We don't really know whether the MMR information they track is a scalar at all. MMR probably doesn't have different values per matchup because of how they do promotions and the relationship of point scores to MMR, but they might track multiple sigmas per player race or per matchup. This or something like it might explain the multiple parameters in that equation they briefly presented at Blizzcon, and if true would possibly provide a hook by which to come up with race-adjusted skill estimates that might be very useful.
On August 05 2011 12:33 Xevious wrote: My question is why do they think statistics from lower leagues matter as much as higher ones? To but it bluntly, everyone is so bad in bronze that race imbalances aren't what's going to decide the winner, plain and simple (contrary to plat and above, where people don't get supply blocked at 11).
I don't think this is really Blizzard's attitude. If you look at the balance changes they've actually made, the vast bulk are to tweak issues at the high end, and very few are to address issues in lower leagues, and the one or maybe two I can think of where they mentioned low level players exclusively concerned how easy it was to execute certain rushes at the very start of the game.
On the basis of that, I suspect they're quite OK with some imbalance in the lower leagues, but they really don't want things to be completely out of whack (like 70/30 in some matchup) at any level of play, because they'd like all the races to be at least somewhat accessible to new players.
(Incidentally, they did say explicitly at one point that they were OK with Zerg being somewhat harder to learn for totally new players. This by itself suggests that lower league imbalances are a lesser concern than higher league imbalances.)
On August 05 2011 19:24 bmn wrote: If this isn't strong enough evidence to suggest that assuming a priori that all races have roughly equal MMR distributions is not justified, I don't know what is.
I don't think that you said what you meant to say. Clearly I'm not assuming a priori that all races have roughly equal MMR distributions -- in fact I'm saying that differences in MMR distribution can yield useful balance information. I am suggesting that it's useful to assume that those differences in MMR distribution come from something else than, say, players of X race being systematically worse than players of Y race.
But won't this assumption just bite you in the ass if it turns out to be wrong?
As you said earlier, you can't trust your results if any part of calculating them relies on assumptions that may not be true.
I'm not even saying that this is the analysis Blizzard is doing -- what I'm saying is that the fact that one can extract useful information from those distributions demonstrates the falsehood of saying that a matching system that tries to make every individual game a 50/50 matchup cannot provide information about differences between races, a falsehood that keeps getting repeated here.
Yes, you're right about that. I'm more interested in whether Blizzard actually extracts useful information than whether it's possible do so, though -- I doubt what they're doing is anywhere near the theoretical limits of what's possible :-)
Incidentally, while it's interesting what percentage of players in bronze play each race, that's not that useful by itself either -- you'd at least have to normalize the distributions to account for racial preferences across all skill levels.
Yes, but if you look at the rest of the page, you'll see that Terrans are not overrepresented by such a huge margin anywhere other than bronze league (at least not up to Grandmaster). That's why I quoted the number. (Unless I'm misunderstanding what you're saying here...)
And, after doing ALL that, maybe you see there's a bulge in the Terran distribution around bronze, and you match that up to people who have shared their experience of playing the game a little bit coming out of the campaign and then quitting the game. Still, there are ways to test that with the data Blizzard have available, to weed out people who don't stick with the game, for example, or look at the subset who have switched races at least once, or whatever.
It's possible to work around that, I never disputed that. I'm disputing that it's simple, and I'm questioning whether Blizzard actually does all that is necessary. Sure, you can explain it by saying that this is people who just played the game a little bit and then quit -- my point is that this is just one example of how people of one race can indeed be systematically worse than people of other races. This was just an example to demonstrate that this *does* happen, and this is the most trivial example. What about the population that keeps playing, then, and doesn't just stop playing after a little bit? There is a myriad of systematic biases I can think of which sound plausible, and there's no easy way to eliminate those from the statistics.
Gut feelings and player feedback are nice data points, but they're not really that great if you're trying to understand systematic skill differences in player skills across a range of skill levels, given that you already don't know how to measure player skill in a fair, race-agnostic way -- so as far as I can tell it'll usually boil down to you making a certain set of assumptions, then fudging the statistics until it looks like they reasonably support your assumption. (This isn't saying that they intentionally do this, but there's no benchmark to verify that their way of measuring player skill across races is accurate.) And even if you're actually wrong, as long as you're not horribly far off, nobody can contradict you -- only you have the data, only you know the assumptions you made, only you know how you crunched the numbers.
Incidentally: the idea that the matching system exists to provide useful balance data makes me shake my head. That is obviously not the case (since the matching system exists for the sole purpose of ensuring that as few players as possible lose more than half their games).
I don't see anyone suggesting that the match-making system is designed to balance the races.
The outcomes of the matching system is the only data that users have to go by, so it's unavoidable that they will judge balance based on those numbers. What other choice is there beyond blind faith?
Also, why do you shake your head at that? You yourself defend the statement that the match-making system's outcome allows providing information about game balance, and it seems pretty clear that the match-making system's outcomes are used by Blizzard to keep track of balance; whether that was the original purpose of the system is entirely irrelevant in this context.
Final thought: We don't really know whether the MMR information they track is a scalar at all. MMR probably doesn't have different values per matchup because of how they do promotions and the relationship of point scores to MMR, but they might track multiple sigmas per player race or per matchup. This or something like it might explain the multiple parameters in that equation they briefly presented at Blizzcon, and if true would possibly provide a hook by which to come up with race-adjusted skill estimates that might be very useful.
Now you're just muddying the waters by saying that we can't possibly know what they do, so we have no reason to criticize anything they say, because maybe they're doing something brilliant and we'll never know. That's a cop-out :-)
On August 05 2011 20:29 bmn wrote: Now you're just muddying the waters by saying that we can't possibly know what they do, so we have no reason to criticize anything they say, because maybe they're doing something brilliant and we'll never know. That's a cop-out :-)
I'm not saying that it's invalid to criticize anything they say or any change they make.
I am saying that it's not possible, on the outside looking in, to criticize their statistical analysis on technical grounds, because we can't possibly know exactly what they're doing in terms of data collection or analysis. Importantly, though, that's a very small part of how they balance the game (as that video points out.)
In fact, they use statistics that appear out of whack as hints to suggest where to follow up in other areas, looking at player feedback, playing the game themselves as David Kim does, using their own testing tools, etc. Then, they make choices about how to change game rules, if they should think it's warranted, based on the totality of all of that combined with their own personal judgements as game designers and what they think feels right when played.
This isn't chess, with two identical sides and a turn-based system where you can instantly tell who has the sole advantage the game offers. It's a game with three asymmetric race choices and complex mechanics where any change, large or small, may have unintended, unanticipated consequences down the line. Every change to the game is completely subjective -- it has to be. You can't simulate what a change's impact is going to be, because the impact may rely on tactics or strategies that have not yet been thought up. You have to try it and hope, which is one reason that they put balance changes on the PTR.
This is why they look at a 55% win ratio in a matchup and say they're reasonably happy with that. It's not that that isn't statistically significant, it's that you can know nearly absolutely that the balance isn't perfect in a matchup and still have no way to tell in advance that a change you think might help won't make things worse.
And that, by the way, goes double when they see a matchup leaning one way in one region and another way in another region.
I don't see anyone suggesting that the match-making system is designed to balance the races.
There are several comments that allude to this, but here's the one that particularly set me off about it, because it's the best-expressed version of an idea that's simply not true, that striving for a 50% win/loss for players somehow obscures racial differences in the aggregate:
A system that ensures a 50% win rating not only in general, but race to race will hide imbalance by virtue of actively seeking that 50% regardless of skill level. This means two players of identical skill with two different races will both be at 50%, but will have very different MMRs if their respective races are imbalanced against one another.
On August 05 2011 20:29 bmn wrote: Now you're just muddying the waters by saying that we can't possibly know what they do, so we have no reason to criticize anything they say, because maybe they're doing something brilliant and we'll never know. That's a cop-out :-)
I'm not saying that it's invalid to criticize anything they say or any change they make.
I am saying that it's not possible to criticize their statistical analysis on technical grounds, because we can't possibly know exactly what they're doing in terms of data collection or analysis. Importantly, though, that's a very small part of how they balance the game (as that video points out.)
I think your statement is far too strong, the amount of available data and man-power Blizzard has here is very limited, and the solution space is nowhere as large in practice as it is in theory, simply because there are only that many proven systems that are sufficiently robust.
But fair enough: If you contend that we can't even criticize the statistical analysis, there is no point in discussing any analysis based on their statistics. You cannot validly argue any further saying "they use statistics that appear out of whack as hints" if you yourself state that we cannot assume anything about the validity of their statistics.
I don't really see the point of the rest of your reply if we start from the assumption that we cannot know what their statistics do and, based on that, cannot argue technically about what they might be measuring or not.
This is why they look at a 55% win ratio in a matchup and say they're reasonably happy with that. It's not that that isn't statistically significant, it's that you can know nearly absolutely that the balance isn't perfect in a matchup and still have no way to tell in advance that a change you think might help won't make things worse.
Since we already started out by saying that we cannot know what their statistics actually mean, trying to defend their decision-making is entirely baseless here too. Yeah, they probably have an idea what they're doing -- but that's an appeal to their authority, there's nothing you or I could actually discuss meaningfully about this.
I know this sounds harsh, but starting with the assumption that we're talking about an entirely unknown black-box system (without fully knowing either input or output of it, let alone the inner workings) leaves the only conclusion that there is no merit in technically discussing how good or bad that system is. I would not have said that we can't know anything about the match-making system, but that's a personal choice based on how confident you are about knowing what general approaches they might use to create such a ladder system.
I don't see anyone suggesting that the match-making system is designed to balance the races.
There are several comments that allude to this, but here's the one that particularly set me off about it, because it's the best-expressed version of an idea that's simply not true, that striving for a 50% win/loss for players somehow obscures racial differences in the aggregate:
A system that ensures a 50% win rating not only in general, but race to race will hide imbalance by virtue of actively seeking that 50% regardless of skill level. This means two players of identical skill with two different races will both be at 50%, but will have very different MMRs if their respective races are imbalanced against one another.
I didn't interpret that comment the way you did. I took it as saying that the match-making system will hide imbalance at the player level by matching you up with "weaker" players who still have an even chance of winning, and that observation is entirely correct (if trivial).
If you take it as saying that it will hide imbalance in that Blizzard cannot detect this skew, then it is wrong.
But I don't think this implies that exposing the balance of races was the goal of the match-making system. As a player, the match-making system is all we see, so that's the only way we can judge the racial balance from ladder play.
On August 05 2011 11:12 carwashguy wrote: It may be instructive to consider how, in chess, white pieces tend to have an advantage over black pieces. Among weaker players, there's not much advantage. At higher levels, it starts to become a factor. Among top rated computers, white scores about 55%. However, in chess, players take turns playing white and black. It seems to me that the best way to do it for Starcraft would be to look at top-rated random race players. If there is a tendency for them to win with certain races, then I believe this would reveal something meaningful. To put it briefly, the best random race players' should be immune to MMR's affect on their races winnability. An obvious problem is that the random matchups are not totally analogous to the standard matchups since the non-random player doesn't know his opponent's race off hand.
This seems like a good point that's been glossed over a bit. Random players' win percentages should be free of any effect of race on a player's MMR. I'd be surprised if this wasn't used as a pretty significant indicator of balance for Blizzard. Actually, working through it in my head, it may be the case that you would need to only take RvR matchups, since only then will the opponents MMR be free of influence from race.
I can see 2 main problems with this. Firstly, it still doesn't address the issue of any systematic bias for or against playing random. The sc2ranks data shows that it's much less common in Master league than any of the lower 5. Secondly, Terran and Protoss are arguably share more similarities than either race does with Zerg. This might mean that skill is more transferable between the 2 and thus a random player's win rate for each race are not independent.
On August 03 2011 14:09 whacks wrote: Thanks for the responses all.
A lot of people have responded with some variant of "If Blizzard sees all the Zerg players have significantly lower MMR, they'll know there's something wrong." If Blizzard is doing this, then what they're basically doing is comparing the average zerg player's MMR with the average Terran player's MMR. This approach can break for so many reasons, which I'm not going to get into now.
It can break for so many reasons, but comparing average MMR and league placements is the only way the adjusted ELO system allows to account for balance.
For example: if Protoss is 20% of the player base and the masters and grandmasters leagues have less than 20% of Protosses while diamond and lower have more than 20%, that's a sign (just a sign) of race (or map) imbalance.
There may be other explanations to this that don't involve race/map imbalance, but still, comparing average MMR is the best we got.
ChickaChuckWally...The kid obviously was trying to ask if the Thor was supposed to be an answer to the mutalisk, and it wasn't, then doesn't make the mutalisk a balance problem (good implied question as it does force terrain to be one dimensional in going mass marine and blizzard had no idea the magic box would be so effective). What's even funnier is it appears David Kim wasn't even paying attention because then he talked about marauders in his answer.
In fact in that interview the Blizzard had a number of interesting things to say... They strongly defended the marauder as a needed answer to zerg. They also felt the marauder wasn't even that great of a unit and the benefits units marauders got from stimming frequently didn't counter the damage done. Oh boy...
On August 05 2011 21:23 bmn wrote: I don't really see the point of the rest of your reply if we start from the assumption that we cannot know what their statistics do and, based on that, cannot argue technically about what they might be measuring or not.
You're going way too far with this. We can speculate all we want about what they might be measuring based on what they've said, and that might be an interesting discussion, but it would be a mistake to turn around and say they're idiots because they're doing some unjustifiable thing or another that we've imagined they might be doing. Also, the question of how a statistical analysis of any kind fits into their larger decision-making is perfectly reasonable to discuss even if we don't know the details.
The only thing we can't do is break down the exact mathematics of their statistical analysis and say it's valid or invalid for this or that reason, and that's what this thread appears to be trying to do.
Blizzard's operation may be small, but they absolutely have one or a couple experienced statisticians on their Battle.net team who are fully capable of performing some kind of reasonable analysis. How useful those results are in a larger sense may be difficult to say, but that's probably not the statisticians' fault. Furthermore, last I checked they didn't need our approval before balancing their game however they saw fit, so I don't see why our opinion on their statistical approach matters beyond entertaining ourselves with speculation, or alternatively to entertain ourselves by complaining just to complain.
My point is this: We can criticize the specific changes to the game based on the difference between the impact you think the change will have vs. the impact they think it will have. We can say we'd rather they have a greater focus on whatever race's issues / whatever league's issues we happen to be in. We can even slice and dice their offhand comments about this or that unit or whatever and call them ill-considered.
What we can't do is assess the accuracy of their specific statistical analysis or the mathematical underpinnings of their matching system beyond what limited information they choose to share with us. That limited info is not nearly enough to say they're doing it wrongly.
Regarding using random players as a control group, that's an interesting idea. I can think of a third problem, though:
On August 05 2011 21:24 IronDoc wrote: I can see 2 main problems with this. Firstly, it still doesn't address the issue of any systematic bias for or against playing random. The sc2ranks data shows that it's much less common in Master league than any of the lower 5. Secondly, Terran and Protoss are arguably share more similarities than either race does with Zerg. This might mean that skill is more transferable between the 2 and thus a random player's win rate for each race are not independent.
A third issue is that it's simply not possible for a random player to practice any one race with the depth that players who prefer one race can devote to theirs. This means that they're likely to have deficient and maybe early-game-centric play with all three races, and that may eliminate their value as a control.
On August 04 2011 11:22 seaofsaturn wrote: The whole purpose of differential equations is to measure things that are constantly changing...
Here is the differential equation from the video:
If you can't make sense of that (I can't!) then I don't know why you're trying to criticize them. The percentages are just simplified representations to present the data to people who aren't math majors, you can't really use them to support random theories.
It's funny they put up some supposedly insane math equation (it's not, you need one year of calculus to understand it), but they don't tell you what anything represents; theta, beta, gamma? Equations are meaningless if that information is omitted. Lol at Blizzard trying to appear transparent, patronization at its best, imo.
"We'll spare you the details, but these are the percentages", sketchy.
you need more than one year of calculus to understand this type of equation. Cal I/II don't even sniff DEs. All the posts in this thread are total crap except those which point out that <1% of the population comprehends the math involved. The details aren't important, because it works. I am amazed at the number of numbskulls who piss and moan about this or that policy from blizzard without doing any anything to contribute to the solution side of things.
Math is extremely hard, the kind of programming talent Blizzard can hire, while not Google/MSFT/AAPL level, is incredibly high. Just be glad that the smartest guys in the gaming industry are working on the IP you love.
No, you don't. The level of ignorance in your post is astonishing, after the first year of Calc, you are be able to solve basic differential equations (in my curriculum at least). Pouring money into something must mean it's the best right (US HEALTHCARE/EDUCATION)? To top it off you cite wikipedia.
On August 05 2011 14:15 Alyoshka wrote: All the posts in this thread are total crap except those which point out that <1% of the population comprehends the math involved.
Really? WTF. There are derivatives and integrals, fractions and exponents. Seriously? 1%? Where did you get this number? How are you coming to your various conclusions?
On August 05 2011 14:34 Disquiet wrote: I agree he is wrong. even without having the variables defined you can still tell what the thing does if you can understand it. Everyone can recognize the formula for a parabola without knowing how x or y is applied.
Actually, the variables are assumed to be defined as the set of all real numbers.
W/e guys, I'm not gonna get into a math debate with people who know how to use google and wikipedia.
i've taken more than my fair share of math in college and I have no idea what that formula exactly means other than the fact that there are some normal distributions involved in the calculation, and it is used to solve some sort of conditional probability problem.
there is no way you can understand what that formula means with only one year of calculus.
i'm pretty sure differential equations isn't even involved in that formula
On August 05 2011 07:05 Wren wrote: ~Ladder stats are essentially meaningless. Blizzard's correction may undo the effect of the matchmaking system gradually, but cannot fix the fact that the matchmaking system generated the data in the first place. You cannot analyze the flaw out of a flawed data set.
The problem is that everyone who's been arguing that the data set is "flawed" somehow have been saying so without any reasoning or explanation behind it, other than to completely misunderstand or misrepresent the impact of the matchmaking system on the data set.
Nobody in this thread knows what their data set is or exactly how they're analyzing it, so all the criticism of it is fantasy based on imagined details to fill in the blanks.
They've said that their data set is every ladder game, repeatedly. My understanding of the match-making and MMR system is that it's essentially the same as every other computerized ranking system: who you played, if you won, how much the other guy wins. All blizzard tracks is who beat who on which map.
This data is flawed because it cannot tell (just an example) if Terran is the best because the best people play Terran or if Terran is the best because the balance is skewed. All it can tell you is that Terran wins x% of their games.
Apply MMR to GSL open 3 and it will tell you that Rain was a better player than IMMvp, because Rain cheesed to the finals while Mvp was cheesed out in Ro16.
disclaimer: I'm not a math expert, just trying to understand things like everyone else. If I've made a mistake, please correct it.
---------------------------- Ok, Lysenko, I've read this thread fairly carefully, and have a question to pose to you.
On August 04 2011 10:55 Lysenko wrote: The way you adjust for skill is to look at overall MMR distribution among each race's population. If one race, let's say Zerg, has a population distribution that's weighted toward lower MMRs, chances are it's the race that's doing it unless there's some external indication that better players systematically favor the other races for some reason.
Is this the only worthwhile balance-related statistic we can get from the ladder?
If so, maybe the OP claim is correct, and even adjusted win-rates aren't very useful.
On August 05 2011 07:05 Wren wrote: ~Ladder stats are essentially meaningless. Blizzard's correction may undo the effect of the matchmaking system gradually, but cannot fix the fact that the matchmaking system generated the data in the first place. You cannot analyze the flaw out of a flawed data set.
The problem is that everyone who's been arguing that the data set is "flawed" somehow have been saying so without any reasoning or explanation behind it, other than to completely misunderstand or misrepresent the impact of the matchmaking system on the data set.
Nobody in this thread knows what their data set is or exactly how they're analyzing it, so all the criticism of it is fantasy based on imagined details to fill in the blanks.
They've said that their data set is every ladder game, repeatedly. My understanding of the match-making and MMR system is that it's essentially the same as every other computerized ranking system: who you played, if you won, how much the other guy wins. All blizzard tracks is who beat who on which map.
This data is flawed because it cannot tell (just an example) if Terran is the best because the best people play Terran or if Terran is the best because the balance is skewed. All it can tell you is that Terran wins x% of their games.
Apply MMR to GSL open 3 and it will tell you that Rain was a better player than IMMvp, because Rain cheesed to the finals while Mvp was cheesed out in Ro16.
disclaimer: I'm not a math expert, just trying to understand things like everyone else. If I've made a mistake, please correct it.
---------------------------- Ok, Lysenko, I've read this thread fairly carefully, and have a question to pose to you.
On August 04 2011 10:55 Lysenko wrote: The way you adjust for skill is to look at overall MMR distribution among each race's population. If one race, let's say Zerg, has a population distribution that's weighted toward lower MMRs, chances are it's the race that's doing it unless there's some external indication that better players systematically favor the other races for some reason.
Is this the only worthwhile balance-related statistic we can get from the ladder?
If so, maybe the OP claim is correct, and even adjusted win-rates aren't very useful.
statistics are about averages, things occuring in the long run. you can't really do that to GSL open 3 since it is an extremely small sample size.
if you watch the video statistics are there to see if there any statistical evidence (significance) for racial imbalance across leagues. they address a lot of factors beyond statistics (such as his comment about TvP, they dont want a game of defend and win even if that led to a 50% winrate matchup average even at top leagues). they also stated that they were careful with balance changes due to the qualitatively different nature of korean ladder.
mathematics only gives positive analysis, for example they cited statistical evidence suggesting that P was too strong against T, however its possible to make a plethora of changes to 'balance' the game. some 'balances' might not be fun, some might, statistics gives you no idea of those types of things. this combined with their insight that 4G was too strong in a myriad of situations and their desire to change PVP is what led to WG nerf. its important to note that its important to balance around both aspects -- if you have statistically significant data across all leagues saying that one race is dominant against another, that it is an important issue to address because there *shouldn't* be huge skill disparity between players of each race.
for example if they found statistical evidence that P was too strong against Z they could just systematically lower the dps of the most popular P unit (the stalker) until win rates adjusted to ~50% even at the highest end, but that wouldn't exactly be what i call good game design.
On August 06 2011 12:14 Wren wrote: All blizzard tracks is who beat who on which map.
That's what the matchmaking system uses. I guarantee you that they store more info than that for each game -- for example, you can go back in someone's game history and look at their build orders. They store information in resources collected over time and units produced over time. Do they use any of that additional information in their analysis? Do they filter their matchmaking data in a way that provides greater insight than just looking at the whole population? You don't know the answer to those questions, and neither do I, so it makes no sense to criticize their quantitative analysis.
Ultimately, how that information gets fed back into changes to the game is a fuzzy process anyway.
On August 03 2011 13:07 whacks wrote: Disclaimer: I’m not concerned about game balance at all. I’m hoping to have a discussion on the math & statistics behind Blizzard's adjusted-win-percentage that they rely on heavily.
Late last year, Blizzard released a bunch of ladder statistics on “skill-adjusted-win-percentages” for the different matchups. The reason I have it in quotes, is because they never really explained how they did the skill-adjustments. I’ve always been skeptical about whether such a “skill-adjustment” is really possible.
Gist of what they said: Raw league matchup numbers aren’t very meaningful because of matchmaking’s system ability to matchup players with equally challenging opponents. The math guy mentions specifically: Not only does the system put players in 50-50 matches, it also tries to keep the race matchups at 50-50 as well. Because of this, we have to adjust for player skill to calculate the true matchup win rates. Example: a ZvP match is about to be played. The Zerg player’s rating (odds of winning) relative to the Protoss player is 55-45. The Zerg race’s rating relative to the Protoss race is 53-47. If the Protoss player ends up winning, the player ratings will then converge to 51-49. The race ratings will also converge to 52-48.
Their explanation just didn’t click with me. Rating systems such as ELO are great when you’re dealing with a single unknown (relative player strength). But can they really work if you’re trying to differentiate between 2 unknowns? Both relative player skill & race balance? I constructed the following scenario which seems to suggest that this is impossible.
It’s important to first establish the following: Any good rating system, including ELO & the point system, relies on the following principle: • Give each agent (could be a player, or a race) a certain rating as an estimate for how strong the agent is • If 2 agents play and one wins at a higher percentage, the more successful agent should eventually end up with a higher rating • If a higher rated agent & a lower rated agent play against each other, and each wins with an equal percentage, the 2 ratings should eventually converge
The ELO system that Blizzard uses for MMR is an optimized algorithm that allows ratings to stabilize much quicker, but other rating systems that utilize the above principle (including the point-system), can achieve the same results in the long run.
Now going back to the scenario, consider the case where Blizzard releases a new patch which nerfs Zerg and makes it UP relative to both Protoss & Terran (eg, drones now cost 60 min). Consider what will happen to the average Zerg player. He will start losing more than 50% of his games, and his MMR will start dropping. Because of his lower MMR, he’ll start playing against weaker opponents. Eventually, his MMR will stabilize at a level where he starts winning 50% of his future games.
Now let’s say Blizzard had assigned each race a rating as well, to track how “strong they think it is.” Suppose that before the patch, all the races were balanced & had equal rating. Immediately after the patch, because the Zerg population goes through a losing streak, the Zerg rating will drop.
But eventually, the Zerg players will have stabilized their MMR and start winning 50% of their games. At this point, because of the last bullet point in the rating system’s principles (ratings will converge at 50% win rates), the Zerg rating will start increasing again. Remember also that the stabilized Zerg players are playing against opponents of the same MMR, so there’s no way to “account for player skill.” Eventually, the zerg rating will once again converge with the other races, even though Zerg is now UP.
Based on this scenario, it seems impossible to determine whether a race is truly UP, using Blizzard’s rating system. Thoughts? Any ideas on how Blizzard could possibly be “accounting for player skill” in calculating race balance?
I'll do my best to explain this.
Blizzard uses Bayesian Inference. This is usually taught as a 3rd year or honors year statistics course at most universities. I only say this to impress upon you that this is not simple stuff.
The formula that is shown in the Youtube video is this:
So, the whole formula is for the Bayesian estimator where the posterior probability is the product of 3 normal distributions (the 3 MMR variables), multiplied over all g (probably stands for games, i.e. takes into account all games played).
Now what does this mean?
What a Bayesian estimator does is it estimates a parameter (in this case the probability of winning) given the evidence (in this case the skill of the player).
Essentially, they have a prior belief about the probability of winning (very likely the simple unadjusted win ratio), this probability is updated by the skill of the player over all games, forming a posterior distribution, and then using this, the probability of winning given the skill of the player is calculated with a Bayesian estimator.
What isn't clear is what each variable stands for, so we don't know if they take into account the map or game length or other variables. Although from the talk, the impression is that only skil, (i.e. MMR) is taken into account to adjust the probabilities of winning.
On August 04 2011 10:32 bamman1108 wrote: I like that part where they're satisfied with 5% differences in W/L when that percent is based off millions of matches. Even a 1% difference with that many matches means that one race very, very significantly favors the other. Wtf are they talking about when a 55% win rate for a specific race matchup is just "borderline?"
Given a sufficiently large sample size, it's possible to make a 0.00001% difference statistically significant, because a 0.00001% difference is a nonzero difference.
But statistically significant doesn't imply an actual appreciable significant difference in everyday language,
The following example from Wikipedia (http://en.wikipedia.org/wiki/Statistical_significance) explains this concept well: As used in statistics, significant does not mean important or meaningful, as it does in everyday speech. For example, a study that included tens of thousands of participants might be able to say with great confidence that residents of one city were more intelligent than residents of another city by 1/20 of an IQ point. This result would be statistically significant, but the difference is small enough to be utterly unimportant.
I just got back from vacation, so forgive me for resurrecting this thread so late
Paralleluniverse, thanks for taking the time to clarify. It sounds like that formula is conceptually pretty similar to ELO, possibly taking into account multiple factors other than player skill, such as racial "scores." This is exactly what I suspected in my OP.
Lysenko, you mention over & over again that the ladder data can yield useful balance information by letting us compare average-MMR difference across the races. I agree completely on this. However, this is NOT what Blizzard is doing. How do I know this?
1) There is actually VERY significant MMR differences between the races. Terran is skewed very heavily towards Bronze, and Zerg is skewed very heavily towards Plat/Diamond/Master's. Blizzard's numbers paint a very rosy picture, but if you compared average MMRs, you'll see very wide differences.
2) Calculating average MMRs is 7th grade math. You sum up all the MMRs of each player, and divide by number of players. You definitely won't need any complicated math like what they announced.
Clearly, when Blizzard presented to us the ladder data, they weren't basing it off average-MMR. You haven't presented any other methods they could be using that works in our ladder system, so you might actually be in agreement with the point I'm trying to make in my OP, and with what others like lhpares & bmn have been saying.
Again, if you have "blind faith" in Blizzard's abilities... I respect that, but that's not what this thread is about.
Bumping this because I found a good article on a bayesian approximation method for online ranking. I have started deciphering the variables in the equation.
So far it appears that they are analyzing the games as if the race the person has chosen is considered an additional player in the match. I'm assuming that the different sigmas represent the sigmas of the 3 races (meaning each race has a MMR). Normally I would just scoff this off as "impossible to calculate", but since they are using the sigmas themselves to calculate the values, it seems a lot more reasonable as it doesn't matter what type of matchmaking system is being used. Like the above post, this is the posterior probability function.
I will provide an update once I figure out all of the variables. I'm mostly having trouble on that Psi.
Edit; In games where ties are not allowed or very infrequent, gamma is typically used for score variance. It's possible that the score at the end of the game is being used to calculate it.
Edit 2; After consideration, the 3 sigmas being used in the equation are "player skill", "matchup skill", and "overall skill". For example. A player might have an mmr of 2000/100. Protoss (vs zerg) has an mmr or 1500/50. and protoss (vs all) has an mmr of 1600/75. That's the conclusion I am coming to so far. I will attempt to verify this hypothesis.
But surely this doesn't work out? Each race will have a different win percentage at each league. Zerg for instance has a high win percentage in lower leagues, because bronzies can't deal with ling runbys, but at high levels where this isn't as viable, the Zerg win rate is far lower.
Perhaps I am wrong about this, but it seems like Blizzard's approach to balancing is to assume that each race has approximately the same skill distribution.
This certainly simplifies the task.
And, personally, I think it is probably the most practical way to approach this matter. The more readily pros switch to races they consider overpowered, the more likely this approach is to yield an outcome close to objective balance.
We could argue that switching races is quite difficult, which means there will be more imbalance.
It's hard to see any other way to simplify the task, which we have already established is intractable without any simplifications.
On August 03 2011 13:07 whacks wrote: Their explanation just didn’t click with me. Rating systems such as ELO are great when you’re dealing with a single unknown (relative player strength). But can they really work if you’re trying to differentiate between 2 unknowns? Both relative player skill & race balance? I constructed the following scenario which seems to suggest that this is impossible.
This has come up a few times, and yes you're right, it is impossible. It's possible that on average players of one race are actually better players than those of another. Based just on game results, there is absolutely no way of distinguishing that from the race being overpowered. Somewhere along the line you have to make an assumption, and I think the assumption they have used is that the player pool for each race is equally "skilled" (another problem is that there's no formal definition of skill), and any further discrepancies in win/loss rates (once matchmaking is accounted for) are due to imbalances in the game. Is it a reasonable assumption? Maybe.
MMR is also a pretty bad indicator of balance, simply because unless Blizzard is using matchup specific MMR, one bad matchup wrecks the reliability of MMR for every other non-mirror matchup. Say TvP is screwed in favor of Terran, this would make PvZ look in favor of Protoss simply because the Protoss player is playing less skilled zerg to force the win percentage towards 50%. Hence, zerg players on the ladder would have the perception that ZvP is Protoss favoured, except that that would be because they're playing opponents of a higher skill.
On September 30 2011 18:47 hummingbird23 wrote: MMR is also a pretty bad indicator of balance, simply because unless Blizzard is using matchup specific MMR, one bad matchup wrecks the reliability of MMR for every other non-mirror matchup. Say TvP is screwed in favor of Terran, this would make PvZ look in favor of Protoss simply because the Protoss player is playing less skilled zerg to force the win percentage towards 50%. Hence, zerg players on the ladder would have the perception that ZvP is Protoss favoured, except that that would be because they're playing opponents of a higher skill.
Who said they are balacing based on MMR, they said they look at specific match ups percentages.
I see a lot of people bashing Blizzard for what system they are using but no one is giving suggestions on whats better.
On September 30 2011 18:47 hummingbird23 wrote: MMR is also a pretty bad indicator of balance, simply because unless Blizzard is using matchup specific MMR, one bad matchup wrecks the reliability of MMR for every other non-mirror matchup. Say TvP is screwed in favor of Terran, this would make PvZ look in favor of Protoss simply because the Protoss player is playing less skilled zerg to force the win percentage towards 50%. Hence, zerg players on the ladder would have the perception that ZvP is Protoss favoured, except that that would be because they're playing opponents of a higher skill.
Who said they are balacing based on MMR, they said they look at specific match ups percentages.
I see a lot of people bashing Blizzard for what system they are using but no one is giving suggestions on whats better.
He is saying that win-percentages are not a good indicator of things like balance because of how they are actually measuring the win-percentages (since who you play is based on MMR)
This is what he means: Let's say Toss has a 40% win rate against Terran, assuming they are of both skill level and things such as EMP make the matchup favor the Terran. Toss then LOSE MMR since they have a hard time playing vT, and so as I said their MMR goes down, but then they get matched up with a Zerg. Well this Toss' MMR is artificially low because PvT is in favor of Terran, so he is playing a zerg who is a lesser-skilled player than him, which then will create PvZ to favor the toss because toss are actually playing worse players since their MMR is artificially low.
On September 30 2011 18:47 hummingbird23 wrote: MMR is also a pretty bad indicator of balance, simply because unless Blizzard is using matchup specific MMR, one bad matchup wrecks the reliability of MMR for every other non-mirror matchup. Say TvP is screwed in favor of Terran, this would make PvZ look in favor of Protoss simply because the Protoss player is playing less skilled zerg to force the win percentage towards 50%. Hence, zerg players on the ladder would have the perception that ZvP is Protoss favoured, except that that would be because they're playing opponents of a higher skill.
Who said they are balacing based on MMR, they said they look at specific match ups percentages.
I see a lot of people bashing Blizzard for what system they are using but no one is giving suggestions on whats better.
My point is that it's pretty hard to use the numbers Blizzard is giving. Each matchup for each player differs. One way to test this system would be to throw every match of a particular matchup (eg. Terran player always quits whenever he sees Zerg), and see if the overall win/lose still tends towards 50%. If it does, it means that whenever the player plays against Protoss or Terran, he's playing inferior opponents. No amount of TvP balance will thus fix a TvZ imbalance, for example.
On August 03 2011 13:34 KiLL_ORdeR wrote: The third and arguably most important factor that they exclude from that though is map balance. They will never get a perfect rating unless the system takes the maps in account.
you already forgot their statement about maps, balance it on small maps and it works everywhere.
and it may be 2 unknowns but they relate to each other. So once you know who did win you can calculate them both. Its actually a clever system to sort out the ladder system searching equal opponents for you. It gives you a good overview of how things changed after a patch at the different levels, which is probably the most important thing. Though you might not see which race has the less effort do do to win, but since they address this as well I guess somehow they know.
Does almost everyone in this thread really think Blizzard is stupid and ignores such obvious things?
They have really, really clever guys working on that, people that are about twice as smart as all posters here combined (including me) and they will have considered everything that is posted here and found ways to work around the limitations.
Example for how it could be, eventhough i'm not half as clever as the guys who thought about those things 8 hours a day for month: Blizzard has 3 MMRs, 1 for each race
Player A (Zerg): vT: 60% Winrate, MMR 1800 vZ: 30% Winrate, MMR 1200 vP: 40% Winrate, MMR 1400
We can assume that his average MMR is somewhere around 1500-1600, he just is very strong vs Terrans and less strong vs Protoss. Against Zerg, he sucks.
Player B (Terran): vZ: 40% Winrate, MMR 1400 vP: 60% Winrate, MMR 1800 vT: 50% Winrate, MMR 1600
This player has has an average MMR of 1600, he is slightly stronger vs Protoss, slightly weaker vs Zerg.
Player C (Terran): vZ: 60% Winrate, MMR 1800 vP: 50% Winrate, MMR 1600 vT: 50% Winrate, MMR 1600
This player has an MMR of about 1600-1700 and is slightly stronger vs Zerg.
If Player A and Player B meet 100 times on the ladder and Player A wins 40% of the matches while Player A and Player C end up in ~55% for Player C, the game should be balanced, eventhough for the point of Player B, it's imbalanced since he loses a lot. Personal skill in different matchups is accounted for and everyone should be happy.
Ofc as mentioned, there are more intelligent people whos job it is to calculate that stuff. They had month to figure everything out, i though of this in about a minute so i might be totally wrong.
On September 30 2011 21:25 Morfildur wrote: Does almost everyone in this thread really think Blizzard is stupid and ignores such obvious things?
They have really, really clever guys working on that, people that are about twice as smart as all posters here combined (including me) and they will have considered everything that is posted here and found ways to work around the limitations.
Example for how it could be, eventhough i'm not half as clever as the guys who thought about those things 8 hours a day for month: Blizzard has 3 MMRs, 1 for each race
Player A (Zerg): vT: 60% Winrate, MMR 1800 vZ: 30% Winrate, MMR 1200 vP: 40% Winrate, MMR 1400
We can assume that his average MMR is somewhere around 1500-1600, he just is very strong vs Terrans and less strong vs Protoss. Against Zerg, he sucks.
Player B (Terran): vZ: 40% Winrate, MMR 1400 vP: 60% Winrate, MMR 1800 vT: 50% Winrate, MMR 1600
This player has has an average MMR of 1600, he is slightly stronger vs Protoss, slightly weaker vs Zerg.
Player C (Terran): vZ: 60% Winrate, MMR 1800 vP: 50% Winrate, MMR 1600 vT: 50% Winrate, MMR 1600
This player has an MMR of about 1600-1700 and is slightly stronger vs Zerg.
If Player A and Player B meet 100 times on the ladder and Player A wins 40% of the matches while Player A and Player C end up in ~55% for Player C, the game should be balanced, eventhough for the point of Player B, it's imbalanced since he loses a lot. Personal skill in different matchups is accounted for and everyone should be happy.
Ofc as mentioned, there are more intelligent people whos job it is to calculate that stuff. They had month to figure everything out, i though of this in about a minute so i might be totally wrong.
This isn't the main problem. The main problem is if race X is UP in all both mathcups. Then blizzards statisctics wont be able to realize that all players of race x actually have to low an mmr. If they used stats like average mmr of races they would have to assume that players of different skills are somewhat evenly distributed over the different leagues, however that is most likely not true.
On September 30 2011 21:25 Morfildur wrote: Does almost everyone in this thread really think Blizzard is stupid and ignores such obvious things?
They have really, really clever guys working on that, people that are about twice as smart as all posters here combined (including me) and they will have considered everything that is posted here and found ways to work around the limitations.
Example for how it could be, eventhough i'm not half as clever as the guys who thought about those things 8 hours a day for month: Blizzard has 3 MMRs, 1 for each race
Player A (Zerg): vT: 60% Winrate, MMR 1800 vZ: 30% Winrate, MMR 1200 vP: 40% Winrate, MMR 1400
We can assume that his average MMR is somewhere around 1500-1600, he just is very strong vs Terrans and less strong vs Protoss. Against Zerg, he sucks.
Player B (Terran): vZ: 40% Winrate, MMR 1400 vP: 60% Winrate, MMR 1800 vT: 50% Winrate, MMR 1600
This player has has an average MMR of 1600, he is slightly stronger vs Protoss, slightly weaker vs Zerg.
Player C (Terran): vZ: 60% Winrate, MMR 1800 vP: 50% Winrate, MMR 1600 vT: 50% Winrate, MMR 1600
This player has an MMR of about 1600-1700 and is slightly stronger vs Zerg.
If Player A and Player B meet 100 times on the ladder and Player A wins 40% of the matches while Player A and Player C end up in ~55% for Player C, the game should be balanced, eventhough for the point of Player B, it's imbalanced since he loses a lot. Personal skill in different matchups is accounted for and everyone should be happy.
Ofc as mentioned, there are more intelligent people whos job it is to calculate that stuff. They had month to figure everything out, i though of this in about a minute so i might be totally wrong.
This isn't the main problem. The main problem is if race X is UP in all both mathcups. Then blizzards statisctics wont be able to realize that all players of race x actually have to low an mmr. If they used stats like average mmr of races they would have to assume that players of different skills are somewhat evenly distributed over the different leagues, however that is most likely not true.
They can still see it when the MMR of many players against one race is higher than against the other races.
Example: We can assume that over all Protoss players, their relative skill in other matchups is about even or the difference is statistically insignificant. Taking all 500'000 protoss into account, some are stronger in other matchups, some are weaker. Some are better in mirror matches, some are worse. With that much data, it will even out.
We can then check the MMR of the mirror match compared to the other players MMR in mirror matches and check the results of the match. For each player we go through each match, show how the person has faired in mirror matches, check the others MMR (again: by mirror) and check who should win. Then we look at who really won and compare the results. If the win was unexpected, we note it and move to the next game.
For a single game, the results will be wrong. Everyone can have a bad day, made a mistake, etc., but statistically, over the huge number of games, the results will even out. Not perfectly, for that the number is still too low, but good enough to show big differences (that might be why they say that between 45% and 55% there is no reason for concern. It's just statistics after all).
If we then find out that Protoss only wins 80% of the matches they should have won, we can put it down as 40% winrate for Protoss and therefor a big imbalance.
Anyways, as i said, there are people who think about it 8 hours a day 5 days a week. They are not stupid, they learned that stuff and know more about it than we do. They have a lot more data than we have access to and more than we might even assume, so we should trust that they don't just pull numbers out of their a**, just because we don't know how they do it.
On September 30 2011 21:25 Morfildur wrote: Does almost everyone in this thread really think Blizzard is stupid and ignores such obvious things?
They have really, really clever guys working on that, people that are about twice as smart as all posters here combined (including me) and they will have considered everything that is posted here and found ways to work around the limitations.
Example for how it could be, eventhough i'm not half as clever as the guys who thought about those things 8 hours a day for month: Blizzard has 3 MMRs, 1 for each race
Player A (Zerg): vT: 60% Winrate, MMR 1800 vZ: 30% Winrate, MMR 1200 vP: 40% Winrate, MMR 1400
We can assume that his average MMR is somewhere around 1500-1600, he just is very strong vs Terrans and less strong vs Protoss. Against Zerg, he sucks.
Player B (Terran): vZ: 40% Winrate, MMR 1400 vP: 60% Winrate, MMR 1800 vT: 50% Winrate, MMR 1600
This player has has an average MMR of 1600, he is slightly stronger vs Protoss, slightly weaker vs Zerg.
Player C (Terran): vZ: 60% Winrate, MMR 1800 vP: 50% Winrate, MMR 1600 vT: 50% Winrate, MMR 1600
This player has an MMR of about 1600-1700 and is slightly stronger vs Zerg.
If Player A and Player B meet 100 times on the ladder and Player A wins 40% of the matches while Player A and Player C end up in ~55% for Player C, the game should be balanced, eventhough for the point of Player B, it's imbalanced since he loses a lot. Personal skill in different matchups is accounted for and everyone should be happy.
Ofc as mentioned, there are more intelligent people whos job it is to calculate that stuff. They had month to figure everything out, i though of this in about a minute so i might be totally wrong.
This isn't the main problem. The main problem is if race X is UP in all both mathcups. Then blizzards statisctics wont be able to realize that all players of race x actually have to low an mmr. If they used stats like average mmr of races they would have to assume that players of different skills are somewhat evenly distributed over the different leagues, however that is most likely not true.
They can still see it when the MMR of many players against one race is higher than against the other races.
Example: We can assume that over all Protoss players, their relative skill in other matchups is about even or the difference is statistically insignificant. Taking all 500'000 protoss into account, some are stronger in other matchups, some are weaker. Some are better in mirror matches, some are worse. With that much data, it will even out.
We can then check the MMR of the mirror match compared to the other players MMR in mirror matches and check the results of the match. For each player we go through each match, show how the person has faired in mirror matches, check the others MMR (again: by mirror) and check who should win. Then we look at who really won and compare the results. If the win was unexpected, we note it and move to the next game.
For a single game, the results will be wrong. Everyone can have a bad day, made a mistake, etc., but statistically, over the huge number of games, the results will even out. Not perfectly, for that the number is still too low, but good enough to show big differences (that might be why they say that between 45% and 55% there is no reason for concern. It's just statistics after all).
If we then find out that Protoss only wins 80% of the matches they should have won, we can put it down as 40% winrate for Protoss and therefor a big imbalance.
Anyways, as i said, there are people who think about it 8 hours a day 5 days a week. They are not stupid, they learned that stuff and know more about it than we do. They have a lot more data than we have access to and more than we might even assume, so we should trust that they don't just pull numbers out of their a**, just because we don't know how they do it.
Those very smart people, thinking about this 8 hours per day, 5 days per week, are also governed by deadlines, budgetting and compromising due to demands of managers, programmers and networking technicians. What they can think up as a perfect system isn't necessarily what they end up shipping. I'd like to point you to the state of bnet 2.0 at launch for a pretty clear example of "very smart people" shipping absolute garbage.
It is not complicated at all to create skill-adjusted-win-percentages.
1. Create separate bell curves for each race. Place every single Zerg player on the the Zerg bell curve based on their MMR. Do the same for Terran and Protoss. 2. Compare the outcome of matches between Protoss and Zerg players at the same place at each race bell cure. 3. If Zerg players that are top 5.5-6.5% in skill on the Zerg bell curve win 60% of their matches against Protoss players that are on top 5.5-6.5% on the Protoss bell curve, there is an imbalance at that specific skill range.
This assumes that that for most of the curve the skill of the players of the races are about the same. This is very likely given the huge amount of players. However, very far out on the curve it is possible this does not hold true due to the numbers of players at those positions on the curve are so few.
EPT
![[image loading]](http://i.imgur.com/6M0gm.jpg)
I honestly don't care that much about race balance. This entire thread is about math-buffs trying the understand how the stuff works. If you're just going to pop in here & say that we should blindly trust Blizzard, you aren't really contributing anything to the thread.
.![[image loading]](http://i.imgur.com/uGmmZ.png)
![[image loading]](http://imgur.com/UkUR2.png)
