|
On August 04 2011 11:22 seaofsaturn wrote:The whole purpose of differential equations is to measure things that are constantly changing... Here is the differential equation from the video: ![[image loading]](http://i.imgur.com/6M0gm.jpg) If you can't make sense of that (I can't!) then I don't know why you're trying to criticize them. The percentages are just simplified representations to present the data to people who aren't math majors, you can't really use them to support random theories.
When I saw the presentation in person, I presumed that that was a joke, because that equation looks a hell of a lot like an orbital wave function in 3D spherical space, from quantum mechanics. However, I'm no expert in multivariate statistics, and it's quite possible that the math looks similar.
Edit: It actually looks like the equation for calculating the probability of a particular quantum state from the wave function. For any real physicists or chemists keeping score.
Edit 2: I'd be very surprised if Blizzard's statisticians worked in spherical coordinates for just about anything. 
|
On August 04 2011 11:34 NATO wrote:
Blizzard just has an a priori for racial skill based on the normal distribution where they assume something about what player skill is supposed to be for each race (probably the same mean). Then if this distribution is off, something is wrong.
Of course that something is more likely to be correlation with player skill or other factors such as race potential, rather than raw racial strength. Furthermore, accounting for meta game this will shift through time as races become more well understood than they were before. Because of this, there is no way for Blizzard to know actual balance through this means.
What you say is true, but I think they know this, which is why they check their plans against player feedback and their own testing tools and David Kim's play experiences.
|
Blizz has people a lot smarter than us working on this all day every day... do any of you really think you can sit here knowing .000001% of what they're doing and criticize them like you know what your're talking about?
No, you can't.
|
On August 04 2011 11:41 Neo.NEt wrote:
Blizz has people a lot smarter than us working on this all day every day... do any of you really think you can sit here knowing .000001% of what they're doing and criticize them like you know what your're talking about?
There are certainly people in this forum with deep technical backgrounds. I think it's reasonable for such people to ask tough questions about what they're doing based on that presentation, and some of this thread is just that. That said, people with deep technical backgrounds often jump to conclusions especially when speaking casually, and that's represented in this thread as well.
|
On August 04 2011 11:37 Lysenko wrote:Show nested quote +On August 04 2011 11:22 seaofsaturn wrote:The whole purpose of differential equations is to measure things that are constantly changing... Here is the differential equation from the video: If you can't make sense of that (I can't!) then I don't know why you're trying to criticize them. The percentages are just simplified representations to present the data to people who aren't math majors, you can't really use them to support random theories. When I saw the presentation in person, I presumed that that was a joke, because that equation looks a hell of a lot like an orbital wave function in 3D spherical space, from quantum mechanics. However, I'm no expert in multivariate statistics, and it's quite possible that the math looks similar. Edit: It actually looks like the equation for calculating the probability of a particular quantum state from the wave function. For any real physicists or chemists keeping score. Edit 2: I'd be very surprised if Blizzard's statisticians worked in spherical coordinates for just about anything.
Hmm... I have never considered Blizzard the sort of people that would make esoteric jokes at a press conference with how much they always try to be kind to noobs and such, but I guess it is possible...
(edit: typo)
|
On August 04 2011 11:46 seaofsaturn wrote:
Hmm... I have never considered Blizzard the sort of people that would make esoteric jokes at a press conference with how much they always try to be kind of noobs and such, but I guess it is possible...
It wasn't at a press conference -- it was a presentation at Blizzcon, and yeah they absolutely make incredibly oblique and geeky jokes. (There was one a few years ago in a Diablo 3 presentation about a "Horadric Companion Cube" that stuck with me as particularly funny on a number of levels.)
Besides, it's not that esoteric a joke, if you imagine the speaker getting on Google with the goal of finding the most hilariously complicated-looking equation he could find.
|
On August 04 2011 11:31 Lysenko wrote:Show nested quote +On August 04 2011 11:20 McFortran wrote:
If you look at Korea, it would appear that there are significantly more terran players at the highest level than other races. So I don't think the assumption is particularly good in theory or in practice. The thing is, that simple fact doesn't say much. In the absence of any other information it could mean: * That Terran is imbalanced for game design reasons for players at the top level of skill. * That players at the top level of skill favor Terran because it plays in a way that rewards that skill for reasons other than a competitive advantage (for example a playstyle that makes them feel like they're making use of their skills.) * That the skill distributions of all three races are identical but for some reason Korean players favor Terran, thus all levels of the game are more populated with Terrans. Edit, sorry, I left an important one out: * That the most skilled players in Korea have decided among themselves that Terran is favored by imbalances, when in fact it's well-balanced. The video the OP linked explains that when the numbers suggest something, they go looking through other sources of evidence (pro player feedback, community feedback, their own play experiences, tournament replay analysis, results from testing tools) to try to distinguish between possible causes. Sounds to me like a very reasonable approach. As for how they create their "skill-adjusted win percentages," it's hard to criticize without a specific understanding of what they're doing. However, I do know that the Battle.net 2.0 team employs at least a few people with a more rigorous statistical background than most of the posters on Team Liquid's site. My point, of course, is to say that it's completely invalid to criticize any statistical analysis without understanding the details of what they're doing.
I agree. I guess the problem I have is that some people simply assume that these statistics are all-powerful because they don't understand the underlying math (it's just Bayesian inference). In reality they use a lot of methods to determine balance, none of them are particularly good by themselves. It's more of an art than a science in the end; it's not like Blizzard has the answer to everything.
|
On August 03 2011 14:37 puppykiller wrote:
did anyone else watch the question and answer section of the video. That was the most tragic thing I have ever seen. Obviously when the majority of the ppl who play ur games have that mindset your going to be in a position where u cant make a well balanced game.
I went back to the video and watched that part... Ahahahahahahahahhahaaha.
Especially the second guy. I bet he's using IE 6 as his browser.
|
To everyone saying that you have blind faith in Blizzard: I respect that, but stay out of this thread!  I honestly don't care that much about race balance. This entire thread is about math-buffs trying the understand how the stuff works. If you're just going to pop in here & say that we should blindly trust Blizzard, you aren't really contributing anything to the thread.
Maddog: I was giving an example where Zerg is OP compared to both Terran & Protoss. I agree that in the situation you described (where Zerg is OP vs T, UP vs P), the win-rates will look weird & raise flags. But for the case where Zerg is OP relative to both T & P, the Zerg player will indeed show up as 2100, and the win-rates won't flag anything weird. I think this is the most fatal flaw behind Blizzard's adjusted-win-rates.
Lysenko: Thanks for taking the time to post intelligent, detailed replies. I can tell we have a common way of thinking, even if we disagree.
The way you adjust for skill is to look at overall MMR distribution among each race's population. If one race, let's say Zerg, has a population distribution that's weighted toward lower MMRs, chances are it's the race that's doing it
I agree. Looking at each race's MMR distribution is a very valid tool for judging balance. If Blizzard came right out & said this is what they were doing, I might even support it. It's so much simpler & more straight forward than their "adjusted-win-rate" and requires almost no number crunching at all.
However, I don't think this is what Blizzard is doing. Why? Because their adjusted-win-ratios make the game seem so much more balanced than the MMR distribution. Check out the Zerg distribution in comparison to the Zerg player base. They're skewed so much more heavily towards Plat, Diamond & Master's, compared to the 27% player base.
That said, I don't think this is that great a tool for objectively determining balance. Why? Because some races might appeal a lot more to hardcore gamers while others appeal a lot more to casuals. Sounds hard to believe, right? But ask anyone who played a lot of battlegrounds in WoW-classic. Even though the 2 factions were 99% identical, the Horde would dominate the battlegrounds the vast majority of the time over Alliance. This is so true, it's not even a joke. A random Horde team would consistently walk over a random Alliance team, despite there being no balance issues. Why? Because the Horde faction simply appealed so much more to Hardcore gamers, compared to Alliance with their huge population of casuals. Looking at the whopping 40% of Bronze players being Terran, I suspect the same thing is happening in SC2 as well. Hence why looking at MMR distribution, though being handy, is not all that useful.
I'm actually planning on making a different thread regarding this topic, focusing on other statistics that could be used in determining balance. But for now, I wanted to have a discussion about whether the "adjusted-win-ratios" that Blizzard posts make any sense at all. And I'm starting to think the answer is no.
|
On August 04 2011 11:45 Lysenko wrote:Show nested quote +On August 04 2011 11:41 Neo.NEt wrote:
Blizz has people a lot smarter than us working on this all day every day... do any of you really think you can sit here knowing .000001% of what they're doing and criticize them like you know what your're talking about? There are certainly people in this forum with deep technical backgrounds. I think it's reasonable for such people to ask tough questions about what they're doing based on that presentation, and some of this thread is just that. That said, people with deep technical backgrounds often jump to conclusions especially when speaking casually, and that's represented in this thread as well.
Well, then those people should present their credentials.
|
On August 04 2011 11:37 Lysenko wrote:Show nested quote +On August 04 2011 11:22 seaofsaturn wrote:The whole purpose of differential equations is to measure things that are constantly changing... Here is the differential equation from the video: If you can't make sense of that (I can't!) then I don't know why you're trying to criticize them. The percentages are just simplified representations to present the data to people who aren't math majors, you can't really use them to support random theories. When I saw the presentation in person, I presumed that that was a joke, because that equation looks a hell of a lot like an orbital wave function in 3D spherical space, from quantum mechanics. However, I'm no expert in multivariate statistics, and it's quite possible that the math looks similar. Edit: It actually looks like the equation for calculating the probability of a particular quantum state from the wave function. For any real physicists or chemists keeping score. Edit 2: I'd be very surprised if Blizzard's statisticians worked in spherical coordinates for just about anything.
Well, why would you be surprised? The wave equation for QM is mainly a probabilistic one, and really even outside of the realm of Physics spherical coordinates are not uncommon and are actually pretty useful.
Errrr, about it being a joke.. well since none of us is really working with Blizzard we kinda have to either believe in that they are using that model or not ...
And besides a problem such as abstract as balance and skill can´t really be tackled solely by statistics, which I have always found it weird why people focus so much on that. I mean, anyone who has worked in statistics can and will know that they can be terribly misleading. But at one point one has to land their work into real life. And Blizzard has acknowledged that just using ladder statistics by itself is meaningless, that is why supposedly they see tourneys and ask pro players|
Oh god, I hope I didn´t butcher english too much.
|
On August 04 2011 10:32 bamman1108 wrote:
I like that part where they're satisfied with 5% differences in W/L when that percent is based off millions of matches. Even a 1% difference with that many matches means that one race very, very significantly favors the other. Wtf are they talking about when a 55% win rate for a specific race matchup is just "borderline?"
Absolutely.
Before looking at win-rate statistics in chess (white vs. black), if you read up on the first-move advantage in chess you get the impression that the discrepancy between white and black pieces must be huge. You read quotes like this: "Teimour Radjabov was the youngest person to beat Gary Kasparov -- and he did so as black!" Yet the statistics lead to "only" a ~55% win-rate for white across high level matches.
There are so many factors that determine the outcome of a game of Starcraft -- skill, randomness (due to imperfect information), psychological factors, other sources of variability -- that whatever inherent imbalance exists will provide a relatively weak signal in win rates.
For example, no one would deny that lag gives you an inherent -- and unfair -- disadvantage. Yet if you have two NesTea clones duking out 1000 matches, where one NesTea is playing with lag and the other isn't, then the win-rate might only be around 55%. Again, this is because a milieu of other factors determine the outcome. Statistically, it's hard to tease apart that one factor in question (whether racial imbalance, or lag).
Also factor in that our brains suck at understanding statistics intuitively. For example, see the birthday problem or that infamous cancer problem that Bayesians are always talking about.
It's easy to get excited about "only" a 55/45% imbalance because those numbers "look" close. Don't trust your gut interpretation of raw statistics.
|
On August 04 2011 14:00 ero wrote:
It's easy to get excited about "only" a 55/45% imbalance because those numbers "look" close. Don't trust your gut interpretation of raw statistics.
Except that from a game design perspective, "does it feel wrong" is a perfectly reasonable question to ask, and maybe even the overriding concern.
In any case, in chess you can make the case that who moves first is the SOLE difference between the two colors, and thus the source of that 5% deviation (or whatever) from even win percentages.
In Starcraft, a 5% difference is likely the result of a large number of different factors each contributing independently -- and correcting such a difference without introducing other issues elsewhere may be a nearly unsolvable problem.
What's "close enough" is always going to be a heuristic, there's no way around it. Also, I'd be very surprised if Blizzard's data has ever suggested a consistent 5% advantage for either side in any matchup.
(Note that that presentation was made three months after the game was released. It's now been about twelve months since the game was released. That's a lot of time for things to change.)
|
On August 04 2011 13:19 whacks wrote:I agree. Looking at each race's MMR distribution is a very valid tool for judging balance. If Blizzard came right out & said this is what they were doing, I might even support it. It's so much simpler & more straight forward than their "adjusted-win-rate" and requires almost no number crunching at all. However, I don't think this is what Blizzard is doing. Why? Because their adjusted-win-ratios make the game seem so much more balanced than the MMR distribution. Check out the Zerg distribution in comparison to the Zerg player base. They're skewed so much more heavily towards Plat, Diamond & Master's, compared to the 27% player base.
This is the key problem, as I see it. There's no way (I can think of) to internalise the fact that the distribution of players in each race is not independent of the other factors.
Also, not sure how much to trust that sc2rank data; it show's that all races have a win % above 90 when sorted by region 
With regard to that equation, it looks like maximum likelihood estimation or something similar. Almost definitely is a genuine statistical equation and not some out of place particle physics lol.
|
On August 03 2011 13:07 whacks wrote:
Now going back to the scenario, consider the case where Blizzard releases a new patch which nerfs Zerg and makes it UP relative to both Protoss & Terran (eg, drones now cost 60 min). Consider what will happen to the average Zerg player. He will start losing more than 50% of his games, and his MMR will start dropping. Because of his lower MMR, he’ll start playing against weaker opponents. Eventually, his MMR will stabilize at a level where he starts winning 50% of his future games.
Now let’s say Blizzard had assigned each race a rating as well, to track how “strong they think it is.” Suppose that before the patch, all the races were balanced & had equal rating. Immediately after the patch, because the Zerg population goes through a losing streak, the Zerg rating will drop.
But eventually, the Zerg players will have stabilized their MMR and start winning 50% of their games. At this point, because of the last bullet point in the rating system’s principles (ratings will converge at 50% win rates), the Zerg rating will start increasing again. Remember also that the stabilized Zerg players are playing against opponents of the same MMR, so there’s no way to “account for player skill.” Eventually, the zerg rating will once again converge with the other races, even though Zerg is now UP. No, Zerg players will switch their race, and the imbalance will show up eventually. Blizzard works with that assumption for race adaptability by the players. It's imperfect, but probably optimal for now.
In general, there's no problem with ELO over multiple variables. I think I've posted before, for example, that TL's ELO system needs to include a general race and map balance coefficient too at any given moment. I'm glad Blizzard thinks about these details. Thanks a lot for the video!
|
I've read through the thread, watched the whole video, etc. but I want to highlight some points that have been highlighted already, just to see if some straighter, simpler discussion could come of it.
- A system that ensures a 50% win rating not only in general, but race to race will hide imbalance by virtue of actively seeking that 50% regardless of skill level. This means two players of identical skill with two different races will both be at 50%, but will have very different MMRs if their respective races are imbalanced against one another.
- The law of large numbers means that in such a system, EVERY matchup should be EXACTLY 50%. ANY margin of error lends itself to either race, map or system imbalance. Remember, as the sample size becomes larger, the acceptable margin of error becomes smaller., and to call 2-5% acceptable is silly.
- The point has been made that because of the flaws in the ladder system, we should probably look to tournament results for balance results. Given a large enough sample, and randomly positioned opponents, the percentage of race wins will give an actual measure of balance.
The last point might not be well accepted, but it goes back to the law of large numbers. If you have enough tournament matches, every player will face every other player multiple times, essentially leveling out the skill factor. LZGamer, for example, may balance out Nestea (Losses to Wins), leading to an even 50% Win/Loss across the races. Because matchups are chosen (relatively) randomly, this non-fixed system can actually give us a decent look at balanced percentages of wins.
All of that said, I think it can be fairly well agreed the ladder, with it's system of insuring 50% wins regardless of imbalance vs skill should be abandoned as a method of seeking balance (especially if after millions of games across all races and the given matchmaking system in place, the rates STILL don't balance to 50%).
|
On August 05 2011 04:09 Ihpares wrote:+ Show Spoiler +I've read through the thread, watched the whole video, etc. but I want to highlight some points that have been highlighted already, just to see if some straighter, simpler discussion could come of it.
- A system that ensures a 50% win rating not only in general, but race to race will hide imbalance by virtue of actively seeking that 50% regardless of skill level. This means two players of identical skill with two different races will both be at 50%, but will have very different MMRs if their respective races are imbalanced against one another.
- The law of large numbers means that in such a system, EVERY matchup should be EXACTLY 50%. ANY margin of error lends itself to either race, map or system imbalance. Remember, as the sample size becomes larger, the acceptable margin of error becomes smaller., and to call 2-5% acceptable is silly.
- The point has been made that because of the flaws in the ladder system, we should probably look to tournament results for balance results. Given a large enough sample, and randomly positioned opponents, the percentage of race wins will give an actual measure of balance.
The last point might not be well accepted, but it goes back to the law of large numbers. If you have enough tournament matches, every player will face every other player multiple times, essentially leveling out the skill factor. LZGamer, for example, may balance out Nestea (Losses to Wins), leading to an even 50% Win/Loss across the races. Because matchups are chosen (relatively) randomly, this non-fixed system can actually give us a decent look at balanced percentages of wins.
All of that said, I think it can be fairly well agreed the ladder, with it's system of insuring 50% wins regardless of imbalance vs skill should be abandoned as a method of seeking balance (especially if after millions of games across all races and the given matchmaking system in place, the rates STILL don't balance to 50%).
Holy hell, this post above me is spectacular. I was going to spend at least 20 minutes trying to write up my analysis of it, but instead will only pile on to these excellent points.
~Ladder stats are essentially meaningless. Blizzard's correction may undo the effect of the matchmaking system gradually, but cannot fix the fact that the matchmaking system generated the data in the first place. You cannot analyze the flaw out of a flawed data set.
~~A potential fix would be to turn off the MMR-based matchmaking system for a period of time, and match players on a simple statistic (maybe points, no bonus pool). The resulting data would be strange, but untainted.
~For the design team, balance changes meaning from 'broken' to 'unfun' once the game is developed. As Browder mentioned in the video, strategies become more important than the raw units once the game matures, and designers switch focus ('T wins all games <12 minutes, loses all longer than' would be a balanced, but unfun scenario).
Thus, to Blizzard, balance really is a subjective thing. The stats they throw up will always be "close enough" in their estimation to call sc2 balanced, while player and developer gameplay opinion will drive changes.
|
greatest post of all time concerning balance in sc2. I knew this 100% a while ago but couldnt put it into words as well as you. I explained this to my friends but if they dont want to believe then they wont. basically nomatter what the balance battle.net statistics will always show close to 50% because the goal and intention of bnet is to make everyone at 50%.
|
On August 04 2011 13:46 windsupernova wrote:Show nested quote +On August 04 2011 11:45 Lysenko wrote:On August 04 2011 11:41 Neo.NEt wrote:
Blizz has people a lot smarter than us working on this all day every day... do any of you really think you can sit here knowing .000001% of what they're doing and criticize them like you know what your're talking about? There are certainly people in this forum with deep technical backgrounds. I think it's reasonable for such people to ask tough questions about what they're doing based on that presentation, and some of this thread is just that. That said, people with deep technical backgrounds often jump to conclusions especially when speaking casually, and that's represented in this thread as well. Well, then those people should present their credentials.
Or, you can just use your brain and see if their points make sense.
I dislike this "well they probably know better than us" approach, it leads people to follow them blindly. Their answer during this conference was half joke, half nonsense, yet people seem to believe they have a way to calculate win percentages that completely negates the MMR effect and are a great indicator of balance.
That's just not the case, and tournament results should be taken more seriously than the numbers they give us. Of course since you then have a more limited number of players, you can then have a Nestea/MC Protoss/Korean Terran effect that you have to take into consideration. Ultimately, you have to analyse the game because statistical analysis will only get you so far.
Disclaimer: quantitative finance major (that does include a lot of stats) who actually thinks Blizzard does a very good job with balance, I don't have a problem with that. But I do think the numbers they give us is a way to tell us "shut up and play" and have little to no value.
|
On August 03 2011 14:37 aksfjh wrote:
As long as the results are linearly independent (where basically one equation isn't a multiple of another), you can get answers for x and y. If you then throw in another equation where the answer creates a "contradiction":
3x+5y=7
Why is that a contradiction?
x= 2/3 , y=1
|
|
|
|
|
|
EPT
