Blizzard's "skill-adjusted-win-percentages" - Page 5

Lysenko

Iceland2128 Posts

August 05 2011 04:54 GMT

#81

On August 05 2011 13:51 kckkryptonite wrote:
It's funny they put up some supposedly insane math equation (it's not, you need one year of calculus to understand it), but they don't tell you what anything represents; theta, beta, gamma? Equations are meaningless if that information is omitted.

It takes a year of calculus to read it, understanding it either requires some statistical knowledge, or they were joking.

In any case, it clearly wasn't presented to convey information, and if it is, as someone else above speculated, something related to a maximum likelihood calculation involved in the matchmaking system, it's quite likely that none of the people speaking on that stage were fully up to speed on the statistics involved.

kckkryptonite

1126 Posts

August 05 2011 05:07 GMT

#82

On August 05 2011 13:54 Lysenko wrote:

Show nested quote +

It takes a year of calculus to read it, understanding it either requires some statistical knowledge, or they were joking.

If you can read it, you can at the very least interpret it. It's like you're saying you might be able to read Korean, but not understand it. Not divulging more than some arbitrary equation clearly tells us that Blizzard doesn't want to tell us anything other than imply they have a big magical formula done by 5 PhDs.

+ Show Spoiler +

Or a monkey.

Drowsy

United States4876 Posts

August 05 2011 05:14 GMT

#83

ohhhh my god David Kim is sooo cute.

I didn't realize this was Oct 2010 at first. I was shocked at the stats they were going over.

Alyoshka

United States10 Posts

August 05 2011 05:15 GMT

#84

On August 05 2011 13:51 kckkryptonite wrote:

Show nested quote +

It's funny they put up some supposedly insane math equation (it's not, you need one year of calculus to understand it), but they don't tell you what anything represents; theta, beta, gamma? Equations are meaningless if that information is omitted. Lol at Blizzard trying to appear transparent, patronization at its best, imo.

"We'll spare you the details, but these are the percentages", sketchy.

you need more than one year of calculus to understand this type of equation. Cal I/II don't even sniff DEs. All the posts in this thread are total crap except those which point out that <1% of the population comprehends the math involved. The details aren't important, because it works. I am amazed at the number of numbskulls who piss and moan about this or that policy from blizzard without doing any anything to contribute to the solution side of things.

Math is extremely hard, the kind of programming talent Blizzard can hire, while not Google/MSFT/AAPL level, is incredibly high. Just be glad that the smartest guys in the gaming industry are working on the IP you love.

citation: http://www.animationarena.com/video-game-salary.html (on Blizz leading the way for pay, which in turn allows them to leverage top talent)

actual data of the survey: https://spreadsheets.google.com/ccc?key=0Aou3k7ExaTQjdHZ0S2dKMjhfY0lmN2tmTDRESEhjbHc&hl=en&authkey=CNDxyJwF#gid=0

wiki post on DiffE: http://en.wikipedia.org/wiki/Differential_equation

LagT_T

Argentina535 Posts

August 05 2011 05:31 GMT

#85

On August 05 2011 13:51 kckkryptonite wrote:

Show nested quote +

Greek letters have defined meanings in statistical analysis, they are not unpredefined variables like for example the common latin/roman letter "x".

Disquiet

Australia628 Posts

August 05 2011 05:34 GMT

#86

On August 05 2011 14:15 Alyoshka wrote:

Show nested quote +

I agree he is wrong. even without having the variables defined you can still tell what the thing does if you can understand it. Everyone can recognize the formula for a parabola without knowing how x or y is applied.

Lysenko

Iceland2128 Posts

August 05 2011 05:44 GMT

#87

On August 05 2011 14:34 Disquiet wrote:
I agree he is wrong. even without having the variables defined you can still tell what the thing does if you can understand it. Everyone can recognize the formula for a parabola without knowing how x or y is applied.

Actually, when it comes to statistics, that's not true. Here's why: every technique that exists for analyzing data sets statistically has implicit assumptions that must be true before the technique tells you what it purports to.

Here's an example: The mean and standard deviation have different meanings if you're looking at a normal vs. a Poisson distribution. When is it appropriate to presume each type of distribution? The answer isn't found in the mathematical equations that describe the distributions' shapes!

Perseverance

Japan2800 Posts

August 05 2011 05:52 GMT

#88

I wonder how blizzard feels about their game now.

Hikari

1914 Posts

August 05 2011 05:58 GMT

#89

Rewatched the vid:
The powerful terran early game is still a problem which DB recognizes. Yet it is not something easily fixable without nerfing mid/late game. By design the race is like that and I think we will have to wait til HotS for the problem to truly be "fixed".

kckkryptonite

1126 Posts

August 05 2011 06:01 GMT

#90

On August 05 2011 14:15 Alyoshka wrote:

Show nested quote +

No, you don't. The level of ignorance in your post is astonishing, after the first year of Calc, you are be able to solve basic differential equations (in my curriculum at least). Pouring money into something must mean it's the best right (US HEALTHCARE/EDUCATION)? To top it off you cite wikipedia.

On August 05 2011 14:15 Alyoshka wrote:
All the posts in this thread are total crap except those which point out that <1% of the population comprehends the math involved.

Really? WTF. There are derivatives and integrals, fractions and exponents. Seriously? 1%? Where did you get this number? How are you coming to your various conclusions?

Actually, the variables are assumed to be defined as the set of all real numbers.

W/e guys, I'm not gonna get into a math debate with people who know how to use google and wikipedia.

Slago

Canada726 Posts

August 05 2011 06:03 GMT

#91

it seems there main concern is making a player satisfied and happy by giving them the 50% win ratio, but aren't all that concerned with how it misconstrues balance data, so their risking some more reliable numbers for a better experience for the consumer, probably a good move by them

Lysenko

Iceland2128 Posts

August 05 2011 06:09 GMT

#92

On August 05 2011 15:03 Slago wrote:
it seems there main concern is making a player satisfied and happy by giving them the 50% win ratio, but aren't all that concerned with how it misconstrues balance data, so their risking some more reliable numbers for a better experience for the consumer, probably a good move by them

This idea that a 50% individual win ratio is not compatible with a statistical data set that can be used to detect race-wide imbalances is just not true, no matter how much people say it. That individuals are being matched against other individuals with whom they have a 50/50 chance to win says nothing about the information that the overall distributions can communicate.

aksfjh

United States4853 Posts

August 05 2011 06:13 GMT

#93

On August 05 2011 13:51 kckkryptonite wrote:

Show nested quote +

This is a cleaned up version:

At the very least, you can infer their equation takes into account standard deviation. Beyond that, there is more there than something "one year of calculus" can help you understand. You might know the symbols and can figure out the "answer" with the numbers, but one year of calculus will never prepare you for what is actually achieved by the equation itself. I would break some of it down for you, but I'm tired of battling morons in this topic who are here for no other purpose than blindly bashing Blizzard for having foresight far beyond anything they can imagine themselves.

In fact, I'm somewhat surprised that this topic is still alive since the TC all but said outright that he doesn't understand how the system can be used and if it strives to match people on a 50-50 basis, then data from those matches is absolutely worthless. The ONLY person I have seen in this topic with any sort of clarity and education is Lysenko. Because of this, I find it worth the time and effort to address a point he is making:

On August 05 2011 14:44 Lysenko wrote:
Actually, when it comes to statistics, that's not true. Here's why: every technique that exists for analyzing data sets statistically has implicit assumptions that must be true before the technique tells you what it purports to.

Here's an example: The mean and standard deviation have different meanings if you're looking at a normal vs. a Poisson distribution. When is it appropriate to presume each type of distribution? The answer isn't found in the mathematical equations that describe the distributions' shapes!

There is indication that Blizzard has made an attempt to correct some of their statistical analysis back at the end of season 1 and beginning of season 2. We went through this shift and scaling of MMR as well as a "fixing" of certain league thresholds because the actual MMR distribution didn't show a Gaussian curve. As far as active population goes, it's assumed that the system actually produced a log-normal pdf, where players who played more often were ending up with a better MMR on average than players who didn't play as much. In essence, casual players would be "feeding" MMR to people who played more, and since there are more casual than hardcore players, a smaller population was ending up on the high end of the scale compared to the low end.

Because of this corrective action in league assignments, we can at least assume that they are constantly watching and correcting equations to measure data obtained through ladder.

OhMyGawd

United States264 Posts

August 05 2011 06:14 GMT

#94

On August 03 2011 13:36 Snaphoo wrote:

Show nested quote +

Even in terms of position. ZvT close positions on Shattered versus far positions, for example, has got to be pretty skewed.

The saddest part about Close Position maps is that even at a High Masters level i have a higher win % on close pos (about 90%) then on cross positions.

This is mainly because terrans do all'ins or cheesy plays which are easy to stop with proper mechanics.

I don't know if i'm the only on out there.

Lysenko

Iceland2128 Posts

August 05 2011 06:27 GMT

#95

On August 05 2011 15:13 aksfjh wrote:
Because of this corrective action in league assignments, we can at least assume that they are constantly watching and correcting equations to measure data obtained through ladder.

That's an interesting change that I'd failed to understand the importance of at the time. Thanks for pointing that out.

Edit: The reason I'd thought that was a quantum-mechanical equation back at Blizzcon is that during the short time they flashed it up on the screen I thought I saw imaginary exponents in there. On looking again right now, I notice that that's not the case.

MilesTeg

France1271 Posts

August 05 2011 09:59 GMT

#96

On August 05 2011 13:54 Lysenko wrote:

Show nested quote +

It takes a year of calculus to read it, understanding it either requires some statistical knowledge, or they were joking.

They were clearly joking...

I really don't think they have any way of calculating a "clean" win percentage, as I said before it's just a way for them to send the right message to the players (stop caring about balance, just play).

On August 05 2011 15:09 Lysenko wrote:

Show nested quote +

Maybe, but that's not at all what Blizzard was trying to say, and has nothing to do with the numbers they gave us which are obviously meaningless.

bmn

886 Posts

August 05 2011 10:24 GMT

#97

On August 05 2011 11:54 Lysenko wrote:

Show nested quote +

But the distributions will be different over a large number of players in the imbalanced case. If it's harder to win as Zerg, Zerg players will have systematically lower MMRs, and that'll be visible in the distributions, unless you can make a convincing case that worse players choose Zerg.

I don't get why you keep harping on this -- do you have any convincing case that we should believe the race choice to be skill-independent?

There is absolutely nothing symmetric that would lead to any obvious justification here.

Terran is the only race you really play in the campaign. Why would that not have any effect on how many people, and how many new people (who did not play any other race in SC/BW) choose that race? That will be the only race they have serious experience with from single-player, and it's clear that transitioning from T to P/Z is far from easy if you never played P/Z.

Why do you think that all three races are equally frustrating for low-level players? Is it so? I don't know, but there's no symmetry that would justify this assumption a priori. Two races can easily wall off, one cannot. Two races can easily start playing with certain builds which require little scouting or other adaptation (e.g. 4gate and 3rax), that is _very_ helpful for beginners to get a handle of the race; as Zerg you need to play differently to learn how to play the race.

As Protoss you may lose to many early cheeses if you don't 1-base forge (which won't get you far) and if you aren't very careful about how you spend your chronoboosts (to get sentries if needed) early on. As Terran it's a lot easier to survive if you have a wall, since you can build a bunker regardless of whatever else you were doing, and you already have a rax anyway to fill the bunker.

As Terran your scans cannot be stopped, and they don't require any scouting by a unit, which may or may not make it a lot easier and/or more palatable for a disproportionate (more than 1/3rd) number of people. As Zerg you can try to 6pool every game, which may or may not work at low levels, but this is a significantly different strategy from 2raxing or 2gating in terms of how it needs to be stopped.

All these things are examples of how the races play differently in ways which may very well affect which players (categorized by skill level) choose to play those races. What is your evidence that none of these actually matter?
It is not reasonable to assume an a priori distribution of players that is equal across all races, *especially* given that early on (beta) certain strategies were often clearly dominant and abusive (roaches vs p at 1 supply, reapers before rax and speed nerfs) -- which may also make players switch races.

[To people other than Lysenko: This is not a balance whine, and don't bother replying if you have nothing to add to the actual point.]

Finally, actual quantitative data: http://sc2ranks.com/stats/all/1/all
(This is just the first link I came across, there may be much better numbers. I don't know of any, feel free to reply with them if you do.)

Race distribution by league in bronze: 8.1% random, 32.7% protoss, 39.2% terran, 20.0% zerg.

For every bronze Zerg, there are two bronze Terrans.

If this isn't strong enough evidence to suggest that assuming a priori that all races have roughly equal MMR distributions is not justified, I don't know what is.

Lysenko

Iceland2128 Posts

August 05 2011 10:37 GMT

#98

On August 05 2011 19:24 bmn wrote:
If this isn't strong enough evidence to suggest that assuming a priori that all races have roughly equal MMR distributions is not justified, I don't know what is.

I don't think that you said what you meant to say. Clearly I'm not assuming a priori that all races have roughly equal MMR distributions -- in fact I'm saying that differences in MMR distribution can yield useful balance information. I am suggesting that it's useful to assume that those differences in MMR distribution come from something else than, say, players of X race being systematically worse than players of Y race.

I'm not even saying that this is the analysis Blizzard is doing -- what I'm saying is that the fact that one can extract useful information from those distributions demonstrates the falsehood of saying that a matching system that tries to make every individual game a 50/50 matchup cannot provide information about differences between races, a falsehood that keeps getting repeated here.

Incidentally, while it's interesting what percentage of players in bronze play each race, that's not that useful by itself either -- you'd at least have to normalize the distributions to account for racial preferences across all skill levels.

And, after doing ALL that, maybe you see there's a bulge in the Terran distribution around bronze, and you match that up to people who have shared their experience of playing the game a little bit coming out of the campaign and then quitting the game. Still, there are ways to test that with the data Blizzard have available, to weed out people who don't stick with the game, for example, or look at the subset who have switched races at least once, or whatever.

Nobody said, least of all me, that interpreting this data would be clean, or could be done without applying qualitative judgement. That said, the idea that the aggregate data is useless for this purpose because the matchmaking system automatically seeks out 50/50 matches for players is simply stupid. There's information there. Sure, interpreting it requires checking it against other sources of information, such as player feedback and the developers' own gut feelings. They said exactly that in the video that's been posted in this thread. It doesn't mean the data's useless -- it's just one input to a much bigger process.

Incidentally: the idea that the matching system exists to provide useful balance data makes me shake my head. That is obviously not the case (since the matching system exists for the sole purpose of ensuring that as few players as possible lose more than half their games).

Final thought: We don't really know whether the MMR information they track is a scalar at all. MMR probably doesn't have different values per matchup because of how they do promotions and the relationship of point scores to MMR, but they might track multiple sigmas per player race or per matchup. This or something like it might explain the multiple parameters in that equation they briefly presented at Blizzcon, and if true would possibly provide a hook by which to come up with race-adjusted skill estimates that might be very useful.

Lysenko

Iceland2128 Posts

August 05 2011 11:13 GMT

#99

On August 05 2011 12:33 Xevious wrote:
My question is why do they think statistics from lower leagues matter as much as higher ones? To but it bluntly, everyone is so bad in bronze that race imbalances aren't what's going to decide the winner, plain and simple (contrary to plat and above, where people don't get supply blocked at 11).

I don't think this is really Blizzard's attitude. If you look at the balance changes they've actually made, the vast bulk are to tweak issues at the high end, and very few are to address issues in lower leagues, and the one or maybe two I can think of where they mentioned low level players exclusively concerned how easy it was to execute certain rushes at the very start of the game.

On the basis of that, I suspect they're quite OK with some imbalance in the lower leagues, but they really don't want things to be completely out of whack (like 70/30 in some matchup) at any level of play, because they'd like all the races to be at least somewhat accessible to new players.

(Incidentally, they did say explicitly at one point that they were OK with Zerg being somewhat harder to learn for totally new players. This by itself suggests that lower league imbalances are a lesser concern than higher league imbalances.)

bmn

886 Posts

August 05 2011 11:29 GMT

#100

On August 05 2011 19:37 Lysenko wrote:

Show nested quote +

But won't this assumption just bite you in the ass if it turns out to be wrong?

As you said earlier, you can't trust your results if any part of calculating them relies on assumptions that may not be true.

I'm not even saying that this is the analysis Blizzard is doing -- what I'm saying is that the fact that one can extract useful information from those distributions demonstrates the falsehood of saying that a matching system that tries to make every individual game a 50/50 matchup cannot provide information about differences between races, a falsehood that keeps getting repeated here.

Yes, you're right about that. I'm more interested in whether Blizzard actually extracts useful information than whether it's possible do so, though -- I doubt what they're doing is anywhere near the theoretical limits of what's possible :-)

Incidentally, while it's interesting what percentage of players in bronze play each race, that's not that useful by itself either -- you'd at least have to normalize the distributions to account for racial preferences across all skill levels.

Yes, but if you look at the rest of the page, you'll see that Terrans are not overrepresented by such a huge margin anywhere other than bronze league (at least not up to Grandmaster). That's why I quoted the number.
(Unless I'm misunderstanding what you're saying here...)

And, after doing ALL that, maybe you see there's a bulge in the Terran distribution around bronze, and you match that up to people who have shared their experience of playing the game a little bit coming out of the campaign and then quitting the game. Still, there are ways to test that with the data Blizzard have available, to weed out people who don't stick with the game, for example, or look at the subset who have switched races at least once, or whatever.

It's possible to work around that, I never disputed that. I'm disputing that it's simple, and I'm questioning whether Blizzard actually does all that is necessary.
Sure, you can explain it by saying that this is people who just played the game a little bit and then quit -- my point is that this is just one example of how people of one race can indeed be systematically worse than people of other races. This was just an example to demonstrate that this *does* happen, and this is the most trivial example.
What about the population that keeps playing, then, and doesn't just stop playing after a little bit? There is a myriad of systematic biases I can think of which sound plausible, and there's no easy way to eliminate those from the statistics.

Gut feelings and player feedback are nice data points, but they're not really that great if you're trying to understand systematic skill differences in player skills across a range of skill levels, given that you already don't know how to measure player skill in a fair, race-agnostic way -- so as far as I can tell it'll usually boil down to you making a certain set of assumptions, then fudging the statistics until it looks like they reasonably support your assumption. (This isn't saying that they intentionally do this, but there's no benchmark to verify that their way of measuring player skill across races is accurate.)
And even if you're actually wrong, as long as you're not horribly far off, nobody can contradict you -- only you have the data, only you know the assumptions you made, only you know how you crunched the numbers.

Incidentally: the idea that the matching system exists to provide useful balance data makes me shake my head. That is obviously not the case (since the matching system exists for the sole purpose of ensuring that as few players as possible lose more than half their games).

I don't see anyone suggesting that the match-making system is designed to balance the races.

The outcomes of the matching system is the only data that users have to go by, so it's unavoidable that they will judge balance based on those numbers. What other choice is there beyond blind faith?

Also, why do you shake your head at that? You yourself defend the statement that the match-making system's outcome allows providing information about game balance, and it seems pretty clear that the match-making system's outcomes are used by Blizzard to keep track of balance; whether that was the original purpose of the system is entirely irrelevant in this context.

Final thought: We don't really know whether the MMR information they track is a scalar at all. MMR probably doesn't have different values per matchup because of how they do promotions and the relationship of point scores to MMR, but they might track multiple sigmas per player race or per matchup. This or something like it might explain the multiple parameters in that equation they briefly presented at Blizzcon, and if true would possibly provide a hook by which to come up with race-adjusted skill estimates that might be very useful.

Now you're just muddying the waters by saying that we can't possibly know what they do, so we have no reason to criticize anything they say, because maybe they're doing something brilliant and we'll never know.
That's a cop-out :-)

Prev 1 2 3 4 5 6 7 Next All

Please or register to reply.

Blizzard's "skill-adjusted-win-percentages" - Page 5

Completed

Ongoing

Upcoming