|
This is a topic that's been buzzing around my head for a while, so I wanted to get some discussion going on this topic.
SC2 Ladder Analysis: What YOU Need to Know
SC2 Ladder Analysis: Part 2
So far, so good - it is a lot more complicated than it has to be, but there is a reason for it. Blizzard didn't want to force players to be confronted with the fact that they are most likely terrible at this game, so instead of letting people know that they are player 973,412 out of 2 milion, they created leagues and divisions. According to Blizzard then, two players with the same rating have an equal chance of winning a game - that's the definition of rating.
Here comes the tricky part. Blizzard recently posted some statistics about relative win rates within divisions, and used those numbers to draw some conclusions about class balance. Due to their skill-rating system, and due to the match-making system that aims at placing similarly rated players against each other, on average, the win % should always be 50%, and any deviations from that are due to statistical noise.
Note, however, that so far we didn't discuss skill at all. For example, assuming that zerg was underpowered against terran prior to 1.1, the average zerg versus terran match up would still have a 50% relative win rate, simply because the average zerg at a rating of X would be more skilled than the average terran at the same rating. The effect of imbalance would be incredibly difficult to notice through match win %, however, what you should see, is a shifting of the curve at the very high end, where the top tier terrans should have a very high win % against the top tier zerg - which, incidentally is what you saw examining the top 100 or 200 players.
Someone made this argument to Blizzard, and a blue poster replied with:
"The system does account for this to an extent. After analyzing thousands of games and compiling trends based on player skill and race balance, our metrics will attempt to predict who the winner in a given match should be based on player skill and what it thinks is the favored race. These calculations will "correct" themselves over time as trends change.
We do welcome feedback about the announced changes, but there are a considerable number of people outright disregarding the win/loss percentages we provided. Saying we conjured up these numbers to support our own ends, or that our numbers must be incredibly flawed, doesn't really add much to the discussion. The math was developed to disregard the way in which the matchmaking system attempts to keep players as close to a 50% win/loss ratio as possible. As stated in the blog, it's certainly not the only way we analyze balance, but it's very useful to us.
This, to me, seems very hard to understand. Player skill and race balance are not something that can be cleaved from each other, especially if you judge player skill by performance. You can get some insight into player skill and race balance but it requires a lot of assumptions - for example, assuming that the skill distribution is evenly spread throughout the races (*) and the match-up imbalances are constant at all levels of skill (**) then you can end up getting some information about race imbalances by calculating the shift in the rating gaussian from an expected distribution, but you can only really work with the tails of the distribution, and the real skill distribution in a game like SC2 is something that has a very very long tail. Basically, as long as you are working with bayesian inference, you are limited in what you can do by what assumptions you make about the prioris, and it seems to me that a whole set of assumptions are incredibly unsafe to make.
Are there any experts on bayesian inference that would know an analytical way of calculating the expected shift? I was playing around with some paper & pencil and the math gets pretty brutal rather quickly, I think it might be easire to do some kind of simulation with a high number of agents and measure the shift. Even if you did measure the shift though, I am not sure how you'd backpropagate the shift to the PSR system or the MMR system. I doubt the shift is uniform.
* Terrans are more likely than not to have more newbie players due to the campaign mode, and the fact that terran and protoss have by far the most intuitive and least punishing game mechanics.
** For example, 4 gate pushes are an easy to execute strategy that is very hard to defend effectively until mid-high diamond.
|
I'm just going to repost one of my replys from another thread because it kind of fits here too. I just think people are going to have to accept Terran may actually be harder to learn to take advantage of and stop with the ignorant, oh yeah all Terran are just new players from campaign. I bet almost everyone just picked their favourite looking race, I know I did.
What is with people finding crazy ideas as to what these statistics mean. I'm really supsrised no one went for the obvious explanation, it just kind of doesn't make sense untill you think about it abit. This is the way I'm reading the statistics, my opinion is actually differant as I think Terran are easier to play. Also I'm assuming the sample size is big and it's not biased in anyway because that changes everything.
I'm only going to do TvP because it'll just get too complicated for a simple concept even if it doesn't feel right.
The simple answer is Protoss is easier to play at equal skill levels when the skill level is low. But when the skill level evens out to a certain degree at higher levels apparently, Terran start becoming over powered over Protoss. Now think about how close the stats are in PvT at dimond, but how many Terran win tornmanets? Well it makes sense if Terran are worse for less skillful players to play and more of an advantage for more skillfull players to play. This advantage should be widening the higher up you go making Terran easier for very top level play. Also with the statistics if you go into the top percentile it should show more Terran in the top 100 or what ever and if you look at the ranks it does.
Look at it this way too. Terran in my opinion have an advantage early game against Protoss and Protoss have the advantage late game.Lower level players don't understand the importance of early pressure so less of them will do it or if they do not as effective, which could be one of the factors that make Protoss easier to play for lower levels.
|
http://us.battle.net/sc2/en/blog/882511
That post by blizzard seems relevant to this discussion.
They say, "Let's take a look at win percentages on a race-versus-race basis. This is something else we look at to see how matchups are faring over many games. These numbers take individual player skill into account, which helps to avoid the 50% win/loss percentage effect that the matchmaking system can impart on straight win/loss ratios."
Now, I agree with you that there are biases which aren't being removed by analyzing the data this way. The blues, at least publicly, appear fine with using that for making the big balance decisions. I would presume they have some more complicated formulas behind the scenes, but there you have it.
|
Right, but that's exactly what I am getting at Gnial. Individual player skill as separate from individual player skill + racial modifiers is something that is incredibly difficult to obtain, and the way that you choose to obtain it will affect the results that you get when comparing match up win %.
How would you calculate individual player skill?
|
On October 08 2010 19:05 Mearis wrote:
Right, but that's exactly what I am getting at Gnial. Individual player skill as separate from individual player skill + racial modifiers is something that is incredibly difficult to obtain, and the way that you choose to obtain it will affect the results that you get when comparing match up win %.
How would you calculate individual player skill?
Would looking at Random players vs Random players and how those matchups end up be a better way of looking at this?
Given enough games, similarly rated Random players should be similarly skilled regardless of racial imbalances.
|
On October 08 2010 19:09 kojinshugi wrote:Show nested quote +On October 08 2010 19:05 Mearis wrote:
Right, but that's exactly what I am getting at Gnial. Individual player skill as separate from individual player skill + racial modifiers is something that is incredibly difficult to obtain, and the way that you choose to obtain it will affect the results that you get when comparing match up win %.
How would you calculate individual player skill? Would looking at Random players vs Random players and how those matchups end up be a better way of looking at this? Given enough games, similarly rated Random players should be similarly skilled regardless of racial imbalances.
That's a really interesting way of looking at it. As always, when we are working with Bayesian inference though, we'd have to assume that:
- All random players are equally skilled with all races.
- All random players at a given PSR have had an equal distribution of races on the way there
That's really cool though, I am tempted to write a numerical simulation to see under what factors this would hold true.
|
On October 08 2010 18:52 Mearis wrote:
** For example, 4 gate pushes are an easy to execute strategy that is very hard to defend effectively until mid-high diamond.
That's why a Protoss ranked 107 on KR won 7 games in a row on Gisado with the same 4gate all-in. Wait what?!
|
@ OP There are a ton of advanced statistical tools and methods to analyse the data ranging from PCA and other econometric methods. Don't look down at Blizzard when it comes to data analysis and mining. Don't assume they are looking at things face value. The fact you brought up Bayesian is more than enough joke that you just took an undergrad course and starting spewing out crap about the assumptions about skills and distributions etc.
|
On October 08 2010 19:31 tournamentnow wrote:
@ OP There are a ton of advanced statistical tools and methods to analyse the data ranging from PCA and other econometric methods. Don't look down at Blizzard when it comes to data analysis and mining. Don't assume they are looking at things face value. The fact you brought up Bayesian is more than enough joke that you just took an undergrad course and starting spewing out crap about the assumptions about skills and distributions etc.
tournamentnow - bayesian inference is the method that blizzard uses to calculate PSR, as such, it would make sense that they'd incorporate their knowledge about racial imbalances into the priors of the inference
I am actually curious if there are any analytical methods to estimate that prior other than calculating the difference from an expected distribution, which relies on a lot of assumptions.
|
Uhm look at their numbers and look at the changes they are going to make. Terran is even slightly <50% vs zerg yet still this is a patch to help out zerg. Blizz is clearly looking at more than just these numbers and not getting misled.
|
I am not discussing balance. I am curious to know how Blizzard can 'correct' the match up % to account for the effect that PSR has in blurring out any racial differences, I care about the mathematics of this.
|
On October 08 2010 19:31 tournamentnow wrote:
@ OP There are a ton of advanced statistical tools and methods to analyse the data ranging from PCA and other econometric methods. Don't look down at Blizzard when it comes to data analysis and mining. Don't assume they are looking at things face value. The fact you brought up Bayesian is more than enough joke that you just took an undergrad course and starting spewing out crap about the assumptions about skills and distributions etc.
Why not question Blizzard? The current state of balance was exactly due to their data analysis being inaccurate. Like what Tyler mention in the State of the Game Podcast, Idra and Artosis picked Zerg as the main race because they thought it had the highest level of skill ceiling and believe that top players would benefit the most from playing Zerg. Apparently a lot of top Korean players thought the same way which is why the Korean Beta ladder was full of Zergs. At a time when Zerg was dominating, Tester(sks) was able to predict that Terran would be the OP race even without the data that Blizzard has. Blizzard seemed to rely solely on stats as they continued to nerf Zerg and buffed Terran. But this is flawed as they assumed that players who chose Terran tend to have the same skill level as those who chose Zerg. So now when retail came along and good players believe what tester said about OPness of Terran, we saw more good players playing Terran which resulted in the imbalance we see today.
|
On October 08 2010 19:13 Mearis wrote:
- All random players are equally skilled with all races.
Given enough players, this should even out. If we're discussing racial imbalance we're really saying that players with good game sense and good strategy and execution still lose to players with less game sense and worse strategies and execution, because the game actually does not provide them with the tools needed to counter other races. It's highly unlikely that (especially in higher leagues) players' mechanical expertise using the races is that much lower for one randomly picked race.
If there is a real phenomenon, you're going to see trends of in some matchups. That can either be unit weakness or an inherently harder playstyle or something caused by the specific matchup. All are real effects that need to be addressed.
If A are inherently harder or less intuitive to play and therefore a lot of Random players lose more in AvX, then that's also a real problem that needs to be addressed, regardless of whether the outlier is adjusted down or the other two races are adjusted up.
- All random players at a given PSR have had an equal distribution of races on the way there
Again, given enough games, it should be 1/9th for each matchup.
I just think looking at Random players would be far more interesting than looking at single-race players.
|
This is exactly what is bothering me.
How could one even be able to "measure" individual player skill? All you can do is assume, that sucess is linked to player skill. As you guys previously stated: success = playerskill * racemodificator * (racedependent) mapmodificator (* luck). (Of course the enemy playerskill is also important, but let's just say you play vs an AI for the sake of simplicity)
So you can meassure the mapmodificator by letting enough players of one race play on different maps.
What we would really need to determine the racemodificator would be "announced random". A player picks "announced random" and his race is announced to his opponent in the loading screen. Bare in mind, that you can't do some race specific openings vs real random, since you have to decide some BOs before you scout.
If we had enough of these kind of random games, we would see if players had an easier time winning with one race. (also we would need to optimize the experiment: e.g. forcing the players to play x training games with each race first, etc.).
The next best thing is the data from the regular random players. So why didn't they show the race specific data exclusively for random players? That would have been the interesting stuff. (assuming that player skill averages out over the three races, i.e. a players have same APM, creativity and game sense with the three races on average)
|
This is funny from Blizzard.
First they are the ones who say that statistics are not that relevant and must be taken carefully (and they are right here)
Second they expect people to believe them without explaining the mathematics they use, while their way of fixing broken maps is to add destructible rocks.
True this parallel is not relevant, but often enough they proved that they had no clue about balancing, or statistics for that matter. For example everyone said the elemental shaman was UP and broken from design at the release of LK, and people showed numbers that proved it. Blizzard said no, that's not the case. The result : at every new major patch, one or two month later, the shaman received buffs when it was no more deniable that they were too far behind. And each time it was the same story, again and again ...
|
I suspect there's a little killing a fly with a sledge hammer going on here. What are you expecting to get out of the analysis?
|
I'm pretty sure Blizzard is just trying to say "Here are some statistics. But it's not like we're looking at them blindly. We're actually considering different possibilities here without just looking at numbers."
So overanalyzing the statistical part of it seems unnecessary.
|
On October 08 2010 23:02 DoubleReed wrote:
I'm pretty sure Blizzard is just trying to say "Here are some statistics. But it's not like we're looking at them blindly. We're actually considering different possibilities here without just looking at numbers."
So overanalyzing the statistical part of it seems unnecessary.
No, they made a specific claim about adjusting win % with racial imbalances in mind, and I cannot really think of an intelligent way they could do this. Isn't anyone else the slight bit curious about the mathematics behind this?
|
On October 08 2010 19:22 Klive5ive wrote:Show nested quote +On October 08 2010 18:52 Mearis wrote:
** For example, 4 gate pushes are an easy to execute strategy that is very hard to defend effectively until mid-high diamond. That's why a Protoss ranked 107 on KR won 7 games in a row on Gisado with the same 4gate all-in. Wait what?!
Yeah. I think like 5 last of them at least were against Zerg, too. Then he lost to Terran.
|
After taking multiple statistic class in collenge, blizzard can make the race win percentage anything they want. Statistics are all lies, and are scewed to make them see want people want to see
|
|
|
|
|
|
EPT
