|
Excellent post.
I'd like to weigh in on a couple of things. (very long winded, sorry -_-)
First, the "favored" notices on the loading screen are almost certainly based on your display rating compared to your opponent's MMR. This is (as previously mentioned) deceptive, because it is not based on the expected probability that you will win or lose. The evidence for this is that these notices give a preview of points at stake during this game. If you see "even match", you can expect there to be ~10-14 points at stake. "slightly favored" will get you something like -6/10 on a loss or +14-18 on a win, likewise vs a "favored" opponent you can expect -2/6 or +18/22 (all ballpark numbers). If everyone watched for this for a few games, we'd have all the evidence we needed.
About the display rating being "meaningless" or "fluff" or "wrong". Think of it this way: MMR is a quick and dirty search, attempting to locate your current skill level as fast as possible. It is likely to be very inaccurate at times, but is very valuable for quickly separating out the people who are several standard deviations outside the pack. If your MMR is extremely stable, your display rating will eventually converge with it. If (and this is more likely) your MMR tends to jump around quite a bit, your display rating acts as a damping mechanism to give you a much more stable indication of your overall performance level.
The problem is that it might take a hundred or a couple hundred games for the (very damped) display rating to reach the appropriate range for your skill level, which leads a lot of people to freak out about it. Once players get to the 1300-1500 range, expect to see more separation between the top players. That being said, I would like to see bonus pool points stop being accumulated after the ~800-1000 rating mark to take away any incentive to abuse them for ladder whores.
About promotions, I'm pretty confident in the "checkpoint" theory at this point, but it's still a pretty big mystery after the first check. I can understand the logic behind this, especially because the MMR sigmas are ridiculously huge for new players (example: guy with 15 games played on the silly and ill-conceived blizzard top 200 list). It seems pretty well established that the first checkpoint is at ~8 losses, but after that it's pretty fuzzy. To offer a data point from myself, I tanked my placements and landed in silver. After a rocky start, I was promoted to gold after my 8th loss (i was 12-8). A series of wins ended in 4 games in a row vs diamond players, ending in 4 losses, and I went to plat at ~20/12. I'm now matching against ~4-500 rated diamond players and reached my 16th loss, but have yet to be promoted at ~38/16 in plat.
|
Hrm I wonder, Excalibur_Z would it be any use to you for finding the checkpoints if you had a mass quantity of data? I could hookup something to the workers so whenever it sees a team change leagues it saves their points/wins/losses. The only difficulty is that number isn't going to be 100% accurate since they might get promoted and immediately play games.
|
United States12240 Posts
On August 13 2010 07:53 Shadowed wrote:
Hrm I wonder, Excalibur_Z would it be any use to you for finding the checkpoints if you had a mass quantity of data? I could hookup something to the workers so whenever it sees a team change leagues it saves their points/wins/losses. The only difficulty is that number isn't going to be 100% accurate since they might get promoted and immediately play games.
I think that would give us a rough estimate as to where these checkpoints and/or league thresholds might be. It's certainly better than nothing. I'm not entirely sure how we would graph it out, or track match history (since that's very important), but throwing around some arbitrary estimated MMRs and breakpoints we might be able to come up with something.
|
An update, with pictures! Shadowed was kind enough to provide me with a data set of promotions as captured by his data scraper. Thanks to him for making these possible.
Shadowed's scraper runs every so often, and when it detects that a player's league has changed it will record the player's state at that update and the previous update. For example, if the last time the scraper grabbed my stats my W/L record was 10-3, and the current time it grabs it my league has changed and my record is now 17-5, it knows that somewhere in there I had a promotion/demotion. It doesn't know precisely where, just the range of 9 games. I've taken this data and made a set of graphs.
The following graphs are of the combined games played. They include ALL league changes, both promotions and demotions. I will graph them out individually when I'm not half asleep, but I do not think they will vary too much.
The easiest 3 ways to look at this are with the previous games played, the current games played (that is, games played at update time), and the average of the two. Here is a chart combining all 3 across all the data we have currently (there are some hard to see outliers past the 600 games point):
![[image loading]](http://s2.kimag.es/share/98538122.gif)
A smaller slice excluding the long tail that decreases past the end of the image:
![[image loading]](http://s2.kimag.es/share/75530059.gif)
The total dataset is roughly 23,000 promotions/demotions. Past the 100 game mark less than 1,800 promotions/demotions occured.
What inferences can we draw from this data?
- Obviously there are no data points at <5 games due to placements.
- There appears to be a definite checkpoint where the system reviews a player's rating at around 22-26 games.
- There is a bump in the promotions at around 15 games played. I'd need to examine it more when I'm not all sleepy.
- It appears that there are no static (or clustered, anyways) checkpoints after the 25ish check.
- After about 100 games played the system appears to have about 95% accuracy with regard to league placement.
|
Great write up! It has a good mathematical foundation and solid reasoning. I think you are very close to the truth here on a lot of this. Makes me wonder: why doesn't Blizzard disclose a lot more about its rating system? Is it to protect their intellectual property, or for the players?
|
HOLY SH1T
never saw that much geekness together
|
That's ex-wow players for ya, with their neat graphs and indepth maths.
|
wow a big hand for good math!
|
Got a new update to the promotions data. This time the dataset is 56.5k league changes, around twice what we had before.
I'm not going to post the full graph unless someone really wants to see it; it looks the same as the first full graph here and all the interesting stuff happens in the 5-60 games slice.
Updated Graph:
![[image loading]](http://s2.kimag.es/share/84673081.gif)
The results are pretty similar to last time. The bump at the 12-15 game mark is a little more pronounced. After hearing an interesting argument, it's possible there is no checkpoint at 30 games. It's possible that the majority of players have their MMR/sigma criteria fall within certain levels by that point as the system converges quickly.
|
That's cool and all, but I really cant read that. I was just wondering out of curiosity how I still cant get out of 2v2 gold even though me and teammate are 300 points above anyone else in north america.... then again, you did explain it probably, its just your explanation is too technical and confusing. Kind of like the fear of picking up calculus 3.
http://sc2ranks.com/char/us/268968/kingJY
|
I have a question for these mathers, in regards to bonus pool. It may have been answered already.
If the hypothesis that bonus pool is discounted in the matchmaking calculation is true, then in order to reach #1 in your division, wouldn't the best way to go about it be to not play games for a significant period of time before you start playing games?
The question basically is, does your MMR and sigma remain constant while you are not playing games. If that were the case, then by stocking up bonus pool, you would be gaining more perceived points, without increasing the skill level of the people you play. This would allow you to quickly rise in your division without risking a streak of losses to higher skilled players.
I imagine being top in division doesn't matter much to the really hardcore people, but to more casual players it probably does. To that end, giving more casual players an advantage in bonus pool seems a neat trick.
As an edit: I have a little bit of familiarity with GLICKO since I do a lot of tabletop wargaming and we've had leagues governed by GLICKO before, and I seem to remember the uncertainty factor in GLICKO increases if you go a long period of time without playing games.
And a second question, while I think of it.
Does bonus pool accumulate from launch, or from when somebody registers an account? I bought my copy launch day, but if new accounts gain bonus pool from the time the account was registered, it may be much harder for the system to track and discount it. Just a thought.
|
United States12240 Posts
On August 15 2010 23:33 Ketara wrote:
I have a question for these mathers, in regards to bonus pool. It may have been answered already.
If the hypothesis that bonus pool is discounted in the matchmaking calculation is true, then in order to reach #1 in your division, wouldn't the best way to go about it be to not play games for a significant period of time before you start playing games?
I'm not sure why you think that is. The fewer games you have, the greater your MMR volatility. It's not like you would skate up the rankings with a massive bonus pool because whoever is in that #1 position has likely (a) fought all the top players that you would inevitably fight with a volatile MMR and (b) consumed all of his bonus pool. Remember that Bonus Pool replenishes at a constant rate for everyone regardless of their activity level (yesterday when I was playing I actually saw my bonus pool increase by one point at 9:19pm).
The question basically is, does your MMR and sigma remain constant while you are not playing games. If that were the case, then by stocking up bonus pool, you would be gaining more perceived points, without increasing the skill level of the people you play. This would allow you to quickly rise in your division without risking a streak of losses to higher skilled players.
MMR remains constant over periods of inactivity but sigma does not. The system doesn't know if you'll come back from a break refreshed and improved or out of practice and sloppy, so an increase in sigma reflects this. Again, ignoring bonus pool for a moment, if you're at an average level then you'll inevitably play against people that give you a 50-50 record. If most of the time these turn out to be even matches with even results, your rating would stagnate. The bonus pool just prevents this stagnation for active players.
I imagine being top in division doesn't matter much to the really hardcore people, but to more casual players it probably does. To that end, giving more casual players an advantage in bonus pool seems a neat trick.
It's not really an advantage though, because relatively speaking you're not advancing any faster than other players in your division. Say you go 7-0 in your first 7 games, and you're a Gold-level player. After 7-0 you may be facing low-mid Diamond players, so you lose maybe one or two games until you start playing Gold players again. From that point on you'd go 50-50, meaning you'd have a record of 57-52... 67-62... 107-102. You wouldn't be the only person in your division to achieve this, and you'd no doubt see others in your division with similar records. If you were to go inactive for an extended period of time, only the variance in potential opponents increases, so while you may get a Silver player to fight when you return, it's just as likely you'd face a Platinum player, and you'd gain or lose points accordingly.
As an edit: I have a little bit of familiarity with GLICKO since I do a lot of tabletop wargaming and we've had leagues governed by GLICKO before, and I seem to remember the uncertainty factor in GLICKO increases if you go a long period of time without playing games.
And a second question, while I think of it.
Does bonus pool accumulate from launch, or from when somebody registers an account? I bought my copy launch day, but if new accounts gain bonus pool from the time the account was registered, it may be much harder for the system to track and discount it. Just a thought.
From launch. The current bonus pool for newly-placed teams/players is around 330 I believe.
|
Just some random thoughts:
The clustering effect of 15 or 22~26 may be due to learning curve instead of checkpoints.
|
Another proposition to explain why people get stucked in a league:
Say B is in Bronze but has high MMR and can get matched with Platinum players. However, because of high sigma, B is often matched against opponents of various skill levels. Furthermore, because of lack of data (the ladder is less than 1 month old), his opponents often have high sigma, too, which lead to sporadic win and losses that never quite reduce his sigma.
Another explanation is B is playing from a different time zone. (Always "Expanding Searches") 
If these are true, the matching system will improve over time.
|
Vanick
+ Show Spoiler +On August 15 2010 18:20 vanick wrote:Got a new update to the promotions data. This time the dataset is 56.5k league changes, around twice what we had before. I'm not going to post the full graph unless someone really wants to see it; it looks the same as the first full graph here and all the interesting stuff happens in the 5-60 games slice. Updated Graph: The results are pretty similar to last time. The bump at the 12-15 game mark is a little more pronounced. After hearing an interesting argument, it's possible there is no checkpoint at 30 games. It's possible that the majority of players have their MMR/sigma criteria fall within certain levels by that point as the system converges quickly.
I don't know what exactly you raw data looks like but I think it would be interesting to see if their are any visible difference in the different leagues when it comes to premotions/demotions. Does the Diamond graph look signicatently different than the Plat or Gold graph?
With your note about only 1800 premotions over 100 games played. I am a High plat player 65-50, but i am Streaky as hell, for example i have gone 19-8 and 3-12. I think it would be interesting to see if i(or someone else in my position) could get Diamond on a second account in under 30 games played or would i(or someone else in my position) place lower because the number of complete clueless poeple has decreased.
|
United States12240 Posts
On August 21 2010 02:02 Calidus wrote:Vanick + Show Spoiler +On August 15 2010 18:20 vanick wrote:Got a new update to the promotions data. This time the dataset is 56.5k league changes, around twice what we had before. I'm not going to post the full graph unless someone really wants to see it; it looks the same as the first full graph here and all the interesting stuff happens in the 5-60 games slice. Updated Graph: The results are pretty similar to last time. The bump at the 12-15 game mark is a little more pronounced. After hearing an interesting argument, it's possible there is no checkpoint at 30 games. It's possible that the majority of players have their MMR/sigma criteria fall within certain levels by that point as the system converges quickly. I don't know what exactly you raw data looks like but I think it would be interesting to see if their are any visible difference in the different leagues when it comes to premotions/demotions. Does the Diamond graph look signicatently different than the Plat or Gold graph? With your note about only 1800 premotions over 100 games played. I am a High plat player 65-50, but i am Streaky as hell, for example i have gone 19-8 and 3-12. I think it would be interesting to see if i(or someone else in my position) could get Diamond on a second account in under 30 games played or would i(or someone else in my position) place lower because the number of complete clueless poeple has decreased.
While we don't have that data broken down, I have a feeling it probably wouldn't look too much different. If the matchmaking system is doing its job, then everyone will start finding fair games after about the 10th game. After a few matches against evenly-matched opponents it should be able to gauge whether you deserve to be promoted or demoted.
|
Do you have the data broken down into demotions and promotions?
I should have left the personal stuff out, considering it more a random interest than about the matchmaking system.
|
How does bonus pool factor in when somebody is promoted/demoted across division?
If a new player has 300 bonus pool, placement takes them to Platinum, plays through their 300 bonus pool there, and is then promoted to diamond, is their 300 used bonus pool taken account of in their new diamond rank?
It seems like it must be, because if it weren't, then a player who placed diamond would automatically be ahead of a player who placed platinum and was then promoted to diamond.
I am really trying hard to wrap my head around bonus pool, but it seems so strange. If they're not counting it for matchmaking, then it seems totally unnecessary.
Since everybody gains bonus pool at the same rate, all it will do is slowly inflate everybodies points and make it so even with perfect 50/50 win/loss ratios, everybodies points will gradually go up.
I also remember reading something on the Blizzard site at one point saying accounts have a cap on how much bonus pool they can have saved up. Why cap it if it's not being counted?
|
On August 21 2010 06:08 Ketara wrote:
I am really trying hard to wrap my head around bonus pool, but it seems so strange. If they're not counting it for matchmaking, then it seems totally unnecessary.
The bonus Pool Filters inactive players to bottom of the rankings that are displayed. they don't do much else as far was we know.
|
I'm glad we finally know just how garbage blizzard's ladder system is.
|
|
|
|
|
|
EPT
