EPTSC2 Ladder Analysis: Part 2 - Page 14
| Forum Index > SC2 General |
|
KillerDucky
United States498 Posts
| ||
|
Mendelfist
Sweden356 Posts
On September 29 2010 00:48 Excalibur_Z wrote: We still think it's MMR-sigma*3 and nothing having to do with actual displayed rating. We can prove this by looking up points across all divisions on SC2Ranks, where some Bronze players have far more points than we would expect the threshold to be. There's a Bronze player with 2400 points, for example, one with 1900, some with 1800. The only way we could determine the league thresholds would be to have full match histories of one player as well as every player he encountered on the way to his promotion and every player they have encountered as well. I examined some EU bronze divisions and for the top 15 players in each division I subtracted those players spent bonus pools. I expect that the resulting numbers are close to or at least somewhat related to the MMR for each player. Here are those numbers sorted from highest to lowest: Thor Zed: 375 366 362 319 302 297 289 287 279 216 184 182 172 113 33 Tal'darim Nu: 366 360 343 327 320 316 292 282 282 264 244 228 208 205 112 Azimar Sierra: 757 519 499 493 489 466 466 438 402 400 369 364 332 325 303 What's going on here? Why are the numbers for the last division so high? Why haven't they been promoted? I mean, its hardly likely that everyone in a randomly picked league have an unusually large sigma. What am I missing? | ||
Excalibur_Z
United States12235 Posts
On October 02 2010 00:13 Mendelfist wrote: I examined some EU bronze divisions and for the top 15 players in each division I subtracted those players spent bonus pools. I expect that the resulting numbers are close to or at least somewhat related to the MMR for each player. Here are those numbers sorted from highest to lowest: Thor Zed: 375 366 362 319 302 297 289 287 279 216 184 182 172 113 33 Tal'darim Nu: 366 360 343 327 320 316 292 282 282 264 244 228 208 205 112 Azimar Sierra: 757 519 499 493 489 466 466 438 402 400 369 364 332 325 303 What's going on here? Why are the numbers for the last division so high? Why haven't they been promoted? I mean, its hardly likely that everyone in a randomly picked league have an unusually large sigma. What am I missing? I don't have an EU account so I can't look in-game, but did you take a look at their match histories? Looks like the first guy in Azimar Sierra has few games played and is losing few points and gaining many points for losses and wins, respectively. He probably bombed his initial placement matches intentionally and now he's won enough to play against Gold/Plat/Diamond players. I don't think I'd say he's stabilized yet, particularly if he's making such a huge leap in leagues. The second guy has played more games which means he's probably close to stable, and he's winning and losing the same amount of points per outcome which means that's probably where he belongs. As for that division having more points in general, that's a good question. I'd say that it's probably older, but I don't know what that would mean because we're speaking in terms of adjusted ratings. Are some divisions actually "better" than others inherently, beyond the caliber of players that reside there? That sure would make things confusing because we've been operating on the theory that points between divisions are comparable. Based on Bashiok's recent quote that people who are affected by the Bronze Zero bug will be moved into a new division, maybe there really is more to divisions than simply acting as a bucket to toss 100 random players. I'm having a hard time pinpointing what their role would be. | ||
|
J7S
Brazil179 Posts
Great job. | ||
|
Mendelfist
Sweden356 Posts
On October 02 2010 00:56 Excalibur_Z wrote: I don't have an EU account so I can't look in-game, but did you take a look at their match histories? Looks like the first guy in Azimar Sierra has few games played and is losing few points and gaining many points for losses and wins, respectively. He probably bombed his initial placement matches intentionally and now he's won enough to play against Gold/Plat/Diamond players. I don't think I'd say he's stabilized yet, particularly if he's making such a huge leap in leagues. The second guy has played more games which means he's probably close to stable, and he's winning and losing the same amount of points per outcome which means that's probably where he belongs. As for that division having more points in general, that's a good question. I'd say that it's probably older, but I don't know what that would mean because we're speaking in terms of adjusted ratings. Are some divisions actually "better" than others inherently, beyond the caliber of players that reside there? That sure would make things confusing because we've been operating on the theory that points between divisions are comparable. Based on Bashiok's recent quote that people who are affected by the Bronze Zero bug will be moved into a new division, maybe there really is more to divisions than simply acting as a bucket to toss 100 random players. I'm having a hard time pinpointing what their role would be. No, I didn't look at their match histories. I certainly can understand a few outliers, for example the first few people in Azimar Sierra, but as you can see the average difference between the first two and the last division is large, and I did pick them randomly (I clicked around randomly on peoples profile pages). I also could not get any clues by looking at the age of the divisions. None of them are new. This bothers me, because the theories you guys have come up with here make a lot of sense. I really thought you had most of it figured out. :-) My idea with all of this was to gain some insight in the promotion thresholds of the leagues, even if the sigma thing makes it much harder to do that. If the divisions are not equal however, it screws everything up. | ||
|
verne
United States43 Posts
| ||
|
Excalibur_Z
United States12235 Posts
On October 02 2010 01:16 Mendelfist wrote: No, I didn't look at their match histories. I certainly can understand a few outliers, for example the first few people in Azimar Sierra, but as you can see the average difference between the first two and the last division is large, and I did pick them randomly (I clicked around randomly on peoples profile pages). I also could not get any clues by looking at the age of the divisions. None of them are new. This bothers me, because the theories you guys have come up with here make a lot of sense. I really thought you had most of it figured out. :-) My idea with all of this was to gain some insight in the promotion thresholds of the leagues, even if the sigma thing makes it much harder to do that. If the divisions are not equal however, it screws everything up. We don't have any inside information, we're just trying to piece together how it works just like everyone else. That's why we're constantly appealing to the community for more information. This is Vanick's take on the matter: "Is it possible that the lower divisions have players who have not stabilized at a rough 50-50 record yet? That is, it is only likely that they are playing at the proper MMR when their record is about 50-50, and they have played >50 games. The number of games played is slightly arbitrary, but in general 50 games should be enough for them to be at their level if they're Bronze." He agrees that there is probably more to divisions, which only serves to complicate things. I suspect we'll find out more once Blizzard rolls out their hotfix to Bronze Zero players. | ||
|
pzeta
Spain106 Posts
 | ||
|
kojinshugi
Estonia2559 Posts
On October 02 2010 01:37 verne wrote: Is there a TLDR version of this is English ? Thx Your signature is highly apt. TLDR version: If you win you gain MMR rating. If you lose you lose MMR rating. The amount you lose or gain is dependent on the MMR value of your opponent (win against higher rated opponent - gain more; loss against higher rated opponent - lose less). The more games you play the more confident the system is that your MMR value is accurate. This confidence is likely based on a number of recent games, if your performance improves over 50 games you will be promoted even if your performance was static for 200 games prior. You are matched against opponents who are near your MMR value if the system's confidence is high. You are matched against opponents who are above your MMR value if the system's confidence is low. (MMR = matchmaking rating, this is hidden and doesn't depend on your displayed "points") If that was also TLDR: It's magic. | ||
|
dingoman
Canada12 Posts
As you play more games, that uncertainty decreases and the system is more “confident” in the rating it has assigned to you Why is that true? I don't like how it's assuming that everyone has a "natural" skill level and thus as you play more games the engine gets more accurate. However, people might change so why would sigma decrease? The probably aren't transitive. But to use the Bayesian method, I think you need to have them to be transitive. What does it mean to be transitive? If A beats B with a 90% probability, B beats C with 90%, then A must beat C with over 90%. Well, obviously in SC2 some build orders work against some type but not others. So it's possible that A beats B, B beats C, but C also beats A. In other words, if the engine says that B is better than C because he won more games, and A is better than B, then the engine must say that A is better than C. But that's not the case in SC2. | ||
|
c0ldfusion
United States8293 Posts
Why is that true? I don't like how it's assuming that everyone has a "natural" skill level and thus as you play more games the engine gets more accurate. However, people might change so why would sigma decrease? Sigma never actually goes to 0. The fact that it is initially very large and decreases somewhat after a few games is that when you first start out, the system really has -no- idea how good you are. In other words, if the engine says that B is better than C because he won more games, and A is better than B, then the engine must say that A is better than C. But that's not the case in SC2. But the game has to be transitive on a pure skill basis if it is assumed to be perfectly balanced in every match-up. Whether or not SCII is currently balanced, certainly the match making system has to imply transitivity. | ||
|
dingoman
Canada12 Posts
On October 05 2010 07:41 c0ldfusion wrote: Long time lurker here, decided to register to respond to this post.. Sigma never actually goes to 0. The fact that it is initially very large and decreases somewhat after a few games is that when you first start out, the system really has -no- idea how good you are. But the game has to be transitive on a pure skill basis if it is assumed to be perfectly balanced in every match-up. Whether or not SCII is currently balanced, certainly the match making system has to imply transitivity. I've never talked about sigma going down to 0. I'm simply saying that perhaps it shouldn't even decrease. But I think the transitivity is a bigger issue. Perhaps at the tournament level of gameplay, it's transitive, but for sub-Platinum players it's probably not transitive. I can give you an example right now. Player A: Overall good macro and unit composition, but does not scout at all during the first 5 minutes Player B: Very good at cannon cheese, but have no idea how to play the game if his cheese fails (bad macro, no idea of unit composition, bad economy) Player C: Good macro (just as good as A), scouts, and good at dealing with cheeses Now, suppose A plays with B. B wins because A is bad against cheese strategies (if he doesn't scout, then he'd probably discover the cheese way too late). If I were to ask you, who would have the highest chance against C, then without knowing their playing styles then you'll probably pick B. Afterall, B had won a game against A. Well obviously given their playing styles, A would have a better chance of beating C (even though he's still worse). However, the system would probably make the opposite prediction given the results of the first game. That's an example of a non-transitivity. There are many other examples as well. | ||
|
NyuNyu
Canada146 Posts
| ||
|
radiatoren
Denmark1907 Posts
On October 02 2010 00:56 Excalibur_Z wrote: I don't have an EU account so I can't look in-game, but did you take a look at their match histories? Looks like the first guy in Azimar Sierra has few games played and is losing few points and gaining many points for losses and wins, respectively. He probably bombed his initial placement matches intentionally and now he's won enough to play against Gold/Plat/Diamond players. I don't think I'd say he's stabilized yet, particularly if he's making such a huge leap in leagues. The second guy has played more games which means he's probably close to stable, and he's winning and losing the same amount of points per outcome which means that's probably where he belongs. As for that division having more points in general, that's a good question. I'd say that it's probably older, but I don't know what that would mean because we're speaking in terms of adjusted ratings. Are some divisions actually "better" than others inherently, beyond the caliber of players that reside there? That sure would make things confusing because we've been operating on the theory that points between divisions are comparable. Based on Bashiok's recent quote that people who are affected by the Bronze Zero bug will be moved into a new division, maybe there really is more to divisions than simply acting as a bucket to toss 100 random players. I'm having a hard time pinpointing what their role would be. Correct about Azimar Sierras number 1. Guy two has had an 8 game winstreak but is still playing bronze/silver = his relatively stable at sigma, though seems to have improved i.e. learned a good tactic. Games among top 15 in the divisions: Thor Zed: 2094 Tal'darim Nu: 2555 Azimar Sierra: 2407 Quite a gap between Thor Zed and Tal'darim Nu with no reasonable explaination! Isn't what is causing the inflation for Azimar Sierra though. Edited in: What is interesting though: Win% +- 1 standard diviation: Thor Zed: 51 % +- 2.2 Tal'darim Nu: 50 % +- 2.9 Azimar Sierra: 53 % +- 5.9 Those data are pretty interesting: Azimar Sierra has a good deal of overperformers pushing the numbers up and blurring that the rest of the players in top of that division have played more games than the other divisions and thus they have had more time to learn the game/get to their MMR. | ||
|
Excalibur_Z
United States12235 Posts
On October 05 2010 09:55 radiatoren wrote: Correct about Azimar Sierras number 1. Guy two has had an 8 game winstreak but is still playing bronze/silver = his relatively stable at sigma, though seems to have improved i.e. learned a good tactic. Games among top 15 in the divisions: Thor Zed: 2094 Tal'darim Nu: 2555 Azimar Sierra: 2407 Quite a gap between Thor Zed and Tal'darim Nu with no reasonable explaination! Isn't what is causing the inflation for Azimar Sierra though. Edited in: What is interesting though: Win% +- 1 standard diviation: Thor Zed: 51 % +- 2.2 Tal'darim Nu: 50 % +- 2.9 Azimar Sierra: 53 % +- 5.9 Those data are pretty interesting: Azimar Sierra has a good deal of overperformers pushing the numbers up and blurring that the rest of the players in top of that division have played more games than the other divisions and thus they have had more time to learn the game/get to their MMR. It's entirely possible that it's blind luck that better players got into Azimar Sierra than the other two divisions. A random collection of players isn't going to be even. It's also possible that there is some division weighting. I wouldn't conclude it as inflation by any means, but we'll need more data in order to determine whether there is any kind of weighting used in division placement. | ||
|
Ironbound
3 Posts
| ||
|
Excalibur_Z
United States12235 Posts
On October 06 2010 17:08 Ironbound wrote: I've played over 1000 league 1v1s, and am STILL rank 1-2 for Platinum. According to this, it will take me many, many more games before I get promoted. ... No, not necessarily. Look at your match history and opponent profiles. | ||
|
BigDatez
Canada434 Posts
User was temp banned for this post. | ||
|
Matrim
United Kingdom16 Posts
Apologies for the long post. I grew interested in the points a few weeks back and off my own back worked out that the div displayed bonus matched the used bonus. I posted (on the eu boards) my own attempt to see if this could match the 'hidden' MMR in a post here http://eu.battle.net/sc2/en/forum/topic/566442621#19 though my original ideas look a tad simplistic now After battling though people thinking I was saying the bonus pool is used and relating the current state of the 'knowledge' on the topic I ref'd back to your original post and found your update on the original and this thread. At this point I decided to track my own data through the system to see if this would help.I rapidly found that promotion/demotion drastically affected my opponents stats clouding the waters somewhat though some interesting facts could be gleaed from my own. For example you will notice that when I hit 150 (or possibly 15,000) I started hitting Silvers. Some of the names are incorrectly spelt the results are as occurred so opponent stats will include the Win or loss that just occurred against me. I originally made some attempt to calc opponent MMR but found this fluctuated wildly (though if you deduct 735 from the Silvers the first group of 6 all make sense with regards to the points won/lost. Seems to drop out against highers though that might be in response to a Silver who drops rather than is promoted. Left hand side relates to me so in order Points are the DIV displayed points, bonus is the utilised bonus pool. MMR is my original attempt at looking at a simple MMR which is Points minus utilised bonus pool. Band is my current league. Result is a win loss column. VS is where I then attempted to track some small data of what they played and in some cases what they did. Score change is the impact on my score. I used Bonus for all these so if you see +14 it actually means my score went up by +28 and my bonus by +14. These are then applied to the left hand side on the following line. Then a misspelt name column. This is followed by their win losses (including presumably the one that has just occurred) the next Div indicates my opponents Division which is then followed by their rank in that division. The final three list my opponents displayed points, utilised bonus pool and Band. If the html will cope my data is in light gray and my opponents is light blue with the game and other data in white. Rather than confuse further I have removed my attempts to work out an MMR for the opponents.. The red text in my results is a check point. The following day after playing a group of games I screenshotted my div rank and this allowed me to double check my maths as I went along ![[image loading]](http://imgur.com/hD5oE.jpg) ![[image loading]](http://imgur.com/uklx2.jpg) What I was interested in was what happened when promoted and it appears as if my MMR as of the last game (218) had 735 added to it to make by strange co-incidence 953 (which happened to be my utilised bonus pool) then a further 73 was added to pump the score to 1026 with a visible utilised bonus pool of 73. I don’t quite know what to make of it but hopefully the data will prove useful. It might be a coincidence or it might not. If not then it might indicate that utilised bonus becomes score with whatever adjustment required to get to the correct div/league rank then becoming the new utilised bonus pool. Oh and the 1143 line is the last line/game as bronze after that win I was promoted to Silver. The 1171 line is what my stats would have been if I had not been promoted. The 1026 line is the visible Blizzard stats immediately after the promotion (and will remain so until tonight when I will probably be playing again..) Other relevant stats http://eu.battle.net/sc2/en/profile/389962/1/Matrim/ Blizzard Profile http://www.sc2ranks.com/team/5493282 SC ranks profile Interesting to see if this helps.. (edited to explain when the promotion occurs and the data at the end) | ||
|
Excalibur_Z
United States12235 Posts
First, while MMR transcends leagues, ratings do not. That means you have to make an important distinction between MMR and adjusted rating (points minus consumed bonus pool) when comparing across leagues. We believe your league threshold is added to your MMR. For example, maybe Bronze has a threshold of 1000 while Silver has one of 2000. Look at your last three entries, around the time of promotion. Your MMR went from 1000 + 218 to 2000 - 735 after promotion. That's only speculation because we don't actually know what the thresholds are, but it's an important consideration to make when looking at the ladder as a whole. Secondly, you have some mistakes in your data (I'm sure you're aware but I'm just pointing it out). After SnakeGreen, you stopped listing opponents by rank and substituted some unknown value. What is this value, and how does it apply? The consumed bonus pool values post-promotion are also quite confusing... can you clarify how many points you have remaining in your pool so we can determine what exactly these numbers should be? I haven't seen wild swings like that anywhere else. On the surface it doesn't seem to add up, so we need to make sure we're catching any bugs in the system that might be reported. I'd also like to make an important clarification here about classifying points minus used bonus pool as MMR. It's not quite that simple, particularly when a player hasn't played very many games. We say that MMR and rating will eventually converge, but that's only a result of sigma shrinking such that there are less wild swings in your MMR, and that only comes after many games played. This means that some players, such as SnakeGreen and Mantton, may not be accurate reference points because their MMRs lack stability. | ||
| ||
