|
On April 30 2012 14:23 skeldark wrote:The script on Custom division give wrong results. I made a quick exel file -based on other versions here- that should be very easy to use. With error messages on bad data or data that this algorithm cant use: XLS FileODS FileHowto: + Show Spoiler +Dont forget, it must be a recent game or you have to calculate the opponents points back... B: Points of Opponent C: Bonuspool of Opponent D: Change he got for this game (- / + , Direct copy WITH possible Bonuspool)+ Show Spoiler +
The file calculates the right amount without bonus.
If its not possible to calculate the amount it will give you an error message.
Calculation:
If opponent lost: nothing to do
If opponent won and have bonus pool : /2 and round down,
If opponent won and have no bonus pool : Error message
If you KNOW the amount (by asking him), you can enter it in AH
: Master or Grandmaster F : Total Bonuspool ( sc2rank xml file ) = I : Possible error messages. K : Your MMR
Can you specify what is wrong with the results? From what I understand looking at your excel file you have 3 entries, and the first 2 are losses. Your MMR predicted from each game result is always your MMR at the beginning of the match. So if you lost games 1 and 2 and your MMR decreases between games 1-2 and between games 2-3 that is expected. Unless you mean something else? Please explain.
|
On April 30 2012 15:40 Not_That wrote:Show nested quote +On April 30 2012 14:23 skeldark wrote:The script on Custom division give wrong results. I made a quick exel file -based on other versions here- that should be very easy to use. With error messages on bad data or data that this algorithm cant use: XLS FileODS FileHowto: + Show Spoiler +Dont forget, it must be a recent game or you have to calculate the opponents points back... B: Points of Opponent C: Bonuspool of Opponent D: Change he got for this game (- / + , Direct copy WITH possible Bonuspool)+ Show Spoiler +
The file calculates the right amount without bonus.
If its not possible to calculate the amount it will give you an error message.
Calculation:
If opponent lost: nothing to do
If opponent won and have bonus pool : /2 and round down,
If opponent won and have no bonus pool : Error message
If you KNOW the amount (by asking him), you can enter it in AH
: Master or Grandmaster F : Total Bonuspool ( sc2rank xml file ) = I : Possible error messages. K : Your MMR Can you specify what is wrong with the results? From what I understand looking at your excel file you have 3 entries, and the first 2 are losses. Your MMR predicted from each game result is always your MMR at the beginning of the match. So if you lost games 1 and 2 and your MMR decreases between games 1-2 and between games 2-3 that is expected. Unless you mean something else? Please explain.
The web script give me different mmr than my hand calculation.
Also master or grandmaster selection of this script dont change the result.
The data in the exel file is only test data no real game results.
My exel-file have nothing to do with the webscript.
Just wanted to warn you guys that this one give wrong results.
I will edit it out and post in the other thread so the creator can fix it.
|
I'm the author of the webscript. Can you specify in which line is there a discrepancy between your excel and the web one? I can't find it. Keep in mind the web version requires as input the amount of points the opponent has after the change from the game has already been applied.
|
On April 30 2012 16:18 Not_That wrote:
I'm the author of the webscript. Can you specify in which line is there a discrepancy between your excel and the web one? I can't find it. Keep in mind the web version requires as input the amount of points the opponent has after the change from the game has already been applied.
can not tell you, i installed your script testet it with 2 datasets and had diffrent results.
Than i clicked on master and grandmaster and the result did not change so i just thought it is broken.
Perhaps i calculated wrong per hand the 2 trys...
I try again
|
The webscript gives your MMR in relation to your opponent league, so the numbers won't change between Master-GM and between Bronze-Diamond. Yes GM->Master modifier is probably about 430 (and possibly different in SEA server), but I have decided to keep it generic since it's important to note that any result you get will always be in relation to the opponent's league/tier modifier.
|
On April 30 2012 16:34 Not_That wrote:
The webscript gives your MMR in relation to your opponent league, so the numbers won't change between Master-GM and between Bronze-Diamond. Yes GM->Master modifier is probably about 430, but I have decided to keep it generic since it's important to note that any result you get will always be in relation to the opponent's league/tier modifier.
PM
|
I'm a Diamond player and I don't understand one thing - what do the different Tiers mean and how do I understand in which one I am?
|
Nice write-up, you obviously put a lot of work into this project! :-)
No I'm wondering, is it not possible to determine every division's tier by collecting a lot of match information? Start at master/grand-master, get those diamond tiers, and then keep going.
As far as I understood the mathematics, all data we'd need is:
[1] Adjusted points of both players at start of match
[2] The gained/lost points
[3] The match's timestamp (to get the size the season's bonus pool)
[4] The player's league and division
It should be possible to implement this data collector as an sc2gears plugin, the start of a match is announed via the windows registry (AFAIR); point gains/losses, league and divisions can be scraped from Battle.net as soon as the game finishes; Take all this data push it to a webservice/database, and continue from there (automatic/manual analysis, filtering, etc.)
In case this would help anyone, and there are enough people who want to collect some data, I'd be happy to help with the plugin's or webserver's implementation and to provide the server to run it on.
EDIT: Some additional thoughts (after I checked the Battle.net HTTP profile view): it looks like the match history visible in battle.net does not contain, the match's opponent. This is not optimal, especially if there is a delay from the match-end to the display on the http-view (there are probably some caching layers). The crawler could always take the first row, which matches the known criteria (map, 1v1 match-made, points-lost/gained sanity check against the data crawled from the opponent). But there is still the possibility for wrong results.
|
wow honestly great job but maybe one day ill use it
|
Respectfully, I'm not sure of your methodology. Graph 3 is supposed to show the robustness of your model, but it seems as if at least 10% of your datapoints are out of the error bars that you should have if your formula's correct. That's majorly statistically significant, and can throw off all future sampling. What you're proposing isn't a physical science: the function F exists. It is some actual function. It is disingenuous as a researcher to submit work with the title "how to calculate MMR" if your final model has serious flaws in a large percentage of cases, especially since you know that the vast majority of readers are going to skip directly to the Excel sheet that they don't really understand anyway.
I'm not saying that what you're doing is wrong. I think this is a good start. But when you propose a mathematical model, it is imperative that you include in that model caveats about when it's a good fit for the data you have, and document when and how it fails to the best of your ability. Otherwise you end up with a large number of people using the model for data that your system can't handle.
edit: I am aware that you document when the model fails to work in a specific case, but my point is that you NEED to put those warnings in the model itself, and in the Excel sheet, in bold face. This doesn't mean you failed to find the "right formula," it means that your formula is good for a specific purpose. Which is what serious researchers look for anyway.
|
United States12224 Posts
On April 30 2012 14:50 skeldark wrote:Show nested quote +On April 30 2012 03:44 Damnight wrote:
Well, this leaves me with one question: How can I estimate if I´m close to a promotion?
My MMR shows me 182-214. I´m Diamond but I can´t tell the tier.
What now? Its all about the tier of your opponents, thats why you only calculate with master/GM opponents! If you have a MMR of 198 based on a master opponent, calculate the MMR of some random, active, master guys and look how close you are. If you can hold your MMR in average for longer time at master level, the system will promote you. PS: now with knowledge of your mmr , it does not care if you are dia or master any more. We can calculate better than this bad bronce-gm system  @Excalibur_Z whats up on the Tier front? With calculating 5-10 MMRs out of each division and take the average it should be possible to get the tiers of most of them. Perhaps start a project where every dia guy here on tl can calculate the average for his own division? On the other hand we can only calculate the guys that play master opponents to start with and than work on from there. Or is there a project that already have a ingame bnet spider done? so you can automatic the work?
Regarding division tiering info it's very hard to say. I'll give you a couple of different scenarios:
In Scenario A, a Diamond player with 500 points plays against primarily Master-level competition. These Master opponents typically range from 330 to 370 points and the point changes per game are even. Because it's known that the Master offset from Diamond's top tier is -150, which is consistent with the point differences between this Diamond player and his Master opponents, we can conclude with relative certainty that the Diamond player's division tier is the highest for that league.
In Scenario B, a Diamond player with 500 points plays against other Diamonds, and let's say the point exchanges are always even as well. So, if other opponents have around 500 points, it's possible they're in the same tier. However, what about an opponent that has around 468 points? That could either mean the opponent is in the same tier (-32 point range) or a lower neighboring tier (-63+32).
That's not even factoring in the unknown league offsets for everything below Diamond.
So yes, I'm sure it is possible to eventually find all of this out, it would just require a lot of data collection and intelligent filtering out of border cases like Scenario B. I started tracking a lot of this stuff manually with the help of michaelhasanalias and we created this very sloppy spreadsheet which hasn't been updated in nearly a year. I tried to get the most recent games possible for each entry but it's possible the system has gone through some internal math changes over the past year. Still, maybe it's helpful to help support Not_That's spreadsheet.
|
On May 01 2012 00:54 Excalibur_Z wrote: it would just require a lot of data collection and intelligent filtering out of border cases like Scenario B. I started tracking a lot of this stuff manually with the help of michaelhasanalias and we created this very sloppy spreadsheet which hasn't been updated in nearly a year. I tried to get the most recent games possible for each entry but it's possible the system has gone through some internal math changes over the past year. Still, maybe it's helpful to help support Not_That's spreadsheet.
As stated above, I'm quite curious about the barriers for automatic information collecting and processing.
timestamp, "real points" (as in "points shown on battle-net"), point losses and gains, league and division name can all be read from battle.net (takes a little effort to get the correct results) or taken from replay information.
As fas as I got into the math behind it, this should be enough to estimate a division's tier, as long as there are enough datapoints, is it not?
|
On May 01 2012 00:54 Excalibur_Z wrote:Show nested quote +On April 30 2012 14:50 skeldark wrote:On April 30 2012 03:44 Damnight wrote:
Well, this leaves me with one question: How can I estimate if I´m close to a promotion?
My MMR shows me 182-214. I´m Diamond but I can´t tell the tier.
What now? Its all about the tier of your opponents, thats why you only calculate with master/GM opponents! If you have a MMR of 198 based on a master opponent, calculate the MMR of some random, active, master guys and look how close you are. If you can hold your MMR in average for longer time at master level, the system will promote you. PS: now with knowledge of your mmr , it does not care if you are dia or master any more. We can calculate better than this bad bronce-gm system @Excalibur_Z whats up on the Tier front? With calculating 5-10 MMRs out of each division and take the average it should be possible to get the tiers of most of them. Perhaps start a project where every dia guy here on tl can calculate the average for his own division? On the other hand we can only calculate the guys that play master opponents to start with and than work on from there. Or is there a project that already have a ingame bnet spider done? so you can automatic the work? These Master opponents typically range from 330 to 370 points and the point changes per game are even. Because it's known that the Master offset from Diamond's top tier is -150, which is consistent with the point differences between this Diamond player and his Master opponents, we can conclude with relative certainty that the Diamond player's division tier is the highest for that league.
I think this is the key. If we find the highest tier for diamond we can go the way down from there. I think, i did not calculate it just a feeling, that when we make mistakes on the way we must find logical mistakes later. So we can than walk back again and solve it with try and error. Dont know if this method is np-hard tho ^^
With allot of data and high enough certainty it should be possible.
|
I've already posted in this thread... but I just have to say this:
This is honestly one of the most valuable contributions to the community since release. It rivals Excalibur's contributions to the ladder league system explanations. It is some excellent work, both from a deductive standpoint and from a "filling a need" standpoint. You have done truly stand-out work here. Extra credit in every way. Thank you for doing this.
Next: A call to the community. Someone needs to make a website (or modify an existing one, like ranks) that can take a persons ladder name and char code, and spit out their mmr. I would do it myself but frankly I lack the necessary knowledge. If this is done, then we can finally abolish the whole "I'm mid diamond" shenanigans, and people can finally begin asking for help and contributing to strategy discussions with "I'm NA Master's with 874 MMR".... so that everyone in the community knows exactly where this person sits and can tailor discussions accordingly.
Yes, I know that people can follow the formula listed in this thread and obtain this information themselves, but I think we can all agree that the use of MMR as a criteria will only become mainstream when it is easily obtained, either through blizzard publishing it, or a simple to use web interface spitting it out.
Thoughts?
-Crush
|
On May 01 2012 02:45 Crushgroove wrote:
I've already posted in this thread... but I just have to say this:
This is honestly one of the most valuable contributions to the community since release. It rivals Excalibur's contributions to the ladder league system explanations. It is some excellent work, both from a deductive standpoint and from a "filling a need" standpoint. You have done truly stand-out work here. Extra credit in every way. Thank you for doing this.
Next: A call to the community. Someone needs to make a website (or modify an existing one, like ranks) that can take a persons ladder name and char code, and spit out their mmr. I would do it myself but frankly I lack the necessary knowledge. If this is done, then we can finally abolish the whole "I'm mid diamond" shenanigans, and people can finally begin asking for help and contributing to strategy discussions with "I'm NA Master's with 874 MMR".... so that everyone in the community knows exactly where this person sits and can tailor discussions accordingly.
Yes, I know that people can follow the formula listed in this thread and obtain this information themselves, but I think we can all agree that the use of MMR as a criteria will only become mainstream when it is easily obtained, either through blizzard publishing it, or a simple to use web interface spitting it out.
Thoughts?
-Crush
soon...
|
@ above
i'd really really like an easy tool to calculate mmr. if i calculate it for myself it has little meaning... unless i can easily compare it to my friends and opponents mmr.
|
On April 30 2012 23:39 nfogravity wrote:
Respectfully, I'm not sure of your methodology. Graph 3 is supposed to show the robustness of your model, but it seems as if at least 10% of your datapoints are out of the error bars that you should have if your formula's correct. That's majorly statistically significant, and can throw off all future sampling. What you're proposing isn't a physical science: the function F exists. It is some actual function. It is disingenuous as a researcher to submit work with the title "how to calculate MMR" if your final model has serious flaws in a large percentage of cases, especially since you know that the vast majority of readers are going to skip directly to the Excel sheet that they don't really understand anyway.
I'm not saying that what you're doing is wrong. I think this is a good start. But when you propose a mathematical model, it is imperative that you include in that model caveats about when it's a good fit for the data you have, and document when and how it fails to the best of your ability. Otherwise you end up with a large number of people using the model for data that your system can't handle.
edit: I am aware that you document when the model fails to work in a specific case, but my point is that you NEED to put those warnings in the model itself, and in the Excel sheet, in bold face. This doesn't mean you failed to find the "right formula," it means that your formula is good for a specific purpose. Which is what serious researchers look for anyway.
I partially agree.
It is not "at least 10%" of the datapoints that are out of error bars. It is 4 points out of 107 (or 92, if you ignore the Diamond opponents):
win6
win13
loss19
loss26
win62 debatable
(remember that it's okay for the MMR to be entirely out of bounds of the MMR of a previous game, so long as it's in the right direction following a win/loss)
Having said that, I agree it's a big problem. Only after publishing the post I realized there's a problem with my data gathering methodology. I explain it in this post. Basically it is impossible to look into the past of a player's match history and know how many adjusted points he had X games ago. You will always have some amount of uncertainty, and this amount grows larger the more games the player has won since the game you are trying to determine his adjusted points during.
Luckily there's a solution to this problem: Look at the present instead of the past. All that's needed is data of players who record it for themselves after each game they play. I asked for it and I'll repeat it again, and I'll add it to the OP so there'll be more chance people will see it:
I am looking for data of Master league players. The data needs to include a lot of games in a row, and it needs to include: match outcome, your opponent's: league, adjusted points (alternatively ladder points, unspent bonus pool and server bonus pool at the time of your match), and points change of opponent for the match. Preferably in your losses, it needs to include all possible ranges for points change of your opponent. Best thing to do if you're gathering the data now is in games you lose simply ask your opponent how many points he gained without bonus pool. If he doesn't reply, keep in mind upwards of 50% of his winnings may be bonus points and record possible range.
Personally I've been keeping track of my games and asking my opponent every game that I lose how many points he won without bonus pool. So far all of my opponents have been kind enough to reply, so that's good. Unfortunately it seems I spend a lot more time in this game doing things that are not playing it, so I only have 10 games recorded at the moment...
Despite all this I stand behind the creation of this thread, even if parts of it may have been better worded differently. I believe what I describe in the thread brings a powerful tool for analyzing the ladder system (and if my F function is wrong, hopefully this thread will bring about finding the correct one). If in the past we were mostly dependent on +12/-12 matches to make sense of league modifiers now we got a better way to go about things. And of course to some players (atm mostly Master league players can use this) it may be nice to get a number for MMR before their adjusted points settle, for example during the first 20-50 games of a season.
|
Hoe does it handle league overlaps?
There is quite a large overlap in lower leagues. There was a graph showing it around at one point.
|
wowzers thats some hardcore theorycrafting.
|
On May 01 2012 08:20 Not_That wrote:
Despite all this I stand behind the creation of this thread, even if parts of it may have been better worded differently. I believe what I describe in the thread brings a powerful tool for analyzing the ladder system (and if my F function is wrong, hopefully this thread will bring about finding the correct one). If in the past we were mostly dependent on +12/-12 matches to make sense of league modifiers now we got a better way to go about things. And of course to some players (atm mostly Master league players can use this) it may be nice to get a number for MMR before their adjusted points settle, for example during the first 20-50 games of a season.
I agree with you; I think this is a good thing. But I also think that you need to phrase more clearly it as a call for help from Masters-level players rather than the last word on the subject. You're serious enough about this that I trust that you're going to continue to do good things with the data you get. But it's really important, very very important, that the community understand that you are not done yet -- partly to keep people from getting complacent with what you have, and partly to drive participation in your project to figure out exactly what's going on. (And you'll need a lot of data points to do that.)
My point is, use this as an advertisement. Get other people to write down all their data, and keep writing it down. That's the first step to finishing this.
edit: I see that you've changed the original post. That's the only point I was making, and I'm glad you responded to it. Good luck!
|
|
|
|
|
|
EPT
