|
On September 24 2012 15:32 skeldark wrote:
I know their are more hiding functions and i can prove that.
If you've made a post describing exactly what you think's going on in respect to this, please point me to it. If you haven't, go ahead and post it. If neither is the case, it's all just talk.
So far what I have been hearing from you is "my data is so confusing, I don't understand what's going on, there must be additional hiding functions," but that could as easily be you having a flawed methodology as them adding complexity to their system to obscure it.
|
HErp derp
User was warned for this post
|
On September 24 2012 15:32 skeldark wrote:
We describe them in detail! check out op and the links in it
I have. I apologize that I find it very difficult to follow your posts because of the language barrier, but I haven't seen anything in the linked posts that addresses what we're talking about here. Can you point me to specific language that I should focus on to try to understand what you're arguing?
The posts that get into numerical details all get bogged down in now-obsolete division tiers, so it's very hard to extract what you're presuming is going on now that they've been removed.
|
@Lysenko
I say it plain and arrogant: I think outsite of blizzard, noone knows about the ladder system more than Not_that and me.
You have some very basic knowledge about this topic and i am not excalibur. I not nice enough to use my time to explain everyone every detail about the system if he wants it. I discuss the math behind it if you are helpful for the process.
Read my op. Read the linked articles.Read Not_thats analyse. He try to explain all the rules with nice graphics.
Read something about skill functions.
Im will discuss this topic with you if you have some inside in it.
|
OK, I went over Not_That's article one more time:
http://www.teamliquid.net/forum/viewmessage.php?topic_id=332391
Questions that come to mind about this: Is what he calls the CAP function the extra obscuring thing that you're talking about? Are you claiming there's more? That function, according to that post, probably exists to help solve a distinct problem other than obscuring skill ratings.
Why does he assume F is an ELO function when the Blizzard developers have stated that they are using something other than ELO to estimate skill? Is this an empirical guess based on fitting the data? If the guess were wrong, and they were using some similar but slightly different function, how would that affect your results?
I don't claim to have detailed knowledge of Blizzard's systems, but you guys throw these assumptions around without any explanation or justification. "Hey, if I try to make an ELO function fit what I think I see, it looks like something like this."
How do you know that you're not just coming up with a totally different model for what they're doing that just happens to approximate it well enough to seem pretty close? If that were the case, the small differences you see that you understand as Blizzard making an extra effort to confuse you might just be a matter of your understanding of the system being off the mark.
I'm not asking these questions because I'm claiming I know you're wrong. I'm asking them because when people ask things like this you guys blow them off and just claim to understand better than everyone else. In the physical sciences in the US, we call such explanations "hand waving."
|
On September 24 2012 15:50 Lysenko wrote:OK, I went over Not_That's article one more time: http://www.teamliquid.net/forum/viewmessage.php?topic_id=332391Questions that come to mind about this: Is what he calls the CAP function the extra obscuring thing that you're talking about? Are you claiming there's more? That function, according to that post, probably exists to help solve a distinct problem other than obscuring skill ratings.
It exist to solve the problem that people who are "bad", see that they are "bad".
Also if its not in place many people would have 0 points on ladder. Its a hotfix to solve problems that exist because they try to build a fake system around a normal skill function.
Why does he assume F is an ELO function when the Blizzard developers have stated that they are using something other than ELO to estimate skill? Is this an empirical guess based on fitting the data? If the guess were wrong, and they were using some similar but slightly different function, how would that affect your results?
In no noticeable way! this should answer the rest of your post.
The function and rules are taken from the data. We never belief blizzard announcements and you should not either.
I back-engenier. I dont have to know the system. I have to know the output of the system.
System - MMR - Hiding - Ladder.
I analyse and backcalculate the hiding not the skill-system! A simple fact that many people misunderstand.
Its nice that you are interested in this topic. And if you want to learn more about it, ask excalibur some questions in his thread. If you want to go into deep and help analyse the ladder you are welcome to do so. Im sorry to say that i dont have the time and the will to explain you the way. You have to take it from our posts. I know they are not always in best format and my english is not very good. Few weeks ago i would invite you to tl teamspeak to discuss it with us. But since we are kind of done with the analyse we dont discuss this topic here any-more.
I calculate for the enduser, the mmr. I calculate all of this so you dont have to do it!
I dont try to explain the ladder system to Teamliquid. Excalibur does that!
So if you have question over the system please post them in his or in Not_thats thread.
|
On September 24 2012 16:01 skeldark wrote:
I back-engenier. I dont have to know the system. I have to know the output of the system
I don't think I made myself clear -- my reading of the post I quoted is that these things are not "back-engineered," they're assumed. It's fitting a model to a dataset that may be a good approximation but isn't necessarily 100% accurate. It seems more likely to me that you've come up with a model with a different underlying structure that might approximate what Blizzard is doing in terms of results but doesn't necessarily bear any theoretical resemblance to it.
Regardless, I respect that your work has helped pin down some information about division (and now league) offsets and is a useful predictive tool for things like promotions. I do not as easily assume that your model for skill estimation explains much useful about how Blizzard models these things internally.
Furthermore, it would probably take one of their developers (or their academic or industry peers who have worked on similar matchmaking systems) to actually walk me through the statistics involved, because that seems to be not the focus of your effort.
Edit: What you read as arrogance in my posts is really just frustration that you've never once actually addressed any question I've ever raised about what you're doing. You just say "read the posts," and they tend to be conclusory with no way to assess how you arrived at what you say.
|
United States12240 Posts
The Blizzard ladder has several obfuscators at work:
- Bonus Pool
- League (and formerly division) offsets
- Minimum Divisional MMR (earning more points than an opponent should be worth)
- League Change Point Reset
- Stability Requirement for League Changes
If there are more, then I'm not aware of them. I think it is fair to say that all our combined research has been put toward the attempted reverse-engineering of the ladder system. Since the days of the WoL beta, players have wanted one unified leaderboard so they can identify their position among the world's competition. Everything we've uncovered we've explained so that people can understand the reasons for the current ladder structure and can figure out how to find where they truly stand, such as with this MMR-Stats tool.
Note that that may not be an exhaustive list either, because we can only create hypotheses using the collected data. It wasn't until we had huge tables like Not_That's and Mendelfist's and SDream's that we were able to make recent discoveries like minimum divisional MMR (this concept sort of operates like the USCF's "rating floor" idea where you can't go lower than a couple hundred points below your current rating else players become discouraged from playing, except it's applied indirectly to you [because it applies your floor to all your opponents instead]).
|
On September 24 2012 16:43 Excalibur_Z wrote:
The Blizzard ladder has several obfuscators at work:
- Bonus Pool
- League (and formerly division) offsets
- Minimum Divisional MMR (earning more points than an opponent should be worth)
- League Change Point Reset
- Stability Requirement for League Changes
I certainly understood all of those factors. I guess part of my issue is that I would not characterize these as specifically intending to obfuscate player skill --- they all have other good justifications.
The things you mention as being benefits of this work I agree are benefits whether the underlying skill model they assume is accurate or just an approximation. To the extent that the purpose of Skeldark's plug-in (which I'm now happily using myself) is to predict promotion and display *a* numeric measure of skill (though perhaps not *the* measure of skill that Blizzard themselves use) it's a great tool.
However, I think that it's reaching too far to assume that the implementation of the Not_that/Skeldark model reflects very well exactly how MMR is calculated on Blizzard's servers. And, there seems to be a lot of that assumption going around. Does it make a difference in practice if people just want to see some reasonable measure of how they're doing? Maybe not?
|
Excalibur sums it up very good.
Looks there is are some big misunderstandings of my work.
You say you understand the hiding functions, even quote excalibur who list leagues as one of them and then say its my goal to find out promotions.
I dont try to find out when you get a different icon in the bnet frontend!
I back calculate the skill-rating. Whatever picture you see when you log on in sc2 is only interesting as long as it helps to find your skill. Leagues are one of the hiding functions!
Yes there are assumptions in the calculation. You think they are not accurate.
I have near 1 million datapoints to check if they are or not and adjust them.
Like i pointed out very often, im not accurate to the last point. I can be wrong by few points ( +-20 in average)
With 1 million datapoints you KNOW if you are wrong. We dont just invent numbers and hope they are good.
Its about calculating the data and know exactly the errormargin of all assumptions.
You dont assume we are correct. I assume. You can now argue that because i bring a theories up, i have to prove it.
That would be correct if i write a since paper. But what i want to do and did is show the average user his skill.
I dont want to write a paper about mmr. This is not a paper i have to defend.
Dont get me wrong. Im always open to critic and if you can show me that im wrong in some point im very happy to correct it. But you dont have the understanding of the system to do so yet. You make statements like:
"too far to assume". That is your right and i could show you on the data how accurate it is, but im just to lazy and dont see a benefit of proving something to you. Also i have the feeling you ignore the basic points of my post and make generell points about that you think something is not correct in my calculation. This is not helpful feedback.
I had discussion like this many times on tl. It always comes down to the point that people think i own them a detailed analyse and prove of my work. I dont! I do this in my free time.
I dont like to document every step so i dont do so. I dont have to prove accuracy or my assumptions. I do them ass good as i can. If you can do better im happy to learn from it. If you cant, take it or leave it.
We put enough information out there to understand it. With a lot of work and time.
With your agreement i would like to end this discussion here.
Its the wrong thread for it and i dont see anything productive coming out it.
|
On September 21 2012 17:01 skeldark wrote:
-You can not compare MMR-points form last season to MMR-points from this season. Offsets changed = MMR values change
What's the reason for this? I thought the offsets affect your visible points and your league, not your MMR.
|
On September 24 2012 17:47 skeldark wrote:
With 1 million datapoints you KNOW if you are wrong. We dont just invent numbers and hope they are good.
Its about calculating the data and know exactly the errormargin of all assumptions.
There's a qualitative difference between fitting your own algorithm to the data you collect and inferring what their algorithm is. You can have an excellent approximation with a different algorithm if you add enough fiddly corrections. It seems like this is what you're doing. This distinction wouldn't be important if all you wanted to do was have a tool that were predictive of promotions.
Let me come at it another way -- imagine that I started out with some brand new, fresh ELO-based ranking system and started ranking players based on their games, without regard to Blizzard's system. I might be able to say "hey, when you get to so and so score in my ranking system, you're likely to make Platinum." That wouldn't mean that my ranking system were much like theirs. In fact, there might be some weird edge cases where my rankings didn't seem to fit theirs, and someone else in that position might say "hey, I think they're screwing with things to make the system opaque" when in fact it just meant my model didn't really reflect exactly how they were measuring skill.
Yes you have a huge dataset. That means that you can come up with a very, very good approximation of when promotions are going to happen. It doesn't mean that your system for doing so matches what Blizzard is doing, and if your system isn't perfectly predictive, it doesn't mean that they're making choices to screw you up.
|
On September 24 2012 17:47 skeldark wrote:
I had discussion like this many times on tl. It always comes down to the point that people think i own them a detailed analyse and prove of my work. I dont! I do this in my free time.
You don't owe them this, but they don't have to believe your inferences from it if you don't do that work. 
If you push your MMR Stats plug-in as a way to see a measure of someone's performance that lets them know when they're getting pretty close to a promotion, I think it probably does that job very well (not counting that the current displayed promotion boundaries are a little out of whack since this season.)
However, if you promote the number it displays as "the secret hidden Blizzard MMR" (I'm not quoting you here, but let's just say you were to use those words) then I think you'd be misrepresenting what you're doing. Reading the OP and the other posts on this matter, it feels like you're mixing up your MMR with Blizzard's MMR, and they're NOT necessarily identical, even if yours is an excellent predictor of promotions and a good way to identify unusual point score behaviors at league boundaries.
Edit: I have been using the MMR Stats plug-in and it's obvious you've put a heck of a lot of work into it. My ONLY issue is the confusion over whether what you're doing is an approximation or a derivation of Blizzard's algorithm.
|
@ Lysenko:
Wouldn`t a near-to-perfect approximation be the same set of mathematical relation, just maybe in another formula?
Example: TL-MMR-formula vs. Blizzard formula: (very easy one to deliver my point)
a² + b² = TL-MMR-formula
(a + b)² - 2 ab = Blizzard formula
Note that TL-MMR-formula equates the Blizzard-formula, while it might not be THE exact same algorithm, it´s mathematical the same.
My assumption: If the output is the same, the algorithm behind it should be the same mathematical-wise too, no?
Please consider that I sucked at math and am just starting to study IT, at the moment the above assumption ist just intuitive, not backed by any science or so. Could some math-knowledgeable guy help out on this please?:D
|
On September 24 2012 18:30 Mendelfist wrote:Show nested quote +On September 21 2012 17:01 skeldark wrote:
-You can not compare MMR-points form last season to MMR-points from this season. Offsets changed = MMR values change
What's the reason for this? I thought the offsets affect your visible points and your league, not your MMR.
True.The whole range is smaller now ( lowest to highest mmr).
So there must be a change in the system itself. A offset change alone would change the promotion lines but not the mmr value.
So they could changed something insite of the skill function or they just changed the hiding function so the points in a league are closer together.
Anyways i have to add the offset value because blizzard removed it first.
So obvious when i add different values you end up at a different mmr. If only the offsets are different now, the new different dmmr would "fix" this change. But it did not. I can not tell you what the exact change is because there is no way to find out.
@Lysenko
In some way we did approximate to the data in other we did not. Also if the approximation is good i dont see any problem it it. My task is to undo the hiding. I could come up with several hiding functions myself. i dont see anything in it that's worth spending time on. Everything that helps undoing it is helpful everything else is uninteresting.
Dont you see this is a pointless discussion? You guess what we did and point possible mistake of possible ways.
You serious think we do not understand problems of approximate a function to a dataset?
It can be that you just write down your thinking process. I do this myself sometimes. But it sound a little bit like you try to explain me basic logic. In first case: yes, you have to make sure you function is not only working on the data you approximate it from. In the second case: I think this is basic knowledge ... but thanks you for pointing it out.
You don't owe them this, but they don't have to believe your inferences from it if you don't do that work
Like i said i dont do so. Again: Feel free to dont believe it! Only a critical mindset is a real open mindset!
I would not believe me either. But i would not point out undetailed critic before understanding the whole material either.
And after study the material, i think i would agree with me 
@GTo7_Panda
He think i approximate from dataset a. Than my function dont work on b and i would wonder that the system is wrong instead of thinking about my approximation is wrong.
To your post: if a function F give the exact same output to the same input than function G. Than yes: F = G
Or: "perfect approximation" is another word for "equal".
|
On September 24 2012 19:18 GTo7_Panda wrote:
Wouldn`t a near-to-perfect approximation be the same set of mathematical relation, just maybe in another formula?
Not really. Imagine that you've measured the height of a ramp along its length. You have a bunch of points that relate horizontal distance (x) with vertical distance (y.) You look at the plot and think it looks a lot like a straight line, so you fit a line to it:
y = ax + b where a and b are constants.
It might be a pretty good fit. You might even be able to use a statistical measure and say it's accurate within the uncertainty of your measurements.
However, the guy who built the ramp comes along and says that (for whatever reason) it was designed to have the shape of a parabola:
y = a(x^2) + bx + c
Had your measurements been more accurate, you might have noticed this.
In fact, if the guy who built the ramp didn't tell you this, you might notice that your straight-line curve fit doesn't QUITE match at the ends, but it's pretty good for most of the length. You might think "well, this is of course designed to be a straight line, and it mostly is, but it's a little out of shape at the ends, it must be sagging!"
But, you simply chose the wrong model for its shape.
In this instance, assuming that skill is measured using ELO vs whatever Blizzard's actual MMR function is could well be like substituting the straight line for the parabola. In other words, that would be making an assumption and then figuring that since the data is very close the assumption must be correct.
I'm not saying this is what has happened here, but I am saying that as an outside observer it may never be possible to tell no matter how large the dataset is. Honestly, unless the model is precisely predictive of promotions, to the game, every time, over a million plus points, it's probably a little off somewhere. The question is how much.
Here's another way to think about it: For almost everything we do in the world, except in electronics, Newtonian physics does the job. We can calculate the stresses on all the parts of a building, fly an airplane, shoot a shell in the air and strike a target many miles away, all with that model. However, look more closely and at high speeds or high masses (or low!) it breaks down, and you need a model that's more accurate. Newtonian physics may be extremely accurately predictive in many cases, but that doesn't mean it's what the universe has programmed into its pocket calculator.
Edit: None of this is a criticism of Skeldark's work as such. It's a great way to give people visibility into their performance over time. My point is that leaping from "this model is not perfectly predictive" to "Blizzard must be screwing with things to obscure our visibility into what they're doing" isn't necessarily a reasonable leap.
Edit 2: They have stated that the design of the point system is intended to encourage people to play through the bonus point system and ladder resets. I guess you could call this obscuring people's skill levels, but that's a different idea than actively trying to obfuscate what's going on under the hood in the face of a determined analyst.
|
@Lysenko
didn't you notice/understand the cap rule? because its exactly this: "to obfuscate what's going on under the hood in the face of a determined analyst."
edit:
in my opinion blizzard statements like "in the next season with removed league-tiers, your positioning in your division is directly related to your skill positioning in your league"
is exactly this: "to obfuscate what's going on under the hood in the face of a determined analyst." too
|
United States12240 Posts
The main problem with this discussion is there's a fundamental disagreement between what analysts want and what customers understand. We're on the super-fringe here so it's important to recognize that. I can explain concretely why every one of the obfuscators is in place.
Leagues
Leagues are supposed to be a "good enough" skill indicator. That much is obvious. A player's league is quickly noticeable and the icons are bright, unique, and catchy. "This guy is Diamond" "I'm Plat" and so forth.
Divisions and Division Tiers
Divisions are supposed to more tightly estimate your skill, essentially making them "sub-leagues". This is tricky because every player's volatility is different, so trying to pigeonhole them into a division can have negative consequences. If you place them when they're performing better than normal, then their new division's offset becomes too high and they get stuck at 0 points. If you place them when they're slumping, then they shoot to the top of their division effortlessly.
The intended design for divisions is pretty straightforward. Get a group of 100 players all around the same skill level and create a mini-ladder and see who does the best. Sounds great, but it's hard to explain to the average player because you're not only playing against people in your division (which makes the name "division" somewhat ironic) and you can change leagues which puts you in another division that may or may not accurately represent where you should be.
Offsets
I think the original idea for the inclusion of offsets was to allow weaker players in a league to earn some points. If this keeps people playing then it's good, it's always good to have more players. The only extra info I have on this is that they knew aggregator sites like SC2Ranks would appear and while it wasn't their intention to invalidate the unified rankings on those sites, the offsets did have that effect.
Minimum Divisional MMR
This wasn't in the original design. It was patched in later when we found examples of players who were so deep into Bronze that no matter what they did, their points anchored to 0. They applied this rating floor concept which accomplished a couple of key things:
- When you're starting a season fresh at 0 points, promote activity by pushing your points upward toward the mean.
- If you were too weak for your division or you're part of the 1% or so who is destined to lose every game causing you to drop to 0 points forever, promote activity by pushing your points upward toward the mean.
You would think this would cause rating inflation, but it actually doesn't which is a pretty clever application of the illusion of the points system. Even if the mean were 100 points, that would only impact the players who were incapable of earning more than 100 points normally. This sucks for analysts but it sucks more for players if it didn't exist because they'd quit out of frustration.
Bonus Pool
This one is obvious, extra points for everyone who's still playing. Incentivizes play while simultaneously punishing people who try to sit at a high rating.
Removing Division Tiers
Blizzard is moving away from the more casual-friendly division system (probably because of the difficulty in accurately placing players) which is why they cut division tiers. However, they needed a way to explain the consequences of this change. Players are going to find themselves with 99 other players whose combined skill range can potentially be much wider than what they're used to, and the effects could be jarring. In Bronze for example, someone who's used to being in the top 8 in his low-tier division may find himself unable to break the top 50 in an active, Day-One Season 9 division. So, they came up with the "rank = percentile" language which only works on the average and suddenly it makes a little more sense to that same Bronze player why he's having such a hard time.
Skeldark's tool gives people a visible rating that attempts to approximate MMR. But it's not Blizzard's MMR. Skeldark has said for a long time that he's wanted a system that didn't have any of the arbitrary stuff. If there were 10 leagues instead of 7, it wouldn't matter any to Skeldark because numerically nothing would change. I don't agree with his approach of "everything Blizzard says is a lie" because the ladder is structured in a very specific way and we're not the target audience of their explanations. They're also not ever "lies", more like "half-truths" or "things that could be true if X and Y and Z conditions are all met".
|
still loving it still using it
to the post above me, i agree with the stuff you said about "removing division tiers"
i think in the longterm it will help people knowing how good they are, currently they might be confused
|
I've recently noticed (since season change and update to most recent version of MMR-Stats) that occasionally my MMR, or the MMR of my opponents, or sometimes both, will jumpy very sharply for several games. For example, my MMR jumped this morning from ~900 to ~1150. My opponents MMR was very close to mine for those two games. The next game my opponents MMR (according to the graph) dropped to about 600, spiked to 1,000 next game, then dropped slightly over the next 3 games before spiking back up to around 1,150. Is this because of the removal of divisions, or is my data just bad?
|
|
|
|
|
|
EPT
