Dustin Browder Wants Accurate Ranking Using MMR... - Page 3

nTzzzz

France30 Posts

August 09 2015 00:13 GMT

#41

Agreed. I wanna see my MMR as well.

The current system is just a bad attempt at making us think we're better than we actually are and it's quite an insult to our intelligence.

If Blizzard thinks noobs can't handle the truth (I don't think so personally), they can keep distorting information for them but at least give us some way to see our true MMR and where we really stand compared to other players.

paralleluniverse

4065 Posts

August 09 2015 04:30 GMT

#42

On August 09 2015 04:03 Roblin wrote:

Show nested quote +

On August 09 2015 03:01 paralleluniverse wrote:

On August 09 2015 02:49 Roblin wrote:

On August 09 2015 02:21 paralleluniverse wrote:

On August 09 2015 01:06 TokO wrote:

On August 09 2015 00:33 paralleluniverse wrote:

On August 09 2015 00:12 TokO wrote:

On August 08 2015 23:52 paralleluniverse wrote:

On August 08 2015 23:34 TokO wrote:

On August 08 2015 23:06 paralleluniverse wrote:
[quote]
No one does. E.g. if you're in gold league in SC2, you're in the top 40% to top 72%. This is 16 times less precision than know you're rank 20 in HotS which is top 40% to top 42%.
[quote]
Why is this good? Why isn't the transparent HotS system that Dustin Browder advocates better?

[quote]
Divisions are meanginless, so who cares if they are stable. The leagues are wrong. You can have higher MMR than your league and not get promoted, or lower MMR and not get demoted. This is one reason why HotS scraps leagues and replaces with an accurate, fluid ranking system based on a continuum of points, not buckets that are discrete, arbitrary and wrong.

Well, the reason I think it is better is that there is more justification for anyone's rank. Looking at games played that season and placements in division, I have a pretty good idea about another player's skill that season. Obviously, I haven't played the latest iterations with 20-30 games placement in Heroes, but when I played, my team would almost always consist of a shotgun shot of different ranks, and I had no idea of how to translate that into anything comprehensive. You kinda have this image of 50 ranks system being a perfect indicator of skill, when there are so many factors that could distort the indication, and distort it quite easily.

I think a lot of people level a lot of hate towards the SC2 system because they aren't getting a promotion when they believe that they "deserve" it. The opposite is actually the case. I think promotion in SC2 is a much stronger indicator because the rigidity means that the person actually deserved his promotion when he finally achieves it. You only move in SC2 when you're consistent, and when the amount of games played is high, it becomes a consistent indicator. On the other hand, in systems where you fluctuate a lot because of unlucky teams, being carried, etc. the ranking means a lot less.

How do you justify a wrong rank? Even if leagues requires "consistency" (ExcaliburZ says that the requirement of low MMR uncertainty to be promoted has been scrapped, so this might not even be true), there are many reasons why someone is a league. They could have just been put there after 5 games, rather due to "consistency", they could so bad they deserve to be demoted but are locked in, they could deserve a promotion but Blizzard forgot to adjust the league boundary (they have calculate this manually each time, it seems) so they don't get one. Moreover, a league can be 32% of players. So really, when you know the league, you don't know anything.

As you point out, a problem with promotions because people think they deserve one and the promotion criteria is not transparent, and leagues given very liltte indication of skills, with up to 16 times less precision than HotS ranks. That's a reason to scrap leagues as has been done in HotS.

You frequently get matched to people in different leagues in SC2. That's because player's distorted ranked is different from the MMR skill rating. That's the problem that Dustin Browder says he will now fix in HotS by making ranks based off what is the percentile of MMR after placement matches.

That's why you need to look at the amount of games played. The more games, the stronger the indication. Fewer games means the rank is a worse indicator of skill. This is going to be the case regardless of system.

No. Looking at the number of games played is not informative. The player could deserve a demotion after many games, and is locked in, distorting his rank. Or the player could deserve to be demoted after playing small number of games because the initial placement was an overestimate.The player could have only a few games played but have very low uncertainty in the MMR due to lots of games played last season. Or the player could h very low uncertainty in the MMR due to lots of games played. So the number of games is not useful. It's also needlessly complicated to take so many things into account, when you could (and the system does) just look at the MMR. HotS uses 1 number.

I don't really accept your argument of ranks in Heroes being 16 times more precise.

They are 16 times more precise because they use groups of size 2%, which is 16 times more precise than groups of size 32%, by definition. Suppose SC2 used groups of size 50%. Does this not give less information? You could be the worst player or the median player, both are equal.

If the placements lack proper justification, and can more easily due to factors outside of player skill, then the ranks cease to be an accurate indicator. It seems that you've made an assumption that Heroes ranking system is always accurate when it places you, and that SC2 is always inaccurate, because of leagues.

You also argue with the assumption that all gold-leaguers are the same, all platinum-leaguers are the same, etc. But they are not. If we compare the two systems fairly, there is actually 700 ranks different ranks in SC2 that players can tend towards, making it a lot more precise.

Ranks will always be distorted to a certain degree, due to skill decay, rank decay, changes in skill of playerbase composition. That's why you meet people of different leagues and ranks in SC2. It's really naive to believe that Heroes will successfully have each of their 2% brackets perfectly represent the distribution of playerbase's skill in Heroes. As long as some people don't play until they are accurately placed, play deliberately worse, stop playing, only play once in a blue moon, etc. there will be inconsistencies established in Heroes. The same things that you are attacking the SC2 system for.

You are confused between the meaning of precision and accuracy.

"It seems that you've made an assumption that Heroes ranking system is always accurate when it places you, and that SC2 is always inaccurate, because of leagues." I have not made such an assumption.

You're argument that factors X, Y and Z affect ranks, so they are wrong. This is basically the fallacy that if we don't know everything so we know nothing.

I'm some factors distort ranks (e.g. skill decay), some factors, contrary to your claims, don't distort ranks (e.g. changes in playerbase). I'm saying that there is more factors that there are far more factors that distort ranks in SC2 that do exist in HotS: e.g. bonus pool, leagues. And Dustin Browder agrees, that's why his removing them.

When I am looking at the information that the SC2 system gives me, games played definitely gives me a relevant indicator of how accurate the given player's position is. If you think that a systems ability to accurately place a player after 5 games is the same as it is after 500 games, then obviously there is no reason of discussing anything. Obviously previous seasons matter, but other things being equal, many games recently tend to give a better indication of recent skill.

Do you understand that the system gives you more information than that a player is in gold? That there is a point count and a rank within each league? You evaluate the system as if the only feedback the system gave you was in which league you're in. A league can't really be compared directly to Heroes' system. So the 16 times more accurate statement is not true at all.

All I'm trying to argue is that the more rigid system makes the ranking evaluation more informative. You move relatively faster between ranks in Heroes so every rank conveys less information. There are a lot of factors that distorts both systems, maybe there are more systemic distortions in SC2, but as long as those are revealed and well understood, it doesn't make SC2 ranking that much worse all.

In doesn't really make a difference, as long as the system used matches you up with people who are the same MMR. That wasn't my experience in Hero League when I played (When you would move around being awarded between 100 and 200 points out of 300 per match). Obviously, as the ranks become more rigid and advancement slower, ranks in Heroes will become more informative. It just seems like your critique is very severe as a result of a really simplified understanding of the SC2 system. It's not that bad as you would have it.

No. Games played does not given an indication of how accurate your rank is and I have given 4 examples above, where games played is not informative.

Your MMR after 500 games is more accurate than after 5 games. But your rank after 500 games is NOT necessarily more accurate than after 5 games because of all the unique SC2 distortionary factors that creates bias, not just standard error, as I've shown in the examples above. So your understanding of ranking systems is very uninformed.

Your statement that HotS ranks contain less information is flat out false. What's more information: You're MMR percentile plus a bunch of biased distortionary factors is in the top 40% to 72% or your MMR percentile in is the top 40% to 42%? The latter, is far more information and far more accurate, which is why the HotS ranking system is being changed in this way.

you ignore a very simple fact. pretty much everyone in heroes jump up and down in ranks pretty wildly. I myself vary between rank 27 and rank 22 depending on how well im playing that particular day and also of course, varing based on luck (aka unpredictable factors favouring one side over the other).
so for me, if I am in rank 25 it is misleading to claim that means I am in percentages 50% to 52% when in reality I vary betweeen percentages 44% to 54%

and besides, it not like SC2 gives no indication of where in your division you are, true its built in leagues which give a wide range, but your placement within the division tells you whether you are closer to 40th percentile rather than 70th percentile.

to any intelligent person that is interested to know one own approximate standing, SC2 is no better or worse than the suggested model for heroes.

Firstly, 44% to 54% is not 40% to 72%, which is what you'll get in SC2.

Secondly, if someone is 44% to 54%, and another person is 48% to 53%, who should be ranked higher? For any ranking system, even the SC2 ranking system, you need to collapse this information into 1 number, such as a mean, 49% vs 50.5%, so the second should be higher. The problem is that the 1 number in SC2 is wrong, and in HotS it's far less wrong.

Thirdly, why does your rank being between 44% to 54% imply it should be biased with distortionary factors like bonus pool and no demotions, and made up to 16 times (a number not chosen based on the data but pulled out of Blizzard's ass) less precise than HotS ranks?

Fourthly, division ranks are meaningless and wrong. They intrinsically decline every hour due to the bonus pool, can't be accurately compared when bonus pool is not spent, and doesn't account for the fact that some people don't deserve to be the division but are stuck there due to rigidities in the league system.

Lastly, if you want less volatility in ranks, the solution is not the SC2 ranking system of wrong ranks, it's to apply a trend filter to the metric that is used to create the percentile ranks (approximate MMR in HotS or points in SC2).

firstly, the size of a league in SC2 is 32% only for gold league, the rest are 20% or less. you are literally quoting the absolute worst case scenario and I think I am completely justfied in accusing you of cherry-picking.

secondly, mmr does not need to be collapsed into 1 number, mmr is collapsed into 2 number, the computer considers a 44-54 person as a 49 +- 5, it considers a 48-53 person as a 50.5 +- 2.5

thirdly, your third point doesn't even make sense. you are right the rank being between 44 and 54 does not imply the things you say, but thats because my rank being between 44 and 54 doesn't imply anything at all about the system at large, this point is moot.
the fact that my skill is a range rather than a number does however imply that any attempt of putting a specific number on it, or for that matter putting a smaller-than-actual range on it (such as a 2% range when the real range is 10%), will be deceptive in its granularity.

fourthly, division ranks are meaningful in comparison to active players, inactive players decaying out of the system is to get rid of unwanted statistical noise for the active players.

fifthly, applying a trend estimation would let you see how you have been doing so far and let you see where you will probably be in the future, but it wouldn't be any more accurate than what we currently have without extremely unnecessary large amounts of processing.
for example, after each game the current system uses perhaps 10-20 operations to record the result and adjust your mmr, is it really worth it to improve the systems accuracy by a tiny amount if doing so will require 1 000 000 - 2 000 000 operations after each game?

lastly, you do know that heroes doesn't actually use your rank to match you with people right? and it still wont after this change.
it has a hidden mmr just the same as in SC2 which is used to match you, have you ever gotten a "skill bonus" in heroes? thats a bonus you get when your hidden mmr has raced far ahead of your displayed rank and the system tries to let your rank catch up.
for example, a rank 45 player that has just started climbing in hero league might have a hidden mmr in the 20ies and when that player wins it gets a skill bonus to accelerate the climb up to 20ish, but when it reaches that rank it stops getting the skill bonus, and wont get skill bonus again unless it somehow becomes extremely much better very quickly.
the matchmaking is completely unaffected by what rank is being displayed.
the rank is literally just there to be an aestethically pleasing indication of your rough position, there is really no need to tell players exactly where in the list of millions of players they are, doing so would just give the deceptive illusion of accuracy.

1. It's not cherry picking. 16 times worse is the precision face by a pluralities of players, because there are more players in gold league than any other league. 20%, which is diamond league, is 10 times worse. The fact is 98% of players are in a system where the precision is 4 to 16 times worse than HotS. This is not cherry picked, it affects virtually all players. And thus, the rank conveys almost no information.

2. That's not what I said. I said a rank needs to be collapsed into 1 number, even when MMR is more than one number. Is (1405,30) > (1490,80)? This question is not possible to answer because R^2 is not a ordered set. It needs to be transformed into one number, then ranked. In SC2 that rank is league by points. It's also wrong due to distortion and bias.

3. Your skill is not a range, it's a probability distribution function. And even if it is a range, leagues are NOT that range, so your discussion of leagues in this context is misleading, irrelevant, and a red harring. The SC2 system does not tell you your range, points trend towards the mean of your MMR, not the range of your MMR. The league represents the percentile of the mean of your MMR, it says NOTHING about the range of your MMR. The range in the league size is arbitrary, it is not the range of your MMR, so it is useless. And it's wrong you to imply it has anything to do with the range of your MMR.

4. Division ranks are not meaningful compared to active players, because they tell you have you rank compared to 100 arbitrary players, not the entire playerbase, unlike the HotS system. It lets you lie to yourself about your rank, because giving you misleading and meanginless information, distorted by league rigidities. Even amongst active players you don't know if that person is suppose to be demoted but can't get kicked out, or promoted but Blizzard hasn't updated the league boundaries. While division ranks are meaningless, HotS ranks have a clear meaning, the percentile of the mean of MMR.

Your argument is: "The percentile of mean of MMR shown in HotS is not the MMR range, therefore, hide it and distort the ranks instead". A non-sequitur. It does not make any sense.

5. Your knowledge of math appears to be lacking. Applying a trend filter does not require "1 000 000 - 2 000 000 operations after each game". It could be as simple as a weighted average of your MMR over the last 10 games, this would require 10 multiplications and 9 additions. It's not about seeing where you are and where you're going (more players it will essentially be a flat line after 20 games or son), it's to remove variability in ranks, i.e. fix what you're complaining about.

6. Games use a skill rating call MMR to match you with people. Heroes will also use what is essentially this skill rating for ranking too. The difference is SC2 rejects the use of this skill rating for ranking, instead it distorts it until it is biased and wrong, and then uses that as a rank. That why the nonsense system has been thoroughly rejected by Dustin Browder in HotS.

ZackAttack

United States884 Posts

August 09 2015 06:31 GMT

#43

I really think that your assertion that people get "stuck" in leagues and can't move up or down even though they should is dead wrong. If you win you move up, and if you lose you move down there is not one person right now in all of SC2 Battlenet that is "stuck" in a league too high or too low. When people say that they are always just wrong about how good they are.

paralleluniverse

4065 Posts

August 09 2015 08:55 GMT

#44

On August 09 2015 15:31 ZackAttack wrote:
I really think that your assertion that people get "stuck" in leagues and can't move up or down even though they should is dead wrong. If you win you move up, and if you lose you move down there is not one person right now in all of SC2 Battlenet that is "stuck" in a league too high or too low. When people say that they are always just wrong about how good they are.

Not true.

You can't get demoted no matter how many games you lose. So leagues are wrong.

Blizzard's league boundaries are also wrong and are only manually updated when sufficiently large numbers of people complain on the forums that they are not in the league that they deserve to be: http://us.battle.net/sc2/en/blog/18141070 So leagues are, again, wrong.

This is why it is better to scrap leagues, which are discrete groups subject to rigid rules regarding promotion and demotion, and replace it with a continuum of ranks free of these distortionary constraints, like HotS does.

ZackAttack

United States884 Posts

August 09 2015 15:18 GMT

#45

On August 09 2015 17:55 paralleluniverse wrote:

Show nested quote +

I agree that it would be much better in every way to have a continuous ranking system, at least to the point of integers, instead of a discrete ranking system with large, arbitrary bins. I don't think anyone would argue against the fact that ranking people only on MMR would be the best way to go. The Heros of the Storm system is a step in that direction, so I think it is better, and would work even better for a 1v1 game than a team one.

However, the system is not as bad as you say it is. People are, by a vast majority, exactly where they belong. The problems with distribution in the past was not caused by the fact that there are no demotions, because it was the lower leagues that were over inflated, not the higher ones. That was a technical problem with the way the system was handling the percentages.

HewTheTitan

Canada331 Posts

August 09 2015 22:36 GMT

#46

This is him talking about heroes, right?

According to this analysis, it takes at least 600 games to have a somewhat-accurate ranking in a 5v5 elo-style system, and before that it will be nonsense. Even at 1000 games, it's a spread and not a tight grouping for 5v5 games (see comments for the 600 number).

Saying that they can place you in only 100 games is demeaning to everybody with complaints, imo.

Elo Hell: Testing Moba 5v5 Rating Systems. Do they work?

Prev 1 2 3 All

Please or register to reply.

Dustin Browder Wants Accurate Ranking Using MMR... - Page 3

Completed

Ongoing

Upcoming