|
The lack of an overall ladder ranking has been discussed to death. A lot of people hate the divisions. The popular sentiment is that they simply hide information which Blizzard should show us by adding an overall ladder ranking stat while leaving the division system they obviously want to keep in place - a compromise of sorts.
I think the problem goes far deeper than these suggestions acknowledge. The division system does not hide information; it destroys information. There are probably other problems that would be more obvious if any of the algorithms for point calculations, bonus point accumulation, rank up/down requirements, etc., but there are at least two that are obvious from what we know now.
First, let's make sure we're all on the same page about how the system works. The Blues haven't told us a whole lot, but here is what we can know for certain about the system:
-It places you in an initial division based on your placement match record, probably weighting each placement match based on opponent skill.
-Once placed, you start at 1000 points and gain/lose them by winning/losing matches.
-The points won or lost vary based on relative skill, like an Elo system: if you are expected to win you gain fewer points for a win and lose more for a loss, and vice versa.
-Bonus points accumulate at the same rate regardless of when and how much you play. If you win a match and have accumulated bonus points, those points get added to your score and removed from your bonus pool, up to a maximum of the number of points you would have won normally.
-At some point after performing better or worse than the system expects for a consistent period, it decides to move you up or down a league. When this happens, your point total resets.
-Numbered divisions are purely there to divide leagues to smaller units and are not in order of skill or anything like that.
-Ranking up or down involves a complex formula Blizzard feels we shouldn't get to know, but it is not based simply on win/loss ratio or points. It seems to involve flagging you for a potential move and then giving you a series of pair-ups or pair-downs as a test, but it's not clear how that works exactly.
-The system intentionally matches people of different skill levels in some cases to help people who are mis-placed move more quickly, and to give a little change of pace perhaps.
Also for reference: the Elo system is a common, popular competitive ranking system, used in competitive chess as well as many competitive video game systems. Each player starts with a set score and gains/loses points through wins/losses. Favored players lose more/gain less. Every point gained by the winner is taken from the loser; no points are ever created or destroyed except by the addition of new players to the system (inactive players are not deleted per se, so the points per player remains fixed, although high levels of inactivity or game abandonment from unsuccessful players can skew the numbers). When run properly, such as with competitive chess, two players scores alone can give pretty accurate statistical odds of the outcome. This accurate prediction makes the algorithm for weighting gains/losses based on relative skill much more precise, as well. In theory, over the long term, each player will reach their approximate 'true' skill and their rating will stabilize, varying only a little bit in the short term without moving up or down in the long term unless the player's skill changes (or the average skill of the competition changes). It's also very resistant to non-random match assignments (e.g., players in the same areas playing in mostly the same tournaments).
On the surface, it looks like you have basically an Elo system for the points, a silly Division system tacked on for no apparent reason, and the problem is that rankings are based on the silly Division system instead of the more rational Elo system. Depending on how divisions are used with the matchmaking algorithm, there may be additional undesired effects there, but at least we have the points system, right?
Wrong. The points system as implemented, while resembling Elo, does not accomplish the mathematical goals Elo sets out to achieve. There are many reasons for this and I'm sure some math major could provide a more detailed proof or explanation, but here are two core concepts that cause the current system to destroy otherwise useful data about player skill.
1) Bonus Points: This one is pretty widely understood to be a cause of point inflation. Since bonus points are basically 'created' out of thin air, the points per player in the system is no longer fixed; there's a natural tendency to inflate points over time, based on how many semi-active or active players there are (remember, it's not like rest exp in wow - it accumulates constantly, no matter how often you play, so all that matters is whether they at least play enough to redeem those points eventually). However, it doesn't just cause the average to rise; it also severely skews the distribution of points. In an Elo-based point system, the idea is that you eventually stabilize once the system has accurately placed you. Blizzard claims they want their system to achieve this, where people will stop moving around chaotically after enough games and have a ranking that somehow reflects their skill level. The bonus point system is clearly designed to do the exact opposite, presumably so 'average' players will continue to improve slowly in ranking/point total even if their skill remains the same.
Obviously, though, not everyone can be moving up at once or they'd all balance out. The problem is, since players continue gaining points faster than losing them even once they reach what *should* be their 'equilibrium value', they get paired against more skilled opponents and start losing more than half their games, shedding those undeserved points. This is part of why we see rapid accumulation at the top of each division while the middle point value moves much more slowly. This undermines the 'carebear' goal of making average players happy, since instead of knowing they're average and winning half the time, they feel like they "should" keep moving up since they don't seem to be stabilizing in rank, but every time they start making real progress they get an 'unexplained' tough game streak and drop down again. The matchmaking system's intentional mispairings to test players to see if they should move up or down more quickly is already good enough to prevent stagnation. Adding the bonus point dynamic creates a host of new problems, all for the impossible goal of trying to create the illusion that everyone is better than average and doing better and better over time, whether this is accurate based on their in-game performance or not.
2) League Changes: this one I hear very little about, but IMO it's the single biggest problem, and the main reason that points between different divisions and leagues are not at all comparable: when you move divisions, your points reset. This is, quite literally, destroying the information about your previous point total. Numerical scores that respond to each win or loss are a much more useful measure of skill than which of five named leagues you're placed in. This creates a host of wildly unpredictable effects based on other algorithms we are not privy to, but it clearly means the system no longer functions as Elo-like and points themselves have no predictive power between divisions or leagues - your point total is ONLY meaningful in the context of the league and, arguably, the specific division you are in.
Speaking of which: as if creating five tiers of player with some incalculable, unspecified, hazy skill differentiation between each wasn't bad enough, the league change point reset system makes divisions within the SAME league impossible to compare! We don't know how the rank up/down algorithm works but we know that it's fairly complex, is not based on a preset number of fixed games and takes a lot of variables into account beyond just win/loss record over the long term weighted by opp skill. In practice, what this means is that the number of players any given division will gain and lose from this procedure over time can vary dramatically. Additionally, the average points per player in the division will change every time a rank up or down happens, since the point total that player had when the rank change happens can vary by quite a bit. Also, remember that bonus points accumulate slowly over time at a more or less fixed rate, meaning the longer players tend to stay in a given division, the more point inflation will occur.
Based on these issues, and no doubt many others that I haven't mentioned or thought of yet, I can't really imagine how they COULD use the current system to generate a meaningful cross-division ranking. This probably helps to explain why the Favored algorithm is so notoriously inaccurate (comparing across divisions accurately is impossible) - which also further distorts point distribution since that's the algorithm that determines the number of points each player stands to gain or lose. Also, this may be why Blizzard opted to make the Pro League just one really big division above all the others - they realized that's the only way to approximate a serious ladder for the top players, since relative rankings are only at all meaningful within a single division in the current system.
Thoughts? Am I way off base? Does this make sense to people? Any ideas for practical solutions to suggest to Blizzard beyond "start over"?
|
tl;dr:
The ladder system is broken for two main reasons: bonus point inflation and point resetting on division change. The combined effect is that, instead of being a quantifiable estimation of player skill as with an Elo ranking system (see: competitive chess, Heroes of Newerth, many other games that aren't coming to mind off hand), points in the current ladder only tell you a bit about how you compare with the rest of your division, and make up part of the mysterious, unspecified rank change algorithm. In practice, this leaves you with Blizzard's unexplained classification into one of five tiers and a wild-assed guess in the form of a numerical score that is only directly relevant in comparison to at most 99 other players - better than nothing, sure, but hardly deserving of the title 'ladder'. It's no wonder Blizzard's "solution" for pro players at the top is to make one giant division with fixed size per ladder cycle - basically, eliminating all the unique aspects of the whole league and division system, since none of them are actually useful for competitive players who want to compare their performance.
Yes, I recognize that my tl;dr is longer than most posts. Deal with it.
|
This is only speculation but in the last big thread on the topic the op assumed that there is a second hidden rating, which is used to match players together. This is probably the "best there is" ranking system that some Blizz guy talked about at some point.
This assumption makes a lot of sense, because the publicly displayed rating is obviously not a good ranking, for the reasons you mentioned (although I think that the second point should be name "leagues and divisions").
Edit: Actually, I just remembered that the op I was referring to had the Idea with two rankings from WoW, where apparently both rankings are public? Idk.
|
As far as I understand it, the way SC2 battlenet is set up for beta is the system will hunt for players to give you a quicker match, so it's tuned to find you a match faster, and not find you someone closely matched to your skill level. Come launch I'm expecting the system to change so it will find you someone closer to your rating and change the ELO "hunting" range when pairing people for matches much more slowly, as well as reducing the max deviation.
As for the ladder itself, the divisions are there to give a sense of "local" competition with each other, and with multiple #1 spots spread across divisions there isn't as much pressure on competitive players to fight for ELO rating all the time if it were one global ranking (or more specifically, regional ranking in the US, EU and Asia), as hundreds or thousands of competitors fight for the top spots.
The one problem I see is the bonus points system, it completely ruins the whole purpose of ELO and seems like an arbitrary addition. It's supposed to try and let players "catch up" to those that play more, but ELO isn't supposed to reward playing more; it rewards winning more matches than you lose and improving your skill over playing match after match.
|
Yea, I think Pro league is going to be straight up ELO without bonus points. It's all set up to be with the weekend gamer in mind. Nice post dude, it was a good read.
|
Very nice post.
I for one am completely dumbfounded as to why there isn't an overall ladder. I have no definitive way of proving that I'm a worse, or better player than anyone close to my ranking. Very frustrating for those of us with a competitive view towards the game.
The fact that Blizzard won't reveal the ladder promotion formula is another baneling in my mineral line.
|
Thanks for the replies, everyone. Some responses:
Asta: I hope that 1) you're right, and 2) that stat becomes public on launch. Based on the problems with the Favored system and anecdotal evidence of how it seems to compare W/L, League and points/div ranking in some fashion to determine that value. This leads me to believe it's actually pairing based on what we see, which is disturbing 
Spaztick: Agreed 100% that bonus undermines any Elo-like value of the points system - it's part 1 of my post  It's not *as* bad as a rested system, at least, since active players get the same bonus accumulation, but it prevents players from reaching an accurate point equilibrium that only changes as their skill relative to their average competition changes, which is the basic goal of Elo.
Part 2 is the big one, though. Since points are reset when you change leagues, and that and the other non-Elo-like characteristics introduce unpredictable changes to the average points per player between divisions within the same league, points are only really comparable within a single division, making real matching and comparison between divisions, let alone leagues, quite difficult to say the least.
baeracaed: That would certainly make a lot of sense, but depending on how big the pro league is, it will probably leave many of us who are competitive but not pro-caliber players stuck slummin' it in the broken League system instead.
The best solution I can see is to rename Pro League into Competitive League, make it available via opt-in for anyone, and have it be as close to a straight-up Elo system as possible; random team matching can't be "pure" elo, so they could either make comp league 1v1/AT only or just make the RT comp league a slightly modified Elo like many team-based games do (see: Heroes of Newerth). I would like to think Blizzard could make something *better* than pure Elo, with changes that add incentives they want to add without destroying the basic functionality of the system, but if the public ladder system we see has anything to do with matchmaking, then I have little faith in them on that department.
|
I'm a bit sorry but remember, it's the beta! Also, there are plenty of tournaments for judging the overall skill pool. I actually like the system.... It's much more encouraging for players to move from spot 28 to spot 12 than spot 1078 to spot 1062 (as has been said before.)
I think it's a much better system considering the wide range of skill than, say, iccup would be for the general public.
|
Blizzard seems to be making several huge mistakes with their ladder system. However, I don't think bonus points are a problem because everyone gets them equally and it stops a player from getting a top ranking, then never playing again and just staying on top even though the level of play is drastically different from when he originally got it. This is why it is hard to compare chess ratings from now to many years ago.
We definitely need an overall ladder, it's just stupid that Blizzard hasn't added one. They made an assumption about players feeling better if their rank is a lower number, they are wrong. At least for competitive players.
Though I think I can partially help with http://starcraftrankings.com 
Imo the worst problem is that the system is complex and the details are secret. You cannot create a system in which anyone will put their faith in if they cannot understand it. This is what happened to WC3 after they overhauled it and added the hidden ratings and such. Quickly everyone stopped trusting it and you will often hear the response "ladder rank doesn't mean anything" on WC3's Battle.net if you try to use it for anything.
It must be simple, complete, and easy to understand or it simply will not succeed.
|
Im just guessing here but Blizzard probably wants the ladder to be "alive", if you do not play regularly they want you to lose your rank to more active people.
The WC3 ladder demanded that you played a certain number of games each week. If you did not reach this number, the remaining games were counted as losses (your record was not affected, just your ladder rating) so you would have a hard time getting your "real" rating and rank back after a period of inactivity
The bonus points system achieves the same goal, but it is far easier for a player who has been inactive for a few weeks to get his old rank back.
|
I really don't think people want to see 25, 000 people in their silver league.
|
On May 02 2010 06:31 Jonoman92 wrote:
I'm a bit sorry but remember, it's the beta! Also, there are plenty of tournaments for judging the overall skill pool. I actually like the system.... It's much more encouraging for players to move from spot 28 to spot 12 than spot 1078 to spot 1062 (as has been said before.)
I think it's a much better system considering the wide range of skill than, say, iccup would be for the general public.
I know it's the beta - that's why I'm trying to detail the flaws of this system now, while there's still some chance it can be fixed before launch. There's no indication that the division and league system is currently due for a substantial change. Blizzard reps give excuses for its shortcomings and to my knowledge have never said anything about them considering or testing any significant changes to the basic system.
I'm very sick of hearing the "it's more encouraging" argument. This is a competitive game. Ladder rankings are not meant to be friggin' participation trophies. Besides, I don't even think I agree with you - I would much rather see that every game moves my ranking noticeably, and that over time I can actually see myself improve by THOUSANDS of rank positions as I pass other players in skill and performance, and knowing that the division ranks are meaningless outside the context of my one division means I really don't care about my rankings anymore. I'm a hyper-competitive person who always has to win and be the best at everything, yet I'm currently #3 in my division and don't really feel any motivation to get to #1. It doesn't mean I'm the best at something; it means I passed two other people in a more or less meaningless score.
People who suck don't need to worry about their ladder ranking. They have a whole achievement system. Heck, they could keep the divisions but change the things I outlined and create a globally comparable point estimation of skill and corresponding global ranking. Or, they could do the comp league thing, making an Elo alternative available to everyone but sticking people in their carebear system by default.
So far, I have heard dozens of experienced players tell me bad players like divisions, but I've yet to hear a Copper player chime in and say "I agree, I love being top 10 copper and would dislike an overall ranking being added."
|
On May 02 2010 06:02 Spaztick wrote:
The one problem I see is the bonus points system, it completely ruins the whole purpose of ELO and seems like an arbitrary addition. It's supposed to try and let players "catch up" to those that play more, but ELO isn't supposed to reward playing more; it rewards winning more matches than you lose and improving your skill over playing match after match.
There are two conflicting sides to the bonus point argument. Casual players are more likely to stop playing for, say, a week, and perhaps for those people the bonus point system is a good thing, so they don't feel like they need to keep playing to keep up their rank.
On the competitive side, however, the ranking system needs to punish people for not playing, to prevent people from camping their spot on the ladder.
|
On May 02 2010 06:38 DrSmoke wrote:
I really don't think people want to see 25, 000 people in their silver league.
So don't have a league. People got on just fine in the SC1 and WC3 ladders with one very long ordered list as a ladder. Your point score would become a lot more meaningful since it wouldn't inflate with bonus points and reset with rank changes. Every single game you ever play on the ladder will impact your score, and that never goes away. You're telling me this would be a less appealing motivation to you to compete in a ranked ladder environment than a system that only compares you to 99 essentially random other players and makes no serious attempt to find where you "belong" as a stable ranking but rather bounces you around through a combination of undisclosed formulas and ill thought-out unpredictable variations within and between divisions? Kay, then they should make the system that doesn't suck optionally available and you can continue to enjoy the one that does.
|
On May 02 2010 06:42 vAltyR wrote:Show nested quote +On May 02 2010 06:02 Spaztick wrote:
The one problem I see is the bonus points system, it completely ruins the whole purpose of ELO and seems like an arbitrary addition. It's supposed to try and let players "catch up" to those that play more, but ELO isn't supposed to reward playing more; it rewards winning more matches than you lose and improving your skill over playing match after match. There are two conflicting sides to the bonus point argument. Casual players are more likely to stop playing for, say, a week, and perhaps for those people the bonus point system is a good thing, so they don't feel like they need to keep playing to keep up their rank. On the competitive side, however, the ranking system needs to punish people for not playing, to prevent people from camping their spot on the ladder.
Those aren't conflicting. The bonus pool DOES punish people for not playing. If you stop playing, other people will overtake you with their bonus points. Otherwise you could just sit on the #1 spot forever and never have to play again.
|
On May 02 2010 06:42 vAltyR wrote:
There are two conflicting sides to the bonus point argument. Casual players are more likely to stop playing for, say, a week, and perhaps for those people the bonus point system is a good thing, so they don't feel like they need to keep playing to keep up their rank.
On the competitive side, however, the ranking system needs to punish people for not playing, to prevent people from camping their spot on the ladder.
Except that bonus points accumulate at the same rate whether you play or not, so it doesn't really help you catch up at all - just gives inactive players a temporary boost in points that causes them to get matched against those better, more active players to shed those points, while active players get their bonus points more evenly divided between games as they mass points at the top of a division. An Elo system does not require activity to maintain your ranking; once you reach your equilibrium rank, your score should not change over the long term until your skill does, whether you play once an hour or once a month.
Bonus point accumulation doesn't differ from activity, but distribution of those points does. Inactive players will get 'streaky' point changes; active ones will get smoother point inflation. Whether this causes problems with those streaky players then getting a frustrating losing streak or not remains to be seen, but I would imagine if an inactive casual player built up a massive bonus pool, any decent winning streak would shoot them up in ranking and start getting some higher league face crush matchups. More bonus points left? Great, as soon as you finally get one win, wash, rinse, repeat. That potential issue aside, I don't see what *good* they do since everyone gets them at the same rate. I guess people who literally never win don't benefit from point inflation that way?
|
I agree that a system to compare divisions would be nice, but I still say that 90% of the people that end up buying and playing sc2 are not going to want to look at their ladder and see 10K+ names to sort thru.
I look at my ladder, i have 100 names to look at. Its simple and un-cluttered that way.
|
On May 02 2010 06:49 DrSmoke wrote:
I agree that a system to compare divisions would be nice, but I still say that 90% of the people that end up buying and playing sc2 are not going to want to look at their ladder and see 10K+ names to sort thru.
I look at my ladder, i have 100 names to look at. Its simple and un-cluttered that way.
So.... Give it pages like WC3's ladder, and search functionality? It's not hard. It would obviously be stupid to make players scroll through a list of every single SC2 account on one page.
|
and the main reason that points between different divisions and leagues are not at all comparable: when you move divisions, your points reset.
I just moved up yesterday to plat, and started at rank 9. your points are not reset (pre-server reset i moved up from silver to gold, and i lost a significant portion of my points but i wasn't set back to 1000 either)
|
On May 02 2010 06:49 DrSmoke wrote:
I agree that a system to compare divisions would be nice, but I still say that 90% of the people that end up buying and playing sc2 are not going to want to look at their ladder and see 10K+ names to sort thru.
I look at my ladder, i have 100 names to look at. Its simple and un-cluttered that way.
i completely agree with this guy.
|
|
|
|
|
|
EPT
