|
Objective: To have a system that quantifies a player's career accomplishments in some universally comparable format.
Just this weekend, we had a conversation in #liquipedia (the best channel on IRC!) about how to best display players' results (top finishes in important tournaments). That lead inevitably to a long conversation about what results actually mean the most. For example, which is most impressive: 4 DreamHack 2nd places or a few top MLGs? Perhaps more difficult, is every GSL championship "worth" as much as the others?
Perhaps the most important conclusion was that tournaments cannot be judged by their identity as a tournament, but by the player pool. Furthermore, a run that dodges the best players, even if it results in a championship, isn't as impressive as one that goes through top players one after another.
So if tournaments must be evaluated on an instance-by-instance basis, what are the criteria? What metric can be invented that might adequately describe the value of a player's career of tournament successes?
With these questions and plenty more in mind, I have come up with the sketches of an idea. Disclaimer: I'm nothing of a statistician.
TLPD has long kept Elo ratings for players as a measure of their current skill. It is a moving average system that works in much the same way as the points system in-game: more drastic changes in ranking for beating players considered much better than you and losing to players considered much worse. While a good measure of current player skill, the system does not immediately lend itself well to a cumulative idea of a player's career.
So what I am thinking might be a good solution is to use the same sort of points system, without the baseline, with appropriately adjusting weighting.
Leenock's MLG win would have some Elo score for the many matches he played en route, just as jjakji's and HerO and PuMa's recent wins would. The Elo rankings of their opponents (as well as the champion's record against them) would create a score for the player for the tournament. I imagine it would be possible that the highest score for a tournament could go to a player who didn't actually win it, if they had an otherwise impressive run.
I believe that subtracting points for losses is pointless, as a loss does not detract from a championship. However, it seems similarly pointless to count every win, as those that don't lead to "impressive" finishes also don't contribute to a good picture of a player's career.
So, TeamLiquid, do any of you have ideas about how to make such a system work?
 
|
Estonia4644 Posts
#liquipedia (the best channel on IRC!)
quoted for truth
and just because i feel the need to post
empty -> c[_]
|
I didn't quite understand if you accept ELO as a decent measure of current player skill or not? If so, you could just accumulate the ELO points of a player (take the integral or sum, mathematically speaking). Whenever a player does well, his ELOsum is going to increase by a lot, if he does bad his ELOsum will be stagnant or grow by only a smaller amount. Of course players play for different timespans so one has to somehow account for that.
|
You have to rank the events based on those who attended and their level and performance for the period; I created a system for the website GotFrag and ranked Counter-Strike teams (and later TF2). Naturally, including the matches and the scores of the matches would lead to a more accurate ranking, but it was far too time-consuming to really be undertaken and events never kept a good record to implement that.
An explanation of the specifics can be read here (it's nothing special, but quite effective): http://www.gotfrag.com/cs/story/44423/?spage=2
It's not completely applicable in regards to StarCraft just because of the nature of how events work and how many players are involved in a one-on-one game versus a team game. That being said, it's not too difficult to tweak the system for StarCraft if one were so inclined. If the process doesn't jump out at you, send me a message and maybe we can talk through some things.
|
On December 06 2011 17:28 surfinbird1 wrote:
I didn't quite understand if you accept ELO as a decent measure of current player skill or not? If so, you could just accumulate the ELO points of a player (take the integral or sum, mathematically speaking). Whenever a player does well, his ELOsum is going to increase by a lot, if he does bad his ELOsum will be stagnant or grow by only a smaller amount. Of course players play for different timespans so one has to somehow account for that.
Elo seems to be a decent way to measure player skill in a situation where the only reliable data is wins and losses. I think a simple integral would include and double-count too much.
On December 06 2011 22:15 divito wrote:You have to rank the events based on those who attended and their level and performance for the period; I created a system for the website GotFrag and ranked Counter-Strike teams (and later TF2). Naturally, including the matches and the scores of the matches would lead to a more accurate ranking, but it was far too time-consuming to really be undertaken and events never kept a good record to implement that. An explanation of the specifics can be read here (it's nothing special, but quite effective): http://www.gotfrag.com/cs/story/44423/?spage=2It's not completely applicable in regards to StarCraft just because of the nature of how events work and how many players are involved in a one-on-one game versus a team game. That being said, it's not too difficult to tweak the system for StarCraft if one were so inclined. If the process doesn't jump out at you, send me a message and maybe we can talk through some things.
Emphasis mine. That phrase nailed that complication of crediting accomplishments.
Your post is a simple solution. However, rather than pick a tournament's point value, I'm wondering if there's some way to consider each individually. If you had the time or access to full statistics, how would you have done your CS ranking?
|
5003 Posts
You have to rank the events based on those who attended and their level and performance for the period;
Haha, exactly what I was thinking and was about to post after reading the OP.
Luckily, the data for SC2 stuff is a lot more accessible so it's much easier to generate points like that once you have a working system.
|
On December 07 2011 01:39 Milkis wrote:Show nested quote +You have to rank the events based on those who attended and their level and performance for the period; Haha, exactly what I was thinking and was about to post after reading the OP. Luckily, the data for SC2 stuff is a lot more accessible so it's much easier to generate points like that once you have a working system.
And for that, I think a system like TLPD's Elo would do a nice job.
Are there other criteria beside opponents? Does prize money have any real impact on players? Perhaps it's only valid for tournaments sure to get a player's best effort?
|
The ELO is great for some games but sucks and represents nothing for Starcraft or eSports in general.
If you want to quantify accomplishments, you should first develop a more adapted rating system.
When you got your system you just add for example all the points a players earnes over time (his carreer here).
If you want some ideas to developp a more accurate system (or say more adapted to Starcraft) you should look into "incorporating" parameters to the ELO system like :
Parameters to the player :
- inactivity
- winning / loosing streakes in account
- jetlag
- stress
- winning / loosing history against your opponent
Parameters related to the event :
- mappool
- meta-game
- players in your pool / bracket
- prize-money on stakes(relates to stress)
- number of people watching (relates to stress)
This elements won't get you a system thath gets the "achievement" but it will mesure the strength of a player against another considering all those factors. There are probably more that you can add to this list, the more you add paramaters the more accurate it will be.
Edit : after thinking a bit about it, even bracketplay isn't really fair (at all)
Try to go into other systems like the "Glicko system" (if you have the math knowledge, wich I don't ^^), "Kespa System" ...and so on, it will give you some more ideas.
- About substracting points for losses, I don't know what it represents mathematicly but psychologicaly it's sure that a loss will affect you for the next games.
|
On December 07 2011 06:11 LunaSea wrote:
The ELO is great for some games but sucks and represents nothing for Starcraft or eSports in general.
If you want to quantify accomplishments, you should first develop a more adapted rating system.
I'm not sure it'd be worth the effort to replace one ratings system with another, unless there was one that contributed to a tournament's "value" directly.
|
On December 07 2011 06:44 Wren wrote:Show nested quote +On December 07 2011 06:11 LunaSea wrote:
The ELO is great for some games but sucks and represents nothing for Starcraft or eSports in general.
If you want to quantify accomplishments, you should first develop a more adapted rating system. I'm not sure it'd be worth the effort to replace one ratings system with another, unless there was one that contributed to a tournament's "value" directly.
I don't really understand what you mean, but in my opinion, if you "rate" badly players like not taking in account metagame and mappool your "source material" will be flawed aswell. So any system using that data is corrupted by design.
|
On December 07 2011 06:51 LunaSea wrote:Show nested quote +On December 07 2011 06:44 Wren wrote:On December 07 2011 06:11 LunaSea wrote:
The ELO is great for some games but sucks and represents nothing for Starcraft or eSports in general.
If you want to quantify accomplishments, you should first develop a more adapted rating system. I'm not sure it'd be worth the effort to replace one ratings system with another, unless there was one that contributed to a tournament's "value" directly. I don't really understand what you mean, but in my opinion, if you "rate" badly players like not taking in account metagame and mappool your "source material" will be flawed aswell. So any system using that data is corrupted by design.
I suppose. Assuming that we were to build a good enough system of rating tournaments by players competing, the measure of player skill would be substitutable, right?
Anyway, how do you propose to take the qualitative/subjective measures like jet lag and turn them objective?
|
- Well yes, you will rate tournaments maybe "effectively" by substituting the player skill but it also means that you can't measure achievement because than your basically reintegrating the players skill.
A tournament can be important, but if a player wins that tournament it doesn't mean that he achieved the same thing as the former winner of that same tournament.
Maybe the litle imablances in the game, the mappol or any of those parameters could influence the games for another player in that tournament preventing him from winning.
- (For that example I will say that you have two players, a Zerg and a Protoss)
If you take metagame for example it would be something like multiplying your final result (for the Zerg) by the odds of every ZvP played on this patch / season.
Than you can do a similar thing with mappool odds...etc
- For qualitative / subjective measures I can't really say, but I think that the best would be to compare "similar situations" for that particular player (like the same tournament, against a players that has the same rating) and see how it turned out.
It's basicly observing a system where no parameters vary besides the qualitative / subjective one (liek jet lag).
Since that takes quite some sample data you could try with the results that team liquid recorded on TLPD or give your calculated rating an error margin.
|
On December 07 2011 07:28 LunaSea wrote:
- (For that example I will say that you have two players, a Zerg and a Protoss)
If you take metagame for example it would be something like multiplying your final result (for the Zerg) by the odds of every ZvP played on this patch / season.
Than you can do a similar thing with mappool odds...etc
Interesting, I like that.
|
Thanks. =)
If you consider doing something like that I would be happy to help you, I find those things really interesting. Don't hesitate to pm me.
|
|
|
|
|
|
EPT
