|
Hey all,
I'm working on my master's degree in statistics. I've wanted to do a project that was Starcraft related. So, for my semester project in my Bayesian Methods class, I thought I'd do a ranking system for the GSL players. I hope to expand on this later, but right now it just takes all the brackets, the win/loss numbers with the player names, processes them using Bayesian magic (I used a Bayesian hierarchical model with binomial prior, p = inverse logit link with 2 player skill parameters, and player skills are distributed Normal(0,sigma^2). I'll post the technical write-up later if anyone is interested), and spits out a skill parameter which can be translated into a probability statement about who will win in a game.
A lot of players didn't play enough games to be able to estimate their skill with very much confidence, Jinro and Choya are examples, but players with high uncertainty are pulled towards the group average.
Forgive the formatting.
Probabilities of each of the top 4 taking the advancing to the next round and also advancing and then winning the whole tournament (Based only on GSL Seasons 1-3 data):
+ Show Spoiler +
Remember these are only based on the data and are based on the chances of each player winning a Bo7 match against each of these other players. You should look at these predictions and say to yourself, "Those are almost all the same, based on just the data, this GSL could realistically go to any one of these 4 players." .
Names ProbWinNext ProbWinFinal
1 Rain - 0.5134 - 0.2443
2 HongUn - 0.4865 - 0.2256
3 MC - 0.4402 - 0.2183
4 Jinro - 0.5597 - 0.3115
Edit : Fixed Coding problem that made people with high variance skill's go incorrectly high.
Top Player Rankings (Final skill chosen to be mean - 2 * std. error)
Rankings on Google Spreadsheet
How to interpret the skill parameters :
The actual number is not important, only the distance between the numbers.
+ Show Spoiler +To get the probability that one player will beat another, the formula is exp(skill1)/(exp(skill1)+exp(skill2)) where skill1 is the skill of the winner and skill2 is the skill of the loser.
So if you wanted to calculate FruitDealer vs NesTea and their skills were 1.48 and 1.32 respectively, calculate exp(1.48)/(exp(1.48)+exp(1.32)) = .5387. Meaning FruitDealer has a 53.87% chance of beating NesTea in a single game. This is not exactly how I did it, but this is a simple approximation.
If this is interesting to anyone, I'd happily provide more information.
I can generate hypothetical match-ups of anyone I have good data for (Code S players I have pretty good data on as a rule of thumb).
Disclaimer
+ Show Spoiler +Take the predictions and rankings with a grain of salt, they are only preliminary and will in crease in accuracy over time. They are only based on GSL data and they also treat all seasons equally, assuming no skill change between seasons (not an assumption I wanted to make, there just isn't enough data). This makes the data more useful as a ranking tool than a prediction tool at this point. There also are other tournaments that I could potentially enter in, but gathering the data and getting it properly formatted is proving to be a pain in the butt.
Future Work
+ Show Spoiler +I have additions I'd like to add to this analysis. If I can get some help with gathering and formatting data, I could adjust for and calculate race advantages or how much of an advantage a certain map gives to a certain race. Also, if I could just get the quantity of the data to increase, I could do a lot more calculation. I'd like to add in more tournaments besides the GSL. Using just the GSL is great to have a fair localized ranking system, but to predict future games, it's best to have as much data about each player as possible.
If I had hundreds of times more data, I could calculate, based on win/loss data, which build orders, strategies, transitions and such are more effective vs which others.
Right now my data looks like this :
NesTea 2
Jys 1
Vines 1
Sen 2
Goma 2
JookToJung 0
Maka 2
Sleep 0
etc.
No race information, no map information, nothing, I just copied it straight from Liquipedia and then organized it for analysis to look like this:
NesTea Jys
NesTea Jys
Jys NesTea
Vines Sen
Sen Vines
Sen Vines
etc.
After that it was just a matter of coding up the model and interpreting the results.
So leave comments if you found this interesting or want to know more or whatever. If you want help implementing something similar, pm me. If you can think of some interesting Starcraft related questions that can be answered through data, post them and I'll see about looking into it. Also, if you are interested in helping me find and/or format data pm me.
|
Very cool stuff. I'd be one of those people interested in the technical write-up.
|
Well I'm no stats or math wiz but I think the fact that NEXLiveForever is 3rd on your rankings list means that your system could be improved a lot, clearly some weighting should be given to more recent results, and possibly qualifiers should be taken into account as well, as someone like Rainbow who can qualify for 3 GSLs and make a semi final and a finals appearance is clearly better than someone like NEXLiveForever who was only able to only qualify once and made one semi, or OGSTop who made one Ro16 and was unable to qualify after that.
|
On December 08 2010 18:53 Wargizmo wrote:
Well I'm no stats or math wiz but I think the fact that NEXLiveForever is 3rd on your rankings list means that your system could be improved a lot, clearly some weighting should be given to more recent results, and possibly qualifiers should be taken into account as well, as someone like Rainbow who can qualify for 3 GSLs and make a semi final and a finals appearance is clearly better than someone like NEXLiveForever who was only able to only qualify once and made one semi, or OGSTop who made one Ro16 and was unable to qualify after that.
Didn't LF skip the second and third season to focus on school?
|
I agree with your concern completely Wargizmo. I saw NEXLiveForever getting very highly ranked, and was like, "wtf". He's boosted, probably unfairly, by having taking out NesTea in his first set. The model, as it is, treats all games equally, regardless of round played in or which season it is in. At this point, however, with only the GSL data, trying to add in a time-effect isn't feasible. There just isn't enough data.
The biggest issue is cross-over, the single elimination bracket format is a statistical nightmare because it doesn't give you a lot of information how players would be doing against other opponents. A round robin into brackets or even double elimination would be so much better for an objective analysis.
Time effects are something I definitely have in mind for future use. I mean, it's pretty clear that a year from now, no one will care what happened in GSL Season 1 as far as predictions are concerned.
As far as fairness, at very worst, my ranking is as bad as the GomTV rankings with their arbitrary point system that doesn't take into account how difficult your bracket is. My ranking system is almost entirely based on the strength of your opponents in how much a win effects your ranking. With more data, this bayesian approach will be infinitely more fair and more reliable than a simple point system.
|
Yeah, I was just looking at the brackets again on LiveForever and Top, they are sandwiched at a high ranking by who they beat and who they lost against.
Top beat Polt, who beat MC, who is getting very highly ranked by the current season, which drags Top up. Then he lost to Fruitdealer, which pushes him back down.
LiveForever knocked out NesTea, and lost to FruitDealer. Ro4 is nothing to scoff at though, he deserves to be fairly high ranked.
I'm sure if I had more data on these guys, they'd get dragged down somewhat. But LF took out bigger names on his way to Ro4 than did Jinro, for example. I think this is a positive thing of the ranking algorithm, and more data will sort out any strange results.
|
You should use the stats to vote for your next gsl liquibet and see how it goes. Cool stuff.
|
Interesting stuff, keep improving it 
|
Liveforever didn't come to GSL2,3.
|
On December 08 2010 18:53 Wargizmo wrote:
Well I'm no stats or math wiz but I think the fact that NEXLiveForever is 3rd on your rankings list means that your system could be improved a lot, clearly some weighting should be given to more recent results, and possibly qualifiers should be taken into account as well, as someone like Rainbow who can qualify for 3 GSLs and make a semi final and a finals appearance is clearly better than someone like NEXLiveForever who was only able to only qualify once and made one semi, or OGSTop who made one Ro16 and was unable to qualify after that. Yeah that's the most glaring flaw so far. I mean Rain as 5th best? - -
Standardizing the scoring for someone who fails to qualify also has big problems (like for Tester who got knocked out in a qualifier by Foxer).
May have to separate it by season until there is more data.
|
u rly got nothing better to do
User was warned for this post
|
On December 08 2010 19:19 Mip wrote:
Yeah, I was just looking at the brackets again on LiveForever and Top, they are sandwiched at a high ranking by who they beat and who they lost against.
Top beat Polt, who beat MC, who is getting very highly ranked by the current season, which drags Top up. Then he lost to Fruitdealer, which pushes him back down.
LiveForever knocked out NesTea, and lost to FruitDealer. Ro4 is nothing to scoff at though, he deserves to be fairly high ranked.
I'm sure if I had more data on these guys, they'd get dragged down somewhat. But LF took out bigger names on his way to Ro4 than did Jinro, for example. I think this is a positive thing of the ranking algorithm, and more data will sort out any strange results.
That makes sense, but having said that, there still needs to be some weighting for recent results, to account for a player improving over time. NesTea for example had only just switched over to Sc2 in GSL1 and if you watched those games against LiveForever you would see that he was an absolute newbie at the game back then.
Also I don't see why a player's performance in in GSL3 would determine his rating for a game that occurred in GSL1 in the first place, that seems kind of silly to me. I mean if someone else LiveForever beat ends up winning a GSL then you could potentially have this guy rising up the rankings even further without even playing a single game.
|
|
|
Could you give us the standard error of your probabilities ?
I'd like to know if they're relevant or not, obviously you don't have a lot of data with just GSL 1-3.
I like the idea though !
|
you have a high chance of winning the LB for gsl4! lol
|
My background: I am studying mathematics to diploma for 12 semesters now.
To be honest, I think your model should imply matchup deviation, therefor needs to be at least 3-dimensional. Of course, considering every map would be even better but then the sample pool will by far not return you any valueable information.
But unless you at least concern matchups I fear the model is too theoretical for any substance.
I sorted around the issue aswell when I wanted to do some Starcraft manager toy and wanted to use winning probability to generate match results.
Adressed to the point of your transititive conclusion (A beats B beats C), you can't nevertheless your amount of data, because as you already figured out with your approach, matches can be interpreted as weighed coinflips, but you cannot achieve any kind of transitive ordering in a competition (which is good, because thats the point of competition after alll *g*)
Nice stuff though, I love mathematical approaches to this.
Edit: If you want to generate forecasts, you should consider reading the book from the guy that wrote that baseball statistics book and developed the "on-base-percentage" stat. I can for heavens sake not recall his name, i'll look after it when i'm home, you americans though might know him right away, he was consulted by the Red Sox before winning MLB.
|
What is the error for the probability that you generated? Its not something ridiculous like +-0.5 right? 
|
|
|
|
|
I know trolls should not be fed but seriously, if someone put work and time into something concerning SC2 he should be credited and not patronized. You are the sad fuck with no life flaming people like him.
|
|
|
|
|
|
EPT
