| Forum Index > StarCraft 2 Tournaments |
|
RoboBob
United States798 Posts
 I don't know if you've thought of this yet, but if you want to test the validity of your model, why not use SC1 data instead of SC2? Yes, the differences between SC1 and SC2 will introduce more uncertainty into your model, however the wealth of data points you'll gain from it might be worth it. Just a thought. | ||
|
Mip
United States63 Posts
The rankings that I posted are the posterior means of the skill parameters - 1 standard deviation. It's somewhat arbitrary. If I made it 2 standard deviations, you'd see players like LiveForever drop down a lot. For people like FruitDealer and NesTea, there are a lot more games used to estimate their skill, so if you penalize uncertainty, players who have played a lot of GSL games will float to the top. Here's a google spreadsheet of the full ranking results : GSL Ranking Results | ||
|
Mip
United States63 Posts
| ||
|
SolonTLG
United States299 Posts
Again, I agree that a point system cannot account for everything, and a richer model would be preferable. All I am saying is that a point system can be informative. | ||
|
Mip
United States63 Posts
For example, in the GSL Season 2, FruitDealer lost to MarineKing in the Ro32. By the point system, FruitDealer lost out on a lot of points by not getting further in the tournament. In the Bayesian model, it takes into account that MarineKing is friggen good, so losing to him isn't really that big of an upset. The point system also gives no way to quantify uncertainty about the players skill. Realistically, either approach works fairly well, the Bayesian approach is just more dynamic in the way it ranks. | ||
|
CrAzEdBaDgEr
Canada166 Posts
Keep up the good work. | ||
|
SolonTLG
United States299 Posts
On a slightly different tack, on the State of the Game podcast a couple of weeks age there was a big discussion about MLG's extended series tiebreaker. The crux of the argument centered around whether different rounds of a tournament should be considered different; that is, does defeating someone in the Ro32 mean something different than defeating someone in the Ro8. In my opinion, yes, and thus the ending point of a tournament is important to incorporate. For instance, if MarineKing beat two great players in Ro64 and Ro32, that is good, but not as good as beating them in the Ro16 and Ro8. I believe proper seeding and points for tournament ending spot takes this situation into account. | ||
|
KillerDucky
United States498 Posts
On December 08 2010 19:10 Mip wrote: Time effects are something I definitely have in mind for future use. I mean, it's pretty clear that a year from now, no one will care what happened in GSL Season 1 as far as predictions are concerned. Here is a paper for accounting for time effects: "Whole-History Rating: A Bayesian Rating System for Players of Time-Varying Strength" http://remi.coulom.free.fr/WHR/ I thought it sounds like a cool concept and I'd like to see it used. On a different game server I play on (KGS - a Go server) they use Bayesian, but to account for time variation they use a simple weight-decay, and it has some strange side effects. | ||
|
Mip
United States63 Posts
To KillerDucky: Thanks for the article. My thought for a time parameter was to have some measurement of the time passed and have the likelihood of past events shrink toward 50/50 as the data becomes older, the past significant upsets will shrink towards non-significance as time passes. | ||
|
beat farm
United States478 Posts
| ||
|
Mip
United States63 Posts
They are both Bayesian approaches... so probably. | ||
|
Sandermatt
Switzerland1365 Posts
Still I think, once enough data is available it would be more accurate to give the players seperate rankings for each match-up. | ||
|
kazansky
Germany931 Posts
On December 09 2010 02:31 Cel.erity wrote: The difference between football and Starcraft is variance, especially in SC2. Football teams have a lot of players, so the impact of one players having a bad/good day is relatively low compared to a team of one. If the solo player has a bad/good day, it skews the results immensely. Also, football teams have faced each other many times in the professional arena, so there is a lot more data to draw upon. SC2 is also a new game with evolving strategies and nobody is at the top level yet, making the data even more inconsistent. Finally, I don't believe the formula accounts properly for player skill difference. In SC2, a player who is just slightly better than another will almost never lose on a favorable map, even though the data says it's 60/40. I think it's a good effort, but I don't believe there is any formula that can rate SC2 players right now with any degree of accuracy. This would be better applied to BW where the data, players, and maps are more consistent. I just wanted to state out that it is possible to build up very good models just on match histories, not that it is in any way comparable, i'm sorry if I didn't point that out enough :-) I totally agree if you that if it should be any accurate, only a very researched game with at least 5 years of history could fit something like that. | ||
|
Mip
United States63 Posts
| ||
|
PROJECTILE
United States226 Posts
| ||
|
Mip
United States63 Posts
On December 09 2010 07:41 kazansky wrote: I just wanted to state out that it is possible to build up very good models just on match histories, not that it is in any way comparable, i'm sorry if I didn't point that out enough :-) I totally agree if you that if it should be any accurate, only a very researched game with at least 5 years of history could fit something like that. I think you guys are kind of off base, I already have a model that can rate Starcraft players with a decent amount of accuracy with only 400 something games. Is it perfect? No. But it has a lot of strength and will learn as it gets more data. Statistical models of this sort are not going to ever give very high prediction accuracy. If you take players with similar skills, you are always going to have difficulty predicting the outcome. But to say that I need 5 years of "research" to start making predictions is just absurd. As for this model and map imbalance, this model averages over all maps. It's primary function is the rate the players objectively based on their performance, which I believe it does quite nicely. If you want to optimize this for prediction, which I believe there is enough data out there that we could start, we need to pull together more data, which I would like help with if there is anyone out there good at parsing webpages. Like I said in the original post, my data look like this [2343,] "MC" "MarineKing" [2344,] "MC" "MarineKing" [2345,] "MC" "MarineKing" [2346,] "Jinro" "Choya" [2347,] "Jinro" "Choya" [2348,] "Jinro" "Choya" [2349,] "Choya" "Jinro" [2350,] "Choya" "Jinro" It actually starts out like this : MarineKing 1 MC 3 Jinro 3 Choya 2 and then I convert it. If instead I could get my data to look more like this: MC Protoss MarineKing Terran Lost Temple MC Protoss MarineKing Terran Blistering Sands MC Protoss MarineKing Terran Jungle Basin I could then start adjusting for those kinds of things. There should already be enough data to start something like this. So long as I have more data than the effective number of parameters that I'm trying to estimate, I can do it no problem. | ||
|
Mip
United States63 Posts
| ||
|
kazansky
Germany931 Posts
On December 09 2010 15:58 Mip wrote: Statistical models of this sort are not going to ever give very high prediction accuracy. If you take players with similar skills, you are always going to have difficulty predicting the outcome. But to say that I need 5 years of "research" to start making predictions is just absurd. As I said, yes statistic models of this kind are able to give very high prediction accuracy. I didn't say yours will yet, and I think if you keep the work up, yours will in about 5 years, or lets say 2 years. That is at least what I meant. To provide high accuracy for the complete outcome of a tournament, and very reliable predictions, you need a huge amount of data to weight, on the one hand. Why I said 5 years was: if you knew every result of the SC2 players right now to base your assumption on, or every result of the BW players, you would highly likely choose the Broodwar players to predict, because the game is far more figured out, so your variation is narrowed down by far, because no every week a new cheese appears. You can start making predictions whenever you want, but if you want to hit +95% over a total GSL (every game) just based on a statistical model, I think you will have to rely on 5 years of tactic development and 2 years of data :-) I didn't want to spoil your fun, I love your work and totally appreciate it. | ||
|
Darkstar_X
United States197 Posts
| ||
|
Mip
United States63 Posts
Each player has an unmeasurable skill parameter, that we can get glimpses of when they win or lose. So the more wins and loses I observe, the more I can nail down exactly what a player's skill parameter is. Over time, I can hope to achieve a fairly high precision with many player's skill levels. But knowing a player's skill is only the parameter that feeds my function that tells me the probability that a player will win, which from the first post is exp(skill1)/(exp(skill1)+exp(skill2)). If in 5 years, I have 2 players of the same skill, the according to this formula, the probability of either winning is 50/50. Which makes sense for players of identical skill. So right now, I might say, well, there's a 30-70% chance player 1 wins (centered at 50-50, but I'm uncertain about exactly what it is), then 5 years from now I can say that there's a 49-51% chance player 1 wins (still 50-50, but I'm certain it's about 50-50 at this point). I'll be able to narrow in only on the probability that a specific player can beat another, not on the actual outcome. What you are saying is that in 5 years, there will only be <5% upsets, and >95% perfect predictability. According to any paired comparison model, that would imply that all player's skill levels are tremendously far apart, which is not likely to be the case. That would imply that no rivalry would exist, no excitement in wondering who will come out on top in any match-up because 95%+ of the time you'd know the victor in advance. I don't understand how one could ever have high predictability of evenly match opponents. I think that would, by definition, make them not evenly matched. | ||
| ||
StarCraft 2 StarCraft: Brood War League of Legends Other Games |
|
Afreeca Starleague
Light vs JyJ
Soulkey vs hero
WardiTV Weekly
Monday Night Weeklies
Afreeca Starleague
Snow vs Shinee
Shine vs EffOrt
GSL
PiGosaur Cup
The PondCast
Kung Fu Cup
Replay Cast
KCM Race Survival
[ Show More ] |
|
|
EPT
