|
Katowice25012 Posts
On June 06 2009 17:29 jtype wrote:
Mine is based a lot more on gut feeling rather than statistics, but I definitely like your thinking on this and look forward to seeing more on your system.
Ultimately gut feeling is probably better right now, but the first few trials I've been running of this show a lot of promise. I'm expecting to be able to outpace any judgment-based systems within a few months.
Remember guys it takes past rounds into account, for example round 1 and 2 CJ are 10-12 KHAN are 15-7 . Doesnt account for recent slumping in the individual leagues by KHAN.
I do this very much on purpose. People are really fast to call something a slump when in reality its just a series of coinflips gone bad. The only time it would be relevant is the psychological effect on the players/coaches themselves if they think they are slumping, but I don't have that kind of info.
I want to run that KHAN game again though, those numbers seem a little wonky. While I think some of their players are under-performing a little, relative to their numbers, its not nearly to the extent people want to think they are.
75% khan over CJ, yea right..... thats soem good number crunching. does that mean you'll bet me 100$ to 300$ ? ?
3-1 is a probably too generous but find an escrow and I'm more than willing.
For individual player stats, consider using PMF. There are good docs online. Alternatively, you can do something very similar using BUGS (or BRUGS) that I've found is pretty useful for these sorts of things.
This is a good idea, thanks.
|
On June 07 2009 01:49 heyoka wrote:Show nested quote +On June 06 2009 17:29 jtype wrote:
Mine is based a lot more on gut feeling rather than statistics, but I definitely like your thinking on this and look forward to seeing more on your system.
Ultimately gut feeling is probably better right now,
Lol! Well, you say that, but then you haven't seen my 5 wrong vote-streak on Liquibet. 
|
United States12607 Posts
So great to have regular PL coverage back! Thanks riptide.
|
Wow, beautiful work, Heyoka. I imagine that took a while to work out :D
Thanks for the writeup, Riptide. GG.
|
Yeah, SPL write-ups are back. This one was really nice because of the new way you approached it. And I usually enjoy your articles, riptide. Thanks.
|
I'm currently working on using machine learning techniques to improve estimation of the probability a given player will win given his past performance (last 10), current elo and map statistics. Collecting the data to train the models though is quite tedious and will take quite some time before a reasonably accurate model is obtained (since they're based on actual data), however, I think such models should be more accurate than the simple rough calculations provided in the OP. At the moment I'm looking only at simple linear models like naïve Bayesian and logistic since they tend to perform better with limited data over non-linear models. I'm hoping to try out non-linear models like neural networks in the future, when I have sufficient data (which may be a while). I'll be sure to post my results on TL when they become interesting, however at the moment, there isn't much to say except data collection is tedious.
|
Wow this is really nice!
Total dedication and belief in numbers =) Thats how I like it
|
By the way, ELO can already be used as a probabilistic estimator for a given matchup. I haven't tested its accuracy or perplexity though, so I have no clue how good it might be. Overall, I don't think these techniques can beat the best human experts though, but they can infer interesting data about how certain data affects the likelihood of a player winning. Probably most interesting for everyone is how much the map matters statistically speaking.
I'm doing my Master's in Machine Learning and Data Mining, so its good practice for me anyway. 
|
When I was reading the statistical analysis that was the first thing that came to mind as well. I've always found that comparing two players ELO (or better yet, their detailed matchup ELO) to be a better indicator of the match outcome than W-L ratios. Hell with the exception of a few S-class players and rising talents, even just looking at the straight up map balance statistics has been more accurate 
|
The comeback by xellos was sick, zerg went alittle crazy tho, i dont know why he didnt keep expoin before 20minute-ish mark.
|
Katowice25012 Posts
On June 07 2009 04:52 Syntax Lost wrote:
I'm currently working on using machine learning techniques to improve estimation of the probability a given player will win given his past performance (last 10), current elo and map statistics. Collecting the data to train the models though is quite tedious and will take quite some time before a reasonably accurate model is obtained (since they're based on actual data), however, I think such models should be more accurate than the simple rough calculations provided in the OP. At the moment I'm looking only at simple linear models like naïve Bayesian and logistic since they tend to perform better with limited data over non-linear models. I'm hoping to try out non-linear models like neural networks in the future, when I have sufficient data (which may be a while). I'll be sure to post my results on TL when they become interesting, however at the moment, there isn't much to say except data collection is tedious.
They definitely should be. What I have here really amounts to an estimation to see if this kind of study has any merit - I think this shows it does. There are hundreds of ways to improve this now, in the future when we look at this way of doing it, it will seem pathetically inaccurate.
I have done some experiments with ELO and the my results generally aren't good. The player pool we're working with is so small that the results I got ended up not being very accurate, I think techniques used for handicapping baseball and football are more suited for this kind of thing.
The KeSPA system might actually work here though, as it does something similar but heavily weighs recent results. It is more suited to telling us how players are right now whereas ELO results seem better for how good a player was at his best or averaged through his career.
|
This system of calculations would probably earn you a good amount of correct bets in the long run. In regards to winning a liquibet season or the sort, you probably wouldn't end up first using this kind of system. Basing decisions on probability is always a good thing in the long run since a majority of the time you'll end up being correct most of the time. (poker anyone?) But then again there's outside factors that a statistical model can't account for. (poker bluffs?)
|
Katowice25012 Posts
The real reason this isn't optimal for liquibet is that liquibet is very heavily influenced by 1v1 matches. Straight probability is (right now) a poor way of determining how a msl/osl will play out in various stages. That will probably continue to be true even with much more advanced models of ranking specific players as well.
|
On June 07 2009 09:44 heyoka wrote:
I have done some experiments with ELO and the my results generally aren't good. The player pool we're working with is so small that the results I got ended up not being very accurate, I think techniques used for handicapping baseball and football are more suited for this kind of thing.
The current limiting factor is the lack of data. Currently, I'm just collecting last 10 statistics for each player for the given matchup, plus their current ELO, matchup ELO and map statistics for the matchup. E.g. for Bisu vs Hiya, I take Bisu's last 10 games vs Terran as a binary vector plus his ELO vs Terran, Hiya's last 10 vs Protoss as another binary vector plus his ELO vs Protoss, both their general ELO scores and the matchup statistic given as a probability for the map they're to play on. This gives the data 25 dimensions, which is quite substantial and therefore requires significant data to properly cover all of these dimensions, but should give reasonable accuracy for estimating the actual distribution once sufficient data is obtained.
The KeSPA system might actually work here though, as it does something similar but heavily weighs recent results. It is more suited to telling us how players are right now whereas ELO results seem better for how good a player was at his best or averaged through his career.
Problem is that KeSPA only update their ranking once per month, so their value decreases over the month. I'm also unsure what kind of distribution they're trying to fit the players to (if they are trying to fit a distribution at all) which makes estimating the actual distribution of player results much more difficult. ELO is nice in that its designed to fit to players to a Normal distribution (which is a reasonable assumption, in my opinion) and assigns a uniform weight to all matches (rather than what seems like an arbitrary weight given by KeSPA) so its properties are easier to understand.
I prefer ELO over KeSPA's system for these reasons and it seems pretty reasonable when coupled with last 10 statistics (so present winning streaks are factored). I took a couple liberties in re-factoring a few statistics (basically, just resetting Xellos' stats since his joining the army) to help compensate for a few short-comings in the system.
Anyhow, I have no clue how well the results will turn out, as I still require far more data.
By the way, if anybody knows how to quickly get player's ELO scores at the time of past matchups quickly and easily, it will greatly speed up the process.
|
Nice stats analysis. Always love to see this. 
On June 06 2009 21:21 3 Lions wrote:Show nested quote +On June 06 2009 20:12 Balfazar wrote:
Have a look at the matchups for Khan vs CJ, Khan are definitely favorites. The only match where Khan don't have a clear advantage is Effort vs Stork. I will be surprised if they don't win. Given Firebathero's current form, I can actually realistically see CJ win 3-1
+ Show Spoiler +Ironically, FBH was the only one to win.
|
On June 08 2009 02:09 eshlow wrote:Nice stats analysis. Always love to see this. Show nested quote +On June 06 2009 21:21 3 Lions wrote:On June 06 2009 20:12 Balfazar wrote:
Have a look at the matchups for Khan vs CJ, Khan are definitely favorites. The only match where Khan don't have a clear advantage is Effort vs Stork. I will be surprised if they don't win. Given Firebathero's current form, I can actually realistically see CJ win 3-1 + Show Spoiler +Ironically, FBH was the only one to win.
FAUUUUUUUUUUUUUK WHY DID I CLICK THAT!?@?!?@!?@!?@
DONT DO IT PEOPLE! it's soooooo not worth it..
sorry for caps, but that just ruined my day.
|
I think I'm going to trade Iris, I want him out anyways.
Who do you think is a good substitute for him?
I've been thinking about the follow for trades:
Luxury, July, Hwasin, Mind
I think im goin to pick luxury, seems like the new maps favor zerg, and July just isn't getting much playing time these days. Although i like STX more then KTF as far as team points goes. That's pretty much the main reason why I'm not choosing Mind, he just won't be getting the team points with Wemade.
What are your guys thoughts on Hwasin, jesus i wish i didn't choose Berserker in the first place, damn donkey is failing big time. can't even beat + Show Spoiler +FBH... FBH's tvt isn't even stellar. sigh.
|
Your analysis is really good but my CJ fanboyism still wont let me say you're right.
|
Katowice25012 Posts
On June 07 2009 21:39 Syntax Lost wrote:
By the way, if anybody knows how to quickly get player's ELO scores at the time of past matchups quickly and easily, it will greatly speed up the process.
I asked PoP about that exact thing, he said it was in his "to do" list for TLPD but not functional yet. I assume he is a busy dude, maybe if you ask too we can get it bumped up on the list
Regarding KHAN vs CJ:
+ Show Spoiler +
I actually had a post written up on how I think KHAN is a more even match than people realize, because FBH had a much higher chance to win his match than people want to admit/will see. I guess that specific point was right, but in the end CJ still blew out KHAN so I can't brag
Sucks for that dude he didnt get back to me on taking the bet
|
I love the great work keep it up.
On June 08 2009 10:24 heyoka wrote:Regarding KHAN vs CJ: + Show Spoiler +
I actually had a post written up on how I think KHAN is a more even match than people realize, because FBH had a much higher chance to win his match than people want to admit/will see. I guess that specific point was right, but in the end CJ still blew out KHAN so I can't brag
Sucks for that dude he didnt get back to me on taking the bet
+ Show Spoiler +I wouldn't be too discouraged because one prediction went bad. One match is not adequate statistical evidence against your method.
Have you considered running the algorithm on past games and data in order to check its accuracy with a large sample?
|
|
|
|
|
|
EPT
