Weekly Roundup By Riptide Unlike the first week of school, the first week of Proleague R5 started off with a bang. With Wemade taking SKT T1 to the cleaners, and Effort performing an elegant double kill on eSTRO to take CJ to 27 wins, last Saturday saw some good games. Although it may seem unusual that a team ranked 10th would take out SKT T1 in their R5 opener, thanks to the heyoka’s statistics we can see that this is not all that surprising after all.
No, your eyes are not deceiving you. Wemade is actually 3 – 1 vs SKT T1 this year. It’s funny but true, and now we know, thanks to the monster statistics that heyoka has been producing. The table is interesting because it highlights a number of other things, including, but not limited to the dominance Oz has shown this season. 4-0 vs Sparkyz, FOX and ACE, they are currently the team to have the most unbeaten runs. Their position at #1, though strengthened by two great wins this week, is now within grabbing distance of CJ, who also had a great week to end up at 28-18, and now tie with SKT T1 for second place.
CJ and SKT are now both hot on the heels of Oz.
Their loss to Oz in their only game this week put the Sparkyz down by one, but at 26-19 they still are very much in contention for a #2 or #3 spot. Although beat Stars to hold on to a shaky #4, they will definitely get a run for their money from a Hite team led by an ever hungry Leta.
Having looked at the team standings, it’s time now to get down to individual games. There were a number of notable matches this week.
What a game. Not because XellOs’ play was particularly brilliant, or because this HoeJJa was severely outclassed, because he wasn’t, but because it reminded many of us of what the Perfect Terran used to look like. A long drawn out TvZ, the former-CJ, now ACE Terran slowly edges back from the brink of defeat to take down the Zerg in a thoroughly entertaining fashion. Cue comments about HoeJJa’s imba A-move mutas.
In Proleague, you get Ace matches, and then you get Ace matches like this. A crazy game, with Leta making everything from marines, to wraiths, dropships and goliaths, and Jaedong adapting to each tech change with the cold determination of a trained assassin, this is a game that you will want to watch, and then re-watch, and then save so you can show it to your grandchildren someday and say my sons and daughters, this is how eSports was played before they made everything ezmode.
In terms of individual standings, Jaedong bagged 3 wins this week, with Bisu netting only two, and therefore has edged his rival in the top gun ranking. In doing so, the Oz powerhouse also took the game off Leta in the Oz vs Sparkyz Ace match, making it that much more difficult for the Terran to move up the rankings.
Before we move on, here’s a summary of this week’s action. + Show Spoiler +
Now, let’s get started on the fun stuff. Earlier on in the week, I asked heyoka if he could run to his magic stats machine and give me some predictions for Week 2. He emerged a few days later, bleary eyed but smiling, and gave me these.
STX SouL < Woongjin Stars (54%)
KT MagicNs < Hite SPARKYZ (62%)
Samsung KHAN (75%) > CJ Entus
Hwaseung Oz (64%) > SK Telecom T1
MBCGame HERO (61%)] > WeMade FOX
Air Force ACE (61%) > eSTRO
STX SouL < Hite SPARKYZ (60%)
KT MagicNs (62%) > Woongjin Stars
Samsung KHAN < SK Telecom T1 (54%)
Hwaseung Oz (55%) > CJ Entus
How did heyoka arrive at these predictions? Why, we thought you’d never ask.
Crunching Numbers By Heyoka I love the new proleague format. KeSPA has made a lot of awful, awful decisions in the past year but changing the proleague format was not one of them. I like the removal of 2v2, but what I really love is the expansion of proleague to a 5 round format, that spans a whole year. Team league is a war of small advantages, the addition or removal of a single solid player can drastically change your playoff expectations.
With 4 rounds already played, and a round robin still left before playoffs, we have a reasonable amount of data to make projections for future performance. Accurate tools for quantifying specific chances a team has have never truly been worked out, and for the first time in Starcraftf history I think we have a large enough set of numbers to make a good attempt. Over the coming weeks I am going to go through my ideas and thoughs for doing so, as well as discussing the process taken to refine the methods used.
Predictors Eventually I will get into how we can model a team's exact chances of finishing in the top spots, and the worth of making the playoffs. Though before that can be done, I need to explain how I estimate a team's chances of winning a specific match (conveniently this is probably also the most interesting part for most people).
When people discuss who they think the favourite is to win a match, you usually end up with a post that looks like this:
Yes, with a bunch of words describing why you think one player is stronger than another. This is an OK way of doing it, but has two problems. The first is that this kind of ranking system doesn't count for the probability of different combinations of wins happening. Is the chance of Really upsetting BeSt greater or less than the chance of Thezerg upsetting hyvaa? What happens when fantasy gets blown out by proxy gates, do we think SKT can still take a win*? The second, and more important, problem is it relies too much on gut feel of how matchups will play out. The mind will always over-value most recent performance when looking at player strength. Past performance as a whole tends to be a better indicator here, and is something we can easily quantify.
*This is a bad example because eSTRO has an awful record in ace games, stay with me.
I decided to see how it would play out if we were to solve for each possible outcome, and then see which team had the best chances overall. There are 10 possible ways for a team to win a best of 5 competition (WWW, LWWW and on) so there are 20 total outcomes for a match. I made an excel spreadsheet that does each of these for me, it looks something like this:
The columns are the chances each team has of winning, and the rows show the different ways they have of winning. In this case, eSTRO has an estimated 2.2% chance to win this match by a 3-0 result. The bottom row is the total chances each team has of winning the match (it adds up all the total column). The 1 is to make sure the two add up correctly. How do we get those numbers?
This all hinges on estimating win percentages for a single player vs player game. This is (unfortunately) the roughest part of this calculation, as the best way we have of predicting future performance is simply past win-loss. To do this, I filled in a small chart that has the total win percentages of each player vs his opponent's race. I haven't decided how to best go about this yet, if I can use total win percentages or if I should use league-specific stats. This one has stats from these players only during 08-09 proleague, except for HyuK. The chart looked like this:
The "set 5" numbers are ace match percentages through rounds 1, 2, and 4. There are ways to do this to get more exact numbers using who we know is likely to be an ace player, but they aren't developed yet and will be refined later.
To then judge how likely it is that either will win a given game, I used the log5 method. You can read an explanation here, basically it is this formula:
A - A * B
WPct
= -----------------
+ B - 2 * A * B
A is team A's winning percentage and B is team B's winning percentage.
Those calculations make their own little chart, that is referenced by the first set of percentages. That 2.2% chance of the game going 3-0 in eSTRO's favour is found by multiplying the log5 estro set 1, 2 and 3 together. It then goes through and does the same thing, for each possible outcome and then adds it together to get a team's total chances of winning the match. Using this method, it shows SKT with this lineup is 60% to beat this exact lineup. I think that number is probably a little low, but given that SKT was 2:1 vs eSTRO so far (at that point in the season) and that the end result was 3 games to 2, it should be well within a pretty small range of what it truly is.
Using a slightly less refined method than this, I picked something like 65% of winners in proleague through round 4. Thats slightly higher than I was doing with a more intuition based method, so while the results weren't phenomenal I'm pretty confident this is still better handicapping than most people are able to do. Last week I went 8-2 with it, with one loss being SKT vs WeMade (a clear favorite losing I think) and the other being MBC vs STX (a flaw in the system). Last was 0-3 TvT and I was unure how to accurately weigh him vs Sea, the match was close enough that it would have made a difference had I messed with it more and I think I probably under-estimated his true chances of winning.
Some other problems with this so far: - It doesn't account for map balance at all. I haven't figured out to correctly weigh that, but it should be possible in the future.
- You never really get ranges outside of 40-60%. It really only tells you who the favourite is, which pretty often is something that is already commonly understood. It can produce surprising results when lower-tier teams play each other which can be cool.
- Individual player percentages are still rough. This will be fixable in the future, but developing a system that more accurately estimates those is going to take considerable time (probably more than I will ever have, realistically). This model will always have the problem that players with few stats are going to be difficult to quantify, and I'm unsure how to correct it. This method as a whole will probably be unable to solve that, which is unfortunate as many matches have 1 player who is a relative unknown (actually this could be partially solved if minor league was larger than it is but I don't see that happening anytime soon).
Actual Predictions OK, now back to the predictions we made earlier in this piece.
STX SouL < Woongjin Stars (54%)
KT MagicNs < Hite SPARKYZ (62%)
Samsung KHAN (75%) > CJ Entus
Hwaseung Oz (64%) > SK Telecom T1
MBCGame HERO (61%)] > WeMade FOX
Air Force ACE (61%) > eSTRO
STX SouL < Hite SPARKYZ (60%)
KT MagicNs (62%) > Woongjin Stars
Samsung KHAN < SK Telecom T1 (54%)
Hwaseung Oz (55%) > CJ Entus
One final thing is that this isn't all that good for predicting ACE matches. Most ACE players don't have past records that are any good in indicating future performance and I haven't found a way to properly weight that yet. That being said, this specific lineup has an OK sample and I feel confident saying ACE is probably slight favorite in this match, but its still likely the less accurate percentage up there.
The Oz vs CJ match was particularly interesting, as it ended in basically a dead 50-50. I played with some different ways of weighting the data and Oz ended up edging out CJ, which seems accurate but I'll need more testing before I can say if it was correct or not.
In the future I plan on taking a similar approach to other team-format related question. I want to quantify things like the actual number of team wins Jaedong create for Oz. How necessary is it for each team to have one good player of each race? Which teams would be hurt the most by removal of a "1 race per match" rule, and who would gain the most? Does KTM's win expecation drop significantly if they play Flash in the 4th set? Can we optimize strategy of who to play on our team if we know which players the opposing team is most likely to send out? Can we eve quantify team depth?
I am confident most of these are things we can calculate to a small degree of certainty, and plan on going into it further through our friend results and standings. The future is wide open for theoretical brood war studies.
This is such a fascinating part of TL's Liquibet and Fantasy sections and, indeed, the competitive SC scene as a whole.
Things that I like to take into account before deciding on my predictions are obviously, past/recent performance in certain matchups, against certain players and play styles, in certain situations (based on the pressure, nerves, etc) but also on certain maps and types of maps.
Mine is based a lot more on gut feeling rather than statistics, but I definitely like your thinking on this and look forward to seeing more on your system.
Remember guys it takes past rounds into account, for example round 1 and 2 CJ are 10-12 KHAN are 15-7 . Doesnt account for recent slumping in the individual leagues by KHAN.
On June 06 2009 18:35 Pokernubz wrote: 75% khan over CJ, yea right..... thats soem good number crunching. does that mean you'll bet me 100$ to 300$ ? ?
The stats about predictions is absolutely great stuff. The explanations as well as the methodology itself are great.
The sad thing is if you believe that there is variance inherent even in a mostly-deterministic game like starcraft (which I do), it means that the statistical approach will give you the right answers in the long run, but is unlikely to consistently give you the right answer (because so many matchups are 60-40 or less), despite being correct. It means that there is a fairly low upper limit to how good you can be at predicting.
Have a look at the matchups for Khan vs CJ, Khan are definitely favorites. The only match where Khan don't have a clear advantage is Effort vs Stork. I will be surprised if they don't win.
On June 06 2009 20:12 Balfazar wrote: Have a look at the matchups for Khan vs CJ, Khan are definitely favorites. The only match where Khan don't have a clear advantage is Effort vs Stork. I will be surprised if they don't win.
Given Firebathero's current form, I can actually realistically see CJ win 3-1
For individual player stats, consider using PMF. There are good docs online. Alternatively, you can do something very similar using BUGS (or BRUGS) that I've found is pretty useful for these sorts of things.
In terms of individual standings, Jaedong bagged 3 wins this week, with Bisu netting only two, and therefore has edged his rival in the top gun ranking. In doing so, the Oz powerhouse also took the game off Leta in the Oz vs Sparkyz Ace match, making it that much more difficult for the Terran to move up the rankings.
3 is ok, 2 is "only"? Thats a tiny difference Especially if you take into account that SKT wins doesnt depend only of Bisu, and he played less than JD.
Fanboism aside, very nice statistics and math, but i dont have that much patience to carefully read it anymore
On June 06 2009 17:29 jtype wrote: Mine is based a lot more on gut feeling rather than statistics, but I definitely like your thinking on this and look forward to seeing more on your system.
Ultimately gut feeling is probably better right now, but the first few trials I've been running of this show a lot of promise. I'm expecting to be able to outpace any judgment-based systems within a few months.
Remember guys it takes past rounds into account, for example round 1 and 2 CJ are 10-12 KHAN are 15-7 . Doesnt account for recent slumping in the individual leagues by KHAN.
I do this very much on purpose. People are really fast to call something a slump when in reality its just a series of coinflips gone bad. The only time it would be relevant is the psychological effect on the players/coaches themselves if they think they are slumping, but I don't have that kind of info.
I want to run that KHAN game again though, those numbers seem a little wonky. While I think some of their players are under-performing a little, relative to their numbers, its not nearly to the extent people want to think they are.
75% khan over CJ, yea right..... thats soem good number crunching. does that mean you'll bet me 100$ to 300$ ? ?
3-1 is a probably too generous but find an escrow and I'm more than willing.
For individual player stats, consider using PMF. There are good docs online. Alternatively, you can do something very similar using BUGS (or BRUGS) that I've found is pretty useful for these sorts of things.
On June 06 2009 17:29 jtype wrote: Mine is based a lot more on gut feeling rather than statistics, but I definitely like your thinking on this and look forward to seeing more on your system.
Ultimately gut feeling is probably better right now,
Lol! Well, you say that, but then you haven't seen my 5 wrong vote-streak on Liquibet.
I'm currently working on using machine learning techniques to improve estimation of the probability a given player will win given his past performance (last 10), current elo and map statistics. Collecting the data to train the models though is quite tedious and will take quite some time before a reasonably accurate model is obtained (since they're based on actual data), however, I think such models should be more accurate than the simple rough calculations provided in the OP. At the moment I'm looking only at simple linear models like naïve Bayesian and logistic since they tend to perform better with limited data over non-linear models. I'm hoping to try out non-linear models like neural networks in the future, when I have sufficient data (which may be a while). I'll be sure to post my results on TL when they become interesting, however at the moment, there isn't much to say except data collection is tedious.
By the way, ELO can already be used as a probabilistic estimator for a given matchup. I haven't tested its accuracy or perplexity though, so I have no clue how good it might be. Overall, I don't think these techniques can beat the best human experts though, but they can infer interesting data about how certain data affects the likelihood of a player winning. Probably most interesting for everyone is how much the map matters statistically speaking.
I'm doing my Master's in Machine Learning and Data Mining, so its good practice for me anyway.
When I was reading the statistical analysis that was the first thing that came to mind as well. I've always found that comparing two players ELO (or better yet, their detailed matchup ELO) to be a better indicator of the match outcome than W-L ratios. Hell with the exception of a few S-class players and rising talents, even just looking at the straight up map balance statistics has been more accurate
On June 07 2009 04:52 Syntax Lost wrote: I'm currently working on using machine learning techniques to improve estimation of the probability a given player will win given his past performance (last 10), current elo and map statistics. Collecting the data to train the models though is quite tedious and will take quite some time before a reasonably accurate model is obtained (since they're based on actual data), however, I think such models should be more accurate than the simple rough calculations provided in the OP. At the moment I'm looking only at simple linear models like naïve Bayesian and logistic since they tend to perform better with limited data over non-linear models. I'm hoping to try out non-linear models like neural networks in the future, when I have sufficient data (which may be a while). I'll be sure to post my results on TL when they become interesting, however at the moment, there isn't much to say except data collection is tedious.
They definitely should be. What I have here really amounts to an estimation to see if this kind of study has any merit - I think this shows it does. There are hundreds of ways to improve this now, in the future when we look at this way of doing it, it will seem pathetically inaccurate.
I have done some experiments with ELO and the my results generally aren't good. The player pool we're working with is so small that the results I got ended up not being very accurate, I think techniques used for handicapping baseball and football are more suited for this kind of thing.
The KeSPA system might actually work here though, as it does something similar but heavily weighs recent results. It is more suited to telling us how players are right now whereas ELO results seem better for how good a player was at his best or averaged through his career.
This system of calculations would probably earn you a good amount of correct bets in the long run. In regards to winning a liquibet season or the sort, you probably wouldn't end up first using this kind of system. Basing decisions on probability is always a good thing in the long run since a majority of the time you'll end up being correct most of the time. (poker anyone?) But then again there's outside factors that a statistical model can't account for. (poker bluffs?)
The real reason this isn't optimal for liquibet is that liquibet is very heavily influenced by 1v1 matches. Straight probability is (right now) a poor way of determining how a msl/osl will play out in various stages. That will probably continue to be true even with much more advanced models of ranking specific players as well.
On June 07 2009 09:44 heyoka wrote: I have done some experiments with ELO and the my results generally aren't good. The player pool we're working with is so small that the results I got ended up not being very accurate, I think techniques used for handicapping baseball and football are more suited for this kind of thing.
The current limiting factor is the lack of data. Currently, I'm just collecting last 10 statistics for each player for the given matchup, plus their current ELO, matchup ELO and map statistics for the matchup. E.g. for Bisu vs Hiya, I take Bisu's last 10 games vs Terran as a binary vector plus his ELO vs Terran, Hiya's last 10 vs Protoss as another binary vector plus his ELO vs Protoss, both their general ELO scores and the matchup statistic given as a probability for the map they're to play on. This gives the data 25 dimensions, which is quite substantial and therefore requires significant data to properly cover all of these dimensions, but should give reasonable accuracy for estimating the actual distribution once sufficient data is obtained.
The KeSPA system might actually work here though, as it does something similar but heavily weighs recent results. It is more suited to telling us how players are right now whereas ELO results seem better for how good a player was at his best or averaged through his career.
Problem is that KeSPA only update their ranking once per month, so their value decreases over the month. I'm also unsure what kind of distribution they're trying to fit the players to (if they are trying to fit a distribution at all) which makes estimating the actual distribution of player results much more difficult. ELO is nice in that its designed to fit to players to a Normal distribution (which is a reasonable assumption, in my opinion) and assigns a uniform weight to all matches (rather than what seems like an arbitrary weight given by KeSPA) so its properties are easier to understand.
I prefer ELO over KeSPA's system for these reasons and it seems pretty reasonable when coupled with last 10 statistics (so present winning streaks are factored). I took a couple liberties in re-factoring a few statistics (basically, just resetting Xellos' stats since his joining the army) to help compensate for a few short-comings in the system.
Anyhow, I have no clue how well the results will turn out, as I still require far more data.
By the way, if anybody knows how to quickly get player's ELO scores at the time of past matchups quickly and easily, it will greatly speed up the process.
On June 06 2009 20:12 Balfazar wrote: Have a look at the matchups for Khan vs CJ, Khan are definitely favorites. The only match where Khan don't have a clear advantage is Effort vs Stork. I will be surprised if they don't win.
Given Firebathero's current form, I can actually realistically see CJ win 3-1
On June 06 2009 20:12 Balfazar wrote: Have a look at the matchups for Khan vs CJ, Khan are definitely favorites. The only match where Khan don't have a clear advantage is Effort vs Stork. I will be surprised if they don't win.
Given Firebathero's current form, I can actually realistically see CJ win 3-1
I think I'm going to trade Iris, I want him out anyways.
Who do you think is a good substitute for him? I've been thinking about the follow for trades: Luxury, July, Hwasin, Mind
I think im goin to pick luxury, seems like the new maps favor zerg, and July just isn't getting much playing time these days. Although i like STX more then KTF as far as team points goes. That's pretty much the main reason why I'm not choosing Mind, he just won't be getting the team points with Wemade.
What are your guys thoughts on Hwasin, jesus i wish i didn't choose Berserker in the first place, damn donkey is failing big time. can't even beat + Show Spoiler +
On June 07 2009 21:39 Syntax Lost wrote: By the way, if anybody knows how to quickly get player's ELO scores at the time of past matchups quickly and easily, it will greatly speed up the process.
I asked PoP about that exact thing, he said it was in his "to do" list for TLPD but not functional yet. I assume he is a busy dude, maybe if you ask too we can get it bumped up on the list
I actually had a post written up on how I think KHAN is a more even match than people realize, because FBH had a much higher chance to win his match than people want to admit/will see. I guess that specific point was right, but in the end CJ still blew out KHAN so I can't brag
Sucks for that dude he didnt get back to me on taking the bet
I actually had a post written up on how I think KHAN is a more even match than people realize, because FBH had a much higher chance to win his match than people want to admit/will see. I guess that specific point was right, but in the end CJ still blew out KHAN so I can't brag
Sucks for that dude he didnt get back to me on taking the bet
I actually had a post written up on how I think KHAN is a more even match than people realize, because FBH had a much higher chance to win his match than people want to admit/will see. I guess that specific point was right, but in the end CJ still blew out KHAN so I can't brag
Sucks for that dude he didnt get back to me on taking the bet
Ya seriously, i was thinkin about it. and the only way i could make the bet, is with pokerstars.com
Cause i am Wayyyyyyyyyy too lazy to open an account at some $$ place for small $.
how much action do people take on side bets anwyays, i just pratically been a lurker for 5 years, now im actually reading forums i use to only just read articles/watch vods.
I actually had a post written up on how I think KHAN is a more even match than people realize, because FBH had a much higher chance to win his match than people want to admit/will see. I guess that specific point was right, but in the end CJ still blew out KHAN so I can't brag
Sucks for that dude he didnt get back to me on taking the bet
I wouldn't be too discouraged because one prediction went bad. One match is not adequate statistical evidence against your method.
Have you considered running the algorithm on past games and data in order to check its accuracy with a large sample?
No, partially because it isn't fully automated yet, and the "past sample size" isn't very large. I can do it for rounds 1 and 2 (and I did it for about half of R2), but that is about it because I'm not really sure how I would do 2v2 ratings for past years. The other problem is rating "strength at time of matchup" when going into the past which can be a little annoying because it puts an extra step into grabbing numbers from TLPD.
Fully automating it should be the next thing I get on doing, because it means for every improvement I make I can then compare results to past performance easily but it keeps getting pushed back.
I don't know how much is taken on side bets and while I'd love to take over that field this probably isn't the right place for it. Maybe I'll take it over to LP.
What I meant, though, was that you may want to try it on matches for an older league (like Shinhan 07), so you can check accuracy without having to wait for results.
By the way, is there any way to get direct access to the database itself? For automated purposes.
I understood exactly what you said and I'm pretty sure I addressed it. Going through older leagues (such as shinhan 07) is difficult because the format included 2v2, in which the "individual formula" I used here would be considerably less accurate and as such would require a major adjustment. I also haven't done it because finding people's records at the time those matches were played is somewhat of a chore the way I do things now, as it ads another step (I have to filter results by date and perhaps format).
My bad, I was the one who misinterpreted what you said. I haven't been seriously following progaming long enough to know that they used to have 2v2 in proleague.
But that is why I am curious if it is possible for general population to access TLPD data directly (without using a browser). It seems like the "chore" involved in running historic matches could be solved with half a page of code.
On June 10 2009 16:26 DogofWar wrote: But that is why I am curious if it is possible for general population to access TLPD data directly (without using a browser). It seems like the "chore" involved in running historic matches could be solved with half a page of code.
I agree code would help! Generally though, I don't think pop/sonuvbob/whoever controls the actual database lets anyone access it. Maybe I can make a convincing presentation when I have a little stronger argument, getting it completely automated is definitely a goal. I'm going to reach a point eventually where it will become a necessity for improvement/advancing ideas, but right now I'm still sort of playing in the sand with scattered thoughts.
EPT![[image loading]](http://www.teamliquid.net/staff/heyoka/TeamvsTeam.png)
![[image loading]](http://www.teamliquid.net/staff/heyoka/R4Stand.png)
HoeJJa <Shades of Twilight> 
XellOs
BeSt < ![[image loading]](http://www.teamliquid.net/staff/heyoka/Predict/chart1.png)
![[image loading]](http://www.teamliquid.net/staff/heyoka/Predict/chart2.png)
.
