On December 10 2010 16:51 Mip wrote:
Take season 2 for example, We know FruitDealer is amazing because he won Season 1. FruitDealer loses to Foxer in Ro32, that's going to tell me that Foxer is at least skillful enough to beat FruitDealer, which is quite substancial. Then Foxer goes on to lose against NesTea in a nearly dead even match. That's going to hint toward thinking Foxer and NesTea are about at the same level.

Is this actually true though? Do the calculations always assume things like if player a > b and player b > c then player a >c, because I know this isn't the case in most competitive gaming. There are always many cases of rock paper scissors relationships like a > b, b > c, c >a .

See.Blue

United States2673 Posts

December 10 2010 22:39 GMT

#42

On December 10 2010 12:34 Mip wrote:
So I've been working on a SC2 player ranking algorithm (see my other post).

So far I've only used the GSL, and I've only included player rankings, no race bias or map bias, or time-based skill evolution (all in progress and will be implemented as my data quantity increases).

Anyway, so I was looking over the list of Code S players and thought to myself that a lot of those players could easily have lost some of their matches and failed to qualify for Code S. So I wanted to see, based on the data, what was the probability of each player actually being in the Top 32.

Here are the results in a Google Spreadsheet

So as you look at that data, bear in mind, this data only obseving the GSL bracket final 64 player wins/losses is all the data in the world on the subject. This makes the algorithm non-ideal for prediction of the top skilled players. But it is ideal for assessing the uncertainty about the point system in actually getting the best players (at least for the top players).

Also bear in mind, this model implicitly assumes that not-qualifying for top 64 and not registering for the tournament are equivalent, which isn't a fair assumption, but there's no data available to fix this. JookToJung gets the raw end of this assumption. He must be very good to qualify all 3 seasons, but the model sees only his losing in the early rounds. This isn't something I like, but I don't have the proper data to correct this problem at this time.

So the table shows a lot of uncertainty about who actually belongs in Code S. There are plenty that could easy have been Code S if things turned out a slightly differently. July is easily Code S caliber, as is Ret, Loner only needed one more set and he'd be S class.

If I had more data on the qualifying rounds, I'm sure that people like JookToJung would look better. I might look into grouping all the players that have 3 or fewer games into one. Because they are hardly estimable with how little data there is on them.

But the higher up on the spreadsheet you go, the results get a lot more accurate since they are based on more games played. There are players that are clearly Top 32, a lot of people are really good, but the uncertainty associated with knowing their skills is fairly high (completely an artifact of not having a lot of data on them). The way the bracket system works, it just doesn't give very good estimates for the people who get knocked out in the first rounds.

Anyway, it is what it is. It should give you an underlying sense on what kind of information is in the data. You don't have to agree with the results, it's just what the data seem to be pointing to (under the constraints of the assumptions I had to make).

Out of curiosity, as a math person, how did you compute the likelihoods?

GeorgeForeman

United States1746 Posts

December 11 2010 00:44 GMT

#43

On December 11 2010 05:46 Mip wrote:
@GeorgeForeman and confusedcrib I'm glad you paid attention in your intro stats classes, but in Bayesian statistics, you can integrate over the uncertainty in your estimates to obtain a single number that takes into account all of the uncertainty you have in your estimate. We can say with Bayesian statistics that based on our current state of knowledge (priors + data provided) that the probability of Player X actually being Top 32 is Y%.

That you would bring up a t-test for this model immediately puts you at an intro stats level in my brain. Your instinct is correct for that level of stats knowledge, but in this case, it should not be a concern to you. You should think of those percentages in terms of what I described at the end of the paragraph above.

However, to appease you guys, I added a column of Standard Errors. If you are using your intro stats knowledge,however, you will misinterpret them because they mean different things if your data are not from a normal/gaussian distribution.

For a binary outcome, the variance is prob * (1 - prob), and then the standard error is the square root of that, but you have to throw away any thoughts that, for example, 3 standard errors gives you a confidence interval or any nonsense like that that you are taught in intro stats. For example, for NesTea, if you tried to do that, you'd get a confidence interval that included probabilities greater than 1. To do it properly, you'd have to convert to a odds ratio, compute confidence intervals, then convert back to a probability metric.

Kid, I'm a 4th year grad student working on my dissertation in statistics. I've TAUGHT an intro class. If you're going to talk down to someone, at least make sure you know more than they do. Asking for uncertainty estimates only connotes a "t-test" if you're too narrow-minded to consider anything else. As far as I can understand (which is difficult, since you didn't exactly explain it in either of your OPs) you've calculated a posterior distribution for each player's "true skill level". Using the means of these distributions as point estimates you constructed a ranking of them. (This was your previous post.) You've reported standard errors for these, though I'm not sure what those are. Are these numbers the posterior estimates for the standard deviation? Because that's not the same thing as a standard error.

Now, as best as I can tell, you took all of this data and calculated for each player, i, the probability that this player is better than all but at most 31 other players. In other words:

P(S_i>S_j | j is in T and T contains at most 31 elements)

Now, this last thing seems extraordinarily difficult to calculate, given that your estimates for each S_i all come with their own associated variances and that the posterior distribution is dependent upon each of the other. Basically, you've got a p-dimensional normal distribution (where p is the number of players in your data set) with a very confusing-looking covariance matrix. Maybe there's software that makes such a calculation trivial that I'm not aware of, but to me, that looks like a difficult problem. Bravo for taking the time to solve it.

Assuming this is your approach (and again, I'll emphasize that I'm forced to do a lot of inferring because your actual approach is nowhere explained with any degree of clarity), what you end up with are posterior probability estimates. If that is indeed what your spreadsheet is reporting, then I understand why you didn't report the standard deviation, as it's completely determined by the posterior probability estimate.

That said, I'm not sure how useful this second list is. I think the first (where you estimate each player's skill and rank them) does a far better job of not only giving us an idea of who the best players are but also give us an idea of how volatile the estimates are. This "are they REALLY top 32" stuff just muddles the issue IMO. Particularly, it's easy for people to confuse whether someone has a high probability of being top 32 because they're really, really good or whether it's because you've just got a lot of data that tells you to be pretty sure the guy is solid.

Just my $.02. I remember when I took Bayesian a couple of classmates did an analysis of SC1 where they tried to predict winners of matches based on maps, races, and the amount of days the players had since their last game. (I guess this was to measure prep time or something.) It was pretty fun stuff.

Mip

United States63 Posts

December 11 2010 07:29 GMT

#44

I've been hesitant to be too technical in these threads because most of the audience doesn't have a stats background.

The data is a list of names in this format:
Winner Loser
--------------------
Player1 Player2
Player1 Player2
Player2 Player1
Player2 Player3
etc.

The likelihood is the Bradley-Terry model f(x) = exp(skill1)/(exp(skill1)+exp(skill2)).

The priors on the skill parameters are Normal(0,sigma^2) (Bradley Terry model is only dependent on the difference of the skills. Players with skills 100 and 101 would yield the same probability comparisons as if we subtracted 100 to make it 0 and 1, so the 0 mean is arbitrary. It's has same theoretical backing that the ELO system is based off of)

My professor said that sigma^2 could probably be fixed, to test, I just gave it a somewhat informative prior around 1 to see if it the data would alter it (they did not).

So the parameters are run through an MCMC algorithm. Had to use Metropolis steps to calculate draws from the posterior distributions of the skill parameters.

My first report was the mean of the posterior draws and the standard deviation of the posterior draws, then the mean - 2 standard deviations to give a sort of, "at their worst" skill parameter.

The second report, I took each draw from the skill parameters and took the top 32 for each one. Then I calculated the proportion of the times each player appeared in the top 32 over all posterior draws.

Vorlik

1522 Posts

December 11 2010 08:02 GMT

#45

This is fascinating. I like it! :-]

Prev 1 2 3 All

Please or register to reply.

Live Events Refresh