|
On July 12 2012 02:17 skeldark wrote:Show nested quote +On July 12 2012 01:53 Lysenko wrote:On July 12 2012 00:12 skeldark wrote:Oh btw: @lolcanoe Based on my understanding of the MMR system, you would not expect normality... As a stats junky, I'm very suspicious of the statistics here. i hope you troll... No, he's right -- Elo, at least, usually follows a logistic distribution across a population. I pointed that out earlier, btw. http://en.wikipedia.org/wiki/Logistic_distribution Beside all of this. People act like they have a problem with the way i analysed the data. But no one had one single point where i did something wrong. You can say: "i did not understand what you did at point X, can you please explain it and i allways did"
You can say : i think you make an mistake at point Y because this will result in this error ... and i will check it.But what i see so far is this. -this is wrong because its wrong -Skill that dont take apm into account is no skill / you only look at win streaks you should check for the skill not the mmr - Lets talk about something total off topic about statistic... Often people start with so obvious wrong arguments that its hard for me to force myself to read the rest. I did not see an single post that point out an mistake. If you want to atack my thesis you are welcome, but you have to bring a point. And with you i dont mean you personal.
Whatever mmr is, it have nothing to do with the analyse result! I think i show pretty simple that the "race-groups average" is to high to be a random mistake. And there is nothing in the data that is depending on the player race. So the race value must be biased. = not balanced with the whole data. If i just come up with a random number that i write behind each account and i accidental get such results, im A ) very very very lucky or B) my random number is not random it shows an biased in the racedata.
I think the main problem people are having is they have no "baseline" to compair your final results too. -26.98 for terran is a fine result, but I don't know what that means compaired to other months or moments in time. If you did the same test for the ladder 3 months after SC2s release and showed the results, side by side, I think people would have a better idea of what the data means. Or if you did it monthly and we could see trends.
I am not claiming the results don't show anything, but people need raw data to be grounded with some real world examples of what it means. Then when it changes, they can apply the data to the changes they see taking place.
|
On July 12 2012 02:38 Plansix wrote:Show nested quote +On July 12 2012 02:17 skeldark wrote:On July 12 2012 01:53 Lysenko wrote:On July 12 2012 00:12 skeldark wrote:Oh btw: @lolcanoe Based on my understanding of the MMR system, you would not expect normality... As a stats junky, I'm very suspicious of the statistics here. i hope you troll... No, he's right -- Elo, at least, usually follows a logistic distribution across a population. I pointed that out earlier, btw. http://en.wikipedia.org/wiki/Logistic_distribution Beside all of this. People act like they have a problem with the way i analysed the data. But no one had one single point where i did something wrong. You can say: "i did not understand what you did at point X, can you please explain it and i allways did"
You can say : i think you make an mistake at point Y because this will result in this error ... and i will check it.But what i see so far is this. -this is wrong because its wrong -Skill that dont take apm into account is no skill / you only look at win streaks you should check for the skill not the mmr - Lets talk about something total off topic about statistic... Often people start with so obvious wrong arguments that its hard for me to force myself to read the rest. I did not see an single post that point out an mistake. If you want to atack my thesis you are welcome, but you have to bring a point. And with you i dont mean you personal.
Whatever mmr is, it have nothing to do with the analyse result! I think i show pretty simple that the "race-groups average" is to high to be a random mistake. And there is nothing in the data that is depending on the player race. So the race value must be biased. = not balanced with the whole data. If i just come up with a random number that i write behind each account and i accidental get such results, im A ) very very very lucky or B) my random number is not random it shows an biased in the racedata. I think the main problem people are having is they have no "baseline" to compair your final results too. -26.98 for terran is a fine result, but I don't know what that means compaired to other months or moments in time. If you did the same test for the ladder 3 months after SC2s release and showed the results, side by side, I think people would have a better idea of what the data means. Or if you did it monthly and we could see trends. I am not claiming the results don't show anything, but people need raw data to be grounded with some real world examples of what it means. Then when it changes, they can apply the data to the changes they see taking place.
good point. I dont know it either.
I will do this. At the moment i only have data of my last patch because all before did not upload the race data.
So from now on i get new data every day. I can do the exact same with a new dataset.
I use this moment to say sorry, if i offended people that make valid points.
But you guys have to understand me. I see a lot of post that are so completely wrong that you start to get really annoyed.
|
Are we sure arithmetic mean is the way to go here? One of the problems with arithmetic mean is that it's vulnerable to being skewed by outliers on the extremes. Just looking at your chart, the top 2 data points are Protoss and the bottom 2 data points are Terran. It seems plausible to me that the races of those outlying data points might be random (might get different outlying races in a different sample), and that they might have undue influence on the results.
Another issue that might be worth looking into is the possibility of different trends at different points in the distribution. People at TL really only care about balance from masters and above, but the majority of the data used in your analysis is coming from diamond and below. SC2 is a totally different game in diamond than it is in high masters, and it seems plausible that the races would have different strength levels at the differently levels accordingly. For example. marine vs baneling balance is hugely dependent on skill level. In diamond and below, banelings crush marines, but as you approach GM, marines perform much better. I don't mean to pile work on you since I'm not willing to do it, but if someone is interested and motivated, it would probably be worth isolating skill level ranges and seeing what you find.
|
On July 12 2012 02:47 skeldark wrote:Show nested quote +On July 12 2012 02:38 Plansix wrote:On July 12 2012 02:17 skeldark wrote:On July 12 2012 01:53 Lysenko wrote:On July 12 2012 00:12 skeldark wrote:Oh btw: @lolcanoe Based on my understanding of the MMR system, you would not expect normality... As a stats junky, I'm very suspicious of the statistics here. i hope you troll... No, he's right -- Elo, at least, usually follows a logistic distribution across a population. I pointed that out earlier, btw. http://en.wikipedia.org/wiki/Logistic_distribution Beside all of this. People act like they have a problem with the way i analysed the data. But no one had one single point where i did something wrong. You can say: "i did not understand what you did at point X, can you please explain it and i allways did"
You can say : i think you make an mistake at point Y because this will result in this error ... and i will check it.But what i see so far is this. -this is wrong because its wrong -Skill that dont take apm into account is no skill / you only look at win streaks you should check for the skill not the mmr - Lets talk about something total off topic about statistic... Often people start with so obvious wrong arguments that its hard for me to force myself to read the rest. I did not see an single post that point out an mistake. If you want to atack my thesis you are welcome, but you have to bring a point. And with you i dont mean you personal.
Whatever mmr is, it have nothing to do with the analyse result! I think i show pretty simple that the "race-groups average" is to high to be a random mistake. And there is nothing in the data that is depending on the player race. So the race value must be biased. = not balanced with the whole data. If i just come up with a random number that i write behind each account and i accidental get such results, im A ) very very very lucky or B) my random number is not random it shows an biased in the racedata. I think the main problem people are having is they have no "baseline" to compair your final results too. -26.98 for terran is a fine result, but I don't know what that means compaired to other months or moments in time. If you did the same test for the ladder 3 months after SC2s release and showed the results, side by side, I think people would have a better idea of what the data means. Or if you did it monthly and we could see trends. I am not claiming the results don't show anything, but people need raw data to be grounded with some real world examples of what it means. Then when it changes, they can apply the data to the changes they see taking place. good point. I dont know it either. I will do this. At the moment i only have data of my last patch because all before did not upload the race data. So from now on i get new data every day. I can do the exact same with a new dataset.
I use this moment to say sorry, if i offended people that make valid points. But you guys have to understand me. I see a lot of post that are so completely wrong that you start to get really annoyed.
If you want a really sharp contrast, try to get results from the 4-5 rax reaper era. Even if you could only get a small amount of data from good players at that time, you could trim the current data down to show the results being compairable to each other.
You could also open with a line like: "You want to see data that proves imbalance? You can't handle the data that proves imbalance!" Then show us the data.
|
On July 12 2012 02:17 skeldark wrote:
Beside all of this. People act like they have a problem with the way i analysed the data. But no one had one single point where i did something wrong.
You were averaging series' of players MMR and then averaging those into your data set, assuming incorrectly that they were independent data points. That's one error. You say it doesn't have a significant impact, but you came back with that answer suspiciously fast, which tells me you didn't really understand the nature of the concern or what you'd have to do to check that you were correctly applying your statistics everywhere. This is such a fundamental type of problem that it calls into question everything else that you've done behind the scenes (particularly since you don't show your work leading to your results.)
I think you need to take a deep breath and think about what we're actually saying.
Statistical methods like you're using are based on assumptions about how and what kind of data you collect. If you do not collect the data in a way that matches the assumptions of your statistics, you will not get meaningful results. You'll either come up with a measure of how good your numbers are that doesn't really mean what you think it does, or you'll get errors that add up as you collect more data, or you'll fit a curve that doesn't apply to the data set you're collecting.
You're looking at concerns about these things, valid concerns, and saying "well the effect is small, I can hand-wave it all away." No, you really can't. You have to go back to where you started, check every assumption, learn about the statistics you don't know, and convince the rest of us that you applied them all correctly by showing us your math.
You said earlier in this conversation that you couldn't show us everything because it would be "200 pages." Well, in my view, that's 200 pages in which to make a fundamental mistake. If you can't share every step, there's no way to check your work or understand what you've done. There's no way to know whether you're onto something great or just pulling meaningless numbers out of nowhere.
Don't take this personally -- you may have a great data set on your hands. But your responses to the concerns we're raising do not inspire confidence that you understand the issues involved in doing what you're doing.
Yes im not not exactly doing a straight-up book statistical analysis. Thats what i said in the op from the beginning.
But i do an statistical analyse.
It's all or nothing. You either double and triple check that your data set matches the assumptions of the statistical method you're using, or your numbers don't mean anything.
Finally: I'm certainly not in any way offended by anything you've said. I want you to succeed. But, if you're going to predict the likelihood of your result being random, you're deep into actual statistical analysis, and you'd better do it correctly or it just doesn't mean anything.
|
It's all or nothing. You either double and triple check that your data set matches the assumptions of the statistical method you're using, or your numbers don't mean anything.
A check can proove if they are meaningful but dont make them meaningful!
I think that is the main problem. You wait for the kind of prove you are used to see.
So you assume its not meaningful even if its obvious that it is.
I think i found a way to convince you.
That was the reason why i did not care for the avg of mmr: it does not care what mmr is:
Forget sc2 , mmr and races for a second:
Given :
Data A
Data A was created without knowledge of P
Property P
Property P was collected without knowledge of A
90.000 Random sorted Data groups of A produced in 99.55% of the cases values between -25 and +25
P sorted Data groups of data A produced T: -53.68 P 11.87 Z 31.52
P sorted Data group of data B subgroub of A produced T: -27.70 P 17.49 Z 3.82
P sorted Data group of data C subgroub of A produced T: -43.51 P 0.37 Z 34.93
Is this not statistic significant that Data A is biased towards P?
|
|
|
On July 12 2012 03:24 skeldark wrote:
So you assume its not meaningful even if its obvious that it is.
If you haven't shown us that your data set fits the assumptions used by the model you're using to analyze it, nothing is "obvious."
|
On July 12 2012 04:54 Lysenko wrote:Show nested quote +On July 12 2012 03:24 skeldark wrote:
So you assume its not meaningful even if its obvious that it is.
If you haven't shown us that your data set fits the assumptions used by the model you're using to analyze it, nothing is "obvious."
Look my example again. Forget sc2. forget mmr.
Data A, any data. I assume its significant. you agree or not?
I think you misunderstand what im doing here. I dont get paid for it and i have no benefit from it at all.
I dont care at all witch race is imba. I have the data and so i publish it.
You can say its meaningless for you because i did not do it in the way you are used to do it.
But in this case why dont you do it your way and publish it here?
|
On July 12 2012 05:22 skeldark wrote:Show nested quote +On July 12 2012 04:54 Lysenko wrote:On July 12 2012 03:24 skeldark wrote:
So you assume its not meaningful even if its obvious that it is.
If you haven't shown us that your data set fits the assumptions used by the model you're using to analyze it, nothing is "obvious." look my example again. Data A any data. 0 Assumptions. You agree its significant or not?
Statistical signficance is quantifiable - this isn't a matter of agreement or disagreement. Set a P-value, do a single greater than/less than comparison, and you either have it or you don't. If you knew statistics well you'd be asking for better tests than accusing me of "trolling".
Can you please just layout what exactly happenned here? Just lay out how you are getting to the .-29.87 etc values. You take a group. Average the MMR... and then? What's this number coming form exactly? Is it the difference from the mean of all groups?
Lastly, statitistical studies are NEVER perfect, whether it be sampling, distribution fitting, or model assumptions. We're not holding you to a perfect standard - what you need to do is clearly layout the ASSUMPTIONS you made while compiling the data so the readers can determine whether or not these assumptions invalidate the conclusion.
|
On July 12 2012 05:26 lolcanoe wrote:Show nested quote +On July 12 2012 05:22 skeldark wrote:On July 12 2012 04:54 Lysenko wrote:On July 12 2012 03:24 skeldark wrote:
So you assume its not meaningful even if its obvious that it is.
If you haven't shown us that your data set fits the assumptions used by the model you're using to analyze it, nothing is "obvious." look my example again. Data A any data. 0 Assumptions. You agree its significant or not? Statistical signficance is quantifiable - this isn't a matter of agreement or disagreement. Set a P-value, do a single greater than/less than comparison, and you either have it or you don't. If you knew statistics well you'd be asking for better tests than accusing me of "trolling". Can you please just layout what exactly happenned here? Just lay out how you are getting to the .-29.87 etc values. You take a group. Average the MMR... and then? What's this number coming form exactly? Is it the difference from the mean of all groups? Lastly, statitistical studies are NEVER perfect, whether it be sampling, distribution fitting, or model assumptions. We're not holding you to a perfect standard - what you need to do is clearly layout the ASSUMPTIONS you made while compiling the data so the readers can determine whether or not these assumptions invalidate the conclusion.
Like i said...
i take a group of an race. Average the mmr substract the total average mmr of the data from it and have an value.
The diffrence between the Race group and expection.
I assume: when there is no imbalance the average of all terran players = the average of all players.
Serious do i own you anything? because you act like it. Read your post in this thread,
You waste all your time attacking my points and dont take a single second on doing something useful yourself.
You say to agree you have to do x and than tell me to do it?
If you come up and say: i calculated it and i can prove your wrong. that would be fine!
We are not at work here or at university. We are on a community forum.
If you are interested in it and think its important to calculate it why dont you just do it?
|
|
|
On July 12 2012 05:57 monkybone wrote:
I think average MMR is a good statistic and does signify balance. But it assumes one key thing: the activity level of the players of each race is equally distributed. I.e. Zerg, Terran and Protoss have proportional amounts of players playing at each activity level. By focusing on diamond and above I believe this is something we can assume. Taking in every player, the activity level distribution will presumably be heavily skewed to the left for Terran, meaning a lower average MMR.
You can think of it as this: suppose each race did have proportional activity levels equal average MMR. Now add a large group of people with a low activity level for one race, say Terran. This will significantly decrease the average MMR for the race.
"the activity level of the players of each race is equally distributed "
I know thats not the case. My data is from active users because i collect the gamedata ingame life.
what you describe is not an mistake to find imbalance. It is imbalance.
If i have less players of one race on higher skillevel than overall this is imbalance.
There can be many reasons for it ( people dont play because race is boring on high level / people dont play because the design is in favour of an other race, there are no people of this race at this level at all ) , but its all imbalance.
Thats why i pointed out in the op. I detect imbalance not the reason for it.
Imbalance of the data dont have to be imbalance of the game design.
But this is no problem with my method this is a general problem whatever method you use. An not unsolvable one.
|
|
|
1) In sc2 at the moment the activity level of the players of each race is NOT equally distributed.
2) my data is only from active players because my tool collects games life, while the user is playing.
"what you describe is not an mistake to find imbalance. It is imbalance." ?
I dont know how to explain this different than i did.
|
On July 12 2012 05:22 skeldark wrote:
But in this case why dont you do it your way and publish it here?
I'm avoiding doing this because I don't currently know the relevant math well enough to do it, and even despite that it's a huge project.
Serious do i own you anything? because you act like it. Read your post in this thread,
Of course you don't owe us anything. However, if you want to use an extremely sophisticated statistical method to convince someone of something, you'll have to go through each step carefully to show that you're doing it correctly.
Note that even the basic idea by itself of computing an Elo rating is a very complex statistical method. Proceeding to take those ratings and make statements about them in the aggregate just compounds the complexity.
You took this all on yourself when you started this project, and it's fine if you don't care to finish it, but it's not ok to use methods like these with less rigor than they require and then turn around and say "Ok, I broke all of this, you guys fix it for me."
|
On July 12 2012 06:17 Lysenko wrote:Show nested quote +On July 12 2012 05:22 skeldark wrote:
But in this case why dont you do it your way and publish it here? I'm avoiding doing this because I don't currently know the relevant math well enough to do it, and even despite that it's a huge project.
I understand. Same for me. this is a site project. The time i have i use to work on the mmr analyser.
lolcanoe ? how long do you need?
On July 12 2012 06:17 Lysenko wrote:Show nested quote +On July 12 2012 05:22 skeldark wrote:
But in this case why dont you do it your way and publish it here? I'm avoiding doing this because I don't currently know the relevant math well enough to do it, and even despite that it's a huge project. Show nested quote +Serious do i own you anything? because you act like it. Read your post in this thread, Of course you don't owe us anything. However, if you want to use an extremely sophisticated statistical method to convince someone of something, you'll have to go through each step carefully to show that you're doing it correctly. Note that even the basic idea by itself of computing an Elo rating is a very complex statistical method. Proceeding to take those ratings and make statements about them in the aggregate just compounds the complexity. You took this all on yourself when you started this project, and it's fine if you don't care to finish it, but it's not ok to use methods like these with less rigor than they require and then turn around and say "Ok, I broke all of this, you guys fix it for me."
Thats not what i did at all and exactly what im talking about.
If you want to put it this way than i would say:
i show the way 
But serious, look at my data and just tell me,
do you belief based on what i posted that the mmr data is depending on the race?
And if you dont want to answer that question because i did not prove it in the way you want it to be proven
Answer this one:
Is this data more accurate than all the other methods to measure unbalance we saw in the past?
tldr winratio, how many of race x are in top y or tournament z...
|
|
|
|
|
basic yes but instead of having 6 overlapping skill- steps i have 3000+
Also grandmaster is the worst example because its not at all top 200.
But that is an different story.
|
|
|
|
|
|
EPT
