We are researchers at the Cognitive Science Lab (Simon Fraser University in Vancouver, Canada) and are conducting a large-scale study on the development of expertise in StarCraft 2. We believe that the analysis of StarCraft 2 performance has the potential to be very valuable to researchers studying attention, learning, and cognition. Our initial project involves using replay files and survey questions to characterize differences across all skill levels, from bronze to professional. With the community’s support, it could be the largest expertise study ever conducted and may eventually generate positive press coverage for e-Sports and the SC2 community.
Our biggest challenge at this point will be to get a wide variety of players to participate, and to get a lot of them. The size of the study will be one of the big selling points with the academic community so more is definitely better. There are 2 simple ways you can help.
1. Complete the survey, and upload a replay file yourself. 2. Encourage players you know to participate, especially newer players who might not be active in the online community.
If we can get 20,000 players across the full range of skill levels to participate, it will be, by several orders of magnitude, the largest expertise study ever conducted in any domain; bigger than any study on traditional games such as chess, or any sport expertise studies.
We hope that you find our project as exciting and interesting as we do. Your willingness to participate and promote our project is much appreciated.
EDIT: PLEASE DON'T FORGET TO UPLOAD YOUR REPLAY! We are noticing that a lot of people do the survey but don't upload the replay. We are studying screen movements, hotkey use, APM and other factors that we can only get from the replay data, so we really do need the replay. We have about 1000 replays so far. Keep them coming and spread the word. Thank you for your help!
[B]On August 12 2011 23:28 Pharaun wrote:[/B I would like if u can tell us where /when to see any results of this.
We will post results as we get them. Our initial plan is to collect data for about 3 weeks (though it will depend on the response), and the data analysis will take some time. Getting demographic data processed will be easy, though, so we can post up those results not too long after we close the survey.
We also have lots of interesting follow up studies planned. We mainly study attentional learning (how people learn what to pay attention to and when), so doing some eye-tracking studies (we have four trackers) is part of the plan. Ultimately I'd like to go to MLG with some eye-trackers. We are also planning on collecting replays from individuals over a long period of time, say three years, to see what the transition from bronze to diamond looks actually looks like. There are also interesting questions around joint attention and communication in team games.
If our initial attempt can show some interesting results, it will open the doors some research grants and to many interesting followups.
Just any sort of replay? Like, would you discourage cheeses, or try to have replays of games lasting longer than ten minutes, or just, actually, anything??
On August 13 2011 00:31 whoopingchow wrote: Just any sort of replay? Like, would you discourage cheeses, or try to have replays of games lasting longer than ten minutes, or just, actually, anything??
Good Question - We want games in the 10-15 min range.
I don't see how one replay from each preson is very viable. What if they upload a replay file of them that they played their absolute best? Their absolute worst? Under the influence? Any player can tell you that their play can vary widely from day to day
On August 13 2011 00:35 Soliduok wrote: I don't see how one replay from each preson is very viable. What if they upload a replay file of them that they played their absolute best? Their absolute worst? Under the influence? Any player can tell you that their play can vary widely from day to day
It's not about winning/losing. Probably 90% of the submitted games are going to be wins. Your SC2 'DNA' will not really change in a few days, what hotkeys you use, how you use your camera's, macro, micro, apm etc etc.
On August 13 2011 00:31 whoopingchow wrote: Just any sort of replay? Like, would you discourage cheeses, or try to have replays of games lasting longer than ten minutes, or just, actually, anything??
Good Question - We want games in the 10-15 min range.
Isn't that usually just a random cheese or build order loss though?
I uploaded a 25 minute game where I thought I played well (not spectacular but I didn't make any really dumb mistakes) and eventually got out macroed.
I have some experience with survey studies, and no offense but, posting an open call for submissions is never going to get you the samplesize you want. If you could incentivize it somehow (ie a raffle/giveaway) and partner with someone like Husky for publicity, you would stand a better chance. Good luck though.
On August 13 2011 00:35 Soliduok wrote: I don't see how one replay from each preson is very viable. What if they upload a replay file of them that they played their absolute best? Their absolute worst? Under the influence? Any player can tell you that their play can vary widely from day to day
Good Point.
A single game is a really a single sample from the full distribution of performance (on any particular variable) that a player might produce. If people's play varies wildly, e.g. Bronze when drunk and Masters when "in the zone", then we might have a hard time saying precisely how good a player is based on this one sample.
Ideally you'd want a perfect estimate of whatever variable you want to measure, but of course that's not easy to do. Even Blizzards matchmaking system, which is doing essentially that with skill, isn't perfect.
One of the advantages that our study will have over all the other expertise studies, is that we have the possibility to get a really large sample of players from each league, so the aggregate estimate (of, say, the mean APM of a gold level player, for instance) is likely to be pretty good, even when our ability to estimate the abilities of a single player might not be so good.
Long term, we hope to find people who are interested in participating in more detailed research and willing to give us 50 games and not just 1. Then we can have better estimates of each player, as you suggest.
On August 13 2011 00:31 whoopingchow wrote: Just any sort of replay? Like, would you discourage cheeses, or try to have replays of games lasting longer than ten minutes, or just, actually, anything??
Good Question - We want games in the 10-15 min range.
Ohhh, I uploaded my last replay which was a 6pool I won. Oh well too late to change now.
"Without your help scientists would be unable to investigate the mysteries of the human mind!" was a bit over the top but there you have it, survey complete. And for the record I have no idea how many hours I played bw lans with my friends growing up but it was probably over the generous number I put.
On August 13 2011 00:46 maruchan wrote: partner with someone like Husky for publicity, you would stand a better chance.
We have tried to partner with as many casters/teams/communities as possible.
If anyone knows Husky, Day[9], Total Biscuit, HD, d.Apollo, Destiny, or anyone else who can help promote the project please pass along the project link and encourage them to help. You can PM me here, or through my university email (Mark Blair, SFU) if you have questions.
On August 13 2011 01:05 tubasteve wrote: You're going to have a lot of selection bias in this data since you aren't using a random sample.
I would be REALLY helpful if people could encourage their Bronze League friends to do the survey as well. I'm pretty sure most of the ballers here on TL are not in Bronze, so please help spread the word.
will do it later, but so glad to read that the survey takes only 5 minutes... finally researchers who understand what length works on the internet... I mean being cognitive scientists you really should!
On August 13 2011 01:49 TBO wrote: will do it later, but so glad to read that the survey takes only 5 minutes... finally researchers who understand what length works on the internet... I mean being cognitive scientists you really should!
On August 13 2011 01:49 TBO wrote: will do it later, but so glad to read that the survey takes only 5 minutes... finally researchers who understand what length works on the internet... I mean being cognitive scientists you really should!
LOL.
And that 5 minute estimate is generous - it assumes you put some thought into # of hours you've played and stuff!
Hoping to see some results in the near future :D this looks really interesting!
On August 13 2011 00:31 whoopingchow wrote: Just any sort of replay? Like, would you discourage cheeses, or try to have replays of games lasting longer than ten minutes, or just, actually, anything??
Good Question - We want games in the 10-15 min range.
Oops, I did the survey before reading this comment. At first I was looking for a replay of a good game which I won, but then I thought that most people will probably send in reps of games they won so I decided to be different and upload one of a loss. I found a good long pvz macro game that I played and lost recently and uploaded that one. It's about 35 minutes. Hope this is OK.
Anyway, I'm really interested in hearing about what you 'discover', best wishes.
On August 13 2011 06:52 jjthomps wrote: I would take a PhD in SC (an SCD?)! Also, I hope it can be proved (once and for all) that Terran is ridiculously OP.
As a Protoss, I worry about experimenter bias. I'll get one of the Random players to run that analysis.
Easy survey, really interested to see what kind of data will come out of this study! I hope you can get the attention of some prominent figures in the community to help promote this.
Submitted mine with a failure, though I think I played my current best. It was a placement match where I was out macroed, but put up a pretty good fight (re-established my base, held off a few attacks and such). Made a bunch of stupid mistakes, but that's my norm
Also subscribed to the e-mail list
Edit: Oh yeah... I mentioned it in my submission but thought i might as well mention it here:
Be careful about interpreting results involving the hotkey display question. Grid users like myself have no need since there's a direct physical mapping between the command card and the keyboard. Looking at the button's position is equivalent to looking at the hotkey.
I feel a lot of the questions should have an option of < 1 instead of having 1 as the lowest unit. But I guess that isn't a reason enough to remake the test.
Love to help but I had some issues with some questions.
For example the sc1 questions, I don't think many beyond the really hardcore people know from what year to what year you played the game.
Also in the RTS section I found it quite hard to estimate an average. I usually crunch through a new rts in a few days or play an old rts a bit if I ever feel like it. Starcraft 2 is the only RTS I've ever had this consistent interest over a long time so I'd have to say no to frequently playing rts but by no means do I rarely play other RTS in general. On the contrary I love RTS games.
Good stuff, done. I was a little surprised that there was no 0-1 h per day option though. I play 1-2 h, but only just. There must be some people who play less than me (surely?).
On August 13 2011 10:35 Amui wrote: Did the survey, also as an SFU student, would it be possible to drop by in a few weeks to sneak a peak at results?
We'll be posting results as we get them. We're expecting to collect data for 2-3 weeks, and then the analysis will take some time. We're pretty well setup for the data crunching (eyetrackers spit out a lot of data) but we are really careful about checking and double checking our analysis scripts and so on. There will also be challenges that turn up in a large data set that we didn't seen in our pilots. Those will take time to sort out.
I am trying to finish this, but I don't have any 1v1 ladder reps from recently at all as I rarely play ladder and it will not let me finish without one...hmmm...anyone? I guess I will play a ladder game later when I get home to assist you guys.
done, was a bit annoying when it came to how many hours total ive played, since i honestly have no idea, esp with sc1 i just know that its a ton of hours T.T at least 5k hrs in sc1 and 1k in sc2
Done! Where else are you guys talking about study? If it's just TL, I'd imagine that the majority of responses would be from higher level players, maybe diamond and above. You'd probably have a better shot at drawing in lower level players from the blizzard official forums and the like..
On August 13 2011 14:17 nalgene wrote: Since you work at SFU, which class do you teach? Just curious.
I teach Cogs100 Exploring the Mind, and Cogs 310 Consciousness. I don't teach again till Sept 2012 though because I am on Sabbatical during the next year. Full time research for me!
Done, but after I submitted I realised that I may have accidentally posted a practice game rather than a ladder game. It's still a 1v1 game on a ladder map though, if that's alright?
On August 13 2011 20:18 Alsn wrote: Done, but after I submitted I realised that I may have accidentally posted a practice game rather than a ladder game. It's still a 1v1 game on a ladder map though, if that's alright?
It will get thrown out if it's a custom game. If you want to resubmit just to be sure, that is ok. We'll only count one valid submission for each participant.
On August 14 2011 03:08 whatthefat wrote: So, sorry, is the fact that there is no 0-1 h of SC2 per day option a mistake or not? The scale starts at 1-2 h and goes up from there.
Yes, it's a mistake. Since we have a bunch of data already, it's too late to fix.
So, problem... maybe xD I filled out the survey, sent in my profile and all that, went to ladder and BAM promotion. Is there a computer program that collected my rank as soon as I submitted it, or will my new rank be represented in my data because it's to be collected later? I don't wanna ruin any data, science + SC2 = pure awesome and I don't wanna be anything but helpful!
On August 14 2011 07:35 UmiNotsuki wrote: FOR SCIENCE!
Is there a computer program that collected my rank as soon as I submitted it, or will my new rank be represented in my data because it's to be collected later?
There is a delay (about 24hours, typically) between submission and rank collection, so you should be good to go. Congratulations on your promotion!
On August 14 2011 03:08 whatthefat wrote: So, sorry, is the fact that there is no 0-1 h of SC2 per day option a mistake or not? The scale starts at 1-2 h and goes up from there.
Yes, it's a mistake. Since we have a bunch of data already, it's too late to fix.
No problem, you can just treat that bin as 0-2 h.
I'm very excited about this study and I look forward to seeing the results when the analysis is completed. Best of luck!
On August 14 2011 06:22 See.Blue wrote: Done. Spread the word yo
Edit: Any word on how many people you've had respond?
Hey See.Blue, were hoping to collate weekend results and get some information on the numbers out to you guys later on today. There are Day 1 numbers posted on the Skillcraft.ca website under the blog section so check it out
On August 14 2011 03:08 whatthefat wrote: So, sorry, is the fact that there is no 0-1 h of SC2 per day option a mistake or not? The scale starts at 1-2 h and goes up from there.
Yes, it's a mistake. Since we have a bunch of data already, it's too late to fix.
No problem, you can just treat that bin as 0-2 h.
I'm very excited about this study and I look forward to seeing the results when the analysis is completed. Best of luck!
Hey whatthefat, we have changed this option and are now treating that first group as a bin of 0-2 hours, this way we can use the data without worrying about any potential conflicts. Thanks for pointing out the mistake in the first place, and if you guys notice anything else strange about the survey while you are doing it let us know and well try to help you out
Just a separate note, thanks to everyone who has participated so far, we have been amazed at the community response. So many people wishing us good luck and giving us support. You guys are awesome! Make sure to let your friends of all skill levels know and get them to participate as well, from bronze to professional every replay counts!
Just to let you guys know, if you have any questions or comments and you post them here then either myself or CrushDog5 (Mark Blair) will try to answer them as best we can. There might be some down time between questions and answers as we are handling a lot of traffic right now. If you haven't already checked out the project, check us out at Skillcraft.ca and fill out a survey
On August 16 2011 10:36 HTODethklok wrote: Just filled out the survey and made a thread on my clan website encouraging everyone to do the same.
Thanks Dethklok, this is the kind of community support that makes our project possible! We should hopefully have some interesting things to show you guys soon, so keep checking back Mark Blair, the professor in charge of the project, will be doing an AMA (ask me anything) on Reddit on Thursday, so if you have any questions for him directly make sure to check that out, we will be sure to let you guys know where to find that. For everyone out there who has already filled out the survey but would like to still help us out, we are lacking in lower level replays, if you have friends who are just starting out in StarCraft 2 and are in Bronze or Silver league get them to complete our survey and submit a replay, and posting on clan sites like Dethklok goes a long way to spreading the word! A big thank you to everyone who has gone that extra mile to help us out, and to everyone who takes the time to fill out our survey, you guys are awesome!
Hey guys just to let you know, there is an update thread here http://www.teamliquid.net/forum/viewmessage.php?topic_id=255578 It contains some information about the demographics of people participating, ie. What countries are we seeing the most responses from and what leagues have the most participation so far. Go check out the graphs for yourself EDIT: It also contains the official announcement for the AMA, and will be used for further updates, so if you have questions be sure to ask them there.
Hey, I was wondering... What sort of replay should I upload? I'm a silver player and therefore my games vary wildly. Do you want good games or bad games? Games where I get trashed/trash them or long drawn out macro games?
On August 19 2011 02:35 FreudianTrip wrote: Hey, I was wondering... What sort of replay should I upload? I'm a silver player and therefore my games vary wildly. Do you want good games or bad games? Games where I get trashed/trash them or long drawn out macro games?
Hey FreudianTrip, were looking for any normal recent games. Preferably not the extremely short games (i.e.. They dropped out or SCV rushed and won) and not the 2-hour long mine out games. It can be either a win or loss, and good or bad. We want to see everything. Just make sure that its a recent replay from quick match ladder play and that it is representative of what a normal game for you would be. If you have any other questions Mark Blair, the professor in charge of the project, is running an AMA on Reddit right now at http://www.reddit.com/r/starcraft/comments/jmwut/iama_cognitive_science_professor_doing_research/ So check it out and you can ask questions there.
On August 19 2011 03:13 Giwoon wrote: im in this sounds awesome! :D
EDIT:nvm cant im not over 16 -.-
Hey Giwoon our apologies that there is an age limit on the study, because all our research is conducted under the Research Ethics Board of the University, they insist that we have to get parental permission for people under 16 participate. Since we have no way to check your age or verify your parent's permission over the internet, we are required to include an age limit statement on the informed consent page. Sorry about that. We are planning on doing more studies in the future so keep looking out for that and hopefully you can get involved in one of our follow up studies! If you still want to help us out, but cannot participate, make sure you spread the news. Let your friends know about the study and get the word out!
Done! Super easy and good luck with your research sir!
I can't wait until it's done
I would love to know all the details about how you analyze the replays. As in what does someone who hotkeys their buildings to 5-6-7-8 versus someone who does 1-2-3-4 mean? Or does it mean anything? etc.
I'm still very interested in learning the results of this study.
Also, just a small question: Why cap the replays a user can upload to one? Wouldn't it be better for the study if a single user could upload as many as five replays?
On August 24 2011 08:29 State wrote: Why cap the replays a user can upload to one? Wouldn't it be better for the study if a single user could upload as many as five replays?
Good question, we debated the decision heavily.
There are several reasons: First, we though that making it easy for people was very important. This study is the first of it's kind, and we thought we'd get a better response if it was simple to do. As it turned out, only half the people that do the study complete all three steps. If we had required multiple replays, it might be even less.
Second, having multiple replays, and particularly different numbers of replays from each person, creates a scenario which is more difficult to analyze for statistical reasons. Because we have never done a mass replay study, we were reluctant to add more complexity to the analysis.
Third, because we are interested primarily in identify league characteristics, rather than individual player characteristics, it can be better to get fewer games from more players.
If we have some success with this project, we will likely do followups where players submit many games over the course of a long time. We'll have better measures of individual progress, in that case.
Thanks for participating, and please spread the word. The more participants, the better.
Great bump! You helped me find it. Participating now! (and yay, as a bronze player I finally get to participate in something worthwhile!)
edit: after doing it, I'm very very very intrigued as to how they will be interpreting the data. There were some pretty (seemingly) unrelevant questions.
I think this is a brilliant Idea from some good old, BC boys, unfortunately I am without a computer that can handle SC2 atm so I will have to take a raincheck and try and remember to do it when I have a better computer.
On September 06 2011 03:12 robih wrote: done i guess there aren't many low tier players here on TL
I reckon there are a lot of fans of e-sports in general that lurk here or haven't bothered to create an account that I'm sure would be more than happy to participate! The only thing is a topic needs a good bump to catch their attention. They might miss things that the people who check TL every 5 minutes see.
Uploaded my nooby little gold league replay. I have a friend who is bronze. I'll see if I can get him to do this and upload a replay if you need bronze players.
Done, id like them to post thier results. I bet its gonna say starcraft palyers are all brainy... because it made me fail my exams, id like somehting back xD
Wow, this is phenomenal! I'll be sending along my replay as soon as I can get on the computer I have SC2 installed on. Please keep us informed of your progress, this is really exciting!
This will be a fascinating study. There are so many players who play thousands of games, yet still aren't very skilled. I think there's a definite self selection bias and there's not going to be many bad players who want to respond though. I'd love to see the results and conclusions of this.
I'm a bit surprised there were no questions about sound settings. I recently started playing with sound after playing without since last October and I didn't realize how much of a difference it can make sometimes.
I did the survey and tried to spread the word as far as my social connections go..
I am quite amazed and fascinated by this initiative and hard work, also it's very nice to see this kind of dedication towards research involving StarCraft.
Wow, surprised that the split between races is so fair, and that there are so many less Terran, since many people I talked to (especially beginners) picked Terran cause they heard it was the easiest to play and they were somewhat used to it cause of the campaign.
On August 24 2011 08:29 State wrote: Why cap the replays a user can upload to one? Wouldn't it be better for the study if a single user could upload as many as five replays?
Good question, we debated the decision heavily.
There are several reasons: First, we though that making it easy for people was very important. This study is the first of it's kind, and we thought we'd get a better response if it was simple to do. As it turned out, only half the people that do the study complete all three steps. If we had required multiple replays, it might be even less.
Second, having multiple replays, and particularly different numbers of replays from each person, creates a scenario which is more difficult to analyze for statistical reasons. Because we have never done a mass replay study, we were reluctant to add more complexity to the analysis.
Third, because we are interested primarily in identify league characteristics, rather than individual player characteristics, it can be better to get fewer games from more players. + Show Spoiler +
If we have some success with this project, we will likely do followups where players submit many games over the course of a long time. We'll have better measures of individual progress, in that case.
Thanks for participating, and please spread the word. The more participants, the better.
Interesting study, you guys have obviously thought about selection bias. My question is what is the distribution of responses that you are receiving within each league. Are you getting getting a normal distribution of players or are your responses skewed towards the higher end of the spectrum? It's a gut feel, but what if people who have just entered masters, don't feel they "represent" masters? Do you have a way of ensuring that you get a decent distribution of replays?
Also do you expect to have similar numbers of participants in each league or would you stratify based on blizzard's estimates for each league. For example, Bronze would be expected to have much larger numbers than masters, how is that dealth with?
Apologies for the 1001 questions, just interested as my guess is that this study would have a hard time getting a sample, that is actually representative.
Good luck!
Edit:
On September 06 2011 10:22 DystopiaX wrote: Wow, surprised that the split between races is so fair, and that there are so many less Terran, since many people I talked to (especially beginners) picked Terran cause they heard it was the easiest to play and they were somewhat used to it cause of the campaign.
The real question is whether that distribution is consistent between leagues. I would imagine that terran is over-represented in the lower leagues and via-versa with zerg.
I would imagine that the range of players on TL is skewed toward the higher end of the players on battle.net since you have to care about a game a fair bit to be an active participant on a forum and to take this survey.
Also as a SEA player I couldn't figure out how to link to my NA profile (where most of my games are). It is also possible that I loaded the wrong replay and you now have a random pro replay.
So besides giving us some interesting trends in the data that might benefit people playing starcraft. How do you think this could affect other fields of science / teaching or even sports?
I think you'd get a lot more feedback from the EU community (at least in Germany) if you'd post on sites like readmore.de and taketv.net
TeamLiquid is absolutely the central gathering point for the Starcraft 2 community, but there is a tremendous amount of Europeans who don't really come here.
Complete, I didn't forget a replay either. I know this is in the spotlight on TL but is there anyway it can get a little more attention? I think this is a really interesting study for everyone at TL and other people too so the more results the better!
On September 06 2011 16:53 MrBitter wrote: I think you'd get a lot more feedback from the EU community (at least in Germany) if you'd post on sites like readmore.de and taketv.net
TeamLiquid is absolutely the central gathering point for the Starcraft 2 community, but there is a tremendous amount of Europeans who don't really come here.
Only reasonable way to get more bronze/silver participation would be to reach them in-game somehow. Can't see too many lower league players browsing community sites and then taking the time to do a survey related to something they're probably not too passionate about.
On September 06 2011 12:38 Probulous wrote: . My question is what is the distribution of responses that you are receiving within each league. Are you getting getting a normal distribution of players or are your responses skewed towards the higher end of the spectrum? It's a gut feel, but what if people who have just entered masters, don't feel they "represent" masters? Do you have a way of ensuring that you get a decent distribution of replays?
Also do you expect to have similar numbers of participants in each league or would you stratify based on blizzard's estimates for each league. For example, Bronze would be expected to have much larger numbers than masters, how is that dealth with?
I'm not sure how you'd tell about the distribution with leagues. You have to pick a specific variable and see how that varies within league. But you'd have no way of knowing if, for example, your sample is mainly high gold vs. low gold. We don't, unfortunately, have access to the MMR for each player, just their league.
My hope isn't for an equal proportion to what Blizzard uses. It's pretty clear that we'll have fewer lower level players than higher level players. It won't be a problem unless we don't have enough low level players to get reliable numbers. It's not ideal that the players that have the most variability in their data are also the least likely to participate. We have around 150 bronze now, and I'll breath a sigh of relief when we hit 200. Unless the data are unexpected noisy (always a possibility), we should be fine. It's true though that we will have better estimates of those leagues with the most participants.
On September 06 2011 06:58 Drowsy wrote: I'm a bit surprised there were no questions about sound settings. I recently started playing with sound after playing without since last October and I didn't realize how much of a difference it can make sometimes.
In case you don't already know, Crushdog5, the APM as measured ingame is based on ingame time, which, on faster, is 1.33 x regular. Therefore, APM must be multiplied by 1.33 to get the actual number of actions per minute.
Interesting how skewed towards the higher leagues the people participating in this are. Granted, people more involved with the community, and hence more likely to see this, tend to be higher ranked. Still I wonder if uploading a replay is keeping lower leaguers out because they are feel bad having someone look at their play.
Anyway, sending this link to my lower ranked buddies to try to get them to do it
On September 07 2011 03:18 Shadrak wrote: Interesting how skewed towards the higher leagues the people participating in this are. Granted, people more involved with the community, and hence more likely to see this, tend to be higher ranked. Still I wonder if uploading a replay is keeping lower leaguers out because they are feel bad having someone look at their play.
Anyway, sending this link to my lower ranked buddies to try to get them to do it
Well, replays are pretty necessary as that's their primary method of determining skill, as it should be.
I'm probably going to do it, just want to make sure , is there's any problem if my replay is from an older patch and that my bnet profile doesnt have any game this season?
On September 07 2011 03:18 Shadrak wrote: Interesting how skewed towards the higher leagues the people participating in this are. Granted, people more involved with the community, and hence more likely to see this, tend to be higher ranked. Still I wonder if uploading a replay is keeping lower leaguers out because they are feel bad having someone look at their play.
Anyway, sending this link to my lower ranked buddies to try to get them to do it
Well, replays are pretty necessary as that's their primary method of determining skill, as it should be.
Oh I'm not saying they shouldn't require them. Just saying it might help explain the high skill-skewed data they are going to get.
On September 07 2011 03:50 CynanMachae wrote: I'm probably going to do it, just want to make sure , is there's any problem if my replay is from an older patch and that my bnet profile doesnt have any game this season?
Well, newer is better, but if you don't have anything else, go ahead and submit what you have.
Just played some games 1v1 to get a replay. First two were ZvPs with stalkers, immortals, and archons shitting on me. In the first game he kept on denying bases until switching to voids and finishing me. In the second game I just lost a big army vs big army battle despite having 10 infestors with fungal and neural. Third and final game was a ZvZ vs some dumbass who went fast roach off one base, but only made 3 roaches, and then tried to expand. I already had my expo up and running and just swarmed him with lings.
On September 06 2011 12:38 Probulous wrote: . My question is what is the distribution of responses that you are receiving within each league. Are you getting getting a normal distribution of players or are your responses skewed towards the higher end of the spectrum? It's a gut feel, but what if people who have just entered masters, don't feel they "represent" masters? Do you have a way of ensuring that you get a decent distribution of replays?
Also do you expect to have similar numbers of participants in each league or would you stratify based on blizzard's estimates for each league. For example, Bronze would be expected to have much larger numbers than masters, how is that dealth with?
I'm not sure how you'd tell about the distribution with leagues. You have to pick a specific variable and see how that varies within league. But you'd have no way of knowing if, for example, your sample is mainly high gold vs. low gold. We don't, unfortunately, have access to the MMR for each player, just their league.
My hope isn't for an equal proportion to what Blizzard uses. It's pretty clear that we'll have fewer lower level players than higher level players. It won't be a problem unless we don't have enough low level players to get reliable numbers. It's not ideal that the players that have the most variability in their data are also the least likely to participate. We have around 150 bronze now, and I'll breath a sigh of relief when we hit 200. Unless the data are unexpected noisy (always a possibility), we should be fine. It's true though that we will have better estimates of those leagues with the most participants.
Thanks for your reply! I am no statistician but stats still fascinate me. Particularly in this kind of study, where the shape of the data can influence the outcomes you achieve. I would assume you have posted something on the BNET forums. How is the response from there?
Thanks for your reply! I am no statistician but stats still fascinate me. Particularly in this kind of study, where the shape of the data can influence the outcomes you achieve. I would assume you have posted something on the BNET forums. How is the response from there?
Replies have been sparse. So far our biggest source of replays has been Reddit, followed by TL. Now that TL has kindly spotlighted this thread, it will generate an even bigger TL response. No luck yet enticing Husky, or Day[9] to give us a plug. We've basically emailed everyone famous, every team, and every major community. Famous people get tons of email, though, so it's hard to break through. We're still hopeful that someone big will notice the project and throw us a helping hand.
1) Do you have any hypotheses that you're testing? (If so, what are they?) 2) Do you verify the league the players are reportedly in? (If not, how do you justify that?) 3) Do you separate the replays by game version? (idem) 4) What do you want to find?
I wonder if this is all just a scam and people are just testing to see how many people post replays of themselves winning over replays of them losing. Idk which to choose @_@.
On September 07 2011 15:32 interpolarity wrote: I wonder if this is all just a scam and people are just testing to see how many people post replays of themselves winning over replays of them losing. Idk which to choose @_@.
I just chose my most recent one, I can't remember if I won or lost...
I'll just wait til I get home and play a game then. Although TBH I'd probably lean more towards posting one of me winning, or at least save one of me winning anyways. I hardly ever save replays and name them "oh my goodness i lost so hard"
On September 07 2011 14:33 Dagobert wrote: A couple of questions that just popped up...
1) Do you have any hypotheses that you're testing? (If so, what are they?) 2) Do you verify the league the players are reportedly in? (If not, how do you justify that?) 3) Do you separate the replays by game version? (idem) 4) What do you want to find?
The OP has links to the AMA and the interview, which basically answers these questions.
I wonder if this is all just a scam and people are just testing to see how many people post replays of themselves winning over replays of them losing. Idk which to choose @_@.
I just used my most recent one. Im pretty sure i won it but i dont really care, i'd rather submit a replay where i played well over one where i played bad but won.
On September 07 2011 15:32 interpolarity wrote: I wonder if this is all just a scam and people are just testing to see how many people post replays of themselves winning over replays of them losing. Idk which to choose @_@.
LOL. I bet if we looked, we'd find you were right. It's a pretty elaborate ruse for such a simple hypothesis though.
Once we did a learning experiment in the lab and we had this participant INSIST that the categories were unlearnable, and that what we were really studying was how long it would take people to get frustrated and quit. It took a while to calm them down and convince them that we weren't trying to trick them at all.
Those damn social psychologists and all their deceptive studies are making everyone paranoid.
I wonder if this is all just a scam and people are just testing to see how many people post replays of themselves winning over replays of them losing. Idk which to choose @_@.
I just used my most recent one. Im pretty sure i won it but i dont really care, i'd rather submit a replay where i played well over one where i played bad but won.
Also just used my most recent one. You don't need to do an elaborate study to find people are going to submit data more favorable to themselves rather than less favorable when given the option.
I didn't read the AMAs or interview, but I got the impression they were solely taking statistics on hotkeys, screen movements, APM, and the like. Whether winning or losing, that stuff doesn't change a whole lot.
I feel like this is going to be highly biased by people selecting which replay they submit. Good luck, though, it's nice to see a serious attempt at something like this. I'm interested to see what you find.
bah im not at home can i do the survery now and upload a rep later? or can i use a youtube vod or old rep thats probably hosted on a replay site?
PS- for the education level question, I technically never completed/graduated HS because I took a proficiency test that said I could be done junior year. And then for college I never completed any degree I just took a bunch of classes. So i'm not really sure what to put.
the amount of played stuff is kinda quirky too because I haven't gotten to play a lot in the last 12 months but before that (even beta which isn't included) I played A LOT.
is health bars normal the one where its off and you press Alt to turn it on? thats what i use.
I filled it out, though I am not anywhere near my main PC so the replay is fairly old. Pulled it off SC2 replayed and I stopped uploading replays a while ago. Really interesting project.
Question: Do we need to upload a recent replay, or can it be a slightly older one? I haven't played a ladder match since before the last patch, but I do have some saved replays from then. Are those ok to upload, or do I need to go play a couple of matches now for it?
On September 09 2011 06:45 Allomancer wrote: Question: Do we need to upload a recent replay, or can it be a slightly older one? I haven't played a ladder match since before the last patch, but I do have some saved replays from then. Are those ok to upload, or do I need to go play a couple of matches now for it?
I think / hope an older one will work as long as it isn't ancient. I sent one from the last patch. Hopefully I am not messing anything up.
This sounds awesome I filled out the survey and uploaded a replay in a few minutes just like it said. Everyone should try and spread the word to get their friends to do this also.
Very interesting study! One statistic, about which my friends and i argued, was the race. I said there are so many Zs, because of the American Idra-Fanboys. ^^ Maybe you can post some "Demographics by Region", so the race diversion in NA compared to the EU, for example. Would LOVE to see that. I know its not the goal of the study, but since you got the data...
I think / hope an older one will work as long as it isn't ancient. I sent one from the last patch. Hopefully I am not messing anything up.
It's not ideal, but if that's all you have that's OK. We don't want a replay that doesn't match your current skill level, though.
I went back and checked and the last replays were older than I thought, so I just went and played another ladder match and uploaded that. Also emailed all my friends that play, hopefully they'll do it too. Anyway, I think this whole project is awesome, can't wait to see the results.
On September 09 2011 21:39 joeyBanana wrote: ^^ Maybe you can post some "Demographics by Region", so the race diversion in NA compared to the EU, for example. Would LOVE to see that. I know its not the goal of the study, but since you got the data...
That should be pretty easy to do. I'll check it out.
I'm really curious to see proportion race by league. The lack of Terrans is surprising. But maybe there are fewer in the middle leagues where we have more replays.
On August 12 2011 23:48 Micket wrote: If I upload a replay of me 1-1-1 ing Protoss and winning easily, how will it be used to determine things?
That would be part of the evolution of the game. i'm working on finding a game where i do a successful, and fairly effective execution of the meta game of PVP, but its the same all around idea.
EPT![[image loading]](http://skillcraft.ca/wp-content/uploads/2011/09/SkillCraft-2011-09-06-Update.png)
![[image loading]](http://www.sfu.ca/psyc/faculty/blair/Dr._Blairs_Site/Publications_files/EU_only_stats.png)
