EPTStarcraft 2 Science: Skillcraft Results - Page 4
| Forum Index > SC2 General |
|
Doodsmack
United States7224 Posts
| ||
|
Budmandude
United States123 Posts
| ||
|
Mongolbonjwa
Finland376 Posts
On March 07 2013 01:18 CrushDog5 wrote: How does expert performance vary throughout a practice session? Which practice schedules lead to the most performance gains? How does age influence reaction times in _complex_ cognitive motor tasks? How does expert performance changes with small changes to the task (e.g., balance patches, expansions)? If you want to do a neurophysiological study of multitasking, what skill level players do you need to recruit? Are some cognitive motor skills necessary to learn before others become important (e.g., A-moving via the minimap is a crazy technique, unless you are already looking at the minimap a lot), and what is the optimal training order? When in the skill spectrum is it optimal to learn hotkeys (expert digital artists have Photoshop hotkeys, for example)? And so on... One final point: The whole history of science is proof that amazing progress can be made by understanding our world, even if we don't know the applications at the time. The invention of the laser was purely a basic science project, with no obvious practical benefits. Without it though, we have no optical media, no laser eye-surgery. We are a large group of scientists: http://cslab-sfu.ca Most other research carefully measures performance on simple tasks in the lab, or uses an expert vs. novice approach, which tests only two points on the skill continuum, leaving thousands of hours of skill development unmeasured. The 7 virtues of studying RTS game replays are these: 1. A rich dynamic task environment. 2. Highly motivated participants. 3. Noninvasive and direct measures of domain performance. 4. Accurate measures of motor performance and attentional allocation. 5. Large datasets. 6. Numerous variables. 7. Many levels of expertise. No other approach has more than 4 of these. Yes there is lots of research on expertise: The Cambridge Handbook of Expertise and Expert Performance, an 800 page summary of research in this area, sits on my desk. Nevertheless, there are large gaps in our knowledge, and many of those gaps can be filled by studying RTS replays across skill development. We do not claim that our approach obviates any other research method. Quite the contrary, our method works well in conjunction with brain imaging work and other contrastive approaches. But replays have exactly the kind of information that is lacking in other studies: precise cognitive motor performance measures across time. No player can tell you this information, and no fMRI will show this information. Think about it this way. Our first study ended up with data from 3,360 players. If a human coach teaches unique 72 players a year, it would take over 40 years to teach that many players. Even then, the human coach will forget lots of players over that time period. Our dataset, not only remembers every player, it remembers every single screen move, and every single mouse click that every player has performed. We're not saying it will tell you everything, but surely you can see how it would be useful. First problem is that you cannot possible find all neccesary information just from replays, good example is that reaction time question, replay does not provide needed informations to really make any conclusions. To get wider and more complete understanding about experise and its development you would absolutely need to fullfill this research with neurophysiological and psychological examinations. | ||
|
Hitch-22
Canada753 Posts
Love it though, good work. | ||
|
Skytt
Scotland333 Posts
| ||
|
CrushDog5
Canada207 Posts
On March 07 2013 14:34 Skytt wrote: How long will you be taking uploads for? I've not been saving my replays since I started only playing on the HotS beta, if you will be taking HotS replays then I'll upload my first batch of replays from the expansion. The idea is that we take everything you've got that's old (WoL, HotS beta), then we'll check in with you in a couple months and get what you've accumulated since you submitted. We want as complete a sample as possible, so if you have a large collection of WoL replays, please do submit them. | ||
|
Deleted User 137586
7859 Posts
As for PAC, you say that ``A PAC basically consists of a shift of the screen to a new location for some time, followed by at least one action (typically 4-6), and then a shift to some other location." And that's roughly what you talked about in the video as well. But what you are really talking about is how quickly a person changes their focus and how quickly they decide what to do within that new focus, right? So, to get the most insane PAC, you'd need to be a good multitasker. Someone who's quickly jumping between, let's say, 3 medivacs and their base, giving very quick movement single commands at each location, would have very high PAC. So, if higher PAC means that you are a better player, then a better player will more quickly start doing something else and the more quickly part is because the the moment their screen locks, they know how to do the actions as fast as they can. (A combination of decision-making and mechanics.) So, in another context, a good chef is one that will very quickly jump from preparing one ingredient to another, with very little pause in deciding the next necessary action, and quick movements in preparing each ingredient (for example, if the next action is chopping onions, their hands would immediately grab the appropriate knife and an onion, do the slicing motions quickly and efficiently, and then move on.) Did I understand that correctly (I should probably read Salvucci & Goldberg [2000] but may-be I can understand it better already here)? On an unrelated point. Are the major differences in PAC between the races? I could imagine that Z has higher PAC than T due to infects and creep spread, and T higher PAC than P due to the more harass-based gameplay (perhaps limited to TvP). | ||
|
quirinus
Croatia2489 Posts
On March 06 2013 04:02 CrushDog5 wrote: Finally, just for kicks, we scanned the chat of each game for gg or any common variant. If you want to be in GM. ![[image loading]](http://i.imgur.com/BqupuX2.jpg) "No GG, no skill." - White-Ra Seems he was right. XD | ||
|
Mongolbonjwa
Finland376 Posts
In contrary, there is real multitasking in fighter pilot training and actually flying the jet plane, and it is far more difficult than starcrafts "multitasking" which is not even real multitasking. So if you wanna study multitasking, fighter pilots and their training programs are something that should be checked. Even the qualifiers for pilot training programs are very difficult. Flying jets also requires great deal of emergency management skills, which again is far more demanding than playing starcraft. Specially when considered that fighter pilots need to learn to deal with high G-forces and they have to study lots of theory of physics of flying and how the jet itself works, learning to fly jet planes is harder than starcraft. | ||
|
CrushDog5
Canada207 Posts
On March 07 2013 16:37 Ghanburighan wrote: Great post! I'm tempted to massively grind out HoTS 1v1s just to participate again (Apparently I have played myself into Diamond over 3 years in under 300 games ... it sucks having to work). Did you have a lot of RTS experience already? If not, then go ahead and submit your replays. Faster learners are very interesting. Did I understand that correctly (I should probably read Salvucci & Goldberg [2000] but may-be I can understand it better already here)? \This is fine. You have the main idea. Salvucci & Goldberg just discribe a particular algorithm for creating fixations in eye-movement data (a dispersion threshold algorithm) that we used to aggregate the raw screen moves into fixations. Fixations with at least one action are PACS. Just to clarify, PAC Action Latency gets SMALLER, that is, it takes less time to respond. On an unrelated point. Are the major differences in PAC between the races? I could imagine that Z has higher PAC than T due to infects and creep spread, and T higher PAC than P due to the more harass-based gameplay (perhaps limited to TvP). We looked at lots of race specific variables, the vast majority were not very good. At one point we had a variable for MicroAPM. This was higher for Terrans, especially the Terran pros. Other than that, race was not that strong. There might still be something that we missed (we weren't keen to have to explain race in a scientific paper). The other interesting thing we didn't find, is Macro. No macro variable outside of workers trained was useful for anything. Surprising, but there it is. | ||
|
Deleted User 137586
7859 Posts
On March 08 2013 00:59 CrushDog5 wrote: Did you have a lot of RTS experience already? If not, then go ahead and submit your replays. Faster learners are very interesting. I had 0 experience with RTS games before SC2. Now I feel all special ^^ I'll make sure to submit. + Show Spoiler + Did I understand that correctly (I should probably read Salvucci & Goldberg [2000] but may-be I can understand it better already here)? \This is fine. You have the main idea. Salvucci & Goldberg just discribe a particular algorithm for creating fixations in eye-movement data (a dispersion threshold algorithm) that we used to aggregate the raw screen moves into fixations. Fixations with at least one action are PACS. Just to clarify, PAC Action Latency gets SMALLER, that is, it takes less time to respond. On an unrelated point. Are the major differences in PAC between the races? I could imagine that Z has higher PAC than T due to infects and creep spread, and T higher PAC than P due to the more harass-based gameplay (perhaps limited to TvP). We looked at lots of race specific variables, the vast majority were not very good. At one point we had a variable for MicroAPM. This was higher for Terrans, especially the Terran pros. Other than that, race was not that strong. There might still be something that we missed (we weren't keen to have to explain race in a scientific paper). The other interesting thing we didn't find, is Macro. No macro variable outside of workers trained was useful for anything. Surprising, but there it is. That is very surprising indeed (especially for injects and SQ). Can't wait to see more results. | ||
|
AveSharia
United States62 Posts
So how would one go about monitoring one's PACs/latency? It sort of cries out for a Gears plugin. | ||
|
dsjoerg
United States384 Posts
On March 08 2013 00:59 CrushDog5 wrote: The other interesting thing we didn't find, is Macro. No macro variable outside of workers trained was useful for anything. Surprising, but there it is. Wild! Did you look at SQ? I presume there is a lot of correlation among the various measures you used, so that aside from your top-ranked criterion, the next-highest-ranked criterion achieves that spot because it has the most useful information that's orthogonal to the top-ranked one? Some more detail about your ranking algorithm/procedure would be welcome of course  | ||
|
CrushDog5
Canada207 Posts
Didn't. SQ is based on the resources graph, which is not in the actions list part of the replay. We only looked (so far) at the actions part. We also want to focus (at the beginning) on Cognitive/Motor sorts of variables. I presume there is a lot of correlation among the various measures you used, so that aside from your top-ranked criterion, the next-highest-ranked criterion achieves that spot because it has the most useful information that's orthogonal to the top-ranked one? Some more detail about your ranking algorithm/procedure would be welcome of course To some extent. Conditional inference forests (which is what we used) use bootstrapping, which serves to de-correlate the variables.This method is commonly used in screening procedures. Obviously this is different than something like linear regression. In cases where there were very strong correlations (action latencies to the first action compared to latencies of the other actions in a PAC) we chose the most powerful of the two and dumped the other. The most highly correlated variables in the analysis we show are Hotkey Assigns and Hotkey Selects. It's in the paper, which we'll post when it's accepted. Basically we use permutation raking; randomly permuting the variable and measure how much worse the classifier gets. We run these 25 times (that is, we create 25 forests using the same procedure) and include a random control predictor. If the median importance is better than the max of the control, it's unlikely due to chance. That's also how we can tell that Action latency is a better predictor than APM is some leagues: There is simply no overlap over 25 runs, APM always looses. Vice versa for Masters/Pro, where APM always wins. Hope that helps. | ||
|
Mongolbonjwa
Finland376 Posts
| ||
|
Gemini_sc2
Norway69 Posts
| ||
|
Zamiel
United States211 Posts
Great post, but I have some criticisms of the "hotkey" section. Specifically, "production building hootkey assigns", "upgrade building hotkey assigns", and "combat unit hotkey designs". These 3 sections extrapolate the numerical control group assignments from the replay. However, when you do so, you fail to recognize or take into account the possibility of control group remapping. To explain what I mean, if you go into the StarCraft 2 options and select "Hotkeys", then select "Global", then select "Control Groups", you will see that it is possible to remap, for example, the "1" button to control group 4. If a player did that, the replay would show that the player was using a lot of control group 4; implicitly, you assume that he is also using the button "4" to do this, and this is obviously not the case. Many players remap control group hotkeys. Most commonly, it is possible to remap ALL possible hotkeys (not just control groups) to one small area of the keyboard. In doing so, the hand doesn't have to be moved whatsoever to perform any arbitrary action. For an excellent example of this, see the Core. A hotkey configuration in which the hand never has to move is optimal for APM purposes, but the specific configuration of each player's setup varies widely. And since there is no way to extrapolate a player's individual remapping's from a replay, it makes the resulting numerical control group assignments completely arbitrary and altogether worthless for any kind of scientific aim. This is the kind of thing that would never be caught in a scientific peer review, but only by a StarCraft player! As for the rest, good work. =) | ||
|
E.L.V.I.S
Belgium458 Posts
| ||
|
Ry2D2
United States429 Posts
| ||
|
CrushDog5
Canada207 Posts
On March 10 2013 04:55 Zamiel wrote: Hi CrushDog5, Great post, but I have some criticisms of the "hotkey" section. Specifically, "production building hootkey assigns", "upgrade building hotkey assigns", and "combat unit hotkey designs". These 3 sections extrapolate the numerical control group assignments from the replay. However, when you do so, you fail to recognize or take into account the possibility of control group remapping. To explain what I mean, if you go into the StarCraft 2 options and select "Hotkeys", then select "Global", then select "Control Groups", you will see that it is possible to remap, for example, the "1" button to control group 4. If a player did that, the replay would show that the player was using a lot of control group 4; implicitly, you assume that he is also using the button "4" to do this, and this is obviously not the case. Many players remap control group hotkeys. Most commonly, it is possible to remap ALL possible hotkeys (not just control groups) to one small area of the keyboard. In doing so, the hand doesn't have to be moved whatsoever to perform any arbitrary action. For an excellent example of this, see the Core. A hotkey configuration in which the hand never has to move is optimal for APM purposes, but the specific configuration of each player's setup varies widely. And since there is no way to extrapolate a player's individual remapping's from a replay, it makes the resulting numerical control group assignments completely arbitrary and altogether worthless for any kind of scientific aim. This is the kind of thing that would never be caught in a scientific peer review, but only by a StarCraft player! As for the rest, good work. =) What do you suppose the is the percentage of players who remap their 1-0 control groups to different keys? If it is large, then you're right, the data don't mean very much, because you don't really know where those keys are. If it's small though, it will just add some noise to those data. My guess is that lots of pros have extensive brood war experience and so haven't changed those keys at all. I also think that the average <platinum player probably hasn't changed them. This is just my intuition, of course. Has anyone ever done a poll, here? | ||
| ||
