Starcraft 2 Science: Skillcraft Results

STARCRAFT II + SCIENCE

FINDINGS AND RESULTS

The first empirical study into StarCraft 2 expertise (here) was the largest study of expert human performance to date. Thanks to the participation of thousands of SC2 players this study we have expanded our knowledge of human skill acquisition, and we believe that the results, when published, will go a long way to convince the scientific community that StarCraft 2 is a genuine domain of expertise. The project received positive press coverage from Scientific American, the Wall Street Journal and from game oriented media such as TeamLiquid. It also spawned a segment on the science show Daily Planet on the Discovery Channel (Canada) featuring none other than the incomparable Liquid`TLO. We can’t release our results before publication, but we can summarize some of the findings we thought would be most interesting to players. We do so below, after an all-important solicitation for participation in an even more important study.

While our first project was trail-blazing in a number of ways, our second StarCraft 2 project will dwarf the first both in the quantity and quality of data collected. We believe that the project has the potential to profoundly change how skill learning and expertise are studied forever. The key to the project is that it involves analyzing the changes in individual players across time. What we need is replay packs from players who have saved most (>75%) of their replays. As long as the missing replays are few and you have a lot of replays to provide (300+). This may be three hundred, a couple thousand, or even 10’s of thousands (White-Ra!). The scientific community simply does not have access to similar data on learning in a complex game like StarCraft 2. It will show us when individual players learn the necessary skills. We will also see to what extent certain skills can compensate for others, and if certain skills are always learned before others. Finally, and perhaps most importantly, we’ll have much better ideas of how to actually help StarCraft 2 players improve.

Please participate here (instruction for finding your replays)

The survey and uploader are here.

If you haven’t saved your replays, please help spread the word to players who might have.

Interesting data from our project so far...

APM versus The Perception Action Cycle

In daily life, we use our attention to focus on objects we want to interact with. That is, looking is for doing. StarCraft is no different, and thus the vast majority of actions happen as part of what we call a “Perception Action Cycle” (PAC). A PAC basically consists of a shift of the screen to a new location for some time, followed by at least one action (typically 4-6), and then a shift to some other location. The delay to the first action in a PAC turns out to be one of the best predictors across all leagues, and the best in certain leagues (beating out the venerable APM, which, despite it’s faults, is a good predictor of league). Other components of the PAC are also important predictors of skill.

Does Variable Importance Change with Skill?

Our primary research question was: to what extent does the importance (i.e. predictive power) of variables change across levels of skill? If it changes more than a little, it is a difficult problem for researchers, because most of the time it’s not feasible to collect detailed data from across many levels of experience, and the alternatives (for example collecting data just from experts and novices) are less useful where variables frequently change in importance. We used a machine learning technique (random forests of conditional inference trees) to build classifiers for each league compared to the one two leagues up (Bronze vs. Gold, Silver versus Platinum, etc.).

This figure describes the ranked importance of each of our key variables in these “skip-league” classifiers. APM was ranked first, for instance, in the Bronze vs Gold classifier, implying that it was our most useful variable for distinguishing those leagues.



Settings Data

Ever wonder how other people have their options set? So did we. The following graphs are results from our original survey about player settings. Note, we did not get surveys from pro players, so we report only GMs. The following figures describe how players set their game settings in each league. For example, roughly 60% of Pros set flyer status (red lines showing the location over which a flyer hovers) to “always on.” Unit Status bars refer to unit health bars and Alerts (obviously) refer to game alerts.







Hotkey Data

These data shows the assignment of hotkeys by pros. This is interesting when compared to Bronze hotkeys. We have two different kinds of figure here. The first figure describes the frequency of Bronze and Pro players’ hotkey assigns per game (broken down by type of unit being assigned). Note how much more the Pros assign hotkeys.



The following figures describe how often particular unit types are assigned to particular hotkeys by professional players. For example, professional players in our sample assigned almost two production structures to hotkey 4 a game.







GGs

Finally, just for kicks, we scanned the chat of each game for gg or any common variant. If you want to be in GM.



Thanks

The participation of players such as yourselves makes our research possible. Thank you.

Questions

I realize this was brief (probably too science-y for some, and not enough for others) I’ll keep an eye on the comments to this thread and answer any questions that I can. I’ll also post the link to the actual scientific paper (which has much more detail about the specific analyses) once it’s available online.