• Log InLog In
  • Register
Liquid`
Team Liquid Liquipedia
EDT 10:34
CEST 16:34
KST 23:34
  • Home
  • Forum
  • Calendar
  • Streams
  • Liquipedia
  • Features
  • Store
  • EPT
  • TL+
  • StarCraft 2
  • Brood War
  • Smash
  • Heroes
  • Counter-Strike
  • Overwatch
  • Liquibet
  • Fantasy StarCraft
  • TLPD
  • StarCraft 2
  • Brood War
  • Blogs
Forum Sidebar
Events/Features
News
Featured News
[ASL19] Finals Recap: Standing Tall10HomeStory Cup 27 - Info & Preview18Classic wins Code S Season 2 (2025)16Code S RO4 & Finals Preview: herO, Rogue, Classic, GuMiho0TL Team Map Contest #5: Presented by Monster Energy6
Community News
Weekly Cups (June 30 - July 6): Classic Doubles5[BSL20] Non-Korean Championship 4x BSL + 4x China9Flash Announces Hiatus From ASL66Weekly Cups (June 23-29): Reynor in world title form?14FEL Cracov 2025 (July 27) - $8000 live event23
StarCraft 2
General
The GOAT ranking of GOAT rankings Weekly Cups (June 30 - July 6): Classic Doubles Jim claims he and Firefly were involved in match-fixing Esports World Cup 2025 - Final Player Roster Statistics for vetoed/disliked maps
Tourneys
FEL Cracov 2025 (July 27) - $8000 live event Sparkling Tuna Cup - Weekly Open Tournament RSL: Revival, a new crowdfunded tournament series WardiTV Mondays Korean Starcraft League Week 77
Strategy
How did i lose this ZvP, whats the proper response Simple Questions Simple Answers
Custom Maps
[UMS] Zillion Zerglings
External Content
Mutation # 481 Fear and Lava Mutation # 480 Moths to the Flame Mutation # 479 Worn Out Welcome Mutation # 478 Instant Karma
Brood War
General
ASL20 Preliminary Maps i aint gon lie to u bruh... BGH Auto Balance -> http://bghmmr.eu/ [ASL19] Finals Recap: Standing Tall SC uni coach streams logging into betting site
Tourneys
[BSL20] Non-Korean Championship 4x BSL + 4x China [BSL20] Grand Finals - Sunday 20:00 CET CSL Xiamen International Invitational The Casual Games of the Week Thread
Strategy
Simple Questions, Simple Answers I am doing this better than progamers do.
Other Games
General Games
Nintendo Switch Thread Stormgate/Frost Giant Megathread Path of Exile What do you want from future RTS games? Beyond All Reason
Dota 2
Official 'what is Dota anymore' discussion
League of Legends
Heroes of the Storm
Simple Questions, Simple Answers Heroes of the Storm 2.0
Hearthstone
Heroes of StarCraft mini-set
TL Mafia
TL Mafia Community Thread Vanilla Mini Mafia
Community
General
US Politics Mega-thread Russo-Ukrainian War Thread Stop Killing Games - European Citizens Initiative Summer Games Done Quick 2025! Summer Games Done Quick 2024!
Fan Clubs
SKT1 Classic Fan Club! Maru Fan Club
Media & Entertainment
Anime Discussion Thread [Manga] One Piece [\m/] Heavy Metal Thread
Sports
2024 - 2025 Football Thread Formula 1 Discussion NBA General Discussion TeamLiquid Health and Fitness Initiative For 2023 NHL Playoffs 2024
World Cup 2022
Tech Support
Computer Build, Upgrade & Buying Resource Thread
TL Community
The Automated Ban List
Blogs
momentary artworks from des…
tankgirl
Culture Clash in Video Games…
TrAiDoS
from making sc maps to makin…
Husyelt
StarCraft improvement
iopq
Customize Sidebar...

Website Feedback

Closed Threads



Active: 613 users

Starcraft 2 Science: Skillcraft Results

Forum Index > SC2 General
Post a Reply
1 2 3 4 5 6 7 Next All
CrushDog5
Profile Joined March 2010
Canada207 Posts
Last Edited: 2013-03-05 19:02:11
March 05 2013 19:02 GMT
#1
STARCRAFT II + SCIENCE
FINDINGS AND RESULTS
BY: MARK BLAIR (CRUSHDOG5)

[image loading]



The first empirical study into StarCraft 2 expertise (here) was the largest study of expert human performance to date. Thanks to the participation of thousands of SC2 players this study we have expanded our knowledge of human skill acquisition, and we believe that the results, when published, will go a long way to convince the scientific community that StarCraft 2 is a genuine domain of expertise. The project received positive press coverage from Scientific American, the Wall Street Journal and from game oriented media such as TeamLiquid. It also spawned a segment on the science show Daily Planet on the Discovery Channel (Canada) featuring none other than the incomparable Liquid`TLO. We can’t release our results before publication, but we can summarize some of the findings we thought would be most interesting to players. We do so below, after an all-important solicitation for participation in an even more important study.

While our first project was trail-blazing in a number of ways, our second StarCraft 2 project will dwarf the first both in the quantity and quality of data collected. We believe that the project has the potential to profoundly change how skill learning and expertise are studied forever. The key to the project is that it involves analyzing the changes in individual players across time. What we need is replay packs from players who have saved most (>75%) of their replays. As long as the missing replays are few and you have a lot of replays to provide (300+). This may be three hundred, a couple thousand, or even 10’s of thousands (White-Ra!). The scientific community simply does not have access to similar data on learning in a complex game like StarCraft 2. It will show us when individual players learn the necessary skills. We will also see to what extent certain skills can compensate for others, and if certain skills are always learned before others. Finally, and perhaps most importantly, we’ll have much better ideas of how to actually help StarCraft 2 players improve.

Please participate here (instruction for finding your replays)

The survey and uploader are here.

If you haven’t saved your replays, please help spread the word to players who might have.

Interesting data from our project so far...

APM versus The Perception Action Cycle

In daily life, we use our attention to focus on objects we want to interact with. That is, looking is for doing. StarCraft is no different, and thus the vast majority of actions happen as part of what we call a “Perception Action Cycle” (PAC). A PAC basically consists of a shift of the screen to a new location for some time, followed by at least one action (typically 4-6), and then a shift to some other location. The delay to the first action in a PAC turns out to be one of the best predictors across all leagues, and the best in certain leagues (beating out the venerable APM, which, despite it’s faults, is a good predictor of league). Other components of the PAC are also important predictors of skill.

Does Variable Importance Change with Skill?

Our primary research question was: to what extent does the importance (i.e. predictive power) of variables change across levels of skill? If it changes more than a little, it is a difficult problem for researchers, because most of the time it’s not feasible to collect detailed data from across many levels of experience, and the alternatives (for example collecting data just from experts and novices) are less useful where variables frequently change in importance. We used a machine learning technique (random forests of conditional inference trees) to build classifiers for each league compared to the one two leagues up (Bronze vs. Gold, Silver versus Platinum, etc.).

This figure describes the ranked importance of each of our key variables in these “skip-league” classifiers. APM was ranked first, for instance, in the Bronze vs Gold classifier, implying that it was our most useful variable for distinguishing those leagues.



Settings Data

Ever wonder how other people have their options set? So did we. The following graphs are results from our original survey about player settings. Note, we did not get surveys from pro players, so we report only GMs. The following figures describe how players set their game settings in each league. For example, roughly 60% of Pros set flyer status (red lines showing the location over which a flyer hovers) to “always on.” Unit Status bars refer to unit health bars and Alerts (obviously) refer to game alerts.







Hotkey Data

These data shows the assignment of hotkeys by pros. This is interesting when compared to Bronze hotkeys. We have two different kinds of figure here. The first figure describes the frequency of Bronze and Pro players’ hotkey assigns per game (broken down by type of unit being assigned). Note how much more the Pros assign hotkeys.



The following figures describe how often particular unit types are assigned to particular hotkeys by professional players. For example, professional players in our sample assigned almost two production structures to hotkey 4 a game.







GGs

Finally, just for kicks, we scanned the chat of each game for gg or any common variant. If you want to be in GM.



Thanks

The participation of players such as yourselves makes our research possible. Thank you.

Questions

I realize this was brief (probably too science-y for some, and not enough for others) I’ll keep an eye on the comments to this thread and answer any questions that I can. I’ll also post the link to the actual scientific paper (which has much more detail about the specific analyses) once it’s available online.

SkillCraft.com - StarCraft + Science
Evolpeac
Profile Joined December 2007
United States34 Posts
March 05 2013 13:27 GMT
#2
Awesome :D I was waiting for these results. Thanks for all the effort.
Any fool can criticize, condemn and complain and most fools do. - Benjamin Franklin
dsjoerg
Profile Joined January 2012
United States384 Posts
March 05 2013 14:28 GMT
#3
Outstanding! What is the median time to publication for a paper such as yours? Can't wait to read it.
card-carrying grubby fan. developer of GGTracker.
dsjoerg
Profile Joined January 2012
United States384 Posts
March 05 2013 14:34 GMT
#4
Is it too late to submit your paper for "peer review" to quantitatively-inclined members of the Starcraft community? I'd be happy to review it.

Meanwhile, some questions:
  • How do you define Action Latency?
  • Is Number of PACs literally that, or Number of PACs per unit time?
  • Is Workers Made literally that, or Workers Made per unit time?
  • In your analysis do you restrict yourself to ladder games? What other restrictions/filters do you impose, such as game length.
  • Would you be OK with software such as GGTracker incorporating analysis of PACs along the lines of your research? If so it would be helpful for you to describe your algorithm for PACs.



card-carrying grubby fan. developer of GGTracker.
CrushDog5
Profile Joined March 2010
Canada207 Posts
March 05 2013 19:21 GMT
#5
On March 05 2013 23:34 dsjoerg wrote:
Is it too late to submit your paper for "peer review" to quantitatively-inclined members of the Starcraft community? I'd be happy to review it.


A revision of the paper is already under review. We're just awaiting a decision. If it doesn't go at that journal, we'll submit to another.


[*] How do you define Action Latency?


It's the time between the start of a PAC (a new fixation, based on the dispersion threshold algorithm) and the time of the first action.

A youtube description of PACS can be found here:

[*] Is Number of PACs literally that, or Number of PACs per unit time?

Per unit time. There were not big differences in game length based on league, so it's basically the same for our purposes.

[*] Is Workers Made literally that, or Workers Made per unit time?

Again, per minute.

[*] In your analysis do you restrict yourself to ladder games? What other restrictions/filters do you impose, such as game length.

Yes. 1v1 ladder games


[*] Would you be OK with software such as GGTracker incorporating analysis of PACs along the lines of your research? If so it would be helpful for you to describe your algorithm for PACs.


Yes, of course. We're grateful for the community support and glad to share. Special shoutout to DakotaFanning and the SC2 Gears project, it was very helpful.

Basically, Salvucci & Goldberg (2000), Thresholds we used are 20 timestamps/6 map units. PACs are fixations with at least one action. You have to actually parse the replay, though, to get all the actions and screen moves.


SkillCraft.com - StarCraft + Science
MCXD
Profile Blog Joined February 2012
Australia2738 Posts
March 05 2013 19:38 GMT
#6
Very interesting. I look forward to seeing more from the study.
CountChocula
Profile Blog Joined January 2011
Canada2068 Posts
March 05 2013 19:51 GMT
#7
I'm curious how you measured the "unique hotkeys" category means in the variable importance table. Could you explain?
Writer我会让他们连馒头都吃不到 Those championships owed me over the years, I will take them back one by one.
CrushDog5
Profile Joined March 2010
Canada207 Posts
March 05 2013 19:58 GMT
#8
On March 06 2013 04:51 CountChocula wrote:
I'm curious how you measured the "unique hotkeys" category means in the variable importance table. Could you explain?


This is the number of different keys used for custom hotkeys. This goes from zero to 10 [1-0]. Sometimes high level players spam hotkeys, put their nexus on all 10, but it's not that common overall.


SkillCraft.com - StarCraft + Science
PedroBlanco
Profile Joined August 2011
Canada65 Posts
March 05 2013 20:00 GMT
#9
It's really frustrating playing in Masters, rage>gg.
xxjcdentonxx
Profile Joined November 2012
Canada163 Posts
Last Edited: 2013-03-05 20:01:37
March 05 2013 20:01 GMT
#10
league names in the GG chart are not in the correct order
"Expand or die." —Ferengi Rule of Acquisition #45
CrushDog5
Profile Joined March 2010
Canada207 Posts
March 05 2013 20:10 GMT
#11
On March 06 2013 05:01 xxjcdentonxx wrote:
league names in the GG chart are not in the correct order


Good catch.
The data are in the correct order, it's only the label that's off. League is coded (and plotted) as 1-7.
SkillCraft.com - StarCraft + Science
JinDesu
Profile Blog Joined August 2010
United States3990 Posts
March 05 2013 20:14 GMT
#12
One small note - I don't know if it's because the sun is glaring in from my window right now, but white text on almost teal background is a little hard to read. Sorry for being a bit picky.
Yargh
Defacer
Profile Blog Joined October 2010
Canada5052 Posts
March 05 2013 20:24 GMT
#13
Congrats on your paper, Crush. Me and a colleague of mine actually met with you to learn more about your work for an exhibition. Sorry that didn't pan out. But it's really exciting to see some of your findings!
Heyoka
Profile Blog Joined March 2008
Katowice25012 Posts
March 05 2013 20:25 GMT
#14
On March 06 2013 05:14 JinDesu wrote:
One small note - I don't know if it's because the sun is glaring in from my window right now, but white text on almost teal background is a little hard to read. Sorry for being a bit picky.


I did the layout for this and agree it's not great I might change it to yellow or something, will try to fix today.

The results are pretty cool, looking forward to seeing it published.
@RealHeyoka | ESL / DreamHack StarCraft Lead
crbox
Profile Joined August 2010
Canada1180 Posts
Last Edited: 2013-03-05 20:26:23
March 05 2013 20:25 GMT
#15
wow incredible post. So much research, very interesting. I use 7 for my evo chambers too, didn't know it was the most common. Also in the "gg chart" you inverted platinium and diamond

edit: funny how masters has the lowest gg ratio. Doesn't surprise me, I used to get bm'd way more back in the days in master league.
CrushDog5
Profile Joined March 2010
Canada207 Posts
March 06 2013 01:39 GMT
#16
On March 06 2013 05:25 crbox wrote:
wow incredible post. So much research, very interesting. I use 7 for my evo chambers too, didn't know it was the most common. Also in the "gg chart" you inverted platinium and diamond

edit: funny how masters has the lowest gg ratio. Doesn't surprise me, I used to get bm'd way more back in the days in master league.



I assumed better players would know how to play a manner game. Guess not O.o

SkillCraft.com - StarCraft + Science
seaofsaturn
Profile Blog Joined September 2010
United States489 Posts
March 06 2013 01:44 GMT
#17
Cool stuff.

In the GG chart you have Diamond before Platinum.
Photoshop is over-powered.
crbox
Profile Joined August 2010
Canada1180 Posts
March 06 2013 01:45 GMT
#18
On March 06 2013 10:39 CrushDog5 wrote:
Show nested quote +
On March 06 2013 05:25 crbox wrote:
wow incredible post. So much research, very interesting. I use 7 for my evo chambers too, didn't know it was the most common. Also in the "gg chart" you inverted platinium and diamond

edit: funny how masters has the lowest gg ratio. Doesn't surprise me, I used to get bm'd way more back in the days in master league.



I assumed better players would know how to play a manner game. Guess not O.o



well most gm players are mannered tbh. I think it has something to do with the frustration of not being recognized even though they consider themselves as good.
c0sm0naut
Profile Joined April 2011
United States1229 Posts
March 06 2013 03:30 GMT
#19
Thank you so much. This is truly interesting;D
BlackPride
Profile Joined July 2012
United States186 Posts
March 06 2013 03:34 GMT
#20
I freaking love stuff like this so much!
I've never waited in line at the DMV [YVNG]
1 2 3 4 5 6 7 Next All
Please log in or register to reply.
Live Events Refresh
Next event in 1h 26m
[ Submit Event ]
Live Streams
Refresh
StarCraft 2
Creator 104
StarCraft: Brood War
Bisu 2022
Shuttle 1332
firebathero 1306
EffOrt 1249
Hyuk 629
Mini 612
GuemChi 569
Larva 503
PianO 395
Soulkey 384
[ Show more ]
Snow 254
Mind 157
Soma 154
ToSsGirL 145
Rush 135
Pusan 114
Hyun 73
Sharp 71
Sea.KH 66
Barracks 64
Movie 38
JYJ36
Aegong 32
soO 24
yabsab 24
HiyA 20
GoRush 20
Sacsri 20
JulyZerg 19
Free 14
Terrorterran 14
Bale 13
Yoon 13
IntoTheRainbow 13
ivOry 4
Dota 2
Gorgc11488
qojqva2604
XcaliburYe260
syndereN220
League of Legends
singsing2349
Dendi1243
Counter-Strike
flusha328
fl0m278
kRYSTAL_187
Super Smash Bros
Mew2King91
Other Games
tarik_tv24921
gofns19446
B2W.Neo1317
hiko906
DeMusliM459
crisheroes375
Lowko333
FrodaN330
Pyrionflax125
ArmadaUGS97
QueenE58
Rex29
Organizations
Other Games
gamesdonequick38680
StarCraft 2
angryscii 50
StarCraft: Brood War
Kim Chul Min (afreeca) 11
StarCraft 2
Blizzard YouTube
StarCraft: Brood War
BSLTrovo
sctven
[ Show 13 non-featured ]
StarCraft 2
• poizon28 2
• AfreecaTV YouTube
• intothetv
• Kozan
• IndyKCrew
• LaughNgamezSOOP
• Migwel
• sooper7s
StarCraft: Brood War
• BSLYoutube
• STPLYoutube
• ZZZeroYoutube
League of Legends
• Nemesis6409
• TFBlade683
Upcoming Events
WardiTV European League
1h 26m
MaNa vs sebesdes
Mixu vs Fjant
ByuN vs HeRoMaRinE
ShoWTimE vs goblin
Gerald vs Babymarine
Krystianer vs YoungYakov
PiGosaur Monday
9h 26m
The PondCast
19h 26m
WardiTV European League
21h 26m
Jumy vs NightPhoenix
Percival vs Nicoract
ArT vs HiGhDrA
MaxPax vs Harstem
Scarlett vs Shameless
SKillous vs uThermal
uThermal 2v2 Circuit
1d 1h
Replay Cast
1d 9h
RSL Revival
1d 19h
ByuN vs SHIN
Clem vs Reynor
Replay Cast
2 days
RSL Revival
2 days
Classic vs Cure
FEL
3 days
[ Show More ]
RSL Revival
3 days
FEL
3 days
FEL
4 days
CSO Cup
4 days
BSL20 Non-Korean Champi…
4 days
Bonyth vs QiaoGege
Dewalt vs Fengzi
Hawk vs Zhanhun
Sziky vs Mihu
Mihu vs QiaoGege
Zhanhun vs Sziky
Fengzi vs Hawk
Sparkling Tuna Cup
4 days
RSL Revival
4 days
FEL
5 days
BSL20 Non-Korean Champi…
5 days
Bonyth vs Dewalt
QiaoGege vs Dewalt
Hawk vs Bonyth
Sziky vs Fengzi
Mihu vs Zhanhun
QiaoGege vs Zhanhun
Fengzi vs Mihu
Replay Cast
6 days
Liquipedia Results

Completed

BSL Season 20
HSC XXVII
Heroes 10 EU

Ongoing

JPL Season 2
BSL 2v2 Season 3
Acropolis #3
KCM Race Survival 2025 Season 2
CSL 17: 2025 SUMMER
Copa Latinoamericana 4
Jiahua Invitational
Championship of Russia 2025
RSL Revival: Season 1
Murky Cup #2
BLAST.tv Austin Major 2025
ESL Impact League Season 7
IEM Dallas 2025
PGL Astana 2025
Asian Champions League '25
BLAST Rivals Spring 2025
MESA Nomadic Masters
CCT Season 2 Global Finals
IEM Melbourne 2025

Upcoming

2025 ACS Season 2: Qualifier
CSLPRO Last Chance 2025
CSL Xiamen Invitational
2025 ACS Season 2
CSLPRO Chat StarLAN 3
K-Championship
uThermal 2v2 Main Event
SEL Season 2 Championship
FEL Cracov 2025
Esports World Cup 2025
Underdog Cup #2
StarSeries Fall 2025
FISSURE Playground #2
BLAST Open Fall 2025
BLAST Open Fall Qual
Esports World Cup 2025
BLAST Bounty Fall 2025
BLAST Bounty Fall Qual
IEM Cologne 2025
FISSURE Playground #1
TLPD

1. ByuN
2. TY
3. Dark
4. Solar
5. Stats
6. Nerchio
7. sOs
8. soO
9. INnoVation
10. Elazer
1. Rain
2. Flash
3. EffOrt
4. Last
5. Bisu
6. Soulkey
7. Mini
8. Sharp
Sidebar Settings...

Advertising | Privacy Policy | Terms Of Use | Contact Us

Original banner artwork: Jim Warren
The contents of this webpage are copyright © 2025 TLnet. All Rights Reserved.