• Log InLog In
  • Register
Liquid`
Team Liquid Liquipedia
EDT 00:44
CEST 06:44
KST 13:44
  • Home
  • Forum
  • Calendar
  • Streams
  • Liquipedia
  • Features
  • Store
  • EPT
  • TL+
  • StarCraft 2
  • Brood War
  • Smash
  • Heroes
  • Counter-Strike
  • Overwatch
  • Liquibet
  • Fantasy StarCraft
  • TLPD
  • StarCraft 2
  • Brood War
  • Blogs
Forum Sidebar
Events/Features
News
Featured News
TL Team Map Contest #5: Presented by Monster Energy4Code S RO8 Preview: herO, Zoun, Bunny, Classic7Code S RO8 Preview: Rogue, GuMiho, Solar, Maru3BGE Stara Zagora 2025: Info & Preview27Code S RO12 Preview: GuMiho, Bunny, SHIN, ByuN3
Community News
Classic & herO RO8 Interviews: "I think it’s time to teach [Rogue] a lesson."0Rogue & GuMiho RO8 interviews: "Lifting that trophy would be a testament to all I’ve had to overcome over the years and how far I’ve come on this journey.2Code S RO8 Results + RO4 Bracket (2025 Season 2)11BGE Stara Zagora 2025 - Replay Pack2Weekly Cups (June 2-8): herO doubles down1
StarCraft 2
General
Classic & herO RO8 Interviews: "I think it’s time to teach [Rogue] a lesson." Rogue & GuMiho RO8 interviews: "Lifting that trophy would be a testament to all I’ve had to overcome over the years and how far I’ve come on this journey. The SCII GOAT: A statistical Evaluation Code S RO8 Results + RO4 Bracket (2025 Season 2) feardragon: Blizzards biggest blunder with SC was…
Tourneys
RSL: Revival, a new crowdfunded tournament series [GSL 2025] Code S: Season 2 - Ro8 - Group A [GSL 2025] Code S: Season 2 - Ro8 - Group B SOOPer7s Showmatches 2025 Sparkling Tuna Cup - Weekly Open Tournament
Strategy
[G] Darkgrid Layout Simple Questions Simple Answers [G] PvT Cheese: 13 Gate Proxy Robo
Custom Maps
[UMS] Zillion Zerglings
External Content
Mutation # 477 Slow and Steady Mutation # 476 Charnel House Mutation # 475 Hard Target Mutation # 474 Futile Resistance
Brood War
General
BGH Auto Balance -> http://bghmmr.eu/ Recent recommended BW games BW General Discussion FlaSh Witnesses SCV Pull Off the Impossible vs Shu StarCraft & BroodWar Campaign Speedrun Quest
Tourneys
[Megathread] Daily Proleagues [BSL 2v2] ProLeague Season 3 - Friday 21:00 CET Small VOD Thread 2.0 [BSL20] ProLeague Bracket Stage - Day 4
Strategy
I am doing this better than progamers do. [G] How to get started on ladder as a new Z player
Other Games
General Games
Path of Exile Nintendo Switch Thread Stormgate/Frost Giant Megathread Beyond All Reason What do you want from future RTS games?
Dota 2
Official 'what is Dota anymore' discussion
League of Legends
Heroes of the Storm
Simple Questions, Simple Answers Heroes of the Storm 2.0
Hearthstone
Heroes of StarCraft mini-set
TL Mafia
TL Mafia Community Thread Vanilla Mini Mafia
Community
General
Things Aren’t Peaceful in Palestine US Politics Mega-thread UK Politics Mega-thread Russo-Ukrainian War Thread Vape Nation Thread
Fan Clubs
Maru Fan Club Serral Fan Club
Media & Entertainment
Korean Music Discussion [Manga] One Piece
Sports
2024 - 2025 Football Thread NHL Playoffs 2024 TeamLiquid Health and Fitness Initiative For 2023 Formula 1 Discussion
World Cup 2022
Tech Support
Computer Build, Upgrade & Buying Resource Thread
TL Community
The Automated Ban List
Blogs
A Better Routine For Progame…
TrAiDoS
StarCraft improvement
iopq
Heero Yuy & the Tax…
KrillinFromwales
I was completely wrong ab…
jameswatts
Need Your Help/Advice
Glider
Trip to the Zoo
micronesia
Customize Sidebar...

Website Feedback

Closed Threads



Active: 27358 users

Replay Classification Project

Blogs > StRyKeR
Post a Reply
1 2 Next All
StRyKeR
Profile Blog Joined January 2006
United States1739 Posts
Last Edited: 2008-11-19 08:06:43
November 19 2008 07:05 GMT
#1
I've embarked on a personal project of mine involving a high-tech way to classify the players in a Starcraft replay.

Basically, imagine training a computer to learn to recognize who played in a replay with an accuracy of prediction better than (paladin)roMAD's.

I was inspired by romad's superhuman ability to recognize who played in a replay just by looking at hotkey signatures. I use his example as a benchmark for my machine.

Since hotkey usage is not that sophisticated to measure and compile, I figured that the high-tech machine learning tools that I've been learning in class this term could easily apply to this domain.

Early results are very promising.

The procedure is rather simple. I take a replay and grab 212 features and put it into a 212-dimensional vector. I note who played in the replay (for the training set, I need to know who played in the replay) and label the vector accordingly. I do this for a whole bunch of replays and I apply a machine learning algorithm that uses this set of vectors and labels to train itself.

Then I go about testing the machine by giving it examples it hasn't seen yet and seeing how accurate it is.

Currently, for all matchups except TvP, I can achieve about 90% accuracy. I have to add though, that as I add more players into the mix, the accuracy might go down. Right now I'm classifying among 10 players and 90% seems to be the resulting accuracy.

For some reason, the machine finds TvP hard to learn. I think this makes sense, since there's not many variations in strategy or unit composition in TvP. I don't play T (I play Z) so it's harder for me to know how to fix TvP learning.

Currently, I'm using Taiche's RepASM library to convert mass replays into mass 212-dimensional vectors. While his library is awesome, I'd like more features. Right now, his library cannot tell which unit is actually being clicked or saved into a hotkey. All I know is the unit's ID, which doesn't tell me anything. Having that information might bump up the accuracy to 95% or even higher.

Having said that, it's quite amazing how even stupid things like hotkey typing frequencies are consistent for one player's games and helps in training the machine. Right now, 10 dimensions (one for each hotkey) in the feature vector simply count the relative frequencies of the hotkeys being used. If someone prefers to use 1 a lot, it would be reflected as a high percentage in one of these 10 dimensions.

Some actions are habitual, even if the player does not recognize them.

I'm going to eventually make a lot of this automated, so that it can be of help to a lot of people wondering who's playing in a replay. This would prove useful especially for the iCCup Who's Who thread.

I'm setting up a server to do, but I'm still learning how to setup an automated system, since I'm using MATLAB to train the machine. I'm thinking that I could manually train the machine and provide an interface that could automatically classify replays. I might retrain the machine every week or so (would take some time retraining, especially with more and more replays).

One important thing is for me to get as many replays as possible, especially the pro replays. I have a ton of TSL replays, so right now I can classify a lot of foreigners with a good amount of accuracy.

One caveat is that I need a lot of replays for training. Preferably over 50. 20 might be okay. 10 is probably not enough.

I'll probably setup a website where you guys can upload replays with labels on them (remember, I need labels for training). The labels better be right though; otherwise, it would confuse the machine (wouldn't be catastrophic but still).

I'll issue a self-challenge. Send me a replay of unknown players and ask me to classify it. I'll post the results and analysis that results from feeding the replay into my machine.

Here's an example. I trained the machine using TSL replays and I just fed the replay of Mondragon playing MistrZZZ (recent replay, look in Replay section). Here are the results:

[image loading]


Okay, ng.stryker is me, and I never played in the TSL. But I thought I might include myself.

The confidence indicates how many replays I had to train that particular classifier. Basically, the more replays I used to train, the more likely it is to be correct.

Positive similarities indicate well, good similarity.
A value of 1 means that the classifier thinks the keystrokes are pretty damn similar.
A value of -1 means that the keystrokes are pretty damn different.

Just to let you know that these features need some work, here is a bad example.

[image loading]


Apparently, the machine thinks the player is very similar to David. However, the player is none other than Jaedong. In its defense, maybe if we had trained a Jaedong classifier, we'd get a Jaedong similarity score of 2 or something which would beat out David's.

I took a look at the replay, and there are some obvious difference between Jaedong and David that the 212 dimensions do not yet capture. That will be work for tomorrow.

Because I have very few replays of progamers, if you give me a progamer replay, the machine may try to fit the player as a foreigner and give strange predictions. However, we would still be able to observe how close the player's signature is compared to the foreigners, even though the player may not be one.

EDIT: I'm adding some interesting things I've found.

Interesting Things
* Some keystrokes are habitual and consistent for a player, even if the player does not know about them.
* Analyzing the entire replay is often worse than just looking at the first 10 minutes or so. Right now, I've capped the replay analysis to 9 minutes. I've filtered out replays that were too short (I think right now I only admit replays over 4 minutes long).
* APM is factored into the algorithm, but I don't really know how useful it is. It's pretty consistent for a player, so I suppose it helps. There are two dimensions for it -- one is the APM average for the first minute, the second is the average for the entire game. I figured that the first-minute average is very unlikely to change regardless of the opponent.
* I do an initial screen based on matchup. For example, if I get a replay that is ZvP, I won't be asking the machine to identify whether Nada played it (unless I trained some examples of Nada playing Zerg).
* Machine's ZvP keystrokes seem to be radically different from all the other players I've seen. Whenever I test Player X's replay, where X is not Machine, Machine consistently ranks last in similarity.

EDIT 2: I'm wondering if I should make this blog day-by-day and have a new entry every time or just keep updating this one. Okay. This entry is getting too long. I will update on new ones from now on.

****
Ars longa, vita brevis, principia aeturna.
Grobyc
Profile Blog Joined June 2008
Canada18410 Posts
November 19 2008 07:08 GMT
#2
Sounds really wicked. If you complete this me <3 you long time!
If you watch Godzilla backwards it's about a benevolent lizard who helps rebuild a city and then moonwalks into the ocean.
EtherealDeath
Profile Blog Joined July 2007
United States8366 Posts
November 19 2008 07:08 GMT
#3
Someone give him a replay of that olympic ranked player o.O
sixghost
Profile Blog Joined November 2007
United States2096 Posts
November 19 2008 07:11 GMT
#4
http://www.teamliquid.net/forum/viewmessage.php?topic_id=65607

There's a bunch of pro gamer replays in there if you havent already found that. I'm not sure if they are up yet though.

mG.sixghost @ iCCup || One ling, two ling, three ling, four... Camp four gas, then ultra-whore . -Saracen
StRyKeR
Profile Blog Joined January 2006
United States1739 Posts
Last Edited: 2008-11-19 07:12:18
November 19 2008 07:11 GMT
#5
On November 19 2008 16:11 lgdDante wrote:
http://www.teamliquid.net/forum/viewmessage.php?topic_id=65607

There's a bunch of pro gamer replays in there if you havent already found that. I'm not sure if they are up yet though.



Yea, I found that a few weeks ago except the links are all dead.

EDIT: Hmm, some are good.
Ars longa, vita brevis, principia aeturna.
sixghost
Profile Blog Joined November 2007
United States2096 Posts
November 19 2008 07:15 GMT
#6
Have you noticed if there's general minimum length the replay has to be for the accuracy to be near the average?
mG.sixghost @ iCCup || One ling, two ling, three ling, four... Camp four gas, then ultra-whore . -Saracen
OneOther
Profile Blog Joined August 2004
United States10774 Posts
November 19 2008 07:31 GMT
#7
Wow this would be so sick.
GHOSTCLAW
Profile Blog Joined February 2008
United States17042 Posts
November 19 2008 07:43 GMT
#8
If you can manage this that would be amazing. I'm not sure how reliable it would be, but it would be very interesting.
PhotographerLiquipedia. Drop me a pm if you've got questions/need help.
MasterReY
Profile Blog Joined August 2007
Germany2708 Posts
November 19 2008 07:44 GMT
#9
wow, i never thought a thing like that can work, but its nice to hear you already have about 90% accuracy.
PLease work hard to complete this program.
People will love you forever lol :D
gj
https://www.twitch.tv/MasterReY/ ~ Biggest Reach fan on TL.net (Don't even dare to mention LR now) ~ R.I.P Violet ~ Developer of SCRChart
TL+ Member
pachi
Profile Joined October 2006
Melbourne5338 Posts
November 19 2008 07:49 GMT
#10
If you want some replays for source material, Heres an old pack of several Babara Seasons collected by yakii, which should contain many korean reps most of which are probably pros by now.

http://rapidshare.com/files/151429614/babara.rar.html

(Also here is the replay pack from Itemmania, where everyone (koreans + draco) played under a number instead of their usual nick. http://rapidshare.com/files/151430307/Itemmania.rar.html)
Moderatorpachi fanclub http://goto.tl/6DI9 。◕‿◕。
magusmind
Profile Blog Joined May 2008
50 Posts
November 19 2008 07:57 GMT
#11
Wow, this idea brilliant.

Out of curiosity, what ML algorithm are you using? I assume it's a variant of perceptron. (You mention noisy data would be bad but not disastrous, and perceptron is somewhat resilient to noise) Also, how are you choosing the 212 features? I'm wondering because you couldn't have hand picked all of them.

As a side note, every replay (assuming 1v1) would actually be 2 examples, right? How are you counting your accuracy? (+1 for guessing both players correctly? +1 for guessing each correct player?) I'm also curious... does classifying one player correctly correlate with classifying the other one correctly?

Anyway, great idea. Keep us updated. I'm very interested in the results.
BanZu
Profile Blog Joined June 2008
United States3329 Posts
November 19 2008 08:02 GMT
#12
Oh snap this is some nifty stuff

I'd like to see how efficient you can get this thing to be
Sun Tzu once said, "Defiler becomes useless at the presences of a vessel."
StRyKeR
Profile Blog Joined January 2006
United States1739 Posts
Last Edited: 2008-11-19 08:38:19
November 19 2008 08:33 GMT
#13
On November 19 2008 16:49 pachi wrote:
If you want some replays for source material, Heres an old pack of several Babara Seasons collected by yakii, which should contain many korean reps most of which are probably pros by now.

http://rapidshare.com/files/151429614/babara.rar.html

(Also here is the replay pack from Itemmania, where everyone (koreans + draco) played under a number instead of their usual nick. http://rapidshare.com/files/151430307/Itemmania.rar.html)


Hehe, this gets fun.

I ran some replays from Itemmania through the machine.

Interestingly, I got the following results for a particular player.

itemmania_29
I found 3 replays, PvT, 2 replays, PvZ. They all point to Draco.

Result for one PvT:
[image loading]


I'm gonna guess that itemmania_29 is Draco.

Also, I got pretty weak results for itemmania_16, but is it mistrzzz?
2 replays, PvT
[image loading]

[image loading]
Ars longa, vita brevis, principia aeturna.
magusmind
Profile Blog Joined May 2008
50 Posts
November 19 2008 09:02 GMT
#14
Dude, I just noticed you can extend your current project to look at how similarly progamers play.

If you train using the multiclass perceptron (one weight vector for each player), at the end, each vector is pretty much a "characteristic" of how that player plays. Since these are just vectors in a high dimensional space, their dot product tells you how similar they are. In addition, if you run some sort of clustering algorithm on the vectors, you might see some interesting groupings.

This can let us ask some really interesting questions, eg:
Who is really boxer's protege? (Which SKT terran plays most similarly to boxer?)
Is there a clear distinction between how aggro zergs (eg: July) and macro zergs (eg: sAviOr) play? (Do all the aggro zergs and macro zergs cluster into distinct groups?)

Of course this requires a lot of training data that probably isn't available. But if you one day get your hands on them, I can totally imagine these types of questions being answered (As well as many others I can't think of right now).

+ Show Spoiler +
If you couldn't tell, I am well versed in the art of machine learning. If you have any questions or run into any difficulties I'd be glad to help.
AttackZerg
Profile Blog Joined January 2003
United States7454 Posts
November 19 2008 11:12 GMT
#15
This project has my stamp of approval.

This is a neat thing your doing, I'm interested to see how hard it is to fake your machine out .
NarutO
Profile Blog Joined December 2006
Germany18839 Posts
November 19 2008 11:39 GMT
#16
Thats pretty damn sick! Keep up the work, awesome dude! Really.
CommentatorPolt | MMA | Jjakji | BoxeR | NaDa | MVP | MKP ... truly inspiring.
roMAD
Profile Blog Joined April 2004
Russia2355 Posts
Last Edited: 2008-11-29 20:29:57
November 29 2008 20:29 GMT
#17
Wait. Itemmania_29 by any means is NOT Draco. It's not even close. It's beast[fOu]. And iteammania_16 is not Mistrzz at all. Your program is very inaccurate
MasterOfChaos
Profile Blog Joined April 2007
Germany2896 Posts
November 29 2008 22:13 GMT
#18
Impressive. What do you use to analyse the vector? And which observables do you use to create it?
LiquipediaOne eye to kill. Two eyes to live.
SpiritoftheTunA
Profile Blog Joined August 2006
United States20903 Posts
November 29 2008 22:27 GMT
#19
romad desperately avoids being outsourced~

jk, i'd trust romad over a program any day
posting on liquid sites in current year
LosingID8
Profile Blog Joined December 2006
CA10826 Posts
November 29 2008 22:40 GMT
#20
On November 30 2008 05:29 (paladin)roMAD wrote:
Wait. Itemmania_29 by any means is NOT Draco. It's not even close. It's beast[fOu]. And iteammania_16 is not Mistrzz at all. Your program is very inaccurate

true it's inaccurate right now, but thats probably because he doesn't have a good amount of reps for beast[fou] and other players.

once he gets a decent amount of reps for all known gamers then i feel that it would be pretty decent.


but i'd still want romad's opinion to be 100% sure
ModeratorResident K-POP Elitist
1 2 Next All
Please log in or register to reply.
Live Events Refresh
Korean StarCraft League
03:00
Week 76
SteadfastSC353
CranKy Ducklings120
EnkiAlexander 76
HKG_Chickenman16
Liquipedia
[ Submit Event ]
Live Streams
Refresh
StarCraft 2
SteadfastSC 353
Nina 190
StarCraft: Brood War
Larva 336
Leta 310
PianO 123
soO 18
Icarus 7
Dota 2
BabyKnight15
LuMiX1
Counter-Strike
Stewie2K305
Heroes of the Storm
Khaldor54
Other Games
summit1g8436
C9.Mang01365
WinterStarcraft338
ViBE214
RuFF_SC284
Fuzer 57
Trikslyr43
Organizations
Dota 2
PGL Dota 2 - Secondary Stream5205
Other Games
gamesdonequick1013
StarCraft 2
IntoTheiNu 7
Blizzard YouTube
StarCraft: Brood War
BSLTrovo
sctven
[ Show 16 non-featured ]
StarCraft 2
• practicex 42
• Sammyuel 7
• AfreecaTV YouTube
• sooper7s
• intothetv
• Kozan
• IndyKCrew
• LaughNgamezSOOP
• Migwel
StarCraft: Brood War
• Diggity2
• STPLYoutube
• ZZZeroYoutube
• BSLYoutube
League of Legends
• Doublelift5262
• Stunt309
Other Games
• Scarra1221
Upcoming Events
SOOP
4h 16m
sOs vs Percival
CranKy Ducklings
5h 16m
WardiTV Invitational
6h 16m
ByuN vs MaNa
MaxPax vs Solar
Reynor vs Creator
Gerald vs Spirit
Cheesadelphia
10h 16m
CSO Cup
12h 16m
BSL: ProLeague
13h 16m
Hawk vs UltrA
Sziky vs spx
TerrOr vs JDConan
GSL Code S
1d 3h
Rogue vs herO
Classic vs GuMiho
Sparkling Tuna Cup
1d 5h
BSL: ProLeague
1d 13h
Bonyth vs Dewalt
Cross vs Doodle
MadiNho vs Dragon
Replay Cast
1d 19h
[ Show More ]
Wardi Open
2 days
Replay Cast
2 days
Replay Cast
3 days
RSL Revival
3 days
Cure vs Percival
ByuN vs Spirit
RSL Revival
4 days
herO vs sOs
Zoun vs Clem
Replay Cast
4 days
The PondCast
5 days
RSL Revival
5 days
Serral vs SHIN
Solar vs Cham
Replay Cast
5 days
RSL Revival
6 days
Reynor vs Scarlett
ShoWTimE vs Classic
uThermal 2v2 Circuit
6 days
Liquipedia Results

Completed

CSL Season 17: Qualifier 2
BGE Stara Zagora 2025
Heroes 10 EU

Ongoing

JPL Season 2
BSL 2v2 Season 3
BSL Season 20
KCM Race Survival 2025 Season 2
NPSL S3
Rose Open S1
CSL 17: 2025 SUMMER
2025 GSL S2
Murky Cup #2
BLAST.tv Austin Major 2025
ESL Impact League Season 7
IEM Dallas 2025
PGL Astana 2025
Asian Champions League '25
BLAST Rivals Spring 2025
MESA Nomadic Masters
CCT Season 2 Global Finals
IEM Melbourne 2025
YaLLa Compass Qatar 2025
PGL Bucharest 2025

Upcoming

Copa Latinoamericana 4
CSLPRO Last Chance 2025
CSLPRO Chat StarLAN 3
K-Championship
SEL Season 2 Championship
Esports World Cup 2025
HSC XXVII
Championship of Russia 2025
BLAST Open Fall 2025
Esports World Cup 2025
BLAST Bounty Fall 2025
BLAST Bounty Fall Qual
IEM Cologne 2025
FISSURE Playground #1
TLPD

1. ByuN
2. TY
3. Dark
4. Solar
5. Stats
6. Nerchio
7. sOs
8. soO
9. INnoVation
10. Elazer
1. Rain
2. Flash
3. EffOrt
4. Last
5. Bisu
6. Soulkey
7. Mini
8. Sharp
Sidebar Settings...

Advertising | Privacy Policy | Terms Of Use | Contact Us

Original banner artwork: Jim Warren
The contents of this webpage are copyright © 2025 TLnet. All Rights Reserved.