• Log InLog In
  • Register
Liquid`
Team Liquid Liquipedia
EDT 18:18
CEST 00:18
KST 07:18
  • Home
  • Forum
  • Calendar
  • Streams
  • Liquipedia
  • Features
  • Store
  • EPT
  • TL+
  • StarCraft 2
  • Brood War
  • Smash
  • Heroes
  • Counter-Strike
  • Overwatch
  • Liquibet
  • Fantasy StarCraft
  • TLPD
  • StarCraft 2
  • Brood War
  • Blogs
Forum Sidebar
Events/Features
News
Featured News
[ASL19] Finals Recap: Standing Tall10HomeStory Cup 27 - Info & Preview18Classic wins Code S Season 2 (2025)16Code S RO4 & Finals Preview: herO, Rogue, Classic, GuMiho0TL Team Map Contest #5: Presented by Monster Energy6
Community News
Firefly given lifetime ban by ESIC following match-fixing investigation11$25,000 Streamerzone StarCraft Pro Series announced6Weekly Cups (June 30 - July 6): Classic Doubles6[BSL20] Non-Korean Championship 4x BSL + 4x China9Flash Announces Hiatus From ASL66
StarCraft 2
General
The SCII GOAT: A statistical Evaluation Firefly given lifetime ban by ESIC following match-fixing investigation TL Team Map Contest #4: Winners Weekly Cups (June 30 - July 6): Classic Doubles The GOAT ranking of GOAT rankings
Tourneys
$25,000 Streamerzone StarCraft Pro Series announced FEL Cracov 2025 (July 27) - $8000 live event Sparkling Tuna Cup - Weekly Open Tournament RSL: Revival, a new crowdfunded tournament series WardiTV Mondays
Strategy
How did i lose this ZvP, whats the proper response Simple Questions Simple Answers
Custom Maps
[UMS] Zillion Zerglings
External Content
Mutation # 481 Fear and Lava Mutation # 480 Moths to the Flame Mutation # 479 Worn Out Welcome Mutation # 478 Instant Karma
Brood War
General
BGH Auto Balance -> http://bghmmr.eu/ ASL20 Preliminary Maps [G] Progamer Settings [ASL19] Finals Recap: Standing Tall SC uni coach streams logging into betting site
Tourneys
[Megathread] Daily Proleagues [BSL20] Non-Korean Championship 4x BSL + 4x China [BSL20] Grand Finals - Sunday 20:00 CET CSL Xiamen International Invitational
Strategy
Simple Questions, Simple Answers I am doing this better than progamers do.
Other Games
General Games
Nintendo Switch Thread Stormgate/Frost Giant Megathread Path of Exile What do you want from future RTS games? Beyond All Reason
Dota 2
Official 'what is Dota anymore' discussion
League of Legends
Positive Thoughts on Setting Up a Dual-Caliber FX
Heroes of the Storm
Simple Questions, Simple Answers Heroes of the Storm 2.0
Hearthstone
Heroes of StarCraft mini-set
TL Mafia
TL Mafia Community Thread Vanilla Mini Mafia
Community
General
Things Aren’t Peaceful in Palestine US Politics Mega-thread Russo-Ukrainian War Thread Summer Games Done Quick 2025! Stop Killing Games - European Citizens Initiative
Fan Clubs
SKT1 Classic Fan Club! Maru Fan Club
Media & Entertainment
Anime Discussion Thread [Manga] One Piece [\m/] Heavy Metal Thread
Sports
Formula 1 Discussion 2024 - 2025 Football Thread NBA General Discussion TeamLiquid Health and Fitness Initiative For 2023 NHL Playoffs 2024
World Cup 2022
Tech Support
Computer Build, Upgrade & Buying Resource Thread
TL Community
The Automated Ban List
Blogs
momentary artworks from des…
tankgirl
Culture Clash in Video Games…
TrAiDoS
from making sc maps to makin…
Husyelt
StarCraft improvement
iopq
Customize Sidebar...

Website Feedback

Closed Threads



Active: 604 users

Making Starcraft 2 Hands-free

Forum Index > Community Content
Post a Reply
Chaosvuistje
Profile Joined April 2010
Netherlands2581 Posts
Last Edited: 2019-07-17 19:53:53
June 23 2019 17:10 GMT
#1
This topic is about me trying to make Starcraft 2 playable without using a mouse or keyboard. I am doing this using an eyetracker, a microphone and the opensource audio recognition program I am developing at https://github.com/chaosparrot/parrot.py. The series is about doing the Heart of the Swarm campaign without using any of the traditional tools.

Demo of build 0.8.0


Latest episode ( Mission 10 on Zerus - Hard difficulty )


Youtube playlist of all the episodes

The challenge I set out for myself is finishing the Heart of the Swarm campaign with increasing difficulty settings. Further limitations are: I cannot use the mouse or keyboard during a mission, I must rely on the eyetracker and the audio recognition that I have built to control the inputs.

I also cannot develop the software to gain advantages that other players do not have ( build hotkey macros, do inputs faster than the 30hz of the keyboard, automated mouse movement, read out game information ). That means it should abide by the Blizzard End User License Agreement. I want to develop accessibility features so that I can play SC2, I have no desire whatsoever to create software that can be construed as cheating. Relevant area of the agreement:
+ Show Spoiler +

License Limitations. Blizzard may suspend or revoke your license to use the Platform, or parts, components and/or single features thereof, if you violate, or assist others in violating, the license limitations set forth below. You agree that you will not, in whole or in part or under any circumstances, do the following:

    Cheating: Create, use, offer, promote, advertise, make available and/or distribute the following or assist therein:

  • cheats; i.e. methods not expressly authorized by Blizzard, influencing and/or facilitating the gameplay, including exploits of any in-game bugs, and thereby granting you and/or any other user an advantage over other players not using such methods;

  • bots; i.e. any code and/or software, not expressly authorized by Blizzard, that allows the automated control of a Game, or any other feature of the Platform, e.g. the automated control of a character in a Game;

  • hacks; i.e. accessing or modifying the software of the Platform in any manner not expressly authorized by Blizzard; and/or

  • any code and/or software, not expressly authorized by Blizzard, that can be used in connection with the Platform and/or any component or feature thereof which changes and/or facilitates the gameplay or other functionality;



Brief history

About two years ago I started getting problems with my arms and hands. During that time I started looking for ways to mitigate the aches by reaching out for software to help me out. Obviously playing SC2 was out of the question considering the amount of keystrokes and mouse movements involved.

I started fooling around with machine learning at the same time, doing simple recognition tasks. One day I had an idea to make browsing more easy on the hands by doing audio recognition, and then binding mouse clicks and scroll actions to them to let my hands rest after a days work.

After a while, I figured I could try the same with games requiring few inputs to play, like Heroes of the Storm. I had moderate success with it and eventually managed to beat the Elite AI. I thought about using it to go on the ladder, but my interest in Heroes waned after the collapse of the pro scene.

That's when my interest in SC2 reignited, and I had the idea to see if it was even possible to make an APM heavy RTS game playable. So I set a goal for myself and figured I would document my way through the development by making youtube videos. That's where we are now.

Current program specifications ( v0.8.0 )
- Use eyetracker to move the mouse around
- Recognize and perform an action in roughly 60 miliseconds

Sound/keybindings
- Hissing to left-mouse drag ( selecting units )
- Tongue clicking to right-mouse click ( and build units / buildings )
- Whistling to attack-move ( patrol, hold position and stop bound to areas of the screen when whistling )
- Oh sound to hold the control key ( for unit selection )
- Sh sound to hold the shift key ( for unit deselection and adding to control groups )
- ü + looking at top left to Select Kerrigan
- ü + looking at the center area to Jump between bases
- ü + looking at the command card to Follow the selection
- ü + looking at the bottom left to Jump to Alert
- iy + looking somewhere in a 3 x 3 grid on the screen to press a number from 1 till 9 for control groups
- ah + looking somewhere in a 4 x 3 grid on the screen to press a letter from q till v for unit creation
- Oah sound to press Q ( first ability + creep spread )
- Ae sound to press W ( second ability )
- F sound to Burrow
- Finger snap to activate speech recognition ( for GL HF and GGs )
imJealous
Profile Joined July 2010
United States1382 Posts
June 24 2019 13:28 GMT
#2
This is really cool! I test accessibility software frequently as part of my job but would never imagine someone making a hands free starcraft 2 actually work. Super impressive!

For what its worth, I think I'd enjoy the video more if it was just the full unedited gameplay with sound turned off so I could hear your audio cues more clearly.
... In life very little goes right. "Right" meaning the way one expected and the way one wanted it. One has no right to want or expect anything.
BisuDagger
Profile Blog Joined October 2009
Bisutopia19230 Posts
June 24 2019 16:02 GMT
#3
You should message looknohands. He would be a great person to collaborate with since he is a quadriplegic who has been playing SC2 for many years and used to stream a bit too. Here's a thread of his https://tl.net/forum/sponsored/318498-looknohands-fundraiser-announcement
ModeratorFormer Afreeca Starleague Caster: http://afreeca.tv/ASL2ENG2
Chaosvuistje
Profile Joined April 2010
Netherlands2581 Posts
June 24 2019 18:15 GMT
#4
On June 24 2019 22:28 imJealous wrote:
This is really cool! I test accessibility software frequently as part of my job but would never imagine someone making a hands free starcraft 2 actually work. Super impressive!

For what its worth, I think I'd enjoy the video more if it was just the full unedited gameplay with sound turned off so I could hear your audio cues more clearly.


Thanks for the kind words! I thought about what kind of format I wanted to use for this series, and I kind of just settled on an explanatory challenge one like the ones I found in the OSRS community. I might upload an unedited video of me playing vs the AI some time.

On June 25 2019 01:02 BisuDagger wrote:
You should message looknohands. He would be a great person to collaborate with since he is a quadriplegic who has been playing SC2 for many years and used to stream a bit too. Here's a thread of his https://tl.net/forum/sponsored/318498-looknohands-fundraiser-announcement


I'll keep that in mind! I'll probably improve the program first before I contact him. I still want to bring down the response times and add more elements of play to the controls. If there are other folks out there who want to mess around with the program however, I did write an installation and setup tutorial for it on the github page.

By the way - I have a new episode released

Chaosvuistje
Profile Joined April 2010
Netherlands2581 Posts
July 01 2019 18:09 GMT
#5
Added another episode - First hard mission of the challenge

Chaosvuistje
Profile Joined April 2010
Netherlands2581 Posts
July 08 2019 17:14 GMT
#6
Another episode is up

Chaosvuistje
Profile Joined April 2010
Netherlands2581 Posts
Last Edited: 2019-07-17 19:54:47
July 12 2019 19:03 GMT
#7
Added another episode - control groups added in this one

https://www.youtube.com/watch?v=fLjgnkS1AM4

I'm going to have a short video up somewhere in the weekend to show off the 0.4.0 build ( which happens on Mission 14 on Brutal difficulty ). Considering these videos take a long time to make, and I would rather just work on improving the controls, I think I'll put the series on hold for now. Won't stop working on this though! Still want to make this thing ready for use on ladder.

Chaosvuistje
Profile Joined April 2010
Netherlands2581 Posts
July 17 2019 17:09 GMT
#8
Made a video of the latest build ( 0.8.0 ) beating the Elite AI

https://www.youtube.com/watch?v=lhQzrZ3PrtU

- Uses neural nets to provide the recognition part
- Now runs at 40 hz instead of 20ish hz
- Added a repeat delay of 0.5s to make 'holding down' a key work similarly to a keyboard
- Added red and blue dot near cursor to show if CTRL or SHIFT are held down ( rendered over the game screen when the game is in windowed mode - No game files/information needs to be read out or modified to do this )

I'll probably fool around with some machine learning algorithms after this, and maybe make it 60 hz ( I found a paper that suggests a 15 to 35ms audio frame for phoneme recognition, which would put it around 60 hz ).

But I'll need to get better at using it first, to see where it needs the most improvement.
LoneYoShi
Profile Blog Joined June 2014
France1348 Posts
July 20 2019 09:36 GMT
#9
Holy hell, I missed this thread until now but I have to say I'm really, REALLY impressed !
That's a huge amount of work you put in, and a noble goal.

Huge props to you man !
WombaT
Profile Blog Joined May 2010
Northern Ireland25054 Posts
Last Edited: 2019-07-20 23:25:55
July 20 2019 23:25 GMT
#10
Great stuff man, do some volunteering at a disability charity and get to muck around and tweak all sorts of accessibility gadgets, it can be a pain in the ass to get even basic stuff doing what you want, never mind playing a game I can barely play with two working hands and a mouse and keyboard!

Commenting more to bump this for others to see than anything, can’t exactly add anything exciting or interesting
'You'll always be the cuddly marsupial of my heart, despite the inherent flaws of your ancestry' - Squat
Chaosvuistje
Profile Joined April 2010
Netherlands2581 Posts
Last Edited: 2020-01-26 17:17:30
January 25 2020 09:04 GMT
#11
So I´ve tinkered around a bit more with pytorch and stuff and managed to get a new build working. I practiced around with it for a little while before heading out to do my first placement match.



Current build ( V0.9 ) has the following changes
- Replaced sklearn neural nets with pytorch neural nets trained for longer. This increased the accuracy and the amount of sounds I can train on to around 28, probably more if I add more training date.
- Now runs at 60 hz instead of 40hz ( using 30ms recordings instead of 50ms )
- Added a rapid fire hotkey which caps out at 10hz, used primarily for ravager shots
- The hold down keys are capped at 30hz instead of 60hz because that's how my regular keyboard works

I've done all my placement matches so far, and I think I'll make a video about that as well. To be continued

Edit - I just did some more laddering with it and achieved gold :D So happy

[image loading]
imJealous
Profile Joined July 2010
United States1382 Posts
January 25 2020 18:20 GMT
#12
Still really impressive! Congrats!
... In life very little goes right. "Right" meaning the way one expected and the way one wanted it. One has no right to want or expect anything.
Chaosvuistje
Profile Joined April 2010
Netherlands2581 Posts
Last Edited: 2020-02-22 16:45:06
February 22 2020 16:41 GMT
#13
I know I said I would upload some placement match content, but I didn't feel like the games were as fun.
I've practiced some more with it and currently I'm tumbling around in platinum league.

Here's a match of me playing vs a platinum protoss:



I can't seem to update the OP anymore with the edit threshold being reached, so I will just continue posting videos in replies.
Chaosvuistje
Profile Joined April 2010
Netherlands2581 Posts
March 22 2020 10:49 GMT
#14
I've practiced some more and ended up winning an automated tournament



Hunting for diamond league now!
Chaosvuistje
Profile Joined April 2010
Netherlands2581 Posts
April 28 2020 07:09 GMT
#15
Added a video of me playing in zergling-baneling into roach plays. Still practicing to get that league promotion

FFW_Rude
Profile Blog Joined November 2010
France10201 Posts
May 03 2020 10:41 GMT
#16
it's just like you are talking zergs which makes it even more funnier
#1 KT Rolster fanboy. KT BEST KT ! Hail to KT playoffs Zergs ! Unofficial french translator for SlayerS_`Boxer` biography "Crazy as me".
[UoN]Sentinel
Profile Blog Joined November 2009
United States11320 Posts
Last Edited: 2020-05-10 01:11:44
May 10 2020 01:11 GMT
#17
Incredible.

EDIT: Nvm misunderstood the commands
Нас зовет дух отцов, память старых бойцов, дух Москвы и твердыня Полтавы
Chaosvuistje
Profile Joined April 2010
Netherlands2581 Posts
June 21 2020 15:42 GMT
#18
Added another video, this time ZvT in diamond league ( 3.3k mmr )

BirdBird
Profile Joined May 2019
34 Posts
June 28 2020 19:34 GMT
#19
Amazing
WombaT
Profile Blog Joined May 2010
Northern Ireland25054 Posts
June 28 2020 21:13 GMT
#20
Really inspiring stuff
'You'll always be the cuddly marsupial of my heart, despite the inherent flaws of your ancestry' - Squat
Please log in or register to reply.
Live Events Refresh
Next event in 1h 42m
[ Submit Event ]
Live Streams
Refresh
StarCraft 2
ZombieGrub182
JuggernautJason135
ProTech71
StarCraft: Brood War
Artosis 275
NaDa 48
LuMiX 1
League of Legends
Grubby4079
Counter-Strike
Fnx 1501
fl0m1452
Stewie2K853
sgares97
Super Smash Bros
Liquid`Ken42
Heroes of the Storm
Khaldor145
Other Games
summit1g9723
C9.Mang0285
Sick45
Maynarde43
Organizations
Other Games
gamesdonequick49985
BasetradeTV100
StarCraft 2
Blizzard YouTube
StarCraft: Brood War
BSLTrovo
sctven
[ Show 19 non-featured ]
StarCraft 2
• HeavenSC 47
• davetesta29
• RyuSc2 9
• IndyKCrew
• sooper7s
• AfreecaTV YouTube
• Migwel
• intothetv
• LaughNgamezSOOP
• Kozan
StarCraft: Brood War
• Eskiya23 25
• STPLYoutube
• ZZZeroYoutube
• BSLYoutube
Dota 2
• masondota22370
League of Legends
• Jankos2222
• TFBlade958
Other Games
• imaqtpie1801
• Shiphtur505
Upcoming Events
Replay Cast
1h 42m
RSL Revival
11h 42m
ByuN vs SHIN
Clem vs Reynor
OSC
14h 42m
Replay Cast
1d 1h
RSL Revival
1d 11h
Classic vs Cure
FEL
1d 17h
OSC
1d 21h
RSL Revival
2 days
FEL
2 days
FEL
2 days
[ Show More ]
CSO Cup
2 days
BSL20 Non-Korean Champi…
2 days
Bonyth vs QiaoGege
Dewalt vs Fengzi
Hawk vs Zhanhun
Sziky vs Mihu
Mihu vs QiaoGege
Zhanhun vs Sziky
Fengzi vs Hawk
Sparkling Tuna Cup
3 days
RSL Revival
3 days
FEL
3 days
BSL20 Non-Korean Champi…
3 days
Bonyth vs Dewalt
QiaoGege vs Dewalt
Hawk vs Bonyth
Sziky vs Fengzi
Mihu vs Zhanhun
QiaoGege vs Zhanhun
Fengzi vs Mihu
Replay Cast
5 days
Liquipedia Results

Completed

Proleague 2025-07-07
HSC XXVII
Heroes 10 EU

Ongoing

JPL Season 2
BSL 2v2 Season 3
Acropolis #3
KCM Race Survival 2025 Season 2
CSL 17: 2025 SUMMER
Copa Latinoamericana 4
Jiahua Invitational
Championship of Russia 2025
RSL Revival: Season 1
Murky Cup #2
BLAST.tv Austin Major 2025
ESL Impact League Season 7
IEM Dallas 2025
PGL Astana 2025
Asian Champions League '25
BLAST Rivals Spring 2025
MESA Nomadic Masters
CCT Season 2 Global Finals
IEM Melbourne 2025

Upcoming

2025 ACS Season 2: Qualifier
CSLPRO Last Chance 2025
CSL Xiamen Invitational
CSL Xiamen Invitational: ShowMatche
2025 ACS Season 2
CSLPRO Chat StarLAN 3
K-Championship
uThermal 2v2 Main Event
SEL Season 2 Championship
FEL Cracov 2025
Esports World Cup 2025
Underdog Cup #2
StarSeries Fall 2025
FISSURE Playground #2
BLAST Open Fall 2025
BLAST Open Fall Qual
Esports World Cup 2025
BLAST Bounty Fall 2025
BLAST Bounty Fall Qual
IEM Cologne 2025
FISSURE Playground #1
TLPD

1. ByuN
2. TY
3. Dark
4. Solar
5. Stats
6. Nerchio
7. sOs
8. soO
9. INnoVation
10. Elazer
1. Rain
2. Flash
3. EffOrt
4. Last
5. Bisu
6. Soulkey
7. Mini
8. Sharp
Sidebar Settings...

Advertising | Privacy Policy | Terms Of Use | Contact Us

Original banner artwork: Jim Warren
The contents of this webpage are copyright © 2025 TLnet. All Rights Reserved.