• Log InLog In
  • Register
Liquid`
Team Liquid Liquipedia
EDT 08:22
CEST 14:22
KST 21:22
  • Home
  • Forum
  • Calendar
  • Streams
  • Liquipedia
  • Features
  • Store
  • EPT
  • TL+
  • StarCraft 2
  • Brood War
  • Smash
  • Heroes
  • Counter-Strike
  • Overwatch
  • Liquibet
  • Fantasy StarCraft
  • TLPD
  • StarCraft 2
  • Brood War
  • Blogs
Forum Sidebar
Events/Features
News
Featured News
RSL Season 1 - Final Week6[ASL19] Finals Recap: Standing Tall12HomeStory Cup 27 - Info & Preview18Classic wins Code S Season 2 (2025)16Code S RO4 & Finals Preview: herO, Rogue, Classic, GuMiho0
Community News
Esports World Cup 2025 - Brackets Revealed10Weekly Cups (July 7-13): Classic continues to roll4Team TLMC #5 - Submission extension3Firefly given lifetime ban by ESIC following match-fixing investigation17$25,000 Streamerzone StarCraft Pro Series announced7
StarCraft 2
General
The GOAT ranking of GOAT rankings RSL Revival patreon money discussion thread Who will win EWC 2025? Weekly Cups (July 7-13): Classic continues to roll Esports World Cup 2025 - Brackets Revealed
Tourneys
RSL: Revival, a new crowdfunded tournament series FEL Cracov 2025 (July 27) - $8000 live event $5,100+ SEL Season 2 Championship (SC: Evo) WardiTV Mondays Sparkling Tuna Cup - Weekly Open Tournament
Strategy
How did i lose this ZvP, whats the proper response Simple Questions Simple Answers
Custom Maps
External Content
Mutation # 482 Wheel of Misfortune Mutation # 481 Fear and Lava Mutation # 480 Moths to the Flame Mutation # 479 Worn Out Welcome
Brood War
General
BW General Discussion Flash Announces (and Retracts) Hiatus From ASL BGH Auto Balance -> http://bghmmr.eu/ Starcraft in widescreen A cwal.gg Extension - Easily keep track of anyone
Tourneys
[Megathread] Daily Proleagues Cosmonarchy Pro Showmatches CSL Xiamen International Invitational [BSL20] Non-Korean Championship 4x BSL + 4x China
Strategy
Simple Questions, Simple Answers I am doing this better than progamers do.
Other Games
General Games
Path of Exile Nintendo Switch Thread Stormgate/Frost Giant Megathread CCLP - Command & Conquer League Project The PlayStation 5
Dota 2
Official 'what is Dota anymore' discussion
League of Legends
Heroes of the Storm
Simple Questions, Simple Answers Heroes of the Storm 2.0
Hearthstone
Heroes of StarCraft mini-set
TL Mafia
TL Mafia Community Thread Vanilla Mini Mafia
Community
General
US Politics Mega-thread Russo-Ukrainian War Thread Stop Killing Games - European Citizens Initiative Summer Games Done Quick 2025! Things Aren’t Peaceful in Palestine
Fan Clubs
SKT1 Classic Fan Club! Maru Fan Club
Media & Entertainment
[Manga] One Piece Movie Discussion! Anime Discussion Thread [\m/] Heavy Metal Thread
Sports
Formula 1 Discussion TeamLiquid Health and Fitness Initiative For 2023 2024 - 2025 Football Thread NBA General Discussion NHL Playoffs 2024
World Cup 2022
Tech Support
Computer Build, Upgrade & Buying Resource Thread
TL Community
The Automated Ban List
Blogs
Men Take Risks, Women Win Ga…
TrAiDoS
momentary artworks from des…
tankgirl
from making sc maps to makin…
Husyelt
StarCraft improvement
iopq
Trip to the Zoo
micronesia
Customize Sidebar...

Website Feedback

Closed Threads



Active: 604 users

Making Starcraft 2 Hands-free

Forum Index > Community Content
Post a Reply
Chaosvuistje
Profile Joined April 2010
Netherlands2581 Posts
Last Edited: 2019-07-17 19:53:53
June 23 2019 17:10 GMT
#1
This topic is about me trying to make Starcraft 2 playable without using a mouse or keyboard. I am doing this using an eyetracker, a microphone and the opensource audio recognition program I am developing at https://github.com/chaosparrot/parrot.py. The series is about doing the Heart of the Swarm campaign without using any of the traditional tools.

Demo of build 0.8.0


Latest episode ( Mission 10 on Zerus - Hard difficulty )


Youtube playlist of all the episodes

The challenge I set out for myself is finishing the Heart of the Swarm campaign with increasing difficulty settings. Further limitations are: I cannot use the mouse or keyboard during a mission, I must rely on the eyetracker and the audio recognition that I have built to control the inputs.

I also cannot develop the software to gain advantages that other players do not have ( build hotkey macros, do inputs faster than the 30hz of the keyboard, automated mouse movement, read out game information ). That means it should abide by the Blizzard End User License Agreement. I want to develop accessibility features so that I can play SC2, I have no desire whatsoever to create software that can be construed as cheating. Relevant area of the agreement:
+ Show Spoiler +

License Limitations. Blizzard may suspend or revoke your license to use the Platform, or parts, components and/or single features thereof, if you violate, or assist others in violating, the license limitations set forth below. You agree that you will not, in whole or in part or under any circumstances, do the following:

    Cheating: Create, use, offer, promote, advertise, make available and/or distribute the following or assist therein:

  • cheats; i.e. methods not expressly authorized by Blizzard, influencing and/or facilitating the gameplay, including exploits of any in-game bugs, and thereby granting you and/or any other user an advantage over other players not using such methods;

  • bots; i.e. any code and/or software, not expressly authorized by Blizzard, that allows the automated control of a Game, or any other feature of the Platform, e.g. the automated control of a character in a Game;

  • hacks; i.e. accessing or modifying the software of the Platform in any manner not expressly authorized by Blizzard; and/or

  • any code and/or software, not expressly authorized by Blizzard, that can be used in connection with the Platform and/or any component or feature thereof which changes and/or facilitates the gameplay or other functionality;



Brief history

About two years ago I started getting problems with my arms and hands. During that time I started looking for ways to mitigate the aches by reaching out for software to help me out. Obviously playing SC2 was out of the question considering the amount of keystrokes and mouse movements involved.

I started fooling around with machine learning at the same time, doing simple recognition tasks. One day I had an idea to make browsing more easy on the hands by doing audio recognition, and then binding mouse clicks and scroll actions to them to let my hands rest after a days work.

After a while, I figured I could try the same with games requiring few inputs to play, like Heroes of the Storm. I had moderate success with it and eventually managed to beat the Elite AI. I thought about using it to go on the ladder, but my interest in Heroes waned after the collapse of the pro scene.

That's when my interest in SC2 reignited, and I had the idea to see if it was even possible to make an APM heavy RTS game playable. So I set a goal for myself and figured I would document my way through the development by making youtube videos. That's where we are now.

Current program specifications ( v0.8.0 )
- Use eyetracker to move the mouse around
- Recognize and perform an action in roughly 60 miliseconds

Sound/keybindings
- Hissing to left-mouse drag ( selecting units )
- Tongue clicking to right-mouse click ( and build units / buildings )
- Whistling to attack-move ( patrol, hold position and stop bound to areas of the screen when whistling )
- Oh sound to hold the control key ( for unit selection )
- Sh sound to hold the shift key ( for unit deselection and adding to control groups )
- ü + looking at top left to Select Kerrigan
- ü + looking at the center area to Jump between bases
- ü + looking at the command card to Follow the selection
- ü + looking at the bottom left to Jump to Alert
- iy + looking somewhere in a 3 x 3 grid on the screen to press a number from 1 till 9 for control groups
- ah + looking somewhere in a 4 x 3 grid on the screen to press a letter from q till v for unit creation
- Oah sound to press Q ( first ability + creep spread )
- Ae sound to press W ( second ability )
- F sound to Burrow
- Finger snap to activate speech recognition ( for GL HF and GGs )
imJealous
Profile Joined July 2010
United States1382 Posts
June 24 2019 13:28 GMT
#2
This is really cool! I test accessibility software frequently as part of my job but would never imagine someone making a hands free starcraft 2 actually work. Super impressive!

For what its worth, I think I'd enjoy the video more if it was just the full unedited gameplay with sound turned off so I could hear your audio cues more clearly.
... In life very little goes right. "Right" meaning the way one expected and the way one wanted it. One has no right to want or expect anything.
BisuDagger
Profile Blog Joined October 2009
Bisutopia19230 Posts
June 24 2019 16:02 GMT
#3
You should message looknohands. He would be a great person to collaborate with since he is a quadriplegic who has been playing SC2 for many years and used to stream a bit too. Here's a thread of his https://tl.net/forum/sponsored/318498-looknohands-fundraiser-announcement
ModeratorFormer Afreeca Starleague Caster: http://afreeca.tv/ASL2ENG2
Chaosvuistje
Profile Joined April 2010
Netherlands2581 Posts
June 24 2019 18:15 GMT
#4
On June 24 2019 22:28 imJealous wrote:
This is really cool! I test accessibility software frequently as part of my job but would never imagine someone making a hands free starcraft 2 actually work. Super impressive!

For what its worth, I think I'd enjoy the video more if it was just the full unedited gameplay with sound turned off so I could hear your audio cues more clearly.


Thanks for the kind words! I thought about what kind of format I wanted to use for this series, and I kind of just settled on an explanatory challenge one like the ones I found in the OSRS community. I might upload an unedited video of me playing vs the AI some time.

On June 25 2019 01:02 BisuDagger wrote:
You should message looknohands. He would be a great person to collaborate with since he is a quadriplegic who has been playing SC2 for many years and used to stream a bit too. Here's a thread of his https://tl.net/forum/sponsored/318498-looknohands-fundraiser-announcement


I'll keep that in mind! I'll probably improve the program first before I contact him. I still want to bring down the response times and add more elements of play to the controls. If there are other folks out there who want to mess around with the program however, I did write an installation and setup tutorial for it on the github page.

By the way - I have a new episode released

Chaosvuistje
Profile Joined April 2010
Netherlands2581 Posts
July 01 2019 18:09 GMT
#5
Added another episode - First hard mission of the challenge

Chaosvuistje
Profile Joined April 2010
Netherlands2581 Posts
July 08 2019 17:14 GMT
#6
Another episode is up

Chaosvuistje
Profile Joined April 2010
Netherlands2581 Posts
Last Edited: 2019-07-17 19:54:47
July 12 2019 19:03 GMT
#7
Added another episode - control groups added in this one

https://www.youtube.com/watch?v=fLjgnkS1AM4

I'm going to have a short video up somewhere in the weekend to show off the 0.4.0 build ( which happens on Mission 14 on Brutal difficulty ). Considering these videos take a long time to make, and I would rather just work on improving the controls, I think I'll put the series on hold for now. Won't stop working on this though! Still want to make this thing ready for use on ladder.

Chaosvuistje
Profile Joined April 2010
Netherlands2581 Posts
July 17 2019 17:09 GMT
#8
Made a video of the latest build ( 0.8.0 ) beating the Elite AI

https://www.youtube.com/watch?v=lhQzrZ3PrtU

- Uses neural nets to provide the recognition part
- Now runs at 40 hz instead of 20ish hz
- Added a repeat delay of 0.5s to make 'holding down' a key work similarly to a keyboard
- Added red and blue dot near cursor to show if CTRL or SHIFT are held down ( rendered over the game screen when the game is in windowed mode - No game files/information needs to be read out or modified to do this )

I'll probably fool around with some machine learning algorithms after this, and maybe make it 60 hz ( I found a paper that suggests a 15 to 35ms audio frame for phoneme recognition, which would put it around 60 hz ).

But I'll need to get better at using it first, to see where it needs the most improvement.
LoneYoShi
Profile Blog Joined June 2014
France1348 Posts
July 20 2019 09:36 GMT
#9
Holy hell, I missed this thread until now but I have to say I'm really, REALLY impressed !
That's a huge amount of work you put in, and a noble goal.

Huge props to you man !
WombaT
Profile Blog Joined May 2010
Northern Ireland25082 Posts
Last Edited: 2019-07-20 23:25:55
July 20 2019 23:25 GMT
#10
Great stuff man, do some volunteering at a disability charity and get to muck around and tweak all sorts of accessibility gadgets, it can be a pain in the ass to get even basic stuff doing what you want, never mind playing a game I can barely play with two working hands and a mouse and keyboard!

Commenting more to bump this for others to see than anything, can’t exactly add anything exciting or interesting
'You'll always be the cuddly marsupial of my heart, despite the inherent flaws of your ancestry' - Squat
Chaosvuistje
Profile Joined April 2010
Netherlands2581 Posts
Last Edited: 2020-01-26 17:17:30
January 25 2020 09:04 GMT
#11
So I´ve tinkered around a bit more with pytorch and stuff and managed to get a new build working. I practiced around with it for a little while before heading out to do my first placement match.



Current build ( V0.9 ) has the following changes
- Replaced sklearn neural nets with pytorch neural nets trained for longer. This increased the accuracy and the amount of sounds I can train on to around 28, probably more if I add more training date.
- Now runs at 60 hz instead of 40hz ( using 30ms recordings instead of 50ms )
- Added a rapid fire hotkey which caps out at 10hz, used primarily for ravager shots
- The hold down keys are capped at 30hz instead of 60hz because that's how my regular keyboard works

I've done all my placement matches so far, and I think I'll make a video about that as well. To be continued

Edit - I just did some more laddering with it and achieved gold :D So happy

[image loading]
imJealous
Profile Joined July 2010
United States1382 Posts
January 25 2020 18:20 GMT
#12
Still really impressive! Congrats!
... In life very little goes right. "Right" meaning the way one expected and the way one wanted it. One has no right to want or expect anything.
Chaosvuistje
Profile Joined April 2010
Netherlands2581 Posts
Last Edited: 2020-02-22 16:45:06
February 22 2020 16:41 GMT
#13
I know I said I would upload some placement match content, but I didn't feel like the games were as fun.
I've practiced some more with it and currently I'm tumbling around in platinum league.

Here's a match of me playing vs a platinum protoss:



I can't seem to update the OP anymore with the edit threshold being reached, so I will just continue posting videos in replies.
Chaosvuistje
Profile Joined April 2010
Netherlands2581 Posts
March 22 2020 10:49 GMT
#14
I've practiced some more and ended up winning an automated tournament



Hunting for diamond league now!
Chaosvuistje
Profile Joined April 2010
Netherlands2581 Posts
April 28 2020 07:09 GMT
#15
Added a video of me playing in zergling-baneling into roach plays. Still practicing to get that league promotion

FFW_Rude
Profile Blog Joined November 2010
France10201 Posts
May 03 2020 10:41 GMT
#16
it's just like you are talking zergs which makes it even more funnier
#1 KT Rolster fanboy. KT BEST KT ! Hail to KT playoffs Zergs ! Unofficial french translator for SlayerS_`Boxer` biography "Crazy as me".
[UoN]Sentinel
Profile Blog Joined November 2009
United States11320 Posts
Last Edited: 2020-05-10 01:11:44
May 10 2020 01:11 GMT
#17
Incredible.

EDIT: Nvm misunderstood the commands
Нас зовет дух отцов, память старых бойцов, дух Москвы и твердыня Полтавы
Chaosvuistje
Profile Joined April 2010
Netherlands2581 Posts
June 21 2020 15:42 GMT
#18
Added another video, this time ZvT in diamond league ( 3.3k mmr )

BirdBird
Profile Joined May 2019
34 Posts
June 28 2020 19:34 GMT
#19
Amazing
WombaT
Profile Blog Joined May 2010
Northern Ireland25082 Posts
June 28 2020 21:13 GMT
#20
Really inspiring stuff
'You'll always be the cuddly marsupial of my heart, despite the inherent flaws of your ancestry' - Squat
Please log in or register to reply.
Live Events Refresh
Replay Cast
10:00
Galaxy Open Cup Season 1
CranKy Ducklings182
Liquipedia
[ Submit Event ]
Live Streams
Refresh
StarCraft 2
Harstem 413
StarCraft: Brood War
Britney 27034
Rain 5591
BeSt 1540
Zeus 558
Larva 493
EffOrt 453
Mini 356
firebathero 348
Light 266
Pusan 215
[ Show more ]
PianO 207
JulyZerg 102
Mind 96
Rush 95
Stork 60
sSak 49
Aegong 49
Shine 31
NaDa 25
scan(afreeca) 23
Icarus 22
Movie 19
yabsab 15
Noble 12
Bale 6
SilentControl 6
Shinee 6
sas.Sziky 2
ivOry 1
Dota 2
qojqva3396
XcaliburYe337
canceldota139
League of Legends
Dendi1085
Counter-Strike
x6flipin626
sgares559
byalli320
PGG 49
Super Smash Bros
Mew2King115
amsayoshi18
Other Games
B2W.Neo1870
singsing1774
DeMusliM577
crisheroes353
XaKoH 306
Fuzer 247
Pyrionflax182
Lowko158
markeloff56
Trikslyr33
ArmadaUGS24
Organizations
Other Games
gamesdonequick3576
StarCraft 2
Blizzard YouTube
StarCraft: Brood War
BSLTrovo
sctven
[ Show 13 non-featured ]
StarCraft 2
• Berry_CruncH329
• AfreecaTV YouTube
• intothetv
• Kozan
• IndyKCrew
• LaughNgamezSOOP
• Migwel
• sooper7s
StarCraft: Brood War
• BSLYoutube
• STPLYoutube
• ZZZeroYoutube
League of Legends
• Nemesis1459
• Jankos907
Upcoming Events
WardiTV European League
3h 38m
ShoWTimE vs sebesdes
Percival vs NightPhoenix
Shameless vs Nicoract
Krystianer vs Scarlett
ByuN vs uThermal
Harstem vs HeRoMaRinE
PiGosaur Monday
11h 38m
uThermal 2v2 Circuit
1d 3h
Replay Cast
1d 11h
The PondCast
1d 21h
WardiTV European League
2 days
Replay Cast
2 days
Epic.LAN
2 days
CranKy Ducklings
3 days
Epic.LAN
3 days
[ Show More ]
CSO Contender
4 days
BSL20 Non-Korean Champi…
4 days
Bonyth vs Sziky
Dewalt vs Hawk
Hawk vs QiaoGege
Sziky vs Dewalt
Mihu vs Bonyth
Zhanhun vs QiaoGege
QiaoGege vs Fengzi
Sparkling Tuna Cup
4 days
Online Event
5 days
BSL20 Non-Korean Champi…
5 days
Bonyth vs Zhanhun
Dewalt vs Mihu
Hawk vs Sziky
Sziky vs QiaoGege
Mihu vs Hawk
Zhanhun vs Dewalt
Fengzi vs Bonyth
Esports World Cup
6 days
ByuN vs Astrea
Lambo vs HeRoMaRinE
Clem vs TBD
Solar vs Zoun
SHIN vs Reynor
Maru vs TriGGeR
herO vs Lancer
Cure vs ShoWTimE
Liquipedia Results

Completed

2025 ACS Season 2: Qualifier
RSL Revival: Season 1
Murky Cup #2

Ongoing

JPL Season 2
BSL 2v2 Season 3
CSL 17: 2025 SUMMER
Copa Latinoamericana 4
Jiahua Invitational
BSL20 Non-Korean Championship
Championship of Russia 2025
FISSURE Playground #1
BLAST.tv Austin Major 2025
ESL Impact League Season 7
IEM Dallas 2025
PGL Astana 2025
Asian Champions League '25
BLAST Rivals Spring 2025
MESA Nomadic Masters

Upcoming

CSL Xiamen Invitational
CSL Xiamen Invitational: ShowMatche
2025 ACS Season 2
CSLPRO Last Chance 2025
CSLPRO Chat StarLAN 3
BSL Season 21
K-Championship
RSL Revival: Season 2
SEL Season 2 Championship
uThermal 2v2 Main Event
FEL Cracov 2025
Esports World Cup 2025
Underdog Cup #2
ESL Pro League S22
StarSeries Fall 2025
FISSURE Playground #2
BLAST Open Fall 2025
BLAST Open Fall Qual
Esports World Cup 2025
BLAST Bounty Fall 2025
BLAST Bounty Fall Qual
IEM Cologne 2025
TLPD

1. ByuN
2. TY
3. Dark
4. Solar
5. Stats
6. Nerchio
7. sOs
8. soO
9. INnoVation
10. Elazer
1. Rain
2. Flash
3. EffOrt
4. Last
5. Bisu
6. Soulkey
7. Mini
8. Sharp
Sidebar Settings...

Advertising | Privacy Policy | Terms Of Use | Contact Us

Original banner artwork: Jim Warren
The contents of this webpage are copyright © 2025 TLnet. All Rights Reserved.