• Log InLog In
  • Register
Liquid`
Team Liquid Liquipedia
EDT 03:40
CEST 09:40
KST 16:40
  • Home
  • Forum
  • Calendar
  • Streams
  • Liquipedia
  • Features
  • Store
  • EPT
  • TL+
  • StarCraft 2
  • Brood War
  • Smash
  • Heroes
  • Counter-Strike
  • Overwatch
  • Liquibet
  • Fantasy StarCraft
  • TLPD
  • StarCraft 2
  • Brood War
  • Blogs
Forum Sidebar
Events/Features
News
Featured News
[ASL19] Finals Recap: Standing Tall9HomeStory Cup 27 - Info & Preview18Classic wins Code S Season 2 (2025)16Code S RO4 & Finals Preview: herO, Rogue, Classic, GuMiho0TL Team Map Contest #5: Presented by Monster Energy6
Community News
Weekly Cups (June 30 - July 6): Classic Doubles0[BSL20] Non-Korean Championship 4x BSL + 4x China3Flash Announces Hiatus From ASL63Weekly Cups (June 23-29): Reynor in world title form?13FEL Cracov 2025 (July 27) - $8000 live event22
StarCraft 2
General
Weekly Cups (June 30 - July 6): Classic Doubles Program: SC2 / XSplit / OBS Scene Switcher The SCII GOAT: A statistical Evaluation Statistics for vetoed/disliked maps Weekly Cups (June 23-29): Reynor in world title form?
Tourneys
RSL: Revival, a new crowdfunded tournament series FEL Cracov 2025 (July 27) - $8000 live event Sparkling Tuna Cup - Weekly Open Tournament WardiTV Mondays Korean Starcraft League Week 77
Strategy
How did i lose this ZvP, whats the proper response Simple Questions Simple Answers
Custom Maps
[UMS] Zillion Zerglings
External Content
Mutation # 481 Fear and Lava Mutation # 480 Moths to the Flame Mutation # 479 Worn Out Welcome Mutation # 478 Instant Karma
Brood War
General
Player “Jedi” cheat on CSL SC uni coach streams logging into betting site Flash Announces Hiatus From ASL BW General Discussion Practice Partners (Official)
Tourneys
CSL Xiamen International Invitational [BSL20] Non-Korean Championship 4x BSL + 4x China The Casual Games of the Week Thread [BSL20] Grand Finals - Sunday 20:00 CET
Strategy
Simple Questions, Simple Answers I am doing this better than progamers do.
Other Games
General Games
Stormgate/Frost Giant Megathread Path of Exile Nintendo Switch Thread What do you want from future RTS games? Beyond All Reason
Dota 2
Official 'what is Dota anymore' discussion
League of Legends
Heroes of the Storm
Simple Questions, Simple Answers Heroes of the Storm 2.0
Hearthstone
Heroes of StarCraft mini-set
TL Mafia
TL Mafia Community Thread Vanilla Mini Mafia
Community
General
Stop Killing Games - European Citizens Initiative US Politics Mega-thread Summer Games Done Quick 2024! Summer Games Done Quick 2025! Russo-Ukrainian War Thread
Fan Clubs
SKT1 Classic Fan Club! Maru Fan Club
Media & Entertainment
Anime Discussion Thread [Manga] One Piece [\m/] Heavy Metal Thread
Sports
2024 - 2025 Football Thread Formula 1 Discussion NBA General Discussion TeamLiquid Health and Fitness Initiative For 2023 NHL Playoffs 2024
World Cup 2022
Tech Support
Computer Build, Upgrade & Buying Resource Thread
TL Community
Blogs
Culture Clash in Video Games…
TrAiDoS
from making sc maps to makin…
Husyelt
Blog #2
tankgirl
StarCraft improvement
iopq
Trip to the Zoo
micronesia
Customize Sidebar...

Website Feedback

Closed Threads



Active: 676 users

HandsFreeSC FAQ

Blogs > feardragon
Post a Reply
feardragon
Profile Blog Joined October 2010
United States970 Posts
Last Edited: 2020-01-23 17:04:29
January 16 2020 06:01 GMT
#1
Hello!

My name is feardragon. When I'm not trying to be a good commentator, I like wasting my time on coding projects that are based on stupid and useless concepts but are fun or funny.

The latest project I imagined is what I call HandsFreeSC. I've been getting a lot of questions about this so I wanted to create one single page that answers most of the common questions regarding it.



So without further ado:

What is HandsFreeSC?
The idea behind this is, as the name suggests, create a way where someone can play StarCraft 2 without using their hands! Instead, it will use a combination of your voice as a "commander" to issue orders and a Tobii Eyetracker(physical hardware) which will replace your mouse for mouse movement.

So you made this for accessibility reasons?
Actually no. Though this is where almost everyone's mind goes to, I actually did not start this project for accessibility reasons. In reality, this is going to be somewhat unwieldy and hard to use because we can click/use our keyboard a LOT faster than you can say what you want to have done. Rather, if I wanted this to be truly a viable option, I would do what this incredibly smart fellow did and make actions single syllable things so you could speed up the execution of things.

Instead what I want to do is fulfill the "commander fantasy" and use something closer to English to issue commands. This is a pet project so to some degree, it's going still going to be specific keywords but it will still at least appear as something close to normal commands you'd tell someone.

What language did you do this in?
I did this in python. If it was a super serious project, I would have done it in something more efficient but this is a for fun project for me so I did it in python.

Is it open source? Where is the github repo?
Currently, I don't have any plans to make the github repo public. The code is open-source per say in that I code a LOT of it on stream and all of the code is public but this is just a fun personal project for me. Making a repo public opens it to a different level of scrutiny and I'm not really interested in that or other people contributing to the repo since I have specific personal goals with the project others might not share. If someone would like to start their own project and copy the code/ideas from my streams they are more than welcome to.

How did you do it
I am primarily using a python library called speech_recognition that let's me call to various speech recognition libraries and turn what I say into text. Once I have the text, I do my own parsing of the text then use another python library called pynput that let's me simulate keyboard/mouse presses to trigger actions that I determined were desired via the parsed speech to text. This does not track any form of game state so it does not know what units you have selected, what hotkey your high templars are on, etc.

Where can I see the progress of it?
I occasionally stream my development of it over on my twitch channel but I usually also upload the vods from those streams to my youtube. Feel free to find what you're looking for at those two places.

When/where can I try this out?
Right now I have no intention to publicly release this unless it gets to a very very solid state. There is a massive difference in requirements for developing something for only my pc + me being the only user that has to put up with bugs or failures in the code, versus developing a public product that other people will try on lots of setups. If there's something I've realized it's how many people are ready to complain about is how upset they are that their free product isn't working how they want it to. So sorry, I don't have any intention to release this. Maybe at some point in the future but for now I don't have much interest in it. If someone wanted to pay me to do it I might consider it as a business venture but even that I'm not sure about because the nature of this is dependent on some somewhat iffy stuff(speech recognition being accurate). If it gets robust enough, I might release it to folks who want it for accessibility related reasons just as an option but it is far enough down the line that I wouldn't want to promise anyone that.

What do you plan to do with it?
The goal right now is to make something that is good enough to ladder with(on a brand new account) and get gold league with. I plan to stream the whole thing and make it some fun and slightly meme'y content.

Is this against Terms of Service to use?
I did reach out to Blizzard about it and I will say up front I was not given an official "ok" for it but was also told "we're not saying no" either. Effectively, this is very unlikely to be considered an issue to them since it is inherently playing *worse* rather than giving an advantage but they are reserving the right to shut this down at any time.


Did I not get your question? Feel free to ask below and I'll add it. Thanks and hope you enjoy what comes from all this.

****
Ok Starcraft 2 Commentator
Harris1st
Profile Blog Joined May 2010
Germany6890 Posts
January 17 2020 09:12 GMT
#2
That sounds interesting Commander Ravi

Will check out some vods when I'm not at work
Go Serral! GG EZ for Ence. Flashbang dance FTW
Chaosvuistje
Profile Joined April 2010
Netherlands2581 Posts
January 17 2020 10:58 GMT
#3
Hey man, have been enjoying the streams about your project. The misinterpertations of the speech recognition ( and having to begrudgedly repeat yourself, yet keep the tone neutral because otherwise the recognition won't work again ) are especially entertaining for me.

I was wondering if you ever did get feedback back from Blizzard wether this kind of system is allowed on the ladder. It's been a long while since I've been able to beat someone on the ladder, and I really want to use my own tool to terrorize silver league.

Best of luck on the project!
feardragon
Profile Blog Joined October 2010
United States970 Posts
January 17 2020 23:10 GMT
#4
On January 17 2020 19:58 Chaosvuistje wrote:
Hey man, have been enjoying the streams about your project. The misinterpertations of the speech recognition ( and having to begrudgedly repeat yourself, yet keep the tone neutral because otherwise the recognition won't work again ) are especially entertaining for me.

I was wondering if you ever did get feedback back from Blizzard wether this kind of system is allowed on the ladder. It's been a long while since I've been able to beat someone on the ladder, and I really want to use my own tool to terrorize silver league.

Best of luck on the project!

Thanks! Glad you've been enjoying it. I will also say that sometimes I've just forgotten some inconsistencies in the exact wording I'm checking for(since I haven't gotten to the point where I want to spend time making it more dynamic). For example, in the current build as of typing this, move command is "move here" but attack move is "attack

I did reach out to Blizzard about it and I will say up front I was not given an official "ok" for it but was also told "we're not saying no" either. Effectively, this is very unlikely to be considered an issue to them since it is inherently playing *worse* rather than giving an advantage but they are reserving the right to shut this down at any time.

Added the answer to the OP! Thanks!
Ok Starcraft 2 Commentator
BisuDagger
Profile Blog Joined October 2009
Bisutopia19229 Posts
Last Edited: 2020-01-20 17:17:43
January 20 2020 11:01 GMT
#5
Have you reached out to looknohands? He was a sc2 streamer paraplegic who had a similar setup. It would be cool if you collaborated with him.

https://tl.net/forum/starcraft-2/329164-looknohands-fundraiser-results
ModeratorFormer Afreeca Starleague Caster: http://afreeca.tv/ASL2ENG2
Chef
Profile Blog Joined August 2005
10810 Posts
January 20 2020 17:52 GMT
#6
I think it would be most interesting to have an AI play 90% of the game but you give voice commands to affect it. Trying to replace the mouse with eye tracking is a novelty and perhaps something you could generalize to other projects, but it doesn't sound like fun and because it requires special hardware it means you pretty much can't reach out to other people for testing or share your project very widely.

It could be interesting to go even further down the road of making the game purely about strategy, without giving up the real time aspect of it. You probably need to label each base, maybe A1 for main, B1 for opponents main, B2 for opponents natural, etc. Focus on things that are easy to say. Then you can say things like "army B1" for attack move, "retreat" for move home. It seems interesting. I'm sure you've already thought of things like this.
LEGEND!! LEGEND!!
feardragon
Profile Blog Joined October 2010
United States970 Posts
Last Edited: 2020-01-20 20:25:24
January 20 2020 20:24 GMT
#7
On January 20 2020 20:01 BisuDagger wrote:
Have you reached out to looknohands? He was a sc2 streamer paraplegic who had a similar setup. It would be cool if you collaborated with him.

https://tl.net/forum/starcraft-2/329164-looknohands-fundraiser-results

I know him but currently am not looking into accessibility stuff for this right now. Maybe at some point in the future but this is pretty far away from being usable by others.

On January 21 2020 02:52 Chef wrote:
I think it would be most interesting to have an AI play 90% of the game but you give voice commands to affect it. Trying to replace the mouse with eye tracking is a novelty and perhaps something you could generalize to other projects, but it doesn't sound like fun and because it requires special hardware it means you pretty much can't reach out to other people for testing or share your project very widely.

It could be interesting to go even further down the road of making the game purely about strategy, without giving up the real time aspect of it. You probably need to label each base, maybe A1 for main, B1 for opponents main, B2 for opponents natural, etc. Focus on things that are easy to say. Then you can say things like "army B1" for attack move, "retreat" for move home. It seems interesting. I'm sure you've already thought of things like this.

Unfortunately, reading the game state is breaking Terms of Service so that is likely never going to be my plan. You could make something like that with the API they have for making your own AI but then it is only ever useable against other AI which is a lot less fun to me and not the goal of this project.

The mouse tracking with eyes is actually available with other existing software and as stated in the OP, I'm actually not very interested in making this widely and extremely available publicly since it's a LOT more work to develop something that is useable by lots of people with lots of hardware/software setups versus just myself. Especially since nobody is paying me to do this but I know that people can be surprisingly demanding about the quality of free stuff.
Ok Starcraft 2 Commentator
WombaT
Profile Blog Joined May 2010
Northern Ireland25045 Posts
January 25 2020 01:38 GMT
#8
Great work man, really interesting project.
'You'll always be the cuddly marsupial of my heart, despite the inherent flaws of your ancestry' - Squat
sneakyfox
Profile Joined January 2017
8216 Posts
January 27 2020 22:48 GMT
#9
Lol, great video. And really cool project too, good luck with it.
"I saw what sneakyfox wrote on TL.net and it made me furious" - PartinG
Please log in or register to reply.
Live Events Refresh
Next event in 3h 20m
[ Submit Event ]
Live Streams
Refresh
StarCraft: Brood War
Mong 205
Leta 187
Zeus 168
Mind 86
Sharp 39
Shine 23
yabsab 13
Bale 7
Dota 2
XaKoH 565
Counter-Strike
shoxiejesuss314
Super Smash Bros
Mew2King175
Other Games
Stewie2K508
SortOf104
NeuroSwarm92
Happy81
ProTech43
Pyrionflax34
Organizations
Other Games
gamesdonequick30276
BasetradeTV25
StarCraft 2
Blizzard YouTube
StarCraft: Brood War
BSLTrovo
sctven
[ Show 14 non-featured ]
StarCraft 2
• Berry_CruncH365
• AfreecaTV YouTube
• intothetv
• Kozan
• IndyKCrew
• LaughNgamezSOOP
• Migwel
• sooper7s
StarCraft: Brood War
• BSLYoutube
• STPLYoutube
• ZZZeroYoutube
Dota 2
• lizZardDota2226
League of Legends
• Rush1461
• HappyZerGling179
Upcoming Events
Wardi Open
3h 20m
Replay Cast
16h 20m
Sparkling Tuna Cup
1d 2h
WardiTV European League
1d 8h
MaNa vs sebesdes
Mixu vs Fjant
ByuN vs HeRoMaRinE
ShoWTimE vs goblin
Gerald vs Babymarine
Krystianer vs YoungYakov
PiGosaur Monday
1d 16h
The PondCast
2 days
WardiTV European League
2 days
Jumy vs NightPhoenix
Percival vs Nicoract
ArT vs HiGhDrA
MaxPax vs Harstem
Scarlett vs Shameless
SKillous vs uThermal
uThermal 2v2 Circuit
2 days
Replay Cast
2 days
RSL Revival
3 days
ByuN vs SHIN
Clem vs Reynor
[ Show More ]
Replay Cast
3 days
RSL Revival
4 days
Classic vs Cure
FEL
4 days
RSL Revival
5 days
FEL
5 days
FEL
5 days
Sparkling Tuna Cup
6 days
RSL Revival
6 days
FEL
6 days
Liquipedia Results

Completed

BSL Season 20
HSC XXVII
Heroes 10 EU

Ongoing

JPL Season 2
BSL 2v2 Season 3
Acropolis #3
KCM Race Survival 2025 Season 2
CSL 17: 2025 SUMMER
Copa Latinoamericana 4
Jiahua Invitational
Championship of Russia 2025
RSL Revival: Season 1
Murky Cup #2
BLAST.tv Austin Major 2025
ESL Impact League Season 7
IEM Dallas 2025
PGL Astana 2025
Asian Champions League '25
BLAST Rivals Spring 2025
MESA Nomadic Masters
CCT Season 2 Global Finals
IEM Melbourne 2025

Upcoming

2025 ACS Season 2: Qualifier
CSLPRO Last Chance 2025
CSL Xiamen Invitational
2025 ACS Season 2
CSLPRO Chat StarLAN 3
K-Championship
uThermal 2v2 Main Event
SEL Season 2 Championship
FEL Cracov 2025
Esports World Cup 2025
StarSeries Fall 2025
FISSURE Playground #2
BLAST Open Fall 2025
BLAST Open Fall Qual
Esports World Cup 2025
BLAST Bounty Fall 2025
BLAST Bounty Fall Qual
IEM Cologne 2025
FISSURE Playground #1
TLPD

1. ByuN
2. TY
3. Dark
4. Solar
5. Stats
6. Nerchio
7. sOs
8. soO
9. INnoVation
10. Elazer
1. Rain
2. Flash
3. EffOrt
4. Last
5. Bisu
6. Soulkey
7. Mini
8. Sharp
Sidebar Settings...

Advertising | Privacy Policy | Terms Of Use | Contact Us

Original banner artwork: Jim Warren
The contents of this webpage are copyright © 2025 TLnet. All Rights Reserved.