EPTMy name is feardragon. When I'm not trying to be a good commentator, I like wasting my time on coding projects that are based on stupid and useless concepts but are fun or funny.
The latest project I imagined is what I call HandsFreeSC. I've been getting a lot of questions about this so I wanted to create one single page that answers most of the common questions regarding it.
So without further ado:
What is HandsFreeSC?
The idea behind this is, as the name suggests, create a way where someone can play StarCraft 2 without using their hands! Instead, it will use a combination of your voice as a "commander" to issue orders and a Tobii Eyetracker(physical hardware) which will replace your mouse for mouse movement.
So you made this for accessibility reasons?
Actually no. Though this is where almost everyone's mind goes to, I actually did not start this project for accessibility reasons. In reality, this is going to be somewhat unwieldy and hard to use because we can click/use our keyboard a LOT faster than you can say what you want to have done. Rather, if I wanted this to be truly a viable option, I would do what this incredibly smart fellow did and make actions single syllable things so you could speed up the execution of things.
Instead what I want to do is fulfill the "commander fantasy" and use something closer to English to issue commands. This is a pet project so to some degree, it's going still going to be specific keywords but it will still at least appear as something close to normal commands you'd tell someone.
What language did you do this in?
I did this in python. If it was a super serious project, I would have done it in something more efficient but this is a for fun project for me so I did it in python.
Is it open source? Where is the github repo?
Currently, I don't have any plans to make the github repo public. The code is open-source per say in that I code a LOT of it on stream and all of the code is public but this is just a fun personal project for me. Making a repo public opens it to a different level of scrutiny and I'm not really interested in that or other people contributing to the repo since I have specific personal goals with the project others might not share. If someone would like to start their own project and copy the code/ideas from my streams they are more than welcome to.
How did you do it
I am primarily using a python library called speech_recognition that let's me call to various speech recognition libraries and turn what I say into text. Once I have the text, I do my own parsing of the text then use another python library called pynput that let's me simulate keyboard/mouse presses to trigger actions that I determined were desired via the parsed speech to text. This does not track any form of game state so it does not know what units you have selected, what hotkey your high templars are on, etc.
Where can I see the progress of it?
I occasionally stream my development of it over on my twitch channel but I usually also upload the vods from those streams to my youtube. Feel free to find what you're looking for at those two places.
When/where can I try this out?
Right now I have no intention to publicly release this unless it gets to a very very solid state. There is a massive difference in requirements for developing something for only my pc + me being the only user that has to put up with bugs or failures in the code, versus developing a public product that other people will try on lots of setups. If there's something I've realized it's how many people are ready to complain about is how upset they are that their free product isn't working how they want it to. So sorry, I don't have any intention to release this. Maybe at some point in the future but for now I don't have much interest in it. If someone wanted to pay me to do it I might consider it as a business venture but even that I'm not sure about because the nature of this is dependent on some somewhat iffy stuff(speech recognition being accurate). If it gets robust enough, I might release it to folks who want it for accessibility related reasons just as an option but it is far enough down the line that I wouldn't want to promise anyone that.
What do you plan to do with it?
The goal right now is to make something that is good enough to ladder with(on a brand new account) and get gold league with. I plan to stream the whole thing and make it some fun and slightly meme'y content.
Is this against Terms of Service to use?
I did reach out to Blizzard about it and I will say up front I was not given an official "ok" for it but was also told "we're not saying no" either. Effectively, this is very unlikely to be considered an issue to them since it is inherently playing *worse* rather than giving an advantage but they are reserving the right to shut this down at any time.
Did I not get your question? Feel free to ask below and I'll add it. Thanks and hope you enjoy what comes from all this.
