This topic is about me trying to make Starcraft 2 playable without using a mouse or keyboard. I am doing this using an eyetracker, a microphone and the opensource audio recognition program I am developing at https://github.com/chaosparrot/parrot.py. The series is about doing the Heart of the Swarm campaign without using any of the traditional tools.
Demo of build 0.8.0
Latest episode ( Mission 10 on Zerus - Hard difficulty )
The challenge I set out for myself is finishing the Heart of the Swarm campaign with increasing difficulty settings. Further limitations are: I cannot use the mouse or keyboard during a mission, I must rely on the eyetracker and the audio recognition that I have built to control the inputs.
I also cannot develop the software to gain advantages that other players do not have ( build hotkey macros, do inputs faster than the 30hz of the keyboard, automated mouse movement, read out game information ). That means it should abide by the Blizzard End User License Agreement. I want to develop accessibility features so that I can play SC2, I have no desire whatsoever to create software that can be construed as cheating. Relevant area of the agreement: + Show Spoiler +
License Limitations. Blizzard may suspend or revoke your license to use the Platform, or parts, components and/or single features thereof, if you violate, or assist others in violating, the license limitations set forth below. You agree that you will not, in whole or in part or under any circumstances, do the following:
Cheating: Create, use, offer, promote, advertise, make available and/or distribute the following or assist therein:
cheats; i.e. methods not expressly authorized by Blizzard, influencing and/or facilitating the gameplay, including exploits of any in-game bugs, and thereby granting you and/or any other user an advantage over other players not using such methods;
bots; i.e. any code and/or software, not expressly authorized by Blizzard, that allows the automated control of a Game, or any other feature of the Platform, e.g. the automated control of a character in a Game;
hacks; i.e. accessing or modifying the software of the Platform in any manner not expressly authorized by Blizzard; and/or
any code and/or software, not expressly authorized by Blizzard, that can be used in connection with the Platform and/or any component or feature thereof which changes and/or facilitates the gameplay or other functionality;
Brief history
About two years ago I started getting problems with my arms and hands. During that time I started looking for ways to mitigate the aches by reaching out for software to help me out. Obviously playing SC2 was out of the question considering the amount of keystrokes and mouse movements involved.
I started fooling around with machine learning at the same time, doing simple recognition tasks. One day I had an idea to make browsing more easy on the hands by doing audio recognition, and then binding mouse clicks and scroll actions to them to let my hands rest after a days work.
After a while, I figured I could try the same with games requiring few inputs to play, like Heroes of the Storm. I had moderate success with it and eventually managed to beat the Elite AI. I thought about using it to go on the ladder, but my interest in Heroes waned after the collapse of the pro scene.
That's when my interest in SC2 reignited, and I had the idea to see if it was even possible to make an APM heavy RTS game playable. So I set a goal for myself and figured I would document my way through the development by making youtube videos. That's where we are now.
Current program specifications ( v0.8.0 ) - Use eyetracker to move the mouse around - Recognize and perform an action in roughly 60 miliseconds
Sound/keybindings - Hissing to left-mouse drag ( selecting units ) - Tongue clicking to right-mouse click ( and build units / buildings ) - Whistling to attack-move ( patrol, hold position and stop bound to areas of the screen when whistling ) - Oh sound to hold the control key ( for unit selection ) - Sh sound to hold the shift key ( for unit deselection and adding to control groups ) - ü + looking at top left to Select Kerrigan - ü + looking at the center area to Jump between bases - ü + looking at the command card to Follow the selection - ü + looking at the bottom left to Jump to Alert - iy + looking somewhere in a 3 x 3 grid on the screen to press a number from 1 till 9 for control groups - ah + looking somewhere in a 4 x 3 grid on the screen to press a letter from q till v for unit creation - Oah sound to press Q ( first ability + creep spread ) - Ae sound to press W ( second ability ) - F sound to Burrow - Finger snap to activate speech recognition ( for GL HF and GGs )
This is really cool! I test accessibility software frequently as part of my job but would never imagine someone making a hands free starcraft 2 actually work. Super impressive!
For what its worth, I think I'd enjoy the video more if it was just the full unedited gameplay with sound turned off so I could hear your audio cues more clearly.
On June 24 2019 22:28 imJealous wrote: This is really cool! I test accessibility software frequently as part of my job but would never imagine someone making a hands free starcraft 2 actually work. Super impressive!
For what its worth, I think I'd enjoy the video more if it was just the full unedited gameplay with sound turned off so I could hear your audio cues more clearly.
Thanks for the kind words! I thought about what kind of format I wanted to use for this series, and I kind of just settled on an explanatory challenge one like the ones I found in the OSRS community. I might upload an unedited video of me playing vs the AI some time.
On June 25 2019 01:02 BisuDagger wrote: You should message looknohands. He would be a great person to collaborate with since he is a quadriplegic who has been playing SC2 for many years and used to stream a bit too. Here's a thread of his https://tl.net/forum/sponsored/318498-looknohands-fundraiser-announcement
I'll keep that in mind! I'll probably improve the program first before I contact him. I still want to bring down the response times and add more elements of play to the controls. If there are other folks out there who want to mess around with the program however, I did write an installation and setup tutorial for it on the github page.
I'm going to have a short video up somewhere in the weekend to show off the 0.4.0 build ( which happens on Mission 14 on Brutal difficulty ). Considering these videos take a long time to make, and I would rather just work on improving the controls, I think I'll put the series on hold for now. Won't stop working on this though! Still want to make this thing ready for use on ladder.
- Uses neural nets to provide the recognition part - Now runs at 40 hz instead of 20ish hz - Added a repeat delay of 0.5s to make 'holding down' a key work similarly to a keyboard - Added red and blue dot near cursor to show if CTRL or SHIFT are held down ( rendered over the game screen when the game is in windowed mode - No game files/information needs to be read out or modified to do this )
I'll probably fool around with some machine learning algorithms after this, and maybe make it 60 hz ( I found a paper that suggests a 15 to 35ms audio frame for phoneme recognition, which would put it around 60 hz ).
But I'll need to get better at using it first, to see where it needs the most improvement.
Great stuff man, do some volunteering at a disability charity and get to muck around and tweak all sorts of accessibility gadgets, it can be a pain in the ass to get even basic stuff doing what you want, never mind playing a game I can barely play with two working hands and a mouse and keyboard!
Commenting more to bump this for others to see than anything, can’t exactly add anything exciting or interesting
So I´ve tinkered around a bit more with pytorch and stuff and managed to get a new build working. I practiced around with it for a little while before heading out to do my first placement match.
Current build ( V0.9 ) has the following changes - Replaced sklearn neural nets with pytorch neural nets trained for longer. This increased the accuracy and the amount of sounds I can train on to around 28, probably more if I add more training date. - Now runs at 60 hz instead of 40hz ( using 30ms recordings instead of 50ms ) - Added a rapid fire hotkey which caps out at 10hz, used primarily for ravager shots - The hold down keys are capped at 30hz instead of 60hz because that's how my regular keyboard works
I've done all my placement matches so far, and I think I'll make a video about that as well. To be continued
Edit - I just did some more laddering with it and achieved gold :D So happy
EPT
![[image loading]](https://i.imgur.com/ZT4gNky.png)
