The AIIDE 2010 StarCraft Competition has come to a close. The challenge given to competitors was to build the best performing bot for an immensely popular, commercial game. The competition consisted of four tournaments of varying complexity. This was the first year the competition was held and it turned out to be a success. Even though no prizes were offered, over twenty-eight teams participated in the competition.
The showcase game of the competition was a bot versus human match. In the exhibition match, =DoGo=, a World Cyber Games 2001 competitor played against the top ranking bot of the competition. The result was an exciting man versus machine match highlighting the state-of-the-art in RTS AI.
While the expert player was capable of defeating the top performing bots in the competition, the results are quite encouraging.
Tournament 1: Micromanagement The first tournament evaluated bots in unit micromanagement scenarios.The winner of this tournament was FreSCBot, which uses multi-agent finite-state machines. The runner-up was Sherbrooke, which also uses state machines. Complete tournament details are available here and the full results are posted here. The video below shows the final match between FreSCBot (red) and Sherbrooke (blue).
Tournament 2: Small-Scale Combat The second tournament evaluated bots in small-scale combat scenarios. The results of this competition mirrored the results of the first tournament: FreSCBot won and Sherbrooke was runner-up. Complete tournament details are available here and the full results are posted here. The video below shows the final match between FreSCBot (purple) and Sherbrooke (orange).
Tournament 3: Tech-Limited Game The third tournament tested bots in a tech-limited StarCraft environment which requires reasoning at strategic and tactical levels, but omits much of the complexity of the full version of StarCraft. The winner of this tournament was Mimic Bot, which attempted to mirror the opponent's strategy while also performing a gas steal and applying fallback strategies. The runner up was Botnik, which executes a Zealot rush strategy. Complete tournament details are available here and the full results are posted here. The video below shows the final match between MimicBot (red) andBotnik (teal).
Tournament 4: Full Gameplay The final tournament was a best-of-five, double-elimination tournament simulating a professional gaming competition. The winner of this tournament was Overmind, a Zerg bot that effectively scouted its opponents, interrupted their economy, and performed Mutalisk harassment until victory was ensured. The runner-up was Krasi0, a Terran bot with excellent defense and pushing capabilities. Complete tournament details are available here and the full results are posted here. The video below shows an exert from one of the games between Overmind (yellow) and Krasi0 (brown).
Highlight #1: Krasi0 versus Skynet During this minute of action between bots Krasi0 and Skynet, it was difficult to distinguish whether the game was being played by skilled human players or the StarCraft equivalent of Deep Blue.
Highlight #2: Berkeley's Mutalisk Evolution Contemporary StarCraft wisdom tells us that the best way to use mutalisks is to clump them. In human versus human battles, this makes it difficult to single out the weaker mutalisks, because the units are stacked on top of each other. However, UC Berkeley's team identified a flaw in this tactic; it reduces the damage output of each individual mutalisk, because not all mutalisks will fire when using this tactic. Instead, they employed a model in which mutalisk are always moving, maximizing damage output while simultaneously maximizing movement.
Conclusion The inaugural StarCraft AI Competition was a huge success! While no prizes were planned originally, Blizzard sponsored the competition and the winner of each tournament received a collectors edition of StarCraft 2 signed by the dev team. I would like to thank Blizzard, BWAPI, AAAI, TL, and all of the participants in making this event a reality. The next competition will be hosted by the University of Alberta at AIIDE 2011.
Update: VODs from the Competition I've recorded a few of the more interesting games from the StarCraft AI Competition. All replays from the competition are available and I encourage people to post additional videos and commentaries: http://eis.ucsc.edu/StarCraftAICompetition#Results
Here's one of the matches from the finals between Krasi0 and Overmind (Replay):
This video shows the initial match between =DoGo= and Krasi0, with DoGo playing as Protoss (Replay):
One of the highlights of the competition, Skynet versus Krasi0. This video shows the match-up on Heartbreak Ridge (Replay):
The semi-finals between Chronos and Overmind (Replay):
Edit: Post in this thread if you upload any videos!
WOW!!! FreSCBot in the Micro Tournament totally dominated the competition. 6 zealots left! It looked like it didn't even lose a mutalisk either but I couldn't really tell. I wish they had the units selected so we could see the health bars.
EDIT: LOL I just watched the small-scale tournament. Same bot(FreSCBot) lost a SINGLE marine out of 20. Embarrassing really.
What the teams managed to accomplish in terms of micro control is fairly impressive, but from what I've seen of the actual game replays they seem to have trouble with keeping decent macro.
This is a bit surprising considering that macro is often though of as what an AI would do best. This makes me wonder what it is that caused a lot of the AIs to run at 1k+ minerals for 5+ minutes.
It's rather funny how there actually are AI scripts (using Blizzards script language) that macro better but would probably loose to one of these bots because the AI script micro is horrendous.
Maybe if I have some time and can figure out how to get Overmind zerg bot setup I'll test how it fairs against one of my Terran anti-zerg scripts.
On October 14 2010 14:56 Equalizer wrote: What the teams managed to accomplish in terms of micro control is fairly impressive, but from what I've seen of the actual game replays they seem to have trouble with keeping decent macro.
This is a bit surprising considering that macro is often though of as what an AI would do best. This makes me wonder what it is that caused a lot of the AIs to run at 1k+ minerals for 5+ minutes.
It's rather funny how there actually are AI scripts (using Blizzards script language) that macro better but would probably loose to one of these bots because the AI script micro is horrendous.
Maybe if I have some time and can figure out how to get Overmind zerg bot setup I'll test how it fairs against one of my Terran anti-zerg scripts.
Overmind does have good macro. It's just we have to account for cheese that may or maynot show up, so we always make some super early sunken colonies that leads to starvation. Also, bots are bad at determine if an attack is life threatening or not, so a lot of times the defenses are being put up prematurely. That combined with our 2 hatch muta opening does not lend to a smooth early game macro. However, if you try to play a long game with Overmind it will actually take over the map while saturating each expansion with drones and, if it is smart enough not to suicide mutalisks, it can max 200/200 in reasonable time.
PvP matchups were unique, and fun. My bot, ZotBot, used one of two openings, each had a 100% win rate vs Protoss. I believe my ZotBot had 9 PvP matches.
It just occured to me that in a way, computers cheat when they micro, in that they take advantage of their more direct interface with the game in a way humans can't. Humans either have to locate a unit by sight and then select it by hand, or keep it hotkeyed to one of just 10 groups, while i bet the computer can just magically call up a unit anywhere on the map. I think for a computer to truly legitimately beat a human player, it'd have to get information from the game only by means of a camera facing the moniter, and operate the mouse with a robotic arm.
On October 15 2010 09:25 zobz wrote: It just occured to me that in a way, computers cheat when they micro, in that they take advantage of their more direct interface with the game in a way humans can't. Humans either have to locate a unit by sight and then select it by hand, or keep it hotkeyed to one of just 10 groups, while i bet the computer can just magically call up a unit anywhere on the map. I think for a computer to truly legitimately beat a human player, it'd have to get information from the game only by means of a camera facing the moniter, and operate the mouse with a robotic arm.
that's ridiculous. the point is whether it's an "AI" that can think well enough to play. which is 99% of the challenge and a ridiculously difficult problem.
you could take your argumentt a level further and say that the robotic arm has an advantage because it never gets tired, or is too precise, or whatever, so to be "fair" you have to emulate a human arm cell by cell.
On October 15 2010 09:25 zobz wrote: it'd have to get information from the game only by means of a camera facing the moniter, and operate the mouse with a robotic arm.
As far as video processing, there's already a pacman competition that requires this process.
On October 15 2010 09:25 zobz wrote: It just occured to me that in a way, computers cheat when they micro, in that they take advantage of their more direct interface with the game in a way humans can't. Humans either have to locate a unit by sight and then select it by hand, or keep it hotkeyed to one of just 10 groups, while i bet the computer can just magically call up a unit anywhere on the map. I think for a computer to truly legitimately beat a human player, it'd have to get information from the game only by means of a camera facing the moniter, and operate the mouse with a robotic arm.
that's ridiculous. the point is whether it's an "AI" that can think well enough to play. which is 99% of the challenge and a ridiculously difficult problem.
you could take your argumentt a level further and say that the robotic arm has an advantage because it never gets tired, or is too precise, or whatever, so to be "fair" you have to emulate a human arm cell by cell.
Of course rescripting to make a computer understand the flow of the game is a momentus challenge in itself ffs. But i'm sure you wouldn't claim that the physical aspect of starcraft is negligable. What i'm asking is, what do you think two robotic arms operating a mouse and keyboard at 20 000 apm would look like? Unimpressive? The implication is both that a robot would have a much harder time beating a human if it had to physically perform its apm, and that it would probably take a substantial amount of reprogramming to get it to use the minimap and hotkeys effectively, so that it didn't have to be impossibly fast. The thing is that an sc robot shouldn't exclusively be compared to a chess robot, but also to a soccer robot because it is a physical sport.
On October 15 2010 09:25 zobz wrote: It just occured to me that in a way, computers cheat when they micro, in that they take advantage of their more direct interface with the game in a way humans can't. Humans either have to locate a unit by sight and then select it by hand, or keep it hotkeyed to one of just 10 groups, while i bet the computer can just magically call up a unit anywhere on the map. I think for a computer to truly legitimately beat a human player, it'd have to get information from the game only by means of a camera facing the moniter, and operate the mouse with a robotic arm.
that's ridiculous. the point is whether it's an "AI" that can think well enough to play. which is 99% of the challenge and a ridiculously difficult problem.
you could take your argumentt a level further and say that the robotic arm has an advantage because it never gets tired, or is too precise, or whatever, so to be "fair" you have to emulate a human arm cell by cell.
Of course rescripting to make a computer understand the flow of the game is a momentus challenge in itself ffs. But i'm sure you wouldn't claim that the physical aspect of starcraft is negligable. What i'm asking is, what do you think two robotic arms operating a mouse and keyboard at 20 000 apm would look like? Unimpressive? The implication is both that a robot would have a much harder time beating a human if it had to physically perform its apm, and that it would probably take a substantial amount of reprogramming to get it to use the minimap and hotkeys effectively, so that it didn't have to be impossibly fast. The thing is that an sc robot shouldn't exclusively be compared to a chess robot, but also to a soccer robot because it is a physical sport.
again, you are missing the point. This is an AI competition. If you want a sensor and actuator competition (that's what you are describing) there are such that exist. But that's not the point of this competition. Your point is valid, as in, they are right, the computer do have advantages that it can query units across the map, but your point is irrelevant to the competition itself.
On October 15 2010 09:25 zobz wrote: It just occured to me that in a way, computers cheat when they micro, in that they take advantage of their more direct interface with the game in a way humans can't. Humans either have to locate a unit by sight and then select it by hand, or keep it hotkeyed to one of just 10 groups, while i bet the computer can just magically call up a unit anywhere on the map. I think for a computer to truly legitimately beat a human player, it'd have to get information from the game only by means of a camera facing the moniter, and operate the mouse with a robotic arm.
that's ridiculous. the point is whether it's an "AI" that can think well enough to play. which is 99% of the challenge and a ridiculously difficult problem.
you could take your argumentt a level further and say that the robotic arm has an advantage because it never gets tired, or is too precise, or whatever, so to be "fair" you have to emulate a human arm cell by cell.
Of course rescripting to make a computer understand the flow of the game is a momentus challenge in itself ffs. But i'm sure you wouldn't claim that the physical aspect of starcraft is negligable. What i'm asking is, what do you think two robotic arms operating a mouse and keyboard at 20 000 apm would look like? Unimpressive? The implication is both that a robot would have a much harder time beating a human if it had to physically perform its apm, and that it would probably take a substantial amount of reprogramming to get it to use the minimap and hotkeys effectively, so that it didn't have to be impossibly fast. The thing is that an sc robot shouldn't exclusively be compared to a chess robot, but also to a soccer robot because it is a physical sport.
again, you are missing the point. This is an AI competition. If you want a sensor and actuator competition (that's what you are describing) there are such that exist. But that's not the point of this competition. Your point is valid, as in, they are right, the computer do have advantages that it can query units across the map, but your point is irrelevant to the competition itself.
I didn't miss anything said by the person to whom i was responding.
EPT
