|
On September 23 2026 12:20 evanthebouncy! wrote:
one thing I noticed after watching more of Pluto's playing is that as soon as it attacks, the opponent loses in 30 seconds.
I think the reasons are as follows:
1. pluto is trained in self-play, so it imagined its human opponent to be itself, i.e. with perfect micro
2. pluto maintains some kind of "army value advantage" (let's call it AVA) estimation, to decide if it can beat or not beat the opponent army
3. the online pluto uses maphack, so it has a perfect information to calculate AVA
4. so when pluto commits to an attack, it is imagining that it can beat the perfectly microed opponent army
5. but human's micro is far worse than pluto, so human will lose a chunk of relative army value
6. pluto's AVA goes wayyy up, and goes into a frenzy and human dies
On the flip side, it's way too conservative as Zerg as a result. It thinks its Terran opponent can click hack and precisely target hurt Mutas, so it never harasses with Mutas. It always either runs away immediately or completely commits to a fight. It never tries to trade.
|
On September 27 2026 04:00 ywxr wrote:
I wonder if the creator is moving away from pure W/L-signal RL + strategy bandit to try to learn efficiently from these human games (which would be a uselessly small sample size otherwise), or from relatively small amounts of high-quality counterfactual-pair training data on things like drops and dark swarm. Given that it's already got a probably pretty well-calibrated win probability/value head, I think fine-tuning it to learn these things could probably be done very cheaply.
The replays are I think not likely useful for training. You don't want to imitate the humans because the humans are worse players. And because the current version (Artosis games, but presumably not the Korean pro games) is a newly trained model, the replays are very off-policy.
|
On September 27 2026 07:20 PurpleWave wrote:Show nested quote +On September 27 2026 04:00 ywxr wrote:
I wonder if the creator is moving away from pure W/L-signal RL + strategy bandit to try to learn efficiently from these human games (which would be a uselessly small sample size otherwise), or from relatively small amounts of high-quality counterfactual-pair training data on things like drops and dark swarm. Given that it's already got a probably pretty well-calibrated win probability/value head, I think fine-tuning it to learn these things could probably be done very cheaply. The replays are I think not likely useful for training. You don't want to imitate the humans because the humans are worse players. And because the current version (Artosis games, but presumably not the Korean pro games) is a newly trained model, the replays are very off-policy.
I wasn't suggesting imitating humans. If the value head is broadly well-calibrated in a generalizable way, specific missing skills like drops/dark swarm could be fine-tuned very efficiently using synthetic data of counterfactual pairs. I agree the actual replays aren't necessarily that useful, but in this case it might not matter too much that they're off-policy. And with generalizable enough calibration, I think there could even be a way for it to be a few-shot online learner, which would be really cool.
|
I’m wondering whether the training model is steer-able deep into mid game.
Self-play training games are able to be steered/ weighed with certain openings.
However for current major weakness of Pluto: dealing with drop (reaver/shuttle/dropship), inability to use drop, building placement (nexus/cc/hatch & wall), understanding darkswarm ability.
These might take millions & billions more games and still not discover them in reinforce selfplay by chance (sounds like evolution ha).
|
is awesome32282 Posts
Just dropping by to say that:
- it's great to see your name here again!
- Good context on how the bot works. What other training strategies could be used. I guess the other extreme would be for it to study a set of curated (high ELO) replays. Is there a way to reach the best of both worlds? Give it a baseline based on current human know ledge and let it push the boundaries of infinite APM + attention?
|
On September 27 2026 11:50 IntoTheWow wrote:
Is there a way to reach the best of both worlds? Give it a baseline based on current human know ledge and let it push the boundaries of infinite APM + attention?
This is what AlphaStar did (minus infinite APM, which like Pluto they somewhat limited):
- Train a model to imitate humans, over 1 million Battle.net replays. This model was ~Gold league as-is
- Copy the model many times over, and train them against one another, with different reward signals for each (reward winning + preference different units),
When they trained with just reinforcement learning they didn't manage to get behavior more sophisticated than worker rushes. The imitation step was required. So Pluto requiring no imitation step is impressive.
|
On September 27 2026 05:50 419 wrote:
The bit about Pluto calculating likes its playing against itself makes me wonder if Pluto having contempt would make it even more effective against human players.
yeah I think to beat humans Pluto needs a kind of "skill check" mechanism, which is something that can be adjusted at inference time.
for instance, it can measure the difference of "predicted reward" based on its neural network, and "observed reward" based on how the game actually panned out. For example, Pluto might have calculated that an engagement it will lose 8 marines worth of "score", however after the actual engagement it only lost 2 marines worth of "score". Then it can realize that "oh wow this opponent micros terribly" and now that can be used as additional signal to steer its behaviour, for instance, being more aggressive.
|
On September 27 2026 06:41 StRyKeR wrote:Show nested quote +On September 23 2026 12:20 evanthebouncy! wrote:
one thing I noticed after watching more of Pluto's playing is that as soon as it attacks, the opponent loses in 30 seconds.
I think the reasons are as follows:
1. pluto is trained in self-play, so it imagined its human opponent to be itself, i.e. with perfect micro
2. pluto maintains some kind of "army value advantage" (let's call it AVA) estimation, to decide if it can beat or not beat the opponent army
3. the online pluto uses maphack, so it has a perfect information to calculate AVA
4. so when pluto commits to an attack, it is imagining that it can beat the perfectly microed opponent army
5. but human's micro is far worse than pluto, so human will lose a chunk of relative army value
6. pluto's AVA goes wayyy up, and goes into a frenzy and human dies On the flip side, it's way too conservative as Zerg as a result. It thinks its Terran opponent can click hack and precisely target hurt Mutas, so it never harasses with Mutas. It always either runs away immediately or completely commits to a fight. It never tries to trade.
exactly! and that's how humans wind up beating Pluto by just making a huge army and never actually fight it :D
pluto will just hallucinate this impossibly strong opponent and in its immediate reward function to "reduce unit score loss" never take a fight
I've not seen Pluto take any hail mary attempts, because it is so caught up in optimizing the short-term gain. so Pluto will be very weak in actual positional plays, where you sacrifice the "now" for the "future". For this same reason it never figures out why upgrades are useful either.
|
On September 27 2026 12:20 evanthebouncy! wrote:Show nested quote +On September 27 2026 06:41 StRyKeR wrote:On September 23 2026 12:20 evanthebouncy! wrote:
one thing I noticed after watching more of Pluto's playing is that as soon as it attacks, the opponent loses in 30 seconds.
I think the reasons are as follows:
1. pluto is trained in self-play, so it imagined its human opponent to be itself, i.e. with perfect micro
2. pluto maintains some kind of "army value advantage" (let's call it AVA) estimation, to decide if it can beat or not beat the opponent army
3. the online pluto uses maphack, so it has a perfect information to calculate AVA
4. so when pluto commits to an attack, it is imagining that it can beat the perfectly microed opponent army
5. but human's micro is far worse than pluto, so human will lose a chunk of relative army value
6. pluto's AVA goes wayyy up, and goes into a frenzy and human dies On the flip side, it's way too conservative as Zerg as a result. It thinks its Terran opponent can click hack and precisely target hurt Mutas, so it never harasses with Mutas. It always either runs away immediately or completely commits to a fight. It never tries to trade. exactly! and that's how humans wind up beating Pluto by just making a huge army and never actually fight it :D pluto will just hallucinate this impossibly strong opponent and in its immediate reward function to "reduce unit score loss" never take a fight I've not seen Pluto take any hail mary attempts, because it is so caught up in optimizing the short-term gain. so Pluto will be very weak in actual positional plays, where you sacrifice the "now" for the "future". For this same reason it never figures out why upgrades are useful either.
Indeed. Despite how strong it is right now, it's beatable, especially if you abuse its blind spots (like not knowing what Dark Swarm is).
There's a lot of ways it can get even stronger. If people actually continue to work on this it's not hard to imagine a version that beats Pluto almost every time.
The creator mentioned publishing a paper on this research. I wonder if he'll be less interested in pursuing further research once he achieves that.
|
On September 27 2026 13:44 StRyKeR wrote:Show nested quote +On September 27 2026 12:20 evanthebouncy! wrote:On September 27 2026 06:41 StRyKeR wrote:On September 23 2026 12:20 evanthebouncy! wrote:
one thing I noticed after watching more of Pluto's playing is that as soon as it attacks, the opponent loses in 30 seconds.
I think the reasons are as follows:
1. pluto is trained in self-play, so it imagined its human opponent to be itself, i.e. with perfect micro
2. pluto maintains some kind of "army value advantage" (let's call it AVA) estimation, to decide if it can beat or not beat the opponent army
3. the online pluto uses maphack, so it has a perfect information to calculate AVA
4. so when pluto commits to an attack, it is imagining that it can beat the perfectly microed opponent army
5. but human's micro is far worse than pluto, so human will lose a chunk of relative army value
6. pluto's AVA goes wayyy up, and goes into a frenzy and human dies On the flip side, it's way too conservative as Zerg as a result. It thinks its Terran opponent can click hack and precisely target hurt Mutas, so it never harasses with Mutas. It always either runs away immediately or completely commits to a fight. It never tries to trade. exactly! and that's how humans wind up beating Pluto by just making a huge army and never actually fight it :D pluto will just hallucinate this impossibly strong opponent and in its immediate reward function to "reduce unit score loss" never take a fight I've not seen Pluto take any hail mary attempts, because it is so caught up in optimizing the short-term gain. so Pluto will be very weak in actual positional plays, where you sacrifice the "now" for the "future". For this same reason it never figures out why upgrades are useful either. Indeed. Despite how strong it is right now, it's beatable, especially if you abuse its blind spots (like not knowing what Dark Swarm is). There's a lot of ways it can get even stronger. If people actually continue to work on this it's not hard to imagine a version that beats Pluto almost every time. The creator mentioned publishing a paper on this research. I wonder if he'll be less interested in pursuing further research once he achieves that.
If I recall correctly, tscmoo has been working on BW AI bots for a decade, if not longer. Even though he is on top of the bot ladder (and briefly the human ladder!), somehow I don't think his interest will fade just because he is on top or he published a paper  The BW AI competition landscape could get very interesting after such a systemic shock (not that it wasn't already!).
E: Definitely more than a decade:  Tscmoo
|
On September 27 2026 04:00 ywxr wrote:
I wonder if the creator is moving away from pure W/L-signal RL + strategy bandit to try to learn efficiently from these human games (which would be a uselessly small sample size otherwise), or from relatively small amounts of high-quality counterfactual-pair training data on things like drops and dark swarm. Given that it's already got a probably pretty well-calibrated win probability/value head, I think fine-tuning it to learn these things could probably be done very cheaply.
I think . . . the easy way to train it to learn about drop/dark-swarm is to find ways to synthetically generate a lot of data of games containing them, in the order of 10s of millions, in order to train pluto.
There is already some exploration intrinsic reward for making more niche units (that's why you see it upgrading ghosts or making a single corsair), except this incentive is not strong enough for it to climb the tech-tree all the way to dark swarm
A rather crude approach is to take some human games and train a kind of unit composition reward function, and use that reward function in its own game. For instance if defilers would give a high reward, the bot would learn to make them first, and then through sheer luck figure out how consume / dark-swarm works through millions of trials and error
on fine-tuning the winprob/value head, I'm not so sure. Clearly it is not calibrated against stuff like dark swarm and drop. I think those things do require significant amount of additional training data to converge
|
On September 27 2026 13:44 StRyKeR wrote:Show nested quote +On September 27 2026 12:20 evanthebouncy! wrote:On September 27 2026 06:41 StRyKeR wrote:On September 23 2026 12:20 evanthebouncy! wrote:
one thing I noticed after watching more of Pluto's playing is that as soon as it attacks, the opponent loses in 30 seconds.
I think the reasons are as follows:
1. pluto is trained in self-play, so it imagined its human opponent to be itself, i.e. with perfect micro
2. pluto maintains some kind of "army value advantage" (let's call it AVA) estimation, to decide if it can beat or not beat the opponent army
3. the online pluto uses maphack, so it has a perfect information to calculate AVA
4. so when pluto commits to an attack, it is imagining that it can beat the perfectly microed opponent army
5. but human's micro is far worse than pluto, so human will lose a chunk of relative army value
6. pluto's AVA goes wayyy up, and goes into a frenzy and human dies On the flip side, it's way too conservative as Zerg as a result. It thinks its Terran opponent can click hack and precisely target hurt Mutas, so it never harasses with Mutas. It always either runs away immediately or completely commits to a fight. It never tries to trade. exactly! and that's how humans wind up beating Pluto by just making a huge army and never actually fight it :D pluto will just hallucinate this impossibly strong opponent and in its immediate reward function to "reduce unit score loss" never take a fight I've not seen Pluto take any hail mary attempts, because it is so caught up in optimizing the short-term gain. so Pluto will be very weak in actual positional plays, where you sacrifice the "now" for the "future". For this same reason it never figures out why upgrades are useful either. Indeed. Despite how strong it is right now, it's beatable, especially if you abuse its blind spots (like not knowing what Dark Swarm is). There's a lot of ways it can get even stronger. If people actually continue to work on this it's not hard to imagine a version that beats Pluto almost every time. The creator mentioned publishing a paper on this research. I wonder if he'll be less interested in pursuing further research once he achieves that.
the author is very passionate about bw I think he'll write a paper but he'll also just keep on pushing this bot further I'm sure.
I think the most immediate thing to make pluto better (vs human) is to add a skill check mechanism so he figures out how strong (or weak really) human is at micro, and adjust its value functions accordingly
|
The cost to run training must be pretty high I imagine. He may be limited by compute for diminishing returns unless he's being funded by Meta for its AI division.
|
On September 27 2026 21:33 StRyKeR wrote:
The cost to run training must be pretty high I imagine. He may be limited by compute for diminishing returns unless he's being funded by Meta for its AI division.
we can only guess haha :D
and yeah these trainings would be expensive for sure. The game space for starcraft is much larger than Go, and it'll take a good amount of compute to "flush out" that space. Pluto has done such an amazing job of flushing out the space, with ostensibly less compute than what deepmind had for playing Go. Even then pluto is yet to comprehend the orange cloud thing 
how to train RL agents efficiently is very much an open problem, that's why it's still paper worthy
|
On September 27 2026 16:08 evanthebouncy! wrote:Show nested quote +On September 27 2026 04:00 ywxr wrote:
I wonder if the creator is moving away from pure W/L-signal RL + strategy bandit to try to learn efficiently from these human games (which would be a uselessly small sample size otherwise), or from relatively small amounts of high-quality counterfactual-pair training data on things like drops and dark swarm. Given that it's already got a probably pretty well-calibrated win probability/value head, I think fine-tuning it to learn these things could probably be done very cheaply. I think . . . the easy way to train it to learn about drop/dark-swarm is to find ways to synthetically generate a lot of data of games containing them, in the order of 10s of millions, in order to train pluto. There is already some exploration intrinsic reward for making more niche units (that's why you see it upgrading ghosts or making a single corsair), except this incentive is not strong enough for it to climb the tech-tree all the way to dark swarm A rather crude approach is to take some human games and train a kind of unit composition reward function, and use that reward function in its own game. For instance if defilers would give a high reward, the bot would learn to make them first, and then through sheer luck figure out how consume / dark-swarm works through millions of trials and error on fine-tuning the winprob/value head, I'm not so sure. Clearly it is not calibrated against stuff like dark swarm and drop. I think those things do require significant amount of additional training data to converge
I agree about generating large amounts of synthetic data. I don't think the value head needs to be recalibrated directly; what I mean is that since it's very likely already quite generalizable to phenomena like "suddenly lose a fight very badly" or "suddenly lose a ton of economy," even if the network function doesn't understand well the underlying cause, with the right SFT architecture it should be possible to learn far more efficiently than millions of games.
Or more speculatively, even learn few-shot online by looking for sharp discontinuities in the value head which will be like high "perplexity" and then tracing back credit.
|
wonder if it would be faster to train something like this by save-stating moments in time and repeatedly repeating from that state trying different alternatives
|
|
|
|
|
|
EPT
