|
Russian Federation4235 Posts
Okay, after seeing a discussion on missrates in SC I became very interested to find them out myself. Like everyone before me, I made a simple test.
Test chamber 1:
Two Protoss Photon cannons attacking each other, one on low ground, another on high ground. Powered by invulnerable pylons to avoid improper targeting, controlled by the AI. Cannon stats:
-8000 hp
-1 damage
-0 shields
-0 armor
Now, this has taken awhile, fortunately, when windowed, SC doesn't need focus to run, so I was free to do other stuff. The results were predictable:
3760/8000 hp which means 4240 hits out of 8000 tries. The resulting percentage is actually so close to 53 (missrate is 46,999999999999999) that it made me totally sure that I just repeated a good experiment and got predictable results. After this, my mind is set on the 53/47 hit/miss chances.
But since I love precision, I decided to conduct another test with slightly different initial conditions.
Test chamber 2:
Instead of one pair of very FAT cannons, I made them 100 hp, but made 13 pairs (coincidence, just that many fit on the map =)). I made 3 rounds of testing and got these results (hp left out of 100):
Round 1:
56, 50, 53, 46, 48, 43, 51, 53, 46, 54, 61, 53, 51
averages at 51.15
Round 2:
50, 51, 46, 48, 50, 47, 67, 47, 50, 57, 45, 58, 55
averages at 51.62
Round 3:
50, 58, 55, 44, 48, 56, 56, 45, 51, 52, 43, 57, 48
averages at 51.0
Total average: 51,26
Cool. Not 53%, but 51%. Interesting already, but wait a minute! That's hp left! It means that the actual hitrate is less than 49%! Idk what made the system behave that way, but it's drastically different and actually below 50.
It was hard to believe such results, even though they look pretty reliable, so I remade the test with 1000 hp cannons.
Results:
526, 515, 540, 463, 511, 529, 525, 533, 526, 512, 549, 515, 512
averages at 519
So it's even 48% chance to hit with such experiment initial conditions.
Possible conclusions:
Well, I don't know. I've already ruled out the global 1% missrate because it shouldn't affect averages when there's two of the same unit shooting each other. The thing we don't know is how global miss and highground miss chances stack, but that should be neglible. The thing that worries me is cooldown creep, that, depending on how it's implemented, might be actually visible regardless of sample size, so I'm redoing the over-time test atm.
But in general, the preliminary conclusions are quite funny - SC is non-ergodical in it's random numbers, meaning that averages over an ensemble differ from those over time. No idea why.
Maybe I've done something wrong with the mapping and just fail to see it. I would very much enjoy you to test them out:
DL should start automatically.
|
do a p with expected values = .5
|
This is pretty interesting actually, I'll try to test it myself.
Also, where have you been lately, I love your posts.
|
motbob
United States12546 Posts
On October 07 2009 22:20 Caller wrote:
do a p with expected values = .5
There's no way statistically that the EV of cannon damage on his first test is .5
|
first off, shouldn't the missed shots be more like 3/10 of 8000? So 2400, not 3760?
second, what about the shields? don't they regenerate even if you started at 0?
third, starcraft is deterministic - which means: if you repeat an experiment with the same conditions you will get the same result (otherwise replays wouldn't work)
|
Russian Federation4235 Posts
Now I have a strong feeling that I fucked up somewhere, that's why I posted this under blogs. But if different people get the same results, it might be very interesting. I'm thinking about increasing the ensemble size to, say, 100, and looking what happens.
The thing is that the jump from 53% to 48% is a 10% decrease in effective DPS which is something you could actually feel in a game.
|
Russian Federation4235 Posts
On October 07 2009 22:25 Konni wrote:
first off, shouldn't the missed shots be more like 3/10 of 8000? So 2400, not 3760?
second, what about the shields? don't they regenerate even if you started at 0?
third, starcraft is deterministic - which means: if you repeat an experiment with the same conditions you will get the same result (otherwise replays wouldn't work)
1) Oh no, the whole thing is about a set of test that made it obvious that the hit rate is very very far from 70%.
Read this thread.
2) Max shields were set to 0, so it was 0/0 all the time. Well, yes, to be perfectly precise I should've found that old .dat editor and disabled shields completely, but I figured that 0/0 would do as well.
3) Starcraft is deterministic, but when you start a game, you get a new random number generator seed that makes you receive different random numbers with unchanged user input. Replays work because they also save and load the RNG seed.
Protoss was chosen because zerg buildings regenerate and terran buildings burn. Both disrupt the experiment. Terran units controlled by the AI don't burn or regen, but they sometimes freak out and start wandering around instead of shooting even though they are in range and have a map revealer (possibly because AI doesn't use fog of war and instead tries to emulate high ground invisibility by some other mechanism). Sieged tanks are an exception, their AI is good, but unfortunately their splash deals fractional damage even when it misses.
|
3760/8000 hp which means 4240 hits out of 8000 tries. The resulting percentage is actually so close to 53 (missrate is 46,999999999999999)
rofl, you need an upgrade on whatever calculator you used for that.. obviously its exactly 53/47, not 46.9999999999999, did you really believe that?
As for replays, the randomness is correlated to game time, or remembers a number that is input into the RNG for that game.
|
Netherlands6142 Posts
Add to LP please 
|
Russian Federation4235 Posts
On October 07 2009 22:47 arcology wrote:Show nested quote +3760/8000 hp which means 4240 hits out of 8000 tries. The resulting percentage is actually so close to 53 (missrate is 46,999999999999999) rofl, you need an upgrade on whatever calculator you used for that.. obviously its exactly 53/47, not 46.9999999999999, did you really believe that? As for replays, the randomness is correlated to game time, or remembers a number that is input into the RNG for that game.
I used IEEE floating point. Well, yeah, it's obviosly a floating point precision error, with integers it's exactly 53, my bad.
|
Could the hit chance be 49%? Blizzard says it is 70%, and if they accidentally checked the hit chance twice instead of once, we'd get 0.7² = 0.49 hit chance.
|
have u tried it with 2 fat cannons? i could imagine that sc can only handle a limited amount of required random number generations per timeframe.
|
Do they have vision of each other?
|
Belgium9942 Posts
Did the lowground cannon have vision of the high ground one with an invulnerable observer or something? I think the sightrange thing might have skewed your results.
|
Russian Federation4235 Posts
On October 07 2009 22:57 Scorch wrote:
Could the hit chance be 49%? Blizzard says it is 70%, and if they accidentally checked the hit chance twice instead of once, we'd get 0.7² = 0.49 hit chance.
That's an obvious thing to think, but most people who did single-unit tests reported numbers around 53%, not 49%. It's not the main point of the thread, even though the nature of this percentage is alone an interesting topic. The puzzling thing is why the numbers differ for different numbers of units.
|
Russian Federation4235 Posts
On October 07 2009 23:02 RaGe wrote:
Did the lowground cannon have vision of the high ground one with an invulnerable observer or something? I think the sightrange thing might have skewed your results.
Good point. I'll doublecheck.
Yes! I looked at the table and yes the initial cannon that had a map revealer over the cliff posts (apparently I placed the other revealers for the player but forgot them for the AI) stable results of lower hp than the other ones (in all tests cannon number 4). I think you've hit the jackpot, thanks.
Well, there's still a conclusion to make:
1) It's still not 70%, something closer to 53%. Source of such percentage unknown.
2) Even though a unit on cliff reveals itself by firing, you have slightly more effective DPS when you have a spotter.
Point two might actually find some use in games. I think that case is closed for the moment, I'll just rerun a test with revealers to clarify. Thank you RaGe.
|
On October 07 2009 22:24 motbob wrote:There's no way statistically that the EV of cannon damage on his first test is .5
oh my god stats... give me a while to remember how to the stat functions on my calculator and i'll try to post some tests.
|
You realize the cannon on the bottom misses the first shot right? There's your 0.5%, so you're actually at something like 50.76 average hp left, which is well within something like a 99% confidence interval for it to be 50%.
|
Hmm, i think Blizzard did state at some point that it was 70%. It may have felt like less in some games, but how was I going to doubt something Blizzard said? From these results, i'd think a plain 50% is most likely, I doubt they'd make it 50.5% or 51%, what's the point?
|
Thanks for clearing my questions and nice detective work there!
|
|
|
|
|
|
EPT
