On July 04 2016 17:19 Freakling wrote:- There will be a difference between different locations and between different mineral formation, however.
Sure, but I wasn't going to test all the different locations and different mineral formations. The most relevant test is for fighting spirit anyway since this is the most played map.
Evidently there will be small differences for the different locations but the purported claims of the workers having a different mining efficiency give no mention at all of being spawning position dependent (I haven't seen anyone say or experience "Protoss mines out faster than terran only when they're in the 1 o'clock position!"). It definitely warrants testing but again how big do you expect the differences to be?
On July 04 2016 17:19 Freakling wrote: [*]Then there is the problems of comsats (or other buildings) placed to influence mining rates (not a factor for basic measurements, but certainly a non-negligible factor in a real game).
Sure, but the claim of a protoss worker being the fastest and a terran worker being the slowest wasn't investigated in such a situation, which was what I was trying to quickly investigate.
On July 04 2016 17:19 Freakling wrote:[*]There is also a problem that worker paths to mineral patches can have either of these properties: - they can be metastable, meaning there is more than one stable path for a worker to take, some of them may be more optimal than others
- they me bugged or periodic, meaning workers on certain patches take longer than normal to mine or worker path cycles through a number of different paths during a series of mining trips. Note that a worker path being bugged for one race does not necessarily imply its being bugged for either of the others, or more generally: workers of the three races do not generally follow the same mining paths (which of course is the root of all the problems listed here ; because all the town halls have a different size, they all have a different effect on pathfinding)
Yes, but such differences would clearly be corrected for if you let the workers mine for a long enough time (and eventually repeat the test multiple times). Also with worker paths being metastable are you saying that having more than 9 workers (influencing each other's mining) might have a substantial effect on the efficiency of mining?
The higher the number of workers mining, the less important the influence of these metastable paths would be, don't you agree? When you reach a certain threshold of workers, almost all of the patches are constantly being mined and workers are waiting for patches anyway, the fact that they might take a slightly less efficient or more efficient path will not have a significant effect on the total amount of minerals mined.
A similar reasoning can be applied to the bug paths of workers. I think such behavior will have an influence at worker numbers > 9 but smaller than the threshold number where all patches are basically constantly being mined. And such behavior will only be significant in my opinion on short mining times.
On July 04 2016 17:19 Freakling wrote:Bottom line: Doing some kind of quick and naïve test tells you absolutely nothing. Your error interval is ± 8 mineralsny the way. That's ± 72 for 9 patches. That is enough to eat up most of your measured differences in a worst-case scenario.
My test actually tells a lot. It has 9 workers mining for 10 minutes for all three races at the same location on fighting spirit. There's essentially no difference between the efficiency of the workers. If there were an obvious difference between the mining efficiency of a single worker on the order of 18% (!!), what is the chance that it wouldn't show up in my test?
We'd have to calculate the statistical power of my test to see how many iterations I would have to do and what the probability is of my conclusion (there being almost no difference) being wrong. We can calculate the type II error.
On July 04 2016 17:19 Freakling wrote:And that's not even starting on the matter of statistic significance. Statistical significance is interesting, but an actual difference is what we would want to see first in this case (again 18%!). It could be that my difference of 2 minerals/minute is statistically significant if we would repeat the test a lot of times, but is it actually a difference that matters?
On July 04 2016 17:19 Freakling wrote:Your results are not the same as the ones stated in the article, by the way. Your maximal relative deviation is only 3.6%, only 1/5 of the 18% claimed by the article, and quite a bit less than the up to ~10% difference I have found between different mains on some former Kespa maps.
I was very skeptical that there would be an 18% difference and my preliminary test indicated in that direction.
On July 04 2016 17:19 Freakling wrote:Basically, generating reliable and statistically significant results is a task for some massive data mining. My plan is actually to make a simple BWAPI bot that does just that. Unfortunately I don't really have the time to take on a prohject like that right now (I am not even much of a programmer...). So if you want to help out with that, contact me.
I would be interested in something like this too, but I have no experience at all with BWAPI (or programming for that matter).
|
On July 04 2016 22:00 B-royal wrote:Sure, but I wasn't going to test all the different locations and different mineral formations. The most relevant test is for fighting spirit anyway since this is the most played map. Which means your results are only specific to single worker saturation on that one location on that one map. Period. You cannot really deduce anything more from that. And there is quite a bit of room for improvement through repeated and longer test runs to minimize error margins even for that.
Evidently there will be small differences for the different locations but the purported claims of the workers having a different mining efficiency give no mention at all of being spawning position dependent (I haven't seen anyone say or experience "Protoss mines out faster than terran only when they're in the 1 o'clock position!"). This only means two things:
- People making any such general claims usually haven't more than a vague (and hence more often than not wrong) idea of what's going on, as they have no objective data that would stand up to scrutiny to support their claim. At best they what they do is an invalid generalization.
- where you start on a map will often have more impact on your harvesting rate than which race you pick
It definitely warrants testing but again how big do you expect the differences to be? Difference between what? Between corresponding expansions on the small map I have found differences as much as ~10% for mineral and up to ~30% for gas mining rates (yes, that's on so called "pro" maps, I usually get differences down to sub 2% for minerals and sub 10% for gas on my own maps). However you put it, systematic differences between races will be overshadowed by this and other effects.
Show nested quote +On July 04 2016 17:19 Freakling wrote: [*]Then there is the problems of comsats (or other buildings) placed to influence mining rates (not a factor for basic measurements, but certainly a non-negligible factor in a real game). Sure, but the claim of a protoss worker being the fastest and a terran worker being the slowest wasn't investigated in such a situation, which was what I was trying to quickly investigate. And the more idealized your testing conditions are the less applicable the result becomes to any real game, making the whole debate purely academic (which is not necessarily a bad thing, depending on what it is you want to achieve or find out).
Show nested quote +On July 04 2016 17:19 Freakling wrote:[*]There is also a problem that worker paths to mineral patches can have either of these properties: - they can be metastable, meaning there is more than one stable path for a worker to take, some of them may be more optimal than others
- they me bugged or periodic, meaning workers on certain patches take longer than normal to mine or worker path cycles through a number of different paths during a series of mining trips. Note that a worker path being bugged for one race does not necessarily imply its being bugged for either of the others, or more generally: workers of the three races do not generally follow the same mining paths (which of course is the root of all the problems listed here ; because all the town halls have a different size, they all have a different effect on pathfinding)
Yes, but such differences would clearly be corrected for if you let the workers mine for a long enough time (and eventually repeat the test multiple times). [/list]More or less (there are some technicalities and potential systematic errors that you'd need to control for, but I don't want to go too much into detail with this right now). Repetition would help more than longer test runs, due to gaining more variation in initial parameters. But this is why I said you'd need to mine a lot of data first before you could say anything at all.
Also with worker paths being metastable are you saying that having more than 9 workers (influencing each other's mining) might have a substantial effect on the efficiency of mining? No that is called worker migration. However, bad metastable worker paths may be self-correcting under worker migration. In physics or chemistry a metastable state is a state that corresponds to a local, but not absolute energetic minimum (and you need to put some activation in to push it over the threshold to another. more stable state). What I mean by it here is simple that there is more than one stable path and that one will not transform into the other unless you manually intervene and micro the worker (i.e. change it's path/angle of approach to the mineral patch once).
The higher the number of workers mining, the less important the influence of these metastable paths would be, don't you agree? When you reach a certain threshold of workers, almost all of the patches are constantly being mined and workers are waiting for patches anyway, the fact that they might take a slightly less efficient or more efficient path will not have a significant effect on the total amount of minerals mined. Under saturation rates >~1.5 worker migration becomes the major factor in limiting mining rates, yes. However, that is not what is being asked here. If you take this into account, Zerg has the upper edge in mining by a significant margin due to the simple fact that Zerg are usually up in expos and have much lower worker saturations, hence a lot less mining time lost to worker migration.
A similar reasoning can be applied to the bug paths of workers. I think such behavior will have an influence at worker numbers > 9 but smaller than the threshold number where all patches are basically constantly being mined. And such behaviour will only be significant in my opinion on short mining times. It is enough to break ZvZ on certain maps, for the reasons stated above. This alone is enough to make this not just a theoretical problem. But in principle you are right, of course: Higher saturation rates make the effects of individual bad worker paths less significant, as worker migration contributes more to wasted worker time.
Show nested quote +On July 04 2016 17:19 Freakling wrote:Bottom line: Doing some kind of quick and naïve test tells you absolutely nothing. Your error interval is ± 8 mineralsny the way. That's ± 72 for 9 patches. That is enough to eat up most of your measured differences in a worst-case scenario. My test actually tells a lot. It has 9 workers mining for 10 minutes for all three races at the same location on fighting spirit. There's essentially no difference between the efficiency of the workers. If there were an obvious difference between the mining efficiency of a single worker on the order of 18% (!!), what is the chance that it wouldn't show up in my test? We are in the same boat here, I think: That article is simply bogus and pretty much any point in it has been rejected by your test. However, some one certainly tested something before writing that (I hope... Maybe they just made it up). If the pictures in the article are any indication, they may have just tested that one patch (at that one location of that one map). However, your test is not sufficient to support the claim of a significant mining rate advantage of any race even under specific conditions (i.e. whatever position on FS you used under single saturation...). However, for the purpose of this thread, yes, you have pretty much shown that some one made a bogus claim which you proved wrong with your test and which can hence be ignored in the debate about racial balance in BW.
We'd have to calculate the statistical power of my test to see how many iterations I would have to do and what the probability is of my conclusion (there being almost no difference) being wrong. We can calculate the type II error. Yes. However, there is no reason to expect significantly different results (at least on FS, which has actually pretty good main mineral lines), so your goal should be to hack away at those error margins to get a reliable mean value. Then you could say : This is the effect I measured (on a certain position on FS...)
Show nested quote +On July 04 2016 17:19 Freakling wrote:And that's not even starting on the matter of statistic significance. Statistical significance is interesting, but an actual difference is what we would want to see first in this case (again 18%!). It could be that my difference of 2 minerals/minute is statistically significant if we would repeat the test a lot of times, but is it actually a difference that matters? The difference would not really matter, knowing that it does not really matter, however, would matter a lot (and stop people from making arcane claims)
I was very skeptical that there would be an 18% difference and my preliminary test indicated in that direction. What was your preliminary test then? could explain the error in the article.
|