Do you macro like a pro?

Economic management (or ‘macro’) is one of the central skills in any real time strategy game, and is frequently cited as the key factor separating players in different leagues on the ladder. However, the relationship between overall skill at SC2 (ladder rank) and economic management has not been quantified. For my 500th post I decided to tackle that problem.

Like others in the community, I am fascinated by many of the questions pertaining to macro and overall skill. Is there a significant difference between Grandmaster and Master level players in how consistently they make workers? What about Bronze and Silver level players? Grandmaster level players are presumably better at spending their money than other players, but how much better are they than Diamond level players, and how do Diamond level players compare to Bronze level players? Finally, how does income relate to unspent resources exactly, and is it possible to define some sort of approximate measure of macro skill to assess self-improvement?

[image loading]

To try to answer these sorts of questions, I set about analyzing key indicators of economic management for players from all different leagues on the North American ladder. Players studied were:

Top 100 players from Grandmaster league
100 players from a randomly selected division
100 players from a randomly selected division
100 players from a randomly selected division
100 players from a randomly selected division
100 players from a randomly selected division
100 players from a randomly selected division

For each player, I viewed their last three 1v1 games through Battlenet, and from each game I recorded the following data for that player:

• Race
• Game result
• Game duration
• Average unspent resources
• Average income (Resource collection rate)
• Number of workers constructed

A small number of games that ended within a few minutes with neither side having any advantage were assumed to be drop-outs and were replaced by the next game by the same player. In a small number of cases, players had played fewer than 3 ladder games. These games were not replaced. A total of 2084 games were recorded from August 28 to September 15.

It may have been nice to include some additional data or multiple divisions from each league, but the task was onerous enough already. By the end, I wanted to reach into my screen and strangle the circular loading symbol until it stopped cycling. More worryingly, I was starting to notice some number plates contained data that I had recorded in previous days. So was it all worth it? Let's see...

[image loading]

Grandmaster league provides a yardstick against which all other leagues can be measured, so we begin our analysis there.

[image loading]

Analysis of number of workers constructed as a function of game duration shows - as ought to be expected - that Grandmasters construct a lot of workers.

Workers constructed by Grandmaster players. Each data point is a single game. The blue line is a sigmoidal best fit line to the data. Note that while this is a measure of the number of workers constructed by the end of a game (rather than an actual measure of workers at any stage of a game) it shows the expected profile for worker production across a game.

However, it is clear that there is a great deal of variability relative to the best fit line. Much of this can be explained by the fact that this graph averages across all types of strategies. Some builds cut workers early and would therefore follow a much flatter line, while heavy macro builds would follow a much steeper line. Nonetheless, much of this variability can be accounted for simply by grouping the data by race.

Workers constructed by Grandmaster players of each race.

Terran and Protoss players tend to construct a similar number of workers (saturating at 90 and 81 workers, respectively, based on the fit) at a similar overall rate (averaging 59 and 53 workers at 15 minutes, respectively, based on the fit), whereas Zerg players tend to construct many more workers (saturating at 132 workers) and do so more quickly (averaging 74 at 15 minutes). This can be understood in terms of Zerg's tendency to more rapidly expand, the different macro and production mechanics, and the need to build more workers to compensate for losing workers to buildings.

[image loading]

Top level players are renowned for their ability to rapidly spend the resources they collect, and it has often been joked that an ideal player would never float above 400 minerals. But do top level players actually approach that figure, and how does their spending vary as their income increases across the game? One might guess that average unspent money is approximately a fixed percentage of average income. However, the data tell a different story.

Average unspent resources as a function of average income for Grandmaster players. Each data point is a single game. The straight line is an exponential fit (note the logarithmic scale on the vertical axis).

Surprisingly, there is a very close to exponential relationship between Average Unspent Resources and Average Income. This means that for Grandmaster players, every time their average income increases by 500, their Average Unspent Resources approximately doubles.

To try to understand what might cause this, I imagined the simple case where income is constant (or slowly varying). In that case,
[image loading]

where M is money as a function of time, I is income, and S(M) is rate of spending as a function of money. The system will come to equilibrium when the rate of spending is equal to the income, i.e., S(M) = I.

Now suppose the rate of spending is a logarithmic function of money (i.e., players spend money rapidly with a low income, but spending starts to tail off as income becomes large),
[image loading]

where k and A are constants. Then, rearranging the equation S(M) = I gives:
[image loading]

In other words, the exponential relationship between income and money (unspent resources) can be explained by Grandmaster players spending in a logarithmic fashion. Why this is the case, I'm not exactly sure. It's probably partly due to the increasing difficulties involved in mechanically keeping up with increasing income, especially once maxed on population. It would be interesting to explore how different population caps would affect this curve, and to test whether this is something that applies to all real time strategy games.

Interestingly, spending profiles are extremely similar for all three races for Grandmaster level players.

Average unspent resources for Grandmaster players of each race.

[image loading]

Having analyzed the economic management of the highest level players, we now examine how the key elements of economic management differ across leagues, ranging from Bronze to Grandmaster.

[image loading]

As we move down through the leagues, worker production slows dramatically. In the first 28 minutes of the game, Grandmaster players produce an average of 3.7 workers per minute, while Bronze players average 1.5 workers per minute. However, the differences in worker production between leagues are not uniform.

Number of workers as a function of game duration for players of all leagues. Data is averaged in 4 minute bins, with error bars showing standard error of the mean. Normalizing data based on number of players of each race in each division made little difference to any of the figures. Differences between leagues are plotted in the panels below.

The difference between Grandmaster and Master league is impressively large; after 28 minutes, a Grandmaster player has on average 19 more workers than a Master level player. Comparable differences exist between Diamond and Platinum, Platinum and Gold, and Silver and Bronze. Meanwhile, I found no significant differences between Master and Diamond, and Gold and Silver.

An important caveat to this analysis is the fact that we are comparing single divisions between leagues. Each league is thought to contain multiple different tiers which may correspond to different MMRs (as explained in detail by Excalibur_Z’s excellent Ladder Guide). To date, Blizzard has not made it possible to determine which divisions belong to which tiers. It is therefore feasible that in the cases of Master and Diamond, and Gold and Silver, I am comparing the higher league to a high tier division from the lower league (note that Master and Grandmaster leagues do not have tiers).

Also, note that differences in worker production could be due to both: (i) differences in macro skill (less workers = less skill), and (ii) differences in types of strategies used (less workers = more games involving strategies where workers are cut early). We cannot directly differentiate these effects here. However, the below analysis of spending provides another measure of macro skill.

[image loading]

Interestingly, the approximately exponential relationship between Income and Unspent Resources (which shows up as a straight line with a logarithmic scale on the Unspent Resources axis) persists for players in lower leagues. This suggests that all players (not just Grandmasters) spend their income in an approximately logarithmic fashion. However, players in different leagues cluster along different lines, representing different rates of spending efficiency.

Average unspent resources as a function of average income for players of all leagues. Each data point is a single game, with colors representing different leagues. Straight lines are exponential best fits to each league for games with Income greater than 600.

[image loading]

Currently there is no reliable method for quantitatively assessing a player’s macro. I can use my subjective impressions – it feels like I was more on top of my macro today than yesterday – but have I improved since last month? And am I doing better or worse than others in the same league as me? I could try to count certain events, like how many times I get supply blocked, but that is rather cumbersome to record. It would be better if I could quickly calculate how well I am macroing based just on numbers on the score screen at the end of each game.

[image loading]

The above plot of Unspent Resources as a function of Income shows that players in lower leagues tend to cluster higher up, i.e., they are worse at spending their money. However, the slopes of the lines for different leagues are quite similar. This suggests that we may be able to approximately measure a player’s macro abilities based on which parallel band they land in.

Diagonal bands approximately corresponding to the positions of average level players for each league.

Based on this, I devised a measure called the Spending Quotient (SQ), which allows players from different leagues to be separated based on their skill in keeping money low relative to their income. The SQ essentially measures which diagonal band the player lies in.

Histograms of Spending Quotient (SQ) for all leagues, as labeled. Players in higher leagues tend to score higher. Each histogram is constructed using a bin width of 8.

SQ does quite a good job of separating the games of players from different leagues. And what’s particularly interesting is that the differences between leagues are greatest at the extremes. Despite Masters and above containing only 2% of the player population, there is a huge divide between the top of Grandmaster league and the bottom of Master league. Similarly, the bottom of Bronze extends a long way below Silver. By contrast, Silver to Diamond are all squeezed in together, with Gold and Silver particularly close to each other. If skill level follows a Gaussian (or other long-tailed) distribution across the player population, we should expect this, because the highest and lowest divisions contain the very tails of the distribution. Nonetheless, it surprised me to see just how much diversity exists within the top 100 players of Grandmaster league.

However, even for middle of the road players, it is clear that within each league there is a lot of variability. What is this variability due to? Is it primarily due to differences from game to game for all players (e.g., the effects of different strategies on SQ), or is it due to differences in macro skills between players within the same league? In other words, is SQ less variable if we look at the games of a single player?

To determine this, I calculated whether SQ was less variable for each player in Grandmaster league than for the group as a whole. (To do this I calculated the RMS deviation from the mean for the entire group, and compared this to the square root of the sum of the squared RMS for each player). SQ was found to be 30% less variable for the games of each player individually than for the group as a whole. The same analysis for Gold league found that SQ was 29% less variable or the games of each player individually than for the group as a whole. In other words, a good portion of the variability can be accounted for by the fact that different players within the same league have very different levels of macro skill.

[image loading]

To show you how to calculate your own SQ, I will use EGIdrA as an example. EGIdrA is known to be one of the best foreigner macro-style Zergs. Below are the last 20 ladder games in which we had an Income of 600 or greater. Note that the SQ scale is not designed to be accurate for games with Income lower than 600 (partly because there is little demand on a player to spend well during the opening stages, especially if they are following a prepared build order). I recommend you use at least 10 of your own games with an Income of 600 or greater; the more games you use, the more accurate the figure will be.
[image loading]

For each game, calculate SQ using this formula:
[image loading]

For those less mathematically inclined, “ln” is log natural (or log base e). You can use the following formula in excel: "= 35*(0.00137*I-LN(U))+240", replacing I and U with the appropriate cells for income (resource collection rate) and unspent resources, respectively.

Once you have calculated the SQ for each game, you can calculate the average value. [Note that you should do the calculation in this order – don’t just calculate SQ from the average income and average unspent resources.] The scale is designed to range from approximately 0 to 100 (although it is possible to go outside these bounds in extreme cases). For EGIdrA, his SQ is 88 (standard error of the mean = 2), an extremely high score.

The SQ Scale, showing average values for all leagues, and EGIdrA.

SQ is of course not a comprehensive measure of skill; it provides a gauge of spending efficiency, while being insensitive to strategic factors – there may well have been a good reason to stockpile resources at one point in the game. It is also important to remember that there is a lot of game-to-game variability in SQ; a Diamond player could feasibly outscore a Grandmaster player in a single game, but they would be highly unlikely to do so when averaged across many games.

Nonetheless, SQ does show robust differences between players of different leagues, and provides a (sorely lacking) quantitative measure of spending efficiency – a key element of macro. Furthermore, it works equally well for all 3 races (unlike measures of worker production). Thus, I see it as being a useful measure for:

• Quantitatively tracking self-improvement.
• Comparing skills between different populations, including different regions, or changes in the same region over time.
• Assessing pro level performances.

[image loading]

This analysis turned up a lot of interesting findings regarding differences in economic management across the ladder. I was inspired to conduct this analysis by the lack of existing tools for assessing my own macro, and some of the other excellent analyses conducted on these forums (e.g., Lalush's Analysis of Macro).

While there are of course many different playing styles for SC2, this analysis confirms that there are robust differences in economic management between the leagues of the ladder. Interestingly, it also shows the enormous range of skill that exists within the very top tiers.

Hopefully some players will find this useful in terms of setting personal benchmarks, and monitoring self-improvement. Others will hopefully find it interesting from a scientific standpoint, or for comparing pro players. It would be particularly interesting to calculate SQ for players in tournaments - I suspect that pro players will score somewhat better in a tournament environment than on ladder. It would also be interesting to calculate SQ for top Korean players on the ladder (sadly, it would be difficult to obtain GSL data). I also hope that this analysis will provide useful information for those conducting more detailed studies, such as the very impressive SkillCraft.ca study currently being conducted (check it out if you haven’t already).

Thanks for reading this far! I look forward to feedback from the TL community, and I encourage you to try calculating your own SQ!

FAQ

Q: Surely it's not fair to compare SQ between races? There are different production cycles, and therefore different rates of spending.
A: I expected going into this project that there would indeed be significant differences between races. As it turns out, race has a very mild effect of SQ at both high and low levels of play. It's perhaps fortuitous, but SQ seems to function equally well for all races.

Q: You haven't taken into account the effects of players having to remake workers after they are lost or how long players are maxed.
A: This is true. I restricted myself to data that could be extracted from the score screen at the end of a game. Neither of these figures are easy to obtain, and may affect SQ. This is partially what I mean by SQ not taking strategic factors into account, and this is why it is necessary to calculate an average based on a decent number of games to get a reliable estimate. Note that the data include long games for all leagues (where players became maxed), and the approximately exponential relationship between income and unspent resources still seems to hold up.

Q: SQ does not take game duration into account. Surely it is easier to spend more efficiently in shorter games than longer ones?
A: While it is true that it is hard to keep up efficient spending in longer games, longer games tend to also have higher income. I found no positive or negative correlation between game duration and SQ for any of the leagues. Including game duration in SQ would probably slightly improve the accuracy, but it does not seem to be a critical factor.

Q: How were these data collected? How long did it take?
A: The data were collected by hand, which took around 20 hours (including double-checking). It would of course be nice to develop more efficient methods, such as an add-on to another program. But now that the hard work's done, the SQ can be calculated for any individual within a few minutes. Just go through the game history of a player, and put the values into excel (or whatever other program you'd like to use).

Q: Why did you use IdrA, when [player X] is better? And why is his head on the Macro Scale?!
A: IdrA is a well known player within the foreigner community, and he is particularly well known for his tendency to play macro games. I could have picked any player for the demo, and I actually hope that others will try analyzing the SQs of other pros. Keep in mind that I analyzed just his last 20 ladder games. Is this a representative sample, I don't know? He may have been having an off day, and it may not be representative of his tournament performances. As for his head being on the scale, it tickled me.

Q: How was the SQ formula determined?
A: I determined the equation empirically, after testing numerous functional forms. An exponential performed best, and the value 0.00137 was picked as a compromise value to yield a good fit across all leagues. A more complicated formula would probably perform fractionally better, but would be much harder for the average reader to understand and implement.

Q: I'm confused, what are U and I in the formula?
A: U = Average Unspent Resources, I = Resource Collection Rate. This can both be taken off the Income tab at the end of a game. The term "Income" is used interchangeably with "Resource Collection Rate" throughout the article.
Some TL users have already created tools to help simplify the task:
SQ Calculator: http://acslayer.webs.com/
Excel spreadsheets: http://www.mediafire.com/?c3gchw5a389t9sa (newest one!)
http://www.mediafire.com/?h1cd1f2kdj16p6f

Q: I'd love it if SQ could be calculated by SC2 Gears! Has somebody done that yet?
A: Believe it or not, replay files do not contain the information that is available on the score screen, i.e., resource collection rate and average unspent resources. As a result, it is currently not feasible to calculate SQ directly from the replay file. Hopefully Blizzard will add this information to the replay files at some point in the near future.

Do you macro like a pro?

Completed

Ongoing

Upcoming