|
Something always bothered me about the way people treat the evaluation of skill using as a basis the amount of wins in the Arena. It seems that this modality is treated like a "challenge against AI" where everyone can succeed simultaneously. They forget that it is a game that generates duels between actual players and - guess what - someone has to lose.
When I saw people complaining of having achieved "only" 4 or 5 wins in Arena I was thinking how would be the distribution of persons by number of victories. How many can achieve 12-0 ? What about 0-3 ?
Without access to the real numbers I decided to generate a computer simulation.
About the program I made:
+ Show Spoiler +I made a small program which performs the following tasks: - Generates a population of 100,000 players with different skill levels that simulate a normal distribution across the entire set.
- Performs crossings players using matchmaking by similar score. Important to note that some matchmaking is important for the final result. Tests with complete randomness in the selection of opponents were executed and although the macro results remain very similar when we look at the result with matchmaking we see important differences as considerable less players with 0-3 result.
- Performs duels between players considering their skills and applying a small margin of RNG in each game, such margin is more relevant to players with less skill.
- Retire players who reach 12 wins or 3 losses.
- With all players retired, collect information about percentages of players who have reached each one of the possible final results.
Note that from now on the conclusions and opinions that I present are based entirely on the results of the computer simulation. If in the future it proves to be excessively flawed by the presentation of real numbers, some or all of the following arguments may become invalids.
The results were stable in several different executions of the program. Almost 300,000 matches are played in each run, and the variations in results between different runs are quite small.
Data :
+ Show Spoiler +Total population: 100,000 ![[image loading]](http://i.imgur.com/B7gpMYM.jpg)
Main Graph:
![[image loading]](http://i.imgur.com/relCV0k.jpg)
The first graph shows the percentage of players at each one of the possible outcomes.
Additional Graph:
+ Show Spoiler +![[image loading]](http://i.imgur.com/TVdX5Pn.jpg) The second graph shows the cumulative result of the percentage of players from 0-3 to 12-0, increasing the total amout until reaching 1, which means 100%. Note that the line bends strongly after the 4-3, where we have passed by 78% of the population and fewer players can move on to better results.
Some interesting data :
- The number of players who won 7 or more games is equal to the number of players that gets 0-3.
- Approximately 50% of the players do not reach their third victory.
- Winning 10 or more games means being among the top 2%, making a comparision with starcraft it's like you're in the masters league.
- Winning 5 matches already puts you among the top 22%.
My Conclusion:
- I believe the way Hearthstone community evaluates the results may be negative for the people's motivation.
- You can't give advices hoping that all people can reach 12 wins, because there will always be many people with low scores for others to achieve better results.
- I think Arena may not be the best place to earn gold for some players. It is a way to have fun, for sure, because the player has access to new cards, but if the results are bad (and for many people they are) directly exchanging gold gain in daily quests for packets seems better.
- We can't consider a score of 5 wins in Arena as mediocre, as you left almost 80% of the population behind.
- It is very important teaching and learning strategies, and have some ambition to improve, but this needs to be done understanding certain limits. Not everyone will be able to reach the top, because it is expected by the rules of the game that a lot of people NEED to get poor results in order to sustain some people with very good results.
UPDATE - from relevant inputs:
Gerenal questions about the program:
+ Show Spoiler +"gerenal questions about the program"
How did you make the normal distribuiton of skill?
How did you make the matchmaking?
How did you decide the winner?
How did you make the RNG?
I really don't want to discuss details of the program, just because it's tiresome. There is no much famous algorithm behind this program, I made myself every step of it. Some overview: Skill is a number, the greater, the better. Normal distribution: More players have skills like 5, less players have skill like 10 or zero. It increases and decreases gradually. Matchmaking: I don't know how blizzard does, I just pick a random player and try to find another with the same (or as close as possible) "win balance" (win - loss). So a player with 7-0 will play against a 9-2 instead a 9-0. The winner is decided by comparing the skills. But I introduced some RNG to allow a worse skilled player to have a chance to win. Lets say player X with skill equals to 6.5 faces a player Y with 7.0. The considered skill in the match is something like this: X: a random number from 5.85 to 7.15 Y: a random number from 6.3 to 7.7 The player Y has more chance to win, but player X still have his chance. If the players have too discrepant skills then the victory of the best player is certain. Important update: I just made some more simulations varying the amount of RNG and find out that it does not make that big of a difference. Also, I mixed the normal distribution of skill and still get similar results. On the other hand matchmaking is much more relevant and affects a lot the results.
Comparing this result with...
+ Show Spoiler +I don't think arenamastery has all necessary data, just the ones people submit. If so, several results, specially the bad ones, may not be present, which ruins the metric. The only way to confirm the results would be comparing with real data from Blizzard, and I would love to do that. I know this simulation is not perfect, but I think it can give us a clue about what happens. Some methods should be done differently, but so far I think nothing is "wrong enough" to invalidate the results. What I'm saying is that we can't learn here that exactly 77.96% of the players reach no more than 4 victories, but we can imagine this number is not 30% neither 90%.
Some aspects the simulation does not consider:
+ Show Spoiler +On March 25 2014 17:23 RenSC2 wrote:
One further consideration is that each "player" in the simulation is not actually an individual, but instead a single run of the arena. A person who averages 3 wins or less in a run will likely only be able to play in the arena once every couple days. A person in that 4-6 range will be able to run arena about 1-2 times per day. People who average 7+ get to run the arena as much as they want. Some of the really top notch arena people are full time streamers/players and so do a very large number of runs every day and account for many runs in the 7+ club.
So the actual % of people who will average 7+ is less than what the simulation will tell you because a larger portion of those runs are being filled by the same people. Meanwhile, the less than 3 club will actually be a much larger % of people since those people don't get to make nearly as many runs as even the 4-6 people.
Essentially, averaging 5 wins or more per run actually puts you in better standing than the top 22% that the initial post calculated.
The only caveat is that real $ infusions are probably highest in the less than 3 club and those people could be balancing things out a bit.
Still, we live on a forum where being "only" in Masters in Starcraft makes you a "bad" player. Masters was initially supposed to be the top 2% of the active playerbase. We're a very elitist community.
Other simulation
+ Show Spoiler +This is simulating the ladder, it's interesting, but I wanted to try Arena which I think is more interesting 
Inspiration
+ Show Spoiler +On March 26 2014 02:33 mikeymoo wrote:I have issues with this methodology, but you've inspired me to construct my own simulation. On March 25 2014 11:02 Came Norrection wrote:
You just made me want to write my own simulation.
Feel free to try your own tests! When blizzard reveals the real numbers we may see who was more accurate 
|
You just made me want to write my own simulation. There is something rather strange about your data since the expected value is not remotely close to 3, which would be what I'd have guess. You seem to have too many 12s, there are more 12 wins than 11 wins. I am not sure the reasoning for that.
|
^
I think that there are more 12s because the 11-2s face each other and so one person is bound to end up 11-3 and one is bound to end up 12-2. Then there's the cases of the people hitting 12-0 and 12-1 which adds a bit more percentage.
Overall cool data though, pretty nice to see.
|
On March 25 2014 11:02 Came Norrection wrote:
You just made me want to write my own simulation. There is something rather strange about your data since the expected value is not remotely close to 3, which would be what I'd have guess. You seem to have too many 12s, there are more 12 wins than 11 wins. I am not sure the reasoning for that.
I didn't exactly understood what you mean by expected value. In my simulation I have (sort of) 50% of the population below 3 wins, and 50% above 3 wins. That was not expected?
About the "total 12's", I think we must consider 12-2 independent from 12-1 the same way it is from 11-3. They are distinct outcomes.
|
On March 25 2014 11:35 MarcoBrei wrote:Show nested quote +On March 25 2014 11:02 Came Norrection wrote:
You just made me want to write my own simulation. There is something rather strange about your data since the expected value is not remotely close to 3, which would be what I'd have guess. You seem to have too many 12s, there are more 12 wins than 11 wins. I am not sure the reasoning for that. I didn't exactly understood what you mean by expected value. In my simulation I have (sort of) 50% of the population below 3 wins, and 50% above 3 wins. That was not expected? About the "total 12's", I think we must consider 12-2 independent from 12-1 the same way it is from 11-3. They are distinct outcomes.
70% of the runs are 3-3 or below which is somewhat counter intuitive to me. The Average run from your data is 2/3 and not 3/3 which is very strange to me.
|
It's actually very easy to do a quick calculation to show that you expect 50% of players to achieve a record of 2/3 or worse.
Suppose that you have a population of N = 100 000 players.
Suppose that all players have an equal probability of winning a game against any other player. This is a simplification for the purposes of estimation. This OP assumed a distribution in skill.
Finally, we impose a condition that players only face other players with the same record.
Round 1:
50 000 - 1/0
50 000 - 0/1
Round 2:
25 000 - 2/0
25 000 - 1/1
25 000 - 1/1
25 000 - 0/2
---------------------------
25 000 - 2/0
50 000 - 1/1
25 000 - 0/2
Round 3:
12 500 - 3/0
12 500 - 2/1
25 000 - 2/1
25 000 - 1/2
12 500 - 1/2
12 500 - 0/3
---------------------------
12 500 - 3/0
37 500 - 2/1
37 500 - 1/2
12 500 - 0/3
Round 4:
6 250 - 4/0
6 250 - 3/1
18 750 - 3/1
18 750 - 2/2
18 750 - 2/2
18 750 - 1/3
12 500 - 0/3
---------------------------
6 250 - 4/0
25 000 - 3/1
37 500 - 2/2
18 750 - 1/3
12 500 - 0/3
Round 5:
3 125 - 5/0
3 125 - 4/1
12 500 - 4/1
12 500 - 3/2
18 750 - 3/2
18 750 - 2/3
18 750 - 1/3
12 500 - 0/3
---------------------------
3 125 - 5/0
15 625 - 4/1
31 250 - 3/2
18 750 - 2/3
18 750 - 1/3
12 500 - 0/3
Thus, we expect 50% of the population to have a record of 2/3 or worse under the conditions specified above.
|
Why do you need a computer simulation when you can just calculate exact probabilities?
All you need 'access' to is that you're 50% likely to win each game.
|
Wow! Great job! You always have the perception that you need to have 6+ wins to keep your gold stack high enough for another arena run. I will keep this in mind when I "only" get 4 wins or less.
|
On March 25 2014 11:02 Came Norrection wrote:
You just made me want to write my own simulation. There is something rather strange about your data since the expected value is not remotely close to 3, which would be what I'd have guess. You seem to have too many 12s, there are more 12 wins than 11 wins. I am not sure the reasoning for that.
I calculate the expected value and it's 2.98 - just as expected (no pun intended) a little bit below 3. this because the number of wins must be the same as the number of losses and the number of losses is always 3 except for arenas that end with 12 wins
|
One further consideration is that each "player" in the simulation is not actually an individual, but instead a single run of the arena. A person who averages 3 wins or less in a run will likely only be able to play in the arena once every couple days. A person in that 4-6 range will be able to run arena about 1-2 times per day. People who average 7+ get to run the arena as much as they want. Some of the really top notch arena people are full time streamers/players and so do a very large number of runs every day and account for many runs in the 7+ club.
So the actual % of people who will average 7+ is less than what the simulation will tell you because a larger portion of those runs are being filled by the same people. Meanwhile, the less than 3 club will actually be a much larger % of people since those people don't get to make nearly as many runs as even the 4-6 people.
Essentially, averaging 5 wins or more per run actually puts you in better standing than the top 22% that the initial post calculated.
The only caveat is that real $ infusions are probably highest in the less than 3 club and those people could be balancing things out a bit.
Still, we live on a forum where being "only" in Masters in Starcraft makes you a "bad" player. Masters was initially supposed to be the top 2% of the active playerbase. We're a very elitist community.
|
Nice job, it would be interesting to see what is the win average for the players with highest/lowest skill level in your simulation.
Otherwise I agree that not everyone can reach 7+ wins consistently since on all players the average will be a bit less than 3 wins. But since some players are able to currently maintain average 7+ wins consistently with current player pool, then if you don't reach this kind of number, either you are very unlucky ( unlikely ) or you have things to improve with your draft / play.
It would be cool if you could simulate deck strength as well. Since a player can have strong deck, that are likely to get high number of wins even if skill of player is low and vice-versa.
|
FREEAGLELAND26781 Posts
Love these sorts of threads.
|
On March 25 2014 15:36 Jinault wrote:
Why do you need a computer simulation when you can just calculate exact probabilities?
All you need 'access' to is that you're 50% likely to win each game.
So you could tell the amount of players in each level without a simulation, just with math. And you can asses the "exact" probabilities? I'd love to see how you make this calculations.
On March 25 2014 17:23 RenSC2 wrote:
One further consideration is that each "player" in the simulation is not actually an individual, but instead a single run of the arena. A person who averages 3 wins or less in a run will likely only be able to play in the arena once every couple days. A person in that 4-6 range will be able to run arena about 1-2 times per day. People who average 7+ get to run the arena as much as they want. Some of the really top notch arena people are full time streamers/players and so do a very large number of runs every day and account for many runs in the 7+ club.
So the actual % of people who will average 7+ is less than what the simulation will tell you because a larger portion of those runs are being filled by the same people. Meanwhile, the less than 3 club will actually be a much larger % of people since those people don't get to make nearly as many runs as even the 4-6 people.
Essentially, averaging 5 wins or more per run actually puts you in better standing than the top 22% that the initial post calculated.
The only caveat is that real $ infusions are probably highest in the less than 3 club and those people could be balancing things out a bit.
Still, we live on a forum where being "only" in Masters in Starcraft makes you a "bad" player. Masters was initially supposed to be the top 2% of the active playerbase. We're a very elitist community.
This actually makes sense. The simulation does not consider the recurrence of players, just consider that each of the 100,000 players have 1 ticket to play in the Arena. In practice, the number of " distinct people" at the top of the pyramid must be even smaller.
|
I think the actual distributions may be slightly different for arena runs.
http://www.arenamastery.com/sitewide.php
Arena mastery shows some great players so their sitewide stats sit around 4 wins average, but they have a lot of 12s compared to your simulation. This could be because some players legitimately have a greater than 50% chance of winning a certain match.
The average # of wins in a run should be around 3 though. 2.9-3.1 ish.
|
On March 25 2014 21:14 obesechicken13 wrote:I think the actual distributions may be slightly different for arena runs. http://www.arenamastery.com/sitewide.phpArena mastery shows some great players so their sitewide stats sit around 4 wins average, but they have a lot of 12s compared to your simulation. This could be because some players legitimately have a greater than 50% chance of winning a certain match. The average # of wins in a run should be around 3 though. 2.9-3.1 ish.
Arena mastery does not have all information, right? Just the ones people submit, if I'm correct. If so, several results, specially the bad ones, may not be present, which ruins the metric.
About "50% of chance", let me clarify that in my simulation we do not have this rule. Players win game based on their skills, and people have different skills simulating a normal distribution over the population.
|
On March 25 2014 21:28 MarcoBrei wrote:Show nested quote +On March 25 2014 21:14 obesechicken13 wrote:I think the actual distributions may be slightly different for arena runs. http://www.arenamastery.com/sitewide.phpArena mastery shows some great players so their sitewide stats sit around 4 wins average, but they have a lot of 12s compared to your simulation. This could be because some players legitimately have a greater than 50% chance of winning a certain match. The average # of wins in a run should be around 3 though. 2.9-3.1 ish. Arena mastery does not have all information, right? Just the ones people submit, if I'm correct. If so, several results, specially the bad ones, may not be present, which ruins the metric. About "50% of chance", let me clarify that in my simulation we do not have this rule. Players win game based on their skills, and people have different skills simulating a normal distribution over the population.
I agree with the first point.
How did you assign skill levels to players? did you give each a point rating and use that to determine their probability of winning? What did the distribution of skill levels look like? Did you use something like Elo distributions for chess or lol?
|
Just because you are in top 20% doesn't mean you are good, it means the other 80% are bad.
Masters league doesn't mean much, even grandmaster to some extent.
|
Depends what you define as good. If good > average -> top20% = good.
@topic
there is a thread on reddit with a similar topic. Don't know if he made calculations or simulations but there were some really good results based on the skill of a player.
|
Canada7170 Posts
Scattered thoughts:
Excellent work. As a complete newb to HS, I generally get 1-4 wins in arena.
I'm curious about your Monte Carlo sim, you mention that skill levels are normally distributed. What is the variance of this distribution? How is player skill determined? I would assume that something like Elo rating would sense in this context.
I don't know if applying a small RNG bonus is mathematically sound- shouldn't the "RNG factor" be implied as a function of skill? Maybe there is some use to genetic algorithms to first generate your pool of players and their "skill/luck" level, then perform the runs?
Could someone enlighten me to how matchmaking in arena works?
This is quite interesting to me and I may come up with my own simulation method, but I doubt it would end up with significantly different results.
|
I noticed this same trend. I think it is the way you "feel" if you go less than about 7 wins.
Technically, 4 wins should "feel" like a victory, because you won more than you lost, right? But the reality is that going 4-3 feels like losing. I think there are 2 main contributors to this, maybe more:
1. The rewards at the 4-5 win level are pretty mediocre/bad. You barely get 150 gold worth of "stuff." So at 4 wins, sometimes you don't even really break even, in terms of what it cost you to enter.
2. Most of the popular streamers play a lot of hearthstone, and as such tend to go 7+ wins. This creates the perception that "everyone" gets 7+ wins, and you should, too.
|
|
|
|
|
|
EPT
