|
I got a PM today from a relatively new member, ilovezil, that contained what I felt was an open-ended and excellent question. Below is the PM and my response. ilovezil, I hope you don't mind me posting this!
I'd love for the community to share their thoughts on this matter!
---
From: ilovezil
Subject: "how good"
Hi. I just wanted to find out, from your insight, a little about how you determine a player's worth. It was always a problem for me when I tried to determine if player A is defintely better than player B, etc. There are just so many factors like had a bad day, luck, or even the level of the players; I cannot discern all of this in my brain. That statement, "I am better than you...", it's not so easily determined, even if player A goes 3-0 vs player B. I'd appreciate it if you can share some of your thoughts.
---
To: ilovezil
Subject: Re: "how good"
Yeah, absolutely. I know precisely what you mean when you talk about the difficulties in gauging skill. It's never exact, and it's rarely easy.
I won't lie: statistics are helpful. The best statistics are by matchup, meaning that you can view how a player performs vT, vZ, and vP. This lets you not just get a feel for their overall record, but for what matchups they are strong/weak in. A summation of how strong a particular player is across all his matchups is what you’re ultimately looking for (“This player is strong vT, about average vP, and below average vZ.”), but it takes a lot more than statistics to achieve even a semi-accurate statement of this nature.
This difficulty comes from the fact that statistics never tell the whole story. It is important to watch *a lot* of games. You can "discount" (what I mean to say is "semi-ignore," and that "semi-" is a big, big qualification; don't ignore them completely!) a lot of games on the basis of cheese or map imbalance. If, for instance, the matchup/map is TvP on R-Point--and the positions are horizontal--you can *usually* overlook a P loss; it's just so, so difficult to win at those positions. If the P does manage to win, it's *probably* because the T made a mistake or the P cheesed. But if the P did an all-in build and crippled the Terran, and the Terran still managed to come back for the win, well, you can make a skill judgment from a game like that. So overlooking wins and losses is something you have to be careful about.
Beyond that, there is a lot of luck in StarCraft, especially in PvP and ZvZ. If you watch Bisu's games against Ra, it should be pretty obvious that Bisu, in games 1 and 2, happened to pick better builds than Ra, and he won very quickly because of these superior build choices. You can argue that those choices are a form of skill, but that's a pretty hollow argument when neither player was able to conclusively scout the other. The 3-0, then, doesn't mean that there's some massive skill gap between the two: Bisu just had a good day (although Bisu is probably the better player overall, Ra is arguably just as good at PvP).
So the best maps for gauging skill are ones that are balanced in a particular matchup, because no map is balanced across all matchups. A TvP/PvT win on RLT, Arcadia, and Longinus (for example) can be considered a "skilled" win (assuming no cheese), because the maps are essentially balanced in that matchup. But even then, all you're left with is a comparison, something like "Anytime is better than Casy at PvT."
The problem with a statement like that, however, the small sample size; we're only dealing with one game, and in a Bo5, it's rare that we'll have a set of completely balanced maps across the matchup in question. If we were able to watch Casy and Anytime play 25 games on each of those maps, it would be much easier to see a trend in the results that we could correlate to a difference in skill, but with the very limited number of results we have, a lot of talk of skill comes down to subjectivity and general knowledge of StarCraft.
I'm a pretty good player, if I may be so immodest, but more than that, I have a good grasp of what *needs* to be done in a particular game situation. It's just that I'm not always able to achieve those goals because, well, I don't multitask well enough. Having a strong grasp of what a player should be doing allows you to watch professional games and say, "Yes, he's playing really well. He's doing everything I would be doing and much, much more." Acquiring a strong grasp of StarCraft requires both playing and studying the game. Too many people are what I call "replay newbies," in that all they do is watch and copy replays. They don't know *why* they do something, just that "this gosu does it, so it must be good." They never challenge actions of these players or test new theories of their own. I encourage you to really consider the *why* in StarCraft: why certain things work in situations and why others do not. If you have a strong grasp of those concepts, it's not *too* difficult to see which player is "getting it done" moreso than the other player, regardless of the win/loss result--but that's not to say that the result is unimportant!
Finally, there are extenuating circumstances. We'll use MBC Hero for our example, because I like them. Let's say that Hero has the Proleague final coming up against SKT1, and Pusan has a game against Jju coming up in the MSL on a map that is terrible for PvT but decent for PvZ. Now then, it's safe to assume that, because the map is terrible for PvT and Hero is playing SKT1 (with their mass of strong Terrans), Hero is *not* having Pusan practice on it for the Proleague final; they're probably using him on a different map. And because a Bo1 in an individual league is far less important than the Proleague final, Pusan's attention is undoubtedly focused on the final. So if he loses to Jju in the MSL, should we really conclude that his PvZ skill is deteriorating, or that Anytime, who beat Jju three weeks ago, is the better at PvZ than Pusan? No, we can't make that judgement--not based upon those games, anyway. Just like Midas' series of jet-lag-induced losses after his trip to Italy, there are circumstances that should cause us not to overreact in our judgments of players. Don't underestimate the power of these circumstances!
I'm sure there are points I’m failing to discuss--and for that I apologize--but the number of factors that go into any win or loss is simply staggering, which means that it's best to look at the long-term when it comes to a player's skill, because inconsistencies filter out over a long enough period of time. The problem is that most people a) don't watch enough games and b) don't have good enough memories when it comes to games. I'm guilty of the latter; I’m not even sure it’s possible to remember every game clearly. And because it's so difficult/impossible to keep all these things in your mind, many, many people simply acquiesce to statistics without making the above considerations. Just don't be one of those people, and you'll already be doing better than the majority of TL.net. ^^
|
wow, great question i've always been wondering about, thx for answering ilovezil/etter !
|
Wow, Etter. I should ask you about life's problems if you write PMs that are that long.
|
Nice post. Take an example from WCG 2005 for instance.
I dont think the foreigners highest level is as high as Silent_Controls highest level. Meaning Control can play pretty shitty, but he can also play very very good for example against Xellos at WCG. He's very inconsistent but comparing him with those players is far fetched imo. It really depends on how you compare players though, through consistency or the ability to reach a certain level at times.
This in addition to matchups, maps, positions, stakes, offline/online and other points mentioned.
|
Beautiful post etter and great question zil, you stated (imo) the biggest or one of the biggest points, the level of pressure in a game/circumstances.
Great post.
|
On March 07 2007 19:48 davidgurt wrote:
Wow, Etter. I should ask you about life's problems if you write PMs that are that long.
I always answer PMs more thoroughly than I do forum posts. The person who wrote the PM took the time to ask a question to me specifically; the least I can do is provide a comprehensive answer.
|
1.) I think your example of Bisu vs Ra and build orders is a bad one. Ra made mistakes in each game that caused him to lose. He didn't play well enough to give his build orders the credit they deserve. In other words, Ra on a different day, or another Protoss, using the same builds, could have made them work.
I guess you've already decided that that's not the case, so I guess it's just my opinion against yours. But the specific example isn't really the issue. The fact that we disagree on what happened in that series shows how difficult it is to judge. We're both level-headed players and yet we draw contrasting conclusions. The problem is that we have no way of knowing how correct we are. We never get to check our answers. To some extent, we can look at future games and see if one trend continues or if the other one does, but that's hardly an exact science.
2.) It's hard to judge how well a player is playing. You say that you have a grasp of what needs to be done, so you can usually detect when a player is lacking. That's true for many a situation, as an experienced player can "see between the frames" into where exactly the progamer is coming up short. However, in the case of specific strategies or tactics, we never know what happened in the players' practice games. Perhaps we're watching a game and screaming "you just need to get more of this unit and you'll win! gees!" But maybe he had already tried that, and for a reason that isn't immediately obvious, it fails miserably.
For somewhat of an example in the Bisu Ra series, look at the last game. Ra had air control with his corsairs, but was skipping detection. Bisu knew this and ended up sacrificing a shuttle to the corsairs, while the DT shuttle went around the back. Maybe in Ra's practice games, he never had a player pull such a move off on him. So while many fans are watching, and thinking, "damn, Ra was just taking a risk by skipping cannons, Bisu got lucky", Ra might be thinking "damn, I never saw that coming" or "damn, I didn't think that'd actually work on me". So in fact it could have been a perfectly fair test of skill, and not some risky move that got called.
3.) I don't have a 3rd point, this is just a summary. Basically, we don't know when a player should be doing something or shouldn't be doing something, we don't know when to give them credit for a move or not. Yes, for the majority of moves in a game, experienced players have a good idea. However, there are frequently huge question marks over particular moves, often game-winning or game-losing moves, that we just don't know about.
PS: With the same type of thinking, I can argue that savior played well in his match against Bisu. As I stated in a different thread, I believe Savior would've beaten any other Protoss that night with his play. He didn't do something that any old Toss could have taken advantage of as dramatically as Bisu did. Bisu did something special to completely crack the monstrous ZvP of Savior.
|
Osaka27154 Posts
A quality post that others should learn from.
Etter pointed out some good points about map balance/positional balance, matchups, and game significance. There are a couple intangibles that I would like to add to the list.
Firstly, about nerves. I think that most of us are desensitized when it comes to the nerves required to play on stage, after all we watch their emotionless faces all the time. However, if you have ever played a good player on bnet, did you ever get nervous? Did your hands shake for the first couple minutes? I know mine did. I think that there are some players who are naturally better at controlling their nerves, and other players are better prepared by their coaches.
When I see first time players play on TV, there are a lot of micro mistakes caused from nervous spam and misclicks. This is minor. And then, there are times where you get people like Gundam or Yooi, who are really great players, but simply cannot perform to their ability on stage (pro.NT_ShinyA is another great protoss who couldnt play offline). Unfortunately, If you saw Yooi's games form his first OSL, he looked like a B+ noob, and despite all of his preperation and hard work, his nerves shot him to pieces and he didnt stand a chance even before the game had started. Conversly, watching someone like savior shows just how valuable a level head can be.
Connected to that, I belive, is the ability to adapt and solve unexpected problems quickly. With such a detached mind, Savior is very able to quickly and accurately react to things unexpected. When he was four gated against vs daezang on peaks, I believe almost every other zerg would have failed in that game. However, with his analytical skills and cool head, savior was able to put up the exact amount of defense needed to repulse the attack and still maintain a winning edge. There are other players who, when faced with certain situations, are simply unable to generate the solutions needed to certain problems in a time and pressure situation.
Of course, the greatest on-the-fly innovator ever would have to be Boxer. He had a special ability to be able to solve small problems properly, easily, and with the least waste. If you look at the other side of the coin, there are many players, even professionals, who will send their entire army to kill a lurker drop. This is the anti-thesis of what I am talking about. It also seperates the A from the S class.
It is, in the end, these players which are able to generate intrest as well, because they can reach down and pull out the unexpected, solutions that we as the audience cannot generate ourselves.
|
GrandInquisitor
New York City13113 Posts
This sort of discussion is far too deep for a forum post (or even a PM =P)
However, I would claim that Starcraft is a uniquely bad game to objectively judge skill in, and I'll try to explain why.
Starcraft is hindered by the fact that there is no objective method of telling how "close" a game was - as a result, the dichotomy forces us to make big judgements we are not always comfortable with.
Win statistics are useful only in the long term and with extreme samples (savior with over 80% ZvP is a reasonably good indicator of ZvP skill). Anything smaller than that, and we run into luck. Not just PvP / ZvZ BO luck, but little stupid things that can change the game so much. The skill level is so close that little things, things that we would never consider part of a player's skill, have enormous impacts on single games - and the format of our OSL does not help, seeing as how much of an impact single games make. NaDa got to the final despite probably not being anywhere close to the 2nd best player in the league - three times he had everything on the line in a deciding make-or-break game.
But say in one of those games something stupid happened. He didn't look at his minimap for one half of a second and missed a drop that while seemingly negligible immediate impact had dire long term consequences. Had any one of those gone the other way, we would be discussing a completely different Power Rank (no NaDa in top 10, definitely), a totally different KeSPA - and yet all because he didn't see his minimap for half a second? Is that really fair? Can we place any value in this system if such small happenings mean the difference between "Second Best in the World" and "Who The Hell is He?"
Look at how badly Reach is slumping. I guarantee you he doesn't actually suck right now. But getting out of his offline qualifiers is total hell, and unless you dominate everyone else (something he obviously cannot do right now), the only way you are qualifying is through almost pure luck. Things just haven't been clicking for him, and so we don't even consider him top 30 in the world - yet if he got into OSL somehow, he almost certainly could have made it to elimination rounds and (with a lucky break) semis or even finals. Then we would label him one of the Top Five Protosses in the World. This does not reek of preciseness.
Little mistakes also hurt Starcraft because you cannot bounce back easily in a single game. At the top level, doing one thing stupid such as misclicking your drones so they stack instead of spread, causing you to lose a couple extra workers in ZvZ - you are fucked. You are not coming back.
Other sports have some version of this problem, but to a much lesser extent. A random small mistake in soccer will, in all likelihood, not change the result. FA Cups are not won and lost because someone slipped in the mud in one game. And especially, in soccer - even if you make a mistake, at worst you give up a goal and you fight to get that goal back. In Starcraft, when you make a mistake, you probably just lose the game - period.
The last thing I want to talk about (but certainly not the last thought on this topic) is matchups. Say Casy beats GoodFriend TvT. Is Casy a better player than GoodFriend? Maybe. What if GoodFriend can beat every Protoss in the world and Casy loses to people like ahk-gosu? Is Casy still better? What if there weren't really that many Protosses in the world at all? Is Casy somehow suddenly a better player because of the scarcity of other people at a particular race? That is offensive to the very definition of skill.
So what, you say. Let's define people in terms of their matchups. Ignoring the fact that this requires the creation of 9 separate Power Rankings, since it becomes absolutely impossible to compare between two different ones, this is still a hazy question because of styles. NaDa might rape Pusan's style; Casy might rape Anytime's style; GoodFriend might rape Bisu's style. Who's best? How do we make the Best of TvP list? Judge by how good Pusan/Anytime/Bisu are at PvT? How do we make that list, without already having had a Best of TvP list?
The solution most other sports have to this sort of style-countering skill is to play a lot of games against a great many people, and see how they do against a common set of opponents. So we compare everyone's performances against Pusan Anytime AND Bisu. The problem in Starcraft is twofold - not only do players not play everyone else, but also the situation is always different each time (maps, pressure on winning the game, the stage you're playing on). You're no longer comparing on a ceteris paribus basis, and it's really hard to get anything out of it at all?
Given all of this, I think subjective rankings are the only possible way we can do it. Objective rankings are results-based and tell you nothing more than who was fortunate enough to win their leagues. And while I do not doubt that in many circumstances the winner of OSL/MSL is probably pretty clearly indisputably the best, sAviOr is an unusual circumstance. Look at the other three semifinalists in OSL - we know that IriS can beat Casy (though, like I said before, we can't really say definitively that IriS is BETTER than Casy, whatever that means). Do we know that NaDa can beat IriS? Can we even really say that NaDa can beat Casy if they play again? How do you objectively judge who among these three is best? Even if IriS 3-0's NaDa as well - if IriS couldn't beat a USEast Protoss noob and NaDa could beat all P's, is NaDa a better player (even though he never showed off his TvP abilities)? (And this is a fortunate instance, where they would all play the same mirror matchup against each other.)
So all we can do is watch the games and guess. And argue, and bicker, and yet remain so far from that elusive holy grail of objectivism.
|
featured thread plz...
Great discussion
|
it's a mind game people
after your hands get used to the speed requirements it's all in your head
i know all about the nerves  ; being a newb i assume that at least 75% of my oponents are better, or more experienced and it took me a while to control my nerves; but i just get cold; 3-5min in to the game my hands are cold-numb which cause me to make serious mistakes;
watching the replay i know almost every time what i did wrong; durnig the game i know what to do or what i sould have done but from some reason i didnt do it; most of the time is the lack of minerals; i need extra lings if teran moves out quickly or extra sunkens, or i teck too fast which means far less drones, stupidly loosing lings or lurks in the beginning and all that (and some more) means my expo will be very late and that spells - defeat
|
For me it's pretty simple, I can determine how skilled a player is by watching his macro/micro/multitask/BOs during a couple of games, (at least 6-7 games).
|
Among pros i find hard to tell. If you gave me a replay of professional gamers, i probably wouldnt be able to rank them.
I am still not fully certain what separates the top pros from the bottom ones.
My understanding of the game is not high enough to capture all the little decisions that go in a game.
I might see some unit choices, strategies, smart positionining, good micro, spending, and logic.
But I think control, timing, and certain macro problems i miss out.
For example the game Draco took from Midas on Gaia, still astonishes me how Midas could pump so many units after having his nat expo dennied 2-3 times...
|
United States20661 Posts
In the old days, skill and good looks were direcly proportional. Good looking gamers like Reach and YellOw and BoxeR did well.
Then, at some point [circa 2004] the opposite became true. Skill and good looks became INVERSELY proportional. Now we have ugly people like sAviOr and iris dominating.
T-T I liked the old method of measuring skill better.
|
Osaka27154 Posts
On March 08 2007 00:01 Last Romantic wrote:
In the old days, skill and good looks were direcly proportional. Good looking gamers like Reach and YellOw and BoxeR did well.
Then, at some point [circa 2004] the opposite became true. Skill and good looks became INVERSELY proportional. Now we have ugly people like sAviOr and iris dominating.
T-T I liked the old method of measuring skill better.
OBVIOUSLY this is the reason we both suck.
|
I'm an old schooler too.
It is VERY obvious why we suck now.
Ugly people just play SC better....
|
On March 08 2007 00:01 Last Romantic wrote:
In the old days, skill and good looks were direcly proportional. Good looking gamers like Reach and YellOw and BoxeR did well.
Then, at some point [circa 2004] the opposite became true. Skill and good looks became INVERSELY proportional. Now we have ugly people like sAviOr and iris dominating.
T-T I liked the old method of measuring skill better.
I will not wash till I got acne. This must help
Great discussion!
|
Hong Kong20321 Posts
On March 07 2007 22:46 GrandInquisitor wrote:
This sort of discussion is far too deep for a forum post (or even a PM =P)
However, I would claim that Starcraft is a uniquely bad game to objectively judge skill in, and I'll try to explain why.
Starcraft is hindered by the fact that there is no objective method of telling how "close" a game was - as a result, the dichotomy forces us to make big judgements we are not always comfortable with.
Win statistics are useful only in the long term and with extreme samples (savior with over 80% ZvP is a reasonably good indicator of ZvP skill). Anything smaller than that, and we run into luck. Not just PvP / ZvZ BO luck, but little stupid things that can change the game so much. The skill level is so close that little things, things that we would never consider part of a player's skill, have enormous impacts on single games - and the format of our OSL does not help, seeing as how much of an impact single games make. NaDa got to the final despite probably not being anywhere close to the 2nd best player in the league - three times he had everything on the line in a deciding make-or-break game.
But say in one of those games something stupid happened. He didn't look at his minimap for one half of a second and missed a drop that while seemingly negligible immediate impact had dire long term consequences. Had any one of those gone the other way, we would be discussing a completely different Power Rank (no NaDa in top 10, definitely), a totally different KeSPA - and yet all because he didn't see his minimap for half a second? Is that really fair? Can we place any value in this system if such small happenings mean the difference between "Second Best in the World" and "Who The Hell is He?"
Look at how badly Reach is slumping. I guarantee you he doesn't actually suck right now. But getting out of his offline qualifiers is total hell, and unless you dominate everyone else (something he obviously cannot do right now), the only way you are qualifying is through almost pure luck. Things just haven't been clicking for him, and so we don't even consider him top 30 in the world - yet if he got into OSL somehow, he almost certainly could have made it to elimination rounds and (with a lucky break) semis or even finals. Then we would label him one of the Top Five Protosses in the World. This does not reek of preciseness.
Little mistakes also hurt Starcraft because you cannot bounce back easily in a single game. At the top level, doing one thing stupid such as misclicking your drones so they stack instead of spread, causing you to lose a couple extra workers in ZvZ - you are fucked. You are not coming back.
Other sports have some version of this problem, but to a much lesser extent. A random small mistake in soccer will, in all likelihood, not change the result. FA Cups are not won and lost because someone slipped in the mud in one game. And especially, in soccer - even if you make a mistake, at worst you give up a goal and you fight to get that goal back. In Starcraft, when you make a mistake, you probably just lose the game - period.
The last thing I want to talk about (but certainly not the last thought on this topic) is matchups. Say Casy beats GoodFriend TvT. Is Casy a better player than GoodFriend? Maybe. What if GoodFriend can beat every Protoss in the world and Casy loses to people like ahk-gosu? Is Casy still better? What if there weren't really that many Protosses in the world at all? Is Casy somehow suddenly a better player because of the scarcity of other people at a particular race? That is offensive to the very definition of skill.
So what, you say. Let's define people in terms of their matchups. Ignoring the fact that this requires the creation of 9 separate Power Rankings, since it becomes absolutely impossible to compare between two different ones, this is still a hazy question because of styles. NaDa might rape Pusan's style; Casy might rape Anytime's style; GoodFriend might rape Bisu's style. Who's best? How do we make the Best of TvP list? Judge by how good Pusan/Anytime/Bisu are at PvT? How do we make that list, without already having had a Best of TvP list?
The solution most other sports have to this sort of style-countering skill is to play a lot of games against a great many people, and see how they do against a common set of opponents. So we compare everyone's performances against Pusan Anytime AND Bisu. The problem in Starcraft is twofold - not only do players not play everyone else, but also the situation is always different each time (maps, pressure on winning the game, the stage you're playing on). You're no longer comparing on a ceteris paribus basis, and it's really hard to get anything out of it at all?
Given all of this, I think subjective rankings are the only possible way we can do it. Objective rankings are results-based and tell you nothing more than who was fortunate enough to win their leagues. And while I do not doubt that in many circumstances the winner of OSL/MSL is probably pretty clearly indisputably the best, sAviOr is an unusual circumstance. Look at the other three semifinalists in OSL - we know that IriS can beat Casy (though, like I said before, we can't really say definitively that IriS is BETTER than Casy, whatever that means). Do we know that NaDa can beat IriS? Can we even really say that NaDa can beat Casy if they play again? How do you objectively judge who among these three is best? Even if IriS 3-0's NaDa as well - if IriS couldn't beat a USEast Protoss noob and NaDa could beat all P's, is NaDa a better player (even though he never showed off his TvP abilities)? (And this is a fortunate instance, where they would all play the same mirror matchup against each other.)
So all we can do is watch the games and guess. And argue, and bicker, and yet remain so far from that elusive holy grail of objectivism.
this is really what i always thought although i'm nowhere near as eloquent (maybe ;P) or motivated enough to write something up so nicely in a sc forum about all of it.
i didnt think of football as an example though,since i play tennis and often watch a lot of pro tennis where different players were clearly better than others in terms of style and mechanics or whatever you want to call it.
i also think tennis is a better comparison since it's also a 1v1 game and the post-match statistics are very similar to the sc post-match statistics minus the APM haha e.g. % of first serves in, % of second serves in, % service games won,
um i dont really know how to express the rest of what i want to say grandinquisitor if you get my drift you can elaborate on it ;P
|
There must be a better way to determine skills.
im thinkin at least for the macro part i.e. who has better macro.
I beleive some1 should make a program that takes like 100 replays of a certain MU of one player and calculate this:
- How many seconds per game there was not enough supply
- how many seconds per game were the production facilities not producing units (perhaps including SCVs for the first 15 minutes of the game)
- How many seconds per game The minerals/gas were above 400 (example)
And then u add one more number into the equation. The number of interventions player had to make because of the actions his opponent did.(countering attacks, harrasing etc.) If some1 made a decent formula out of this, i believe we could take Luna games and tell who has good macro.
I believe this would work pretty good for distancing players not _too_ close in skills, which means Korean Kespa players are out of the question.
On Kespa level, i believe its all about Self confidence(determination), good strategy play(just like Liverpool beat Barca with good strategy even tho they have 'weaker' players, i think any good toss could've beaten Savior if he played instead of Bisu and used his BO) and luck.
Those things cannot really be measured so the results are all we are left with to compare the players.
|
On March 08 2007 00:01 Last Romantic wrote:
In the old days, skill and good looks were direcly proportional. Good looking gamers like Reach and YellOw and BoxeR did well.
Then, at some point [circa 2004] the opposite became true. Skill and good looks became INVERSELY proportional. Now we have ugly people like sAviOr and iris dominating.
T-T I liked the old method of measuring skill better.
sAviOr is not ugly...
|
|
|
|
|
|
EPT
