|
I think the yes/no/if you have time options are not too good. Same with the star system - imo people (including me) are too influenced by players playing.
Example:
Let's take the game between JangBi and Leenock in the OP. JangBi is playing. JangBi wins. Do I recommend the game? Fuck yeah cause JangBi. But do I think it was a great game? Not really. It was average. That's why I think the the polls need to have words that indicate we are talking about the game itself, not players not anything else.
So, if polls with checkboxes would be possible, I think that is the best way how to do it. If not, I would say something like this is better:
Poll: Was JangBi vs Leenock game one good?Average. (1) 100% Amazing! (0) 0% Good game. (0) 0% Pretty bad. (0) 0% Don't even try to watch. (0) 0% 1 total votes Your vote: Was JangBi vs Leenock game one good? (Vote): Amazing!
(Vote): Good game.
(Vote): Average.
(Vote): Pretty bad.
(Vote): Don't even try to watch.
It is basically the same as star system, but using words that are kinda 'forcing' people to think about the quality of the game and not the circumstances surrounding it is imo better in order to get better/more fair results.
EDIT:
Also, one more thing. If we take into account that these polls are aimed mostly towards people who want to watch VoDs and only want to watch the best games, the poll may take that into account and ask it directly, like this:
Poll: Would you recommend watching JangBi vs Leenock G1 VoD?No, just don't bother, it was bad. (1) 100% Amazing game, must watch! (0) 0% Pretty good game, should watch. (0) 0% Watch if you really like one of the players. (0) 0% Meh, no point in watching really. (0) 0% 1 total votes Your vote: Would you recommend watching JangBi vs Leenock G1 VoD? (Vote): Amazing game, must watch!
(Vote): Pretty good game, should watch.
(Vote): Watch if you really like one of the players.
(Vote): Meh, no point in watching really.
(Vote): No, just don't bother, it was bad.
tl;dr = I think 5 star system is best to determine the quality of the games, but needs to have some words in it to indicate to people we are talking about the game quality and not circumstances surrounding it at all.
|
I think the 5 star system is fine, with context. Everyone should know what a 5 star game looks like. A lot of voting is done immediately after a game, so a lot of hype is involved. But a game should stand on its own merit. If it's a good enough game, people will vote it higher no matter what.
That said, I think there needs to be a system of tags for the criteria based part. Even something as 5 universal tags to add on if necessary (close, one sided, weird, cheese, new meta) based on the game for the sale of description
Regardless, simplicity is the best. Bonus for being able to deal with the inmediate postgame hype (y/n/if time is too rigid, I think).
|
I think it would be very helpful if the options in the recommended game where something along the lines of
No
Good
Great game
You can't miss this to save your life
This way it makes it easier to know which games you should actually spend time on watching the VOD, No one cares about the difference between a meh and a good game.
Also for me, the gameplay recommendation option is to much of a spoiler.
|
I think that the 5-Star Poll is the best system.
The yes/no/if you have time system is not only inaccurate, but also more influenced by personal preference (favorite player lost => Vote no). Also, the "if you have time" option is pretty useless in my opinion. If you have enough time, you could also watch 100 hours of nyan cat, but that doesnt mean that i would recommend you to do it.
The 5 Star Poll however, is probably the best way to measure the quality of games, allows to include personal preference without distorting the result to much and is also easy to use and to understand.
All other options are to clumsy and complex and would probably discourage people to vote in the poll. Also, it would probably mean extra work for you LR people, and you already do enough stuff.
also, great thread. ty lichter
|
On July 29 2013 23:46 Ammanas wrote:Poll: Would you recommend watching JangBi vs Leenock G1 VoD?No, just don't bother, it was bad. (1) 100% Amazing game, must watch! (0) 0% Pretty good game, should watch. (0) 0% Watch if you really like one of the players. (0) 0% Meh, no point in watching really. (0) 0% 1 total votes Your vote: Would you recommend watching JangBi vs Leenock G1 VoD? (Vote): Amazing game, must watch!
(Vote): Pretty good game, should watch.
(Vote): Watch if you really like one of the players.
(Vote): Meh, no point in watching really.
(Vote): No, just don't bother, it was bad.
I really like this concept, I'd prefer this over the 5-star system.
Problems could be, that it could get really awkward when having only a few people voting on it and everyone chooses a different option.
I'm also not sure how JuicyJames could implement this into his weekly roundup, finding out which amount of votes on which options would make it a game worth recommending would need some time to figure out.
It depends a lot on how people react to this kind of poll but I think this idea is worth giving it a try.
|
1001 YEARS KESPAJAIL22272 Posts
On July 30 2013 01:19 Yello wrote:Show nested quote +On July 29 2013 23:46 Ammanas wrote:Poll: Would you recommend watching JangBi vs Leenock G1 VoD?No, just don't bother, it was bad. (1) 100% Amazing game, must watch! (0) 0% Pretty good game, should watch. (0) 0% Watch if you really like one of the players. (0) 0% Meh, no point in watching really. (0) 0% 1 total votes Your vote: Would you recommend watching JangBi vs Leenock G1 VoD? (Vote): Amazing game, must watch!
(Vote): Pretty good game, should watch.
(Vote): Watch if you really like one of the players.
(Vote): Meh, no point in watching really.
(Vote): No, just don't bother, it was bad.
I really like this concept, I'd prefer this over the 5-star system. Problems could be, that it could get really awkward when having only a few people voting on it and everyone chooses a different option. I'm also not sure how JuicyJames could implement this into his weekly roundup, finding out which amount of votes on which options would make it a game worth recommending would need some time to figure out. It depends a lot on how people react to this kind of poll but I think this idea is worth giving it a try.
Purely descriptive polls are difficult to turn into a score, yes. Especially when they have many options. That's why I want to avoid it. It's also a slippery slope to assign each option with a number value (5,4,3,2,1, essentially the same as 5-star) since the 'description' might not follow that kind of distribution of 'recommendedness'.
I will reply to the other comments in the morning.
|
I think the whole OP, while obviously a lot of effort was put into it, was written with a faulty presumption, i.e. a poll can be accurate. A poll with fixed answers will always be imprecise, that is the nature of fixed answers.
Recommended game polls are intended as a quick reference to decide whether or not to watch a game, the current system does just that and any added level of distinction will only increase the time needed to make this decision, while still falling short in accuracy.
|
On July 30 2013 01:36 nimbim wrote:
I think the whole OP, while obviously a lot of effort was put into it, was written with a faulty presumption, i.e. a poll can be accurate. A poll with fixed answers will always be imprecise, that is the nature of fixed answers.
Recommended game polls are intended as a quick reference to decide whether or not to watch a game, the current system does just that and any added level of distinction will only increase the time needed to make this decision, while still falling short in accuracy.
I think a poll can be accurate enough. It doesn't need to be perfect, but it needs to provide enough differentiation to distinguish a decent game from a great game. I think the current system is good at weeding good from bad, but not great at weeding decent games from great games. Of course the tension is between providing enough differentiation while maintaining enough accuracy. I think expanding the system from a three point system to a five point system of some kind could help make the recommendations more useful (for the most part - it will likely make things worse for fairly inactive LR threads with low votes).
I actually like Ammanas' suggestion of having a descriptor based 5-point system rather than just a star system. I realize that it doesn't translate nicely into numeric values, but I personally have no problem ham-fisting the results from the ordinal scale into a numeric scale. Yeah, we all know it isn't perfect, but it doesn't need to be perfect, it just needs to be able to accomplish it's goal which I think it can do. I don't think a 5-point scale is more difficult to read than a 3-point. I think the biggest thing is that you can't have a balanced 5-point scale where 1 and 2 are negative, 3 is neutral, and 4, 5 are positive. I would bet that people would see very little differentiation between 1 and 2 and so it would essentially be a 4 point system. I mean, does anyone really care about if the game is 'pretty bad' vs 'really bad'? Again, it makes translating to a numeric score less accurate, but I do think that it should be good enough for most purposes while maintaining an easy to use system for both the voter and the person deciding if they should watch a game.
|
I personally prefer the yes/no/if you have time option. It's clear, and while it doesn't tell you why a game is worth watching, it does tell enough. If its due to casting, gameplay, closeness of the game, or whatever, doesn't really matter to me. Another benefit of the yes/no/if you have time system that its pretty easy to judge whether a large percentage of votes on one side or the other is biased in favor of one or the other player. .
The star system on the other hand is already way too complicated. I have no idea what it means when a game has 10% one star, 20% two star, 30% three star 30% four star and 20% five star. What I end up doing is adding up the 4 star and 5 star bar and count it as 'yes' in the old poll.
Keep in mind also that the more complicated you make the poll and the more complicated you make the questions, you generally get an even lower turnout and your entire poll becomes less accurate because more people have more questions to misinterpret.
|
On July 30 2013 02:17 jakethesnake wrote:Show nested quote +On July 30 2013 01:36 nimbim wrote:
I think the whole OP, while obviously a lot of effort was put into it, was written with a faulty presumption, i.e. a poll can be accurate. A poll with fixed answers will always be imprecise, that is the nature of fixed answers.
Recommended game polls are intended as a quick reference to decide whether or not to watch a game, the current system does just that and any added level of distinction will only increase the time needed to make this decision, while still falling short in accuracy. I think a poll can be accurate enough. It doesn't need to be perfect, but it needs to provide enough differentiation to distinguish a decent game from a great game. I think the current system is good at weeding good from bad, but not great at weeding decent games from great games. Of course the tension is between providing enough differentiation while maintaining enough accuracy. I think expanding the system from a three point system to a five point system of some kind could help make the recommendations more useful (for the most part - it will likely make things worse for fairly inactive LR threads with low votes). I actually like Ammanas' suggestion of having a descriptor based 5-point system rather than just a star system. I realize that it doesn't translate nicely into numeric values, but I personally have no problem ham-fisting the results from the ordinal scale into a numeric scale. Yeah, we all know it isn't perfect, but it doesn't need to be perfect, it just needs to be able to accomplish it's goal which I think it can do. I don't think a 5-point scale is more difficult to read than a 3-point. I think the biggest thing is that you can't have a balanced 5-point scale where 1 and 2 are negative, 3 is neutral, and 4, 5 are positive. I would bet that people would see very little differentiation between 1 and 2 and so it would essentially be a 4 point system. I mean, does anyone really care about if the game is 'pretty bad' vs 'really bad'? Again, it makes translating to a numeric score less accurate, but I do think that it should be good enough for most purposes while maintaining an easy to use system for both the voter and the person deciding if they should watch a game.
^ I actually think 4 point system would work much better then 3 or 5 point one. Just wanted to stick to the '5 stars' example. Reason being precisely what you said - for this specific issue, there is almost no difference between 'pretty bad' and 'very bad'.
|
Norway25712 Posts
I like the yes/no/if you have time polls.
It's simple and an easy way to know if something's worth watching or not. The only issue is that people tend to auto-vote "if you have time" when the games are somewhat long.
Any point-based system gets too complicated imo. It hink people will use 5 stars for yes, 1 for no and 3 for meh. I don't see 2 stars being used a whole lot, but 4 might get used a lot for decent games. The problem is that some people will think of 5 as an AMAZING game, while others will thik of it as a "you should watch this"-game, which could mean a lot of games will have a pretty even split between 5-star and 4-star votes. Yes/No/Maybe is nice because with a poll that has sufficient amounts of votes it's easy to tell if a game is worth watching or not instantly.
|
I prefer the simple yes/no/if you have the time with variations on the wordings. It is simple, quick, and informative. The latter, because it gets differentiated by the percentage. A 95% Yes is obviously a recommendation (5 stars), while a 60% Yes is rather average (3 stars). More options/complexity does not add more information for me, really, but tends to discourage voting.
|
|
|
United States97276 Posts
|
United States33390 Posts
popular polls will always be flawed and limited as a measure of a game's real quality
just take them for what they are and keep them simple
|
I believe that the ideal solution is a combination of Y/N/I and star rating polls. Having an initial Y/N/I poll at the top, then maybe put the star rating poll underneath, in spoilers. That way, one could simply vote and be done with it, or if they desire, they could go into greater detail in the star rating poll. Furthermore, since both polls gather different information, you could combine them into one rating.
For example:
90% recommend game 1 [★★★★★]
60% recommend game 2 [★★★]
10% recommend game 3 [★]
Additionally, a system such as this would help eliminate bias in poll results. One common observation is that when the more popular player loses, less people recommend the game.
Say that Jaedong is facing Sniper at the finals of a major tournament in the last game of a BO7. Sniper decides to 10 pool, and after a long and exciting back-and-forth micro battle, Sniper barely wins. Chances are, most of the people in the LR thread won't be happy, so they don't recommend that game. However, the micro battle made for a great game.
Such a result could look like this:
20% recommend Sniper vs Jaedong [★★★★]
Even though less people would recommend the game, it was still a good game. Hopefully, this would help eliminate some of the fans' bias, and would encourage more thought after such a simple poll. Especially if the star rating poll were in spoilers, since then most people wouldn't vote for it.
Of course such a system has it's faults. If the star rating poll is in spoilers, then perhaps too few would vote in it to make a decision. Additionally, people could still vote down the rating due to the outcome. Finally, making the polls would be twice as much work for the LR thread.
Even with those faults, I believe that this would be the most adequate way to quickly judge a game.
|
opterown
Australia54784 Posts
some of the polls here are rather complex :s
|
1001 YEARS KESPAJAIL22272 Posts
On July 29 2013 23:22 juicyjames wrote:+ Show Spoiler +On July 29 2013 14:51 lichter wrote:Show nested quote +On July 29 2013 14:37 juicyjames wrote:On July 29 2013 11:48 lichter wrote:
Why We Want A Better Recommendation System
2.) The This Week in SC2 thread will have more accurate recommended games and he won't have a hard time figuring out which games really should be recommended; On July 29 2013 14:17 lichter wrote:
The reason I prefer the polls to be uniformly done is so that TWiSC2's recommended games will have consistency, and that somewhere down the line maybe we can have a database of recommended games--an incredibly useful utility. If the polls aren't standard, there's no way to make one. As the person who does This Week in Starcraft 2, I appreciate your mentioning of my thread(s) when doing this. I rely almost exclusively on recommended game polls to determine which matches get listed. With the old Y/N/I system I had two requirements: 1) 80%+ Recommended 2) 10+ People Voted Yes See: http://www.teamliquid.net/forum/viewmessage.php?topic_id=422910I reasoned that any match worth watching would be able to surpass both requirements. If polls are not available I will either post in the thread asking if anyone recommended any games, or see if anyone made Reddit threads about any specific. Now, I have no problems with changing the recommended games polls. All I ask is that if something new gets decided upon, everyone also decides on what threshold needs to be broken in order to be included in This Week in Starcraft 2. For example, in a criteria-based poll, what exactly will determine if its a good enough game to be recommended? This is not a criticism of this particular poll, just something for everyone to consider once you decide which poll to use. I was going to name you specifically but wasn't sure since some of the threads were being made by shiroiusagi :D I have a few more questions for you: do you think that the results you get from looking at recommended game polls are accurate? Do the best games get the best ratings? Are the best games and merely good games graded differently? Do duds get into the list because of their ratings? Do some good, should be recommended games miss out because their rating isn't high enough? The reason I started questioning the accuracy of the poll was really because of Solar vs Rain compared to Flash vs Fantasy. I love both games but don't believe they are equal. I'm also pretty anal about ratings being accurate lol In the criteria based poll, ideally we need to get R1CH to give us checkbox polls. A radio button for Yes/No, then a checkbox to recommend something in particular. So you'd get info such as 85% Yes 80% Level of Play 80% Entertainment value 50% Build Orders 60% Sick Micro 15 % No So the game is 85% recommended, with 80% recommending it because of level of play and entertainment value. Of course this is dependent on being able to make hybrid radio/checkbox polls T_T They are accurate enough for most people to use as an at a glance tool to see whether or not they should go back to watch VODs, but they are not without their flaws. The greatest indicator of a "must see" game for me is probably when it has 100+ people voting Yes, but the problem is any match that has that many votes will almost never be 100% recommended. The 100% recommended games are often matches with barely 10 votes. http://www.teamliquid.net/forum/viewmessage.php?topic_id=278126http://www.teamliquid.net/forum/viewmessage.php?topic_id=422034So in an absolute sense, no, the best games do not get the best ratings. The fewer people that vote, the more inaccurate the ratings will be. 1-3 people voting No or If you have time can dramatically drag down a low-voted game, or, conversely, give you 100% recommended games that really are not better than games where 98% of like 120+ people voted yes. This is why I include both percents and absolute numbers. I agree, it is hard to separate "the best" vs "good" games. If I had to separate them currently I would go back to what I said and say anything with 100+ people voting Yes would be "the best." Do good games not make the list? Yes. If a tournament is on at a bad time (Code A), has a lot of unknowns in it (Code A), or has a dead LR (WCS AM, which can have anywhere from 15k-25k viewers but a largely inactive LR) there will be a lot of games that may not even have 10 votes total. http://www.teamliquid.net/forum/viewmessage.php?topic_id=420595http://www.teamliquid.net/forum/viewmessage.php?topic_id=421947However, most of the time good games will cause people to vote on the polls (I hope?). Polt vs TaeJa is a good example of this. http://www.teamliquid.net/forum/viewmessage.php?topic_id=422354Do duds make it no the list? Sure, but I would wager anyone actually looking at recommended game polls probably are only going to watch certain matchups or certain pros to begin with. There are way too many choices for people to simply watch random games recommended games. Honestly, WCS created so much SC2 content I actually made my requirements for being listed as a recommended game slightly more strict. I used to list 75%+, 10+ recommended, but now bumped it up to 80%+, 10+ recommended. I still toy with the idea of even going up to 80%+, 15+, but it would probably destroy WCS AM. http://www.teamliquid.net/forum/viewmessage.php?topic_id=422034So, in the end, yes it would be nice if there were a better system of recommended games to clearly separate "must watch" games from merely "good" games. At the very least maybe somehow differentiate between games recommended as a spectator and games recommended as a player of X or Y race .
I think that what we can gather definitively is that the number of votes makes a big, big impact on how games are recommended. On the one hand, more votes means the game will have a rating that averages out better; on the other, more votes also means more idiots screwing up results. However as you mentioned, by combining the score and number of votes you are able to tell good from great. If only there were a way to combine it somehow. Perhaps if the effect of the number of votes on the score were exponential instead of linear (more votes leads to a quicker rise in rating), we could remove both problems? I just woke up from my morning nap so I will think about this further later.
On July 29 2013 23:46 Ammanas wrote:
On July 29 2013 23:47 Juliette wrote:
On July 30 2013 00:32 Paljas wrote:
I think a lot of people agree with you guys that 5-star is nice, if only we could remove that added level of bias and subjectivity. I guess I will try to use 5-star+description in my future polls and see how people like it.
On July 30 2013 01:36 nimbim wrote:
Polls can never be perfectly accurate, especially when there is nothing great at stake forcing people to really educate themselves about their votes. But that doesn't mean we shouldn't try to improve what we already have. I am not advocating for great change--because perhaps it is not necessary--but if we can find small improvements, such as making polls more visible or attractive, then they should be implemented if possible. While I do mention accuracy, the polls about polls asks about effectiveness, which takes into consideration all the flaws and circumstances surrounding the poll. You make the faulty assumption that accuracy is all we want. If we did then we'd have a 100 item 5-star poll that all TL users must answer after each game or get banned. Accurate, but ineffective. :p
On July 30 2013 02:22 Derez wrote:
The star system on the other hand is already way too complicated. I have no idea what it means when a game has 10% one star, 20% two star, 30% three star 30% four star and 20% five star. What I end up doing is adding up the 4 star and 5 star bar and count it as 'yes' in the old poll.
What if the results are shown not as a distribution but as an average? In your example it will show 3.6 stars out of 5. Would that help?
On July 30 2013 02:24 KristofferAG wrote:
Any point-based system gets too complicated imo. It hink people will use 5 stars for yes, 1 for no and 3 for meh. I don't see 2 stars being used a whole lot, but 4 might get used a lot for decent games. The problem is that some people will think of 5 as an AMAZING game, while others will thik of it as a "you should watch this"-game, which could mean a lot of games will have a pretty even split between 5-star and 4-star votes. Yes/No/Maybe is nice because with a poll that has sufficient amounts of votes it's easy to tell if a game is worth watching or not instantly.
Would this problem be solved by adding descriptors beside each star?
On July 30 2013 07:53 Monochromatic wrote:
I believe that the ideal solution is a combination of Y/N/I and star rating polls. Having an initial Y/N/I poll at the top, then maybe put the star rating poll underneath, in spoilers. That way, one could simply vote and be done with it, or if they desire, they could go into greater detail in the star rating poll. Furthermore, since both polls gather different information, you could combine them into one rating.
For example:
90% recommend game 1 [★★★★★]
60% recommend game 2 [★★★]
10% recommend game 3 [★]
Additionally, a system such as this would help eliminate bias in poll results. One common observation is that when the more popular player loses, less people recommend the game.
Say that Jaedong is facing Sniper at the finals of a major tournament in the last game of a BO7. Sniper decides to 10 pool, and after a long and exciting back-and-forth micro battle, Sniper barely wins. Chances are, most of the people in the LR thread won't be happy, so they don't recommend that game. However, the micro battle made for a great game.
Such a result could look like this:
20% recommend Sniper vs Jaedong [★★★★]
Even though less people would recommend the game, it was still a good game. Hopefully, this would help eliminate some of the fans' bias, and would encourage more thought after such a simple poll. Especially if the star rating poll were in spoilers, since then most people wouldn't vote for it.
Of course such a system has it's faults. If the star rating poll is in spoilers, then perhaps too few would vote in it to make a decision. Additionally, people could still vote down the rating due to the outcome. Finally, making the polls would be twice as much work for the LR thread.
Even with those faults, I believe that this would be the most adequate way to quickly judge a game.
That's an interesting idea but I'm not sure how it removes bias. Angry anti-fans fueled by their rage can easily ruin the results of the spoilered 5-star poll. And as you said, fewer people are likely to vote on it because it is hidden and just an 'added layer' of voting. Less votes; more variance. Anti-fans have greater motivation in ruining game recs (it happens often  ) so their votes might end up skewing results even more in such a poll. It is an idea though and maybe it can be explored further.
|
I think you are fighting against the way people use the polls, YouTube used 5 star ratings but dropped it because people overwhelmingly just voted 5s and 1s. Also I think making the polls more complicated is going to lower participation too.
|
1001 YEARS KESPAJAIL22272 Posts
On July 30 2013 13:34 BathTubNZ wrote:
I think you are fighting against the way people use the polls, YouTube used 5 star ratings but dropped it because people overwhelmingly just voted 5s and 1s. Also I think making the polls more complicated is going to lower participation too.
That's what this thread is for: to determine if people overwhelmingly prefer Y/N/I, or if there is some way to improve other poll formats to make them acceptable. I am not prescribing we use one method or another. I present all that is available as well as possibilities and ask you to decide what you think it best. If that means we stick with Y/N/I then sure, I'm fine with that. But discussion can lead to improvements so I hope people continue to talk about it and not just assume Y/N/I is best. 
|
|
|
|
|
|
EPT
![[image loading]](http://www.teamliquid.net/staff/flamewheel/cheep.gif)
![[image loading]](http://www.teamliquid.net/staff/flamewheel/halfsheep.gif)
![[image loading]](http://www.teamliquid.net/staff/flamewheel/emptysheep.gif)
![[image loading]](http://i.imgur.com/LZs2lNC.png)
![[image loading]](http://i.imgur.com/O6t7RJX.png)
![[image loading]](http://i.imgur.com/sVnSssP.png)
![[image loading]](http://i.imgur.com/T9aScv5.png)
