|
On August 02 2009 07:04 azndsh wrote:Show nested quote +On August 02 2009 06:58 VIB wrote:On August 02 2009 06:38 azndsh wrote:
if anything, this thread has proved that people are overconfident in their reading comprehension skills and/or ability to understand social experiments. but after all this is the internet, so I guess this is anything but surprising. And when you came to that conclusion your own overconfidence blinded you from considering that maybe, just maybe... It is you who cannot understand how inefficient this experiment is for measuring confidence, and not the ones saying it's wrong.  on the contrary... people not being able to read correctly and this experiment being inefficient can both be true at the same time. maybe, just maybe... I do understand the inadequacies of asking people online to follow directions and evaluating themselves, but was nevertheless a little disappointed by how badly people misunderstood it and the number of smartass comments that followed.
Well you did kind of sabotage yourself with that introduction. Stop blaming your test subjects.
|
The point of this thread isn't for me to measure how "overconfident" TL is... it's to share with TL an interesting facet of human behavior and a simple way to demonstrate it. This test doesn't measure confidence so much as it reveals how people can make estimates that are far worse than what they think it is.
People claim that the answers were too surprising or irrelevant. It doesn't matter if people have no clue, they should factor that in when they're giving a 90% sure answer. In any case, it seems to me that lots of people make estimates about things that they don't know that much about.
Others claimed that it wasn't using the metric system. Well it's really not that hard to do some simple arithmetic and figure out your 90% confidence interval on what the conversion ratio actually is (or what your arithmetic ability is). That's part of the test as well.
For those who "got it", I can only hope that they were able to take away something from it. Simple as that.
|
On August 02 2009 07:11 Chef wrote:Show nested quote +On August 02 2009 07:04 azndsh wrote:On August 02 2009 06:58 VIB wrote:On August 02 2009 06:38 azndsh wrote:
if anything, this thread has proved that people are overconfident in their reading comprehension skills and/or ability to understand social experiments. but after all this is the internet, so I guess this is anything but surprising. And when you came to that conclusion your own overconfidence blinded you from considering that maybe, just maybe... It is you who cannot understand how inefficient this experiment is for measuring confidence, and not the ones saying it's wrong. on the contrary... people not being able to read correctly and this experiment being inefficient can both be true at the same time. maybe, just maybe... I do understand the inadequacies of asking people online to follow directions and evaluating themselves, but was nevertheless a little disappointed by how badly people misunderstood it and the number of smartass comments that followed. Well you did kind of sabotage yourself with that introduction. Stop blaming your test subjects.
I said that
a) the OP was misunderstood a lot
b) it seems to me that this thread did, after all, show people are overconfident
can't really figure out the part where i'm blaming anything
|
I didn't realize there were so many books in the old testament. O_O
|
On August 02 2009 07:20 azndsh wrote:
The point of this thread isn't for me to measure how "overconfident" TL is... it's to share with TL an interesting facet of human behavior and a simple way to demonstrate it. But that's what I'm pointing out. You're not even demonstrating overconfidence. You're demonstrating something else and presenting it as a simple way to demonstrate overconfidence. That's what is "wrong" about it. You could have rolled a dice 10 times and told people you're demonstrating overconfidence. No, you're just demonstrating a dice rolling and presenting it as something else.
But if your goal was just to show people that apples exist by presenting them with oranges then you succeeded I guess.
|
"overconfidence" is a psychology term. from wiki: The overconfidence effect is a bias in which people are correct in their judgments far less often than they think they are.
|
On July 31 2009 00:58 Chill wrote:
This is stupid because the answers in themselves are shocking. Sure you should account for that in your interval, but the test is pitted against you. If they were questions with reasonable answers in everyday things this would hold a lot more weight for me.
That's basically what I was thinking. It's written better by Chill though, I couldn't figure out how to say it.
|
South Africa4316 Posts
On August 02 2009 07:31 VIB wrote:Show nested quote +On August 02 2009 07:20 azndsh wrote:
The point of this thread isn't for me to measure how "overconfident" TL is... it's to share with TL an interesting facet of human behavior and a simple way to demonstrate it. But that's what I'm pointing out. You're not even demonstrating overconfidence. You're demonstrating something else and presenting it as a simple way to demonstrate overconfidence. That's what is "wrong" about it. You could have rolled a dice 10 times and told people you're demonstrating overconfidence. No, you're just demonstrating a dice rolling and presenting it as something else. But if your goal was just to show people that apples exist by presenting them with oranges then you succeeded I guess.
VIB, do you mind explaining what is wrong with the test. If possible, make a list, so that your points can be addressed one by one. Also, if possible, could you explain how the test works in a few words, just so that we are sure we're discussing the same test.
|
bullshit on anyone answering all 10 correctly
|
And another pseudo-intellectual thread on TL totally blows up.
|
With how you phrased the task it seems like everyone would just put 0 - one trillion
|
On August 03 2009 09:06 Kaysin wrote:
With how you phrased the task it seems like everyone would just put 0 - one trillion
No it doesn't. You would only answer this way if you didn't read the entire post, or if you don't understand what a 90% C.I. is.
You should select your ranges such that you think it is exactly 9 times more likely that the correct answer falls within your predicted range than outside of it.
|
On August 03 2009 03:40 Daigomi wrote:Show nested quote +On August 02 2009 07:31 VIB wrote:On August 02 2009 07:20 azndsh wrote:
The point of this thread isn't for me to measure how "overconfident" TL is... it's to share with TL an interesting facet of human behavior and a simple way to demonstrate it. But that's what I'm pointing out. You're not even demonstrating overconfidence. You're demonstrating something else and presenting it as a simple way to demonstrate overconfidence. That's what is "wrong" about it. You could have rolled a dice 10 times and told people you're demonstrating overconfidence. No, you're just demonstrating a dice rolling and presenting it as something else. But if your goal was just to show people that apples exist by presenting them with oranges then you succeeded I guess. VIB, do you mind explaining what is wrong with the test. If possible, make a list, so that your points can be addressed one by one. Also, if possible, could you explain how the test works in a few words, just so that we are sure we're discussing the same test. I already posted it some pages ago, maybe you missed it. It's past bed time for me now so I'll just be lazy and copy/paste what I already posted:
+ Show Spoiler +On July 31 2009 06:48 Daigomi wrote:Show nested quote +On July 31 2009 06:36 VIB wrote:
I'm amazed at the amount of people taking this test seriously O.O
Oh less than 1% of the people cannot calculate precisely off the top of their head what 90% off something I don't know exactly is. That is sooo shocking. I definitely proved something new and important there. ¬¬ If you didn't look at my profile, and I asked you to guess my age, but provide a minimum and maximum age, would you be able to come up with a range that you'd be 90% sure about? No I wouldn't, no human being possibly would. That is what you're missing. I have no way to calculate if I could be 90% sure of. There are many missing variables. I would have to guess what 90% is. And a guess is just that. A guess. Nothing else. You're trying to read too much into it. If you think you have any slightest idea of what the 90% of something you don't know is then you're over-confident already. And that I'm 100% sure of  Roll a 6-sided dice 10 times and the chance of guessing it right 9 out of 10 times is still less than 1%. This is what the test is accomplishing. The only difference is that the dice size varies randomly from one person to another. I could tell you the results would be less than 1% before anyone did that test, it's not shocking nor surpring nor new nor important nor anything other than a bunch of people being unlucky.
|
On July 31 2009 00:58 Chill wrote:
This is stupid because the answers in themselves are shocking. Sure you should account for that in your interval, but the test is pitted against you. If they were questions with reasonable answers in everyday things this would hold a lot more weight for me.
Which question, might I ask, had a shocking answer?
The elephant?
Large mammals have, as you might expect, longer gestation periods.
For example: a giraffe is 14 months...
|
|
|
On August 02 2009 07:29 Mortality wrote:
I didn't realize there were so many books in the old testament. O_O
Lol god visits more frequently in the good ol days and each time he visits someone writes a book
|
I am surprised so many people fail at this test, no feeling for percentages or probability?
I would be more likely to overestimate the intervals, but maybe having a scientific background changes my perception of numbers and possible errors.
P.S.: I don´t care for any excuses, use the metric system 
|
This test is actually quite interesting and revealing if you bother to understand what it's about. It requires the subject to understand what a 90% confidence interval is, which is already a pretty serious demand, and it requires the subject furthermore to think very carefully about his answers, because coming up with a satisfactory 90% confidence interval is a hell of a lot harder than just a plain "do you know the answer" question. So it's hard to get a decent sample of responses.
I tried this on my friend as well; I came up with 10 quantitative questions and researched the answers online, and gave him the questions after explaining the test. He got 6 out of 10 and his incorrect intervals were always barely too small, meaning that he's consistently a tiny bit overconfident. He then tested me in a similar way and I proved quite overconfident of my answers, despite knowing already that overconfidence was the general tendency.
Anyway, this was fun and educational so thanks for making this topic
|
South Africa4316 Posts
Show nested quote +On July 31 2009 06:48 Daigomi wrote:On July 31 2009 06:36 VIB wrote:
I'm amazed at the amount of people taking this test seriously O.O
Oh less than 1% of the people cannot calculate precisely off the top of their head what 90% off something I don't know exactly is. That is sooo shocking. I definitely proved something new and important there. ¬¬ If you didn't look at my profile, and I asked you to guess my age, but provide a minimum and maximum age, would you be able to come up with a range that you'd be 90% sure about? No I wouldn't, no human being possibly would. That is what you're missing. I have no way to calculate if I could be 90% sure of. There are many missing variables. I would have to guess what 90% is. And a guess is just that. A guess. Nothing else. You're trying to read too much into it. If you think you have any slightest idea of what the 90% of something you don't know is then you're over-confident already. And that I'm 100% sure of Roll a 6-sided dice 10 times and the chance of guessing it right 9 out of 10 times is still less than 1%. This is what the test is accomplishing. The only difference is that the dice size varies randomly from one person to another. I could tell you the results would be less than 1% before anyone did that test, it's not shocking nor surpring nor new nor important nor anything other than a bunch of people being unlucky.
Well, I still fail to see your problem to be honest. If I asked you what your chances are of a rolling a 6 on a die roll, you would say 17% right? That's basically the same as saying that if you'd be willing to bet on 1:6 odds of rolling a six.
Now, if I asked you what Brazil's chances are of losing a football game against Japan, would you be willing to give me odds on that? Let's say I say it's 50:50, you would probably say those odds are way too high, it should be roughly 20:80, or maybe even 10 0, because you think that Brazil would draw or win 9 of the 10 matches they play against Japan. This test does exactly the same thing. It doesn't measure if you know the answer or not. It assumes that you don't know the answer, and then asks to you to give a range which you would be 90% sure that the answers falls into, or in other words, a range on which you would be willing to take a 9:1 bet that the answers falls inside of. For example, I have no idea how old you are, but I am willing to accept a 9:1 bet that the you are between 15 and 45 years old. I'm not necessarily going to be right, but I am 90% confident that I am right with that range.
The test is also not a measure of how good your ranges are. Some people are less confident and they choose huge ranges. Other people are more confident, and they choose small ranges. The point of the test is that when we say we are 90% sure of something, we tend to be wrong way more than 10% of the time. For example, if I had to guess the age of 10 people like you, and I chose a range of 18-30 that I feel 90% confident in (because they sound educated, so I assume they're at university, while at the same time, this is a SC forum so most people won't be too old) then I might get it right with you, and with 6 other guys, but get it wrong for 3 guys. What this means is that I was 20% too confident in my prediction, and I should have increased the range to 15-35, or 12-45.
Realise that in this situation there are thousands of variables missing. I don't know you, I don't know how you look, I don't know if you even come from Brazil. The only data I have is the way you chat on a forum. Using that data, I must then choose a range I'm confident in. 90% simply means 9 out of 10 times I should be right.
Also, even though the test is called a measure of overconfidence, it is rather an illustration that people are overconfident. It doesn't measure how much people are overconfident. What the purpose of the test really is, is to show that when people say they are 90% confident, they really should only be 50% confident, because they're wrong 50% of the time on things they are 90% confident about.
And just so that you know, the test was designed by Prof. Russo and Prof. Schoemaker. Russo is a prof at Cornell, and if I remember correctly, he did his BA in maths, his masters in statistics, and his PhD in cognitive psychology. Schoemaker did a BS in physics, then an masters in management, an MBA in finance, and a PhD in decision making. So with that in mind, I'm 90% (or at least 70%) confident that the people in this thread that say the test is useless are misunderstanding some part of it, rather than it being a mistake on the test's part. That's not to say that the test is flawless, I commented on a few of the problems with the test in a previous post. However, what I can say is that the test is a basic measure of what it intends to measure.
|
Daigomi, try to notice the difference between your 2 first examples:
A)
On August 03 2009 18:10 Daigomi wrote:
If I asked you what your chances are of a rolling a 6 on a die roll, you would say 17% right? That's basically the same as saying that if you'd be willing to bet on 1:6 odds of rolling a six.
B)
On August 03 2009 18:10 Daigomi wrote:
Now, if I asked you what Brazil's chances are of losing a football game against Japan, would you be willing to give me odds on that?
[...]
I'm not necessarily going to be right, but I am 90% confident that I am right with that range. On A) you know the odds are beforehand. The dice has 6 sides, so betting on one is 1:6. That is an objective assumption. You calculated that, so you're safe. You can picture exactly in your head what 16.6% is. But on B), when you say "90% confident that I am right". Where does that number came from? You cannot calculate that what "90% confidence" is. You don't have the variables for that. So you guessed a range that gives you a warm and fuzzy feeling inside that sounds 90%'ish. If I asked you the same question at a different time. Depending on your mood, depending on what random memories are going through your head right now. Your guess could change. Asking you "what do you think my age is", right NOW you could give me a 15-35. But maybe tomorrow you'd guess 18-40, maybe after drinking coffee you would give me a 12-35. Maybe after remembering my last troll post on TL you'd give me a 10-30.
What I'm trying to say is. You don't know what 90% confidence is. You cannot give me a trustworthy safe data to work on my sampling if you don't know how to provide me such data.
To make this more clear let me try to give you a very over-exaggerated bad analogy:
- Assume a citizen that isn't either over or underconfident. He's a fair person, and theoretically a good "confidence test" should rate him us such. If this guy made the test you proposed, assuming the test is good. He should get 9 answers correct all the time.
- Ask this honest guy this question "Give me an interval that you are exactly 83.76% sure the winning number of Lottery #1 is. You don't know what ranges the lottery number are picked from." Ask this same question 10000 times for 10000 different lotteries, with 10000 different winning numbers.
- Assuming my theory is right and my test is good, this fellow citizen should get 8376 out of 10000 ranges correct.
- Now, do you really think he would get anywhere near 8376 answers correct regardless of how honest he his? If he misses by 20%, does that means he is 20% over/under-confident, or does it means he simply got unlucky guesses?
Of course this is a big exaggeration. But it shows you the two points where I disagree with the test's method. There are two axioms that you build at the start of the test that I disagree with:
1) The test subject can calculate off the top of his head what 90% of an unknown value is.
2) The test subject can provide a reliable confidence range off the top of his head to an unknown answer.
You cannot prove the above two mathematically. You don't have the variables for that. Those are axioms your test is assuming from the start. If those are true then the test is valid. But I don't honestly agree too much with those. On the standard scientific method you build your axioms -> calculate a theory based on those axioms -> create a test to try the theory -> test the theory. If the test fails the intended results = the theory is invalid. But if the test matches the theory, then it doesn't mean the theory is necessarily right. Instead, it could mean two things: 1) The theory is right for all possible tests or 2) The theory is only right for that test, but could fail other tests, so it's globally invalid. You will consider 1) to be correct for practical uses until someone comes up with a different test that disproves your theory.
In this case. The theory passed the test. If the theory is right, then a very low number of people should get 9/10 answers correct, and according to the OP, only 1% did. But that doesn't mean the test is correct. I'm pretty sure that if you had asked the test subjects to roll a dice of a random size 10 times instead of asking those 10 questions. The results would be very similar. Does that mean rollings a dice is effectively measuring how confident one can be? I think not.
Actually, I'm pretty sure that if you repeated the same test with the same person, but with completely different questions, at completely different days and times. The results from the first and the test would vary a lot more often then not. That would objectively disprove the test, I think
Now about the authors' credibility. I'm gonna say something is not completely relevant to what we're talking. But it's so funny that I'm gonna post it anyway:+ Show Spoiler +
On August 03 2009 18:10 Daigomi wrote:And just so that you know, the test was designed by Prof. Russo and Prof. Schoemaker. Russo is a prof at Cornell, and if I remember correctly, he did his BA in maths, his masters in statistics, and his PhD in cognitive psychology. Schoemaker did a BS in physics, then an masters in management, an MBA in finance, and a PhD in decision making.
I've had this professor some years ago who was a phd in statistics. He was pretty well known around here because of his veeeery unconventional style and his. He often bragged about all his awards on mathematics "contests" and "olympic tests" (not sure how those are called in english) and how he could solve any complex trigonometry problem using only Tales and Pythagoras. So anyway, we happen to have heard many that he used to have serious money problems because of gambling. But for someone who is phd is fucking statistics that sounded more like gossips. Until one day, during class he was trying to prove that the odds of a specific sequence to happen was so rare. That he pulled a dice he had in his pocket, asked a girl in the front row to roll the dice x times and said that if numbers matched such sequence he would he would approve everyone in the final exams. Well, the girl rolled the dices, got the numbers correct and now he is all desperate begging us not to tell this to anyone because he could get fired and all and how he needed money because he lost so much to gambling already lol And that's how I passed in statistics Not trying to imply anything about the authors of the test. I don't know them. Just saying you should always be skeptical about anyone
|
|
|
|
|
|
EPT
