Smarter people more prone to cognitive bias - Page 8

Hmm, so I had a look at the paper (thanks) and it's hard to articulate how much of a mess this seems to be. The authors write like vicious weasels and I have never seen a more potentially biased paper in any respectable publication.

The selection of data presented gives the impression of a really suspect pattern of entirely avoiding any results that could possibly be harmful to the authors' conclusion. The bit on the second hastily addressed study lacks what would seem to be very necessary and relevant discussion on why and how the participants differ so much from the participants of the first study of all college students, and avoids the majority of the "bias questions" the authors purport to study in the first place.

The first thing I took away, and cannot stress enough, is how subpar the students involved in the study are. To anyone who wasn't sure or thought this was an "ivy-league" school or anything similar - rest assured it's not. It stands out glaringly, to many Americans that would be as soon as one learns this was done at JMU, but the authors mention the necessary statistics regardless. The participants in the study reported SAT scores close to the institution means - 1140 math+reading total. For those not familiar with the SAT, that's completely unimpressive compared to many, many other American universities. JMU students are by and large the rejects of any more prestigious universities of the several nearby states. These are community college level students, especially as the majority of students in the study were freshmen (first year at the university, and as the authors report, median age arond 19).

Likewise the results on the CRT, the three trick-sounding questions which everyone is making a big deal about, are abysmal for these participants. The authors report a mean of 0.79, with 50% or 241 of the participants getting none correct, (lol) and only 27 or 5.6% getting all 3 correct. This is an extremely poor performance compared to other students at other universities on this same CRT test from previous studies (some probably referenced by authors in this paper).

The first objection I'd raise is the worry that the raw number of students who had high performance on any cognitive metric is too small a sample size. There's no data on the real distribution of SAT scores - whether any participants at all had scores high enough they would be good at a more prestigious university - and only 27 total individuals is of course worrisome for the CRT results. The other cognitive measures were not objective, better-or-worse measures of intelligence or cognition such that they could be used in comparisons towards the authors' overall thesis. Considering individuals had to be split into experimental groups for the bias measures, having an insufficient number of individuals with high performance on the cognitive measures could result in no statistical significance.

What's ridiculously annoying if not scandalous, is that authors did not give any data, NO DATA AT ALL, on the relationship between the students' CRT score and performance on their full set of "bias measures." Performance on the CRT was found to be positively correlated with scores on the other cognitive measures and self-reported test scores, at least that's consistent with other studies. However, to draw their headline conclusions, they only found a lack of correlation between the "bias measures" tested to the bias-blind spot generated from self reports and self-reported SAT scores. The correlations to other cognitive measures were also not tabled. This is questionable in and of itself, and is ridiculous given the the authors' report of "Study 2" where the only data they present on a different group of internet-survey answerers is the CRT results. (Also, the internet survey folks perform far better than the pathetic JMU students on the CRT)

It's worth summarizing one of the bias questions examined to see the obvious, intuitive objections any reasonable person would raise to the specifics of this study. Glossing over the specifics to present final conclusions based on a few statistical correlations and omitting others is not acceptable.

So this one bias question on "anchoring" was based around having participants guess how tall the tallest trees in the world are. The experiment for bias was performed by "framing" the question in two ways (two experimental groups), such as the suggestion the tree was 1000 feet tall, versus framing with a smaller number - 85. That's a tree. A tree. The result for the one group was a mean of guessing 884 ft tall with a standard deviation of 572, and to no real surprise the group framed with the smaller number gave much smaller answers, and presumably almost none of them giving objectively idiotic answers. (As expected the authors found the presence of bias on all the bias measures they tested, consistent with previous studies, for the set of participants as a whole. Nothing interesting there. Remember the whole point of the authors' conclusion is whether the "smarter" participants had less bias than the others.) It's also very annoying that, while "base rate" bias and other related biases are supposedly studied in other questions, the attempt to study a DIFFERENT BIAS is done with an absolutely awful experimental question that mixes in these biases as well. But there's nothing to be said for people who think a tree is over a thousand feet tall. Complete morons.

I really wish the authors had actually provided the data, and we could see whether the handful of students who got all 3 CRT questions correct gave such ridiculous answers on several of the bias questions. There were so few of these students amongst the throng of JMU morons (only 27 total) that the sample size might still not be enough, but it's worth knowing. The authors only report a small correlation between overall SAT scores and bias on this question, as with others, but find it's not significant, which still could be relatively meaningless given the awful SAT scores of all participants all around. If the students who got all of the CRT questions correct, also gave answers such as > 1000 ft trees the same as their peers, then at least the authors are being honest in presenting the results they obtained. Somehow I doubt that, I suspect their performance on the bias questions was much better and they were justified in labeling their peers as morons. (After all, the answer to whether the median JMU student is more susceptible than average to cognitive biases should be very high. It's true as the base rate, y'know)