US Politics Mega-thread - Page 10052

This book by John Hattie – Professor of Education at the University of Auckland – is the culmination of more than a decade of research during which he and his team have set out to summarise and synthesise the empirical research on the effects of various educational influences and interventions on student achievement. Probably due to the huge scope of this project – comprising 800 meta-analyses, more than 50,000 smaller studies and more than 80 million pupils – this study has been widely acclaimed. According to a review in the Times Educational Supplement, Hattie’s work “reveals teaching’s Holy Grail”.

Hattie starts from the observation that in education “everything seems to work”, as educational interventions of almost any kind seem to have a positive effect on student achievement. He then proposes to move beyond “everything goes”, towards the development of a barometer of “what works best”. To this end he applies the tools of meta-analysis to a huge body of empirical research and calculates effect sizes (denoted d) for 138 influences in the following domains: student, home, school, teacher, curricula and teaching approaches. Hattie neatly presents the effect sizes in a graphical barometer and convincingly argues that only effect sizes higher than 0.4 are in the so-called zone of desired effects (in other words, are worth the effort). Prior to presenting the barometers and effect size rankings, Hattie develops his visible learning story, which is summarised in the following quote: “Visible teaching and learning occurs when learning is the explicit goal, when it is appropriately challenging, when the teacher and student both seek to ascertain whether and to what degree the challenging goal is attained, when there is deliberate practice aimed at attaining mastery of the goal, when there is feedback given and sought, and when there are active, passionate and engaging people participating in the act of learning” (p. 22). The visible learning story is illustrated using the example of outdoor training. An instructor teaching rock-climbing will have continuous visual feedback on the success of his teaching efforts (pupils climbing high or falling down) and be able to adjust his teaching accordingly.

I find the visible learning story a convincing story. I believe most teachers will agree with the book’s main message that effective instruction cannot take place without proper feedback from student to teacher on the effectiveness of the instruction. Hattie also convincingly argues that the effectiveness of teaching increases when teachers act as activator instead of as facilitator, a view which I find refreshing in a time when teaching approaches such as problem-based learning have the effect of sidelining the instructor. My problem with the book is, however, that I would have been convinced even without the empirical analysis. If anything, Hattie’s meta-meta-analysis casts a few doubts on the validity of his research, as I will explain below.

My first comment, however, relates to Hattie’s goal in writing this book. He states that his aim is “to develop an explanatory story about key influences on student learning”, not to build another “what works recipe”. Yet this aim fits uneasily with the barometers and rankings which are scattered across the book. By presenting these measures so prominently, the author automatically invites the reader to make a clear distinction between what works and what doesn’t work. If Hattie doesn’t want us to draw such conclusions, he should not have presented the material in this way. Related to this is the tension between story-telling and ranking influences. The visible learning story is told in Chapter 3 and naturally refers to some of the effect sizes calculated in the remainder of the book. Yet the relationship between story and effect sizes remains implicit and qualitative. The reader has no indication or test result of how well the effect sizes fit the visible learning story.

I next turn to the way in which the meta-meta-analysis has been conducted. Hattie discusses the various pros and cons of meta-analysis extensively and concludes that this is a valid research methodology. I will not take issue with this point, as meta-analysis is a generally accepted tool of academic research. As a general statistical point, however, I was surprised that Hattie has chosen to summarise the effect sizes of the 800 meta-analyses using unweighted averages. Small and large meta-analyses have equal weight, while I would assume that the number of studies on which a meta-analysis is based indicates its validity and importance. Instead I would have opted for weighted averaging by number of studies, students or effect sizes. At a minimum, it would be interesting to see whether the results are robust to the choice of averaging.

A great asset of Hattie’s book is the reference list, which allows the inquisitive reader to dig a little bit deeper, by moving from the rankings to the underlying meta-studies. I have done this for the top-ranking influence, which is “self-reported grades” (d = 1.44). This result is dominated by the Kuncel et al. (2005) meta-analysis (d = 3.1) (Kuncel et al. 2005). This paper is about the validity of ex-post self-reported grades (due to imperfect storage and retrieval from memory or intentional deception), not about students’ expectations or their predictive power of their own study performance, as Hattie claims. The paper thus should not have been included in the analysis. My N = 1 sampling obviously has its limits, but this example does raise questions regarding the remaining average effect sizes.

Two final comments relate to the application of Hattie’s work. While it is certainly valuable to know “what works best” in education, educational institutions will need to know not just the benefit of educational interventions, but also their cost. So the question which really needs to be answered is “what works best per monetary unit spent”. On the cost side, however, Hattie’s book is silent. Also, given the importance of two-way feedback in teaching, a major challenge for large-scale educational institutions (such as universities) is to organise feedback in a cost-effective manner.

Visible learning should be lauded for emphasising the importance of the student–teacher relationship and of adequate feedback, but at the same time presents managers with the challenge of organising this feedback in large scale educational settings.

On March 13 2018 07:52 Simberto wrote:
"I would argue raising standards and holding people back a few times would help."

I would like to mention that this is in no way supported by empirical evidence. Holding people back a grade actually reduces the amount of stuff they learn during their next year. Retention is roughly as bad as corporal punishment (very bad) at home with regards to its effect on learning, and way worse than for example television at home.