January 15, 2017

Criticisms of mindset research

The replication crisis within psychology has been rather big news over the last few years (read more). Findings of well-known psychologists about topics like willpower/ego depletion, powerposes, priming, and facial feedback have been scrutinized and to some extent debunked. Recently, criticism has also emerged of the research into mindsets by Carol Dweck and her colleagues. I have written a lot about mindset and I think it is a valuable and useful concept. But, as I write here:

"[...] anything can be criticized and I think that nothing is above criticism. [...] the growth mindset will also be criticized. No method or concept is final or perfect. The good thing about a rational and scientific approach to any truth claims is that we may find out what is wrong with specific claims so that we can gradually refine and improve them. The growth mindset is a valuable concept but it is not perfect. It is inevitable that we will improve on it by discovering what can be improved in its conceptualization.

Post by Nick Brown about Mueller & Dweck (1998)

Nick Brown, the person who played a big role in exposing mistakes in Babara Fredrickson’s publication about the positivity ratio, has analyzed a study by Mueller & Dweck (1998). He has written a post about this entitled In which science actually self-corrects, for once. Brown has developed a mathematical technique, GRIM, with which he can detect irregularities in tables of findings in publications. With this technique he has detected many mistakes in many publications. He has also applied it to the publication by Mueller and Dweck and found inconsistencies in this publication as well.

He contacted Dweck and mentions that, to his surprise, he received a reply from Dweck within 24 hours in which she said she would look into it. Within two weeks, he received a 16-page PDF document in which Dweck, with Claudia Mueller and David Yeager had gone through the entire article, line by line, and answered every one of Brown's points.

Some of the inconsistencies could be explained, others turned out to be the result of mistakes. Brown says that "Dweck’s “openness and willingness to address the problems” has been “exemplary”, and that she has done a “thorough job of owning up to the problems” of the paper. She has publicly acknowledged these errors and says she is “considering” writing to the journal to ask them to formally correct the paper." Brown remains skeptical about the concept of mindset but says: "at least I feel confident that the main people researching it are dedicated to doing the most careful reporting of their science that they can."

Buzzfeed article

Another critical article about mindset was published on Buzzfeed. This article mentions Brown but also some other critics of mindset such as Scott Alexander, Timothy Bates, Stuart Richie, and Andrew Gelman. Alexander, who says he is very skeptical about the mindset concept, is surprised of the big effect sizes in mindset research and wonders whether Dweck has "make a pact with the Devil in which she offered her eternal soul in exchange for spectacular study results?"

Bates is currently doing large scale research into mindsets in schools in the UK. Previously he attempted to replicate the Mueller & Dweck (1998) finding and failed. In a response Dweck says that replications are important but they need to be done genuinely and thoughtfully by skilled people (read more). Nick Brown counters: “The question I have is: If your effect is so fragile that it can only be reproduced [under strictly controlled conditions], then why do you think it can be reproduced by schoolteachers?”

The article contains some other critical and interesting issues so I recommend reading it. Brown says that doing statistics properly requires more specialized training than many researchers have had. Bates says the mindset hypothesis should not be thrown out but warns against expecting too much from it.

Gelman: why psychology?

Is psychology such a mess that it is hit by so much criticism? I think the situation is more nuanced than that. Recently a befriended professor told me that similar problems exist in the natural sciences: “[…] people who have once taken a position (often) have great difficulty in acknowledging it may not be entirely right. […] Especially in my area of expertise (simulations of molecules) there are some methods about which not enough questions are asked. To change that a lot of testing and comparing is needed."

Andrew Gelman, statistics professor at Columbia University, mentions a paradox. He says that some positive characteristics of psychology make this science so suitable for the type of critical reflection that now takes place (read here). Firstly, he explains that psychology is more sophisticated than for example economics and biology in it consideration of topics like validity, reliability, and latent constructs. Secondly, he say that psychology nearly exclusively works with clean and easily comparable methods (experiments and surveys).

Thirdly, he says psychology is characterized by a culture of openness and is relatively unaffected by restrictions and big money interested (as medicine and biology are). Fourthly, the current criticism involves the heart of (social) psychology, some of the leaders of the field, whereas in other fields, research controversies typically seem to involve fringe elements in their professions. Fifthly, psychological is interesting and to some extent accessible to the general public.

Reflection

It is a good thing that the demands for good research are becoming stricter. The bar for good psychological research is raised which is necessary for building a robust body of psychological knowledge. It is ironic that the very strengths of psychology of focus on good methods and relative openness make it more vulnerable to criticism than other fields with are less rigorous and open. Other fields may be less transparent but no less problematic.

Even more importantly: we are all, on a daily basis, confronted with countless methods and claims which aren't even attempted to be substantiated. Let's not exclusively be skeptical about a field of science which is trying hard to substantiate its findings and is open to criticism.

Improving research into mindset is important. No concept and no researcher is above criticism. For now, I agree with Bates. We must not abandon the concept of mindset but also not have unrealistic expectations of it.

Also read: 

2 comments:

  1. Note that the posts by "Carol" in that comment thread on Andrew Gelman's blog are almost certainly not by Carol Dweck. This is certainly true of all of the exchanges between "Carol", "mark", and "Andrew" (who is Gelman); I presume it is also true of the comment near the end of the discussion. I don't know if the blog software enforces uniqueness of poster's names --- somehow I doubt it --- but in any case I have discussed several of "Carol's" posts with their author (whose real first name is Carol, entirely by coincidence vis-à-vis Carol Dweck).

    ReplyDelete
    Replies
    1. Thank you, Nick. I've updated the post to remove that comment.

      Delete

Enter your email address:

Delivered by FeedBurner