Month: January 2012

How to discover scientific fraud – the case of Diederik Stapel

The junior researchers who revealed the most striking science fraud of last year shared their side of the story this weekend in the Dutch daily Volkskrant (LINK). What lessons are to be learned for young researchers?
In September last year, Professor Diederik Stapel, social psychologist at the University of Tilburg, was found out to fabricate his data. With over 100 publications which were cited over 1700 times he was one of the most prominent figures in his field. His findings were a collection of the weird and the wonderful: A dirty train station increases racist discrimination (LINK). Meat eaters are more selfish. One has better table manners in a restaurant (LINK). All retracted, prevented from publication or under investigation. How could three young researchers challenge the biggest name in their field? The answer is simple: with good scientific practice.
After being presented with results that seeing fruits on trees influences materialistic thinking differently than fruits in the grass, one of the young researchers grew curious/suspicious and joined Stapel’s data factory to do an unrelated study. After receiving a seemingly perfect data set from Stapel – which confirmed all predictions – he calculated the Cronbach’s alpha for his questionnaires. This measure tells researchers what the internal consistency of questionnaires is. For example, answering yes to ‘Are you a vegetarian?’ should correlate highly with ‘Do you avoid eating meat?’. When the consistency is low this is indicative of poor data quality. Reasons can be numerous including participants who don’t care, bad questionnaire design or mistakes in analysis. The young researcher’s data set turned out to have an internal consistency so low that he concluded chance responding. Such a questionnaire should really not confirm predictions. That was a year ago.
A decision was taken to join together as a team of three young investigators to reveal the truth about Stapel’s practice. In order to strengthen their case they tried to replicate a study twice and failed. Through the next six months they collected a whole dossier about calculations and weird occurrences which all pointed in the same direction.
Then they informed the Head of Department, Professor Zeelenberg who had published with Stapel. Zeelenberg believed them and within a week Stapel sat in front of the university principal to explain himself. Another week later Stapel admitted it all before the press.
So, what can a young researcher learn from the whistleblower’s investigation?
1) Extraordinary claims require extraordinary evidence
If you don’t believe a finding yourself, replicate it and find out. Science is objective, i.e. its results are not tied to who produced them. However, when you want to convince the world that a research star is a huge fraud – an equally extraordinary claim – you also need extraordinary evidence. One failed replication, for example, would not be enough because failed replications happen more often than data fabrication of the scale Stapel practiced it (I hope). So, if you want to make sure that you convince the doubters make sure it will stand up to scrutiny.
2) Know your data
A narrow look at p-values obscured the very low quality of the questionnaire data. How could the other researchers around Stapel simply live with that? Or did they not know their data? A researcher should really delve into his data before drawing conclusions. The reported analysis in the publication is often just the tip of the iceberg in terms of all the analyses done beforehand.
3) Have courage
One thing that becomes really clear is that the three young researchers were somewhat unlikely heroes. They did not boast, they got drunk after telling the Head of Department, they want to remain anonymous now. But despite their own doubts and the real risk of a backlash from the Stapel camp, they went through with it. Scientists sometimes do need real courage.
Surprisingly, the three whistleblowers do not appear to have turned away from science. That’s fortunate because they have truly proven themselves. Their careful, time consuming work revealed a truth which no one suspected. That’s how science is done.