Searching for sloppy science
Do we want to know what goes wrong?
Is the UG scared people will air its dirty laundry? That was the question on many people’s minds ten days ago after it turned out that the university refused to participate in what should have been the biggest study into sloppy science ever, the National Survey on Research Integrity (NSRI).
Of the twenty-three institutes the survey writers approached, only eight were willing to share their staff’s email addresses and actively ask them to participate, head researcher Gowro Gopalakrishna, with the Academic Medical Centre in Amsterdam, told UKrant. This meant she and her colleagues had to look up scientists’ email addresses themselves, which resulted in the invitations not arriving or being marked as spam.
The University of Amsterdam is one of the few institutes who did participate. Their reasons were threefold: ‘Because this study is important, because privacy is guaranteed, and because we had no substantive objections after we had experts look at it and some adjustments were made.’
The University of Maastricht is participating ‘because there isn’t a lot of empirical evidence about scientific integrity, and because this survey offers a unique opportunity to use concrete data to evaluate how we can improve proper research practices.’ Or, as the Radboud University in Nijmegen puts it more simply: ‘Because it can lead to information on things that are going well and things that aren’t.’
Far-fetched
But the UG doesn’t want to participate because, as spokesperson Jorien Bakker said, they had some questions about the study’s set-up and its research question.
‘That didn’t come across as a very strong argument’, says Rink Hoekstra, who works at the UG and studies research practices. Assistant professor Casper Albers called the argument ‘far-fetched’. He’s on the university council and says he will be demanding a better answer during the next meeting.
Universities apparently care more about their image
UG post-doctoral researcher Vera Heininga, a great proponent of open science, is ‘extremely disappointed’ that the universities ‘apparently care more about their image than about coming together to improve research integrity’.
Serge Horbach, who got his PhD from the Radboud University for his research into the way universities deal with reports of scientific fraud, sees that people mainly think the institutes are worried about dirty laundry being aired. ‘As far as I can tell, the survey writers aren’t particularly afraid to promote this idea’, he says from his new office in Denmark, at the University of Aarhus.
He thinks there are several perfectly legitimate reasons not to participate, but the fear of their image suffering does appear to be the most prominent reason, he says. ‘I don’t want to insinuate that this is what’s happening here, but the recurring discussion about institutes’ reputation and how willing they are to openly address integrity issues is particularly interesting.’
Grey area
Hoekstra says it’s not even about such matters as the Diederik Stapel case, in which the Tilburg professor of psychology turned out to have falsified research data on a large scale. These are extremely rare cases, and people like that will always find a way to game the system, he thinks.
He’s more interested in the grey area of things that are allowed but not entirely honest. ‘Like leaving out results that don’t quite work for you or using data differently than you’d originally planned.’
The survey is based on the current trend of populism
What about statistical significance, says Michiel de Boer, researcher at the UMCG and a close acquaintance of the NSRI researchers. ‘Saying something is true when the p-value is below 0.05, or that something isn’t true when it’s above that is questionable. It’s technically incorrect.’ But that doesn’t mean it’s not allowed.
These things are debatable, and yet they happen regularly, says Hoekstra. ‘Probably as a result of work stress and the pressure to publish as much as possible. It’s important to understand that.’
Individual issues
Retired associate professor of mathematical genetics Gerard te Meerman says the survey didn’t seem to be geared towards finding flaws in the system. ‘All the questions were about the kind of fraud Stapel committed.’
He brings up American researcher John Ioannidis, who claims that a lot of published research doesn’t meet scientific standards. Supposedly, the results are often based on statistical happenstance and can’t be replicated. ‘The most important aspects, quantitatively speaking, of what makes scientific research so unreliable aren’t even included in the survey’, he says. He means the survey focused too much on individual issues.
But since the universities’ interference, the survey does include questions on the system, says De Boer. According to Heininga, the survey now also tackles open science, competition, and work stress. She’s happy with the development.
Methodology
After the media made a fuss about the issue, Faculty of Behavioural and Social Sciences vice dean Klaas van Veen, as well as a thousand other people, checked his inbox for the NSRI invitation. After less than ten minutes, he was sick of the survey: ‘What a waste of time.’
‘The questions read like the ones you get when you’re flying to the US’, Van Veen grumbles. ‘Where they ask you if you’ve ever committed a terrorist act. No one in their right mind would answer yes to that.’ He also came across statements that could be interpreted in different ways. ‘That’s the first thing you learn not to do when writing a questionnaire.’
The questionnaire was thorough and professional
The idea that researchers lack integrity rubs him the wrong way: according to him, 99.9% of them are honest. ‘There are so many checks in place that they have no choice. This survey is a result of the current populist, anti-university trend. I completely understand that universities figured they have better things to do right now than participate in it.’
Albers felt the survey’s methodology was solid. ‘As a statistician, I’m often annoyed by questionnaires asking leading questions. But here I specifically noted how thorough and professional it was.’
This difference in opinion may be due to the way the survey is set up. Not everyone gets the same questions, in an effort to keep the survey short; researchers shouldn’t spend more than twenty minutes on it.
Promotion
A few days after the UKrant article was published, the UG says they did in fact do something to ‘promote’ the survey: in late October, they published a news item on the research data management section of the UG website. The item was also included in the department’s newsletter.
‘No one ever checks that section’, Heininga scoffs. ‘Except when they have an issue to do with data. I can’t even remember what else is on that page.’
Besides, she says, if she didn’t know the research leader’s name, she probably wouldn’t have opened the email in the first place. Albers never even got the email, and others deleted it, because they are inundated with survey requests and are worried about phishing scams. But that’s exactly why the survey writers needed the university’s help: to alert their staff to the email.
‘The only way the survey can work is if universities provide their staff members’ email addresses’, the University of Amsterdam determined, sharing its database with research agency TNS-NIPO (now called Kantar). ‘Our employees’ privacy is guaranteed’, the university said.
So why does the UG refuse to cooperate? UKrant‘s repeated requests to speak to rector magnificus Cisca Wijmenga about the matter were rebuffed. By the time the matter is discussed during the next university council meeting in two weeks, it will be too late: the survey is only available until December 7.