VUmc Basispresentatie
1
2
Trust is the core value of science, obviously in close connection to honesty.
KNAW – Vertrouwen in wetenschap – Amsterdam, 2013
Tiemeijer W, De Jonge J – Hoeveel vertouwen hebben Nederlanders in wetenschap? –
WRR en Rathenau Instituut; Den Haag, 2013
3
4
Loosely defined breaches of trust are breaches of research integrity.
Of course honest errors are made all the time and most knowledge turns out to be
wrong after some time, but we should do science the best we can and not submit to the temptations of QRP and RM.
5
Academical integrity concerns norms for responsible professional behavior that should
be taught in universities.
Research integrity concerns the professional norms of scientists.
Breaches of legal and ethical requirements of research involving human subjects and
animals are important examples of disciplne‐specific research integrity issues.
6
http://www.vsnu.nl/files/documenten/Domeinen/Onderzoek/The%20Netherlands%20C
ode%20of%20Conduct%20for%20Academic%20Practice%202004%20(version%202014).
pdf
Focus on RCR and the principles that form its basis  normative framework
Not a law‐like approach describing what is permissable and what not, including
applicable sanctions
7
www.nsf.gov/od/iia/ise/Code_Conduct_ResearchIntegrity.pdf
8
www.emgo.nl/kc
This is an example of the most practical level of Standard Operating Procedures
The central principle is: ‘comply or explain’
9
10
The Poldermans case may have had the largest impact so far among the infamous Dutch cases
Complex case  you should study the reports of the investigation and read the articles
mentioned below
It serves here as an example that research misconduct may do real harm to many
patients
The calculation was disputed and the original article was only two days online in the European Heart Journal owned by European Society of Cardiology (ESC), but is now is again available at the BMJ website.
The estimate of 800,000 is the upper boundary of a wide range of possible values dependant on many assumptions. But even 10% of that number would be quite substantial.
http://www.erasmusmc.nl/cs‐research/bijlagen/integriteit/rapport‐poldermans‐2011
http://www.erasmusmc.nl/cs‐research/bijlagen/integriteit/rapportpoldermans2012
http://www.erasmusmc.nl/cs‐research/bijlagen/integriteit/eindrapport2014nl
11
Bouri S, Shun‐Shin S, Cole GD, Mayet J, Francis DP. Meta‐analysis of secure randomized
controlled trials of beta‐blockade to prevent perioperative death in non‐cardiac surgery. Heart 2014; 100: 456‐64
Cole GD, Francis DP. Perioperative beta blockade: guidelines do not reflect the problems
with the evidence from the DECREASE trials. BMJ 2014; 349: g5210
Cole GD, Francis DP. The challenge of delivering reliable science and guidelines: opportunities for all to participate. European Heart Journal 2014; 35: 2435‐40
11
This is the integrity universum: many shades of grey!
12
Necker S. Scientific misbehavior in economics ‐ Research Policy 2014; 43: 1747‐59
Fanelli D. How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta‐Analysis of Survey Data. PLoS ONE 2009; 4(5): e5738  21 surveys are summarized
Plagiarism and other professional conduct excluded  no bias (distortion of the truth)
Misconduct more frequent among medical and pharmacological researchers  maybe due to large financial interests
Conservative estimates of true prevalences
Difficult to translate to % papers with both categories of misconduct
Self report of fabrication, falsification or alteration implies intention to deceive
Self report of QRP and non‐self reports concern uncertain intention to deceive
UNCLEAR: are economist more honest in reporting or less honest in their research practices??
13
14
63 candidate items
based on literature, experience and survey questions
10 members working group Fostering RCR
three 5‐point scales for each item
also two product scales
leads to five rankings (mean score and SD)
preliminary pilot study
25 international experts participating in 4th WCRI will do the ‘real thing’
One of us (NA) listed all questionable research practices that he came across during his many years of studying research misconduct and teaching responsible conduct of research.
This list was supplemented by the other authors from their own experiences and from carefully studying survey questions used to assess the occurrence of specific forms of research misconduct and questionable research practices. We tried to make the list as complete as possible, without losing ourselves in items that are too detailed or redundant.
Additionally we tried to make the formulation as clear as possible with a minimum of 15
words. And the wording ought to be such that the item is applicable to a wide range of scientific disciplines. In December 2014 we applied the three 5‐point scales to an version of the list of candidate items in a session of the working group (N=10) that is drafting a research program Fostering Responsible Conduct of Research for the Netherlands Organization for Health Research and Development and Netherlands Organisation for Scientific Research.
That experience let to some changes in the wording of the items and to the ranking procedure.
15
In trust top 5 only 2 items that were also in truth top 5.
16
17
Dit is de andere klassieker die ik vandaag onder uw aandacht wil brengen.
Het gaat over de enorme suboptimale efficiëntie in het klinisch onderzoek. De oorzaken daarvan zijn overwegend gelegen in methodologische questionable research practices. Iain Chalmers and Paul Glaziou are framing Questionable Research Practices (implicitly) as Avoidable Waste. That may be useful to shift the debate a bit from the emotional topic of integrity to the more rational topic of efficiency.
Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet 2009; 374: 86‐9
Chalmers I et al. How to increase value and reduce waste when research priorities are set. Lancet 2014; 383; 156‐65
Ioannidis IPA et al. Increasing value and reducing waste in research design, conduct and analysis. Lancet 2014; 383; 166‐75
Salman RA et al. Increasing value and reducing waste in biomedical research regulation
and management. Lancet 2014; 383; 176‐85
Chan A et al. Increasing value and reducing waste: adressing inaccessible research. Lancet 2014; 383: 257‐66
18
Glaziou P et al. Reducing waste from incomplete or unusable reports of biomedical
research. Lancet 2014; 383: 267‐76
Zie ook de website behorende bij de Lancet serie in 2014: http://researchwaste.net/
18
Incentives om minder relevant onderzoek te doen gaan vaak uit van de financiers van het onderzoek: tegendraads en innovatief onderzoek heeft minder kans op subsidie en commerciële opdrachtgevers (farmaceutische industrie) stellen vaak het bedrijfsbelang voorop. Daardoor zijn bijvoorbeeld directe vergelijkingen van geneesmiddelen van verschillende fabrikanten zeldzaam.
19
20
21
22
The phenomena described in the red circles concern selective reporting.
23
Reporting bias is relevant for studies that focus on hypotheses testing (confirmatory
studies). The main evils are undisclosed deviations from the study protocol, including
data‐driven hypothesis testing and leaving out less welcome findings.
In explorative studies and data‐analyses this is not an issue. ‘Anything goes’ as long as it
is clear from the paper that the authors focus on what they found when playing around
with the data and leave out everything they did not find interesting.
Little is known about the determinants of selective reporting  we currently work n a Systematic Review. The factors mentioned my concern authors, reviewers, editors and
sponsors
24
Transparancy is the norm. Only when all relevant information is available the study can
be checked and replicated.
It is important that all information is registered prospectively and cannot be changed at a later stage.
25
http://www.equator‐network.org/
26
Begley CG, Ellis LM. Raise standards for preclinical cancer research. Nature 2012; 483: 531‐3
This example illustrates an important consequence of publication bias and selective
reporting: many published positive findings are false‐positive and cannot be replicated.
27
Selection reporting among animal studies is a huge problem and lead to a lack of replicability.
The authors went into the original labs and tried to replicate the study there together
with the original PIs if needed.
28
Begley CG, Ioannidis JPA. Reproducibility in science. Circulation Research 2015; 116 116‐26
There are many more examples of serious replication problems of preclinical
studies.
29
And investigators doing replications are not at all unimaginative second rate scientists.
In clinical drug research replication seems to be the norm. That is probably driven by
FDA and EMA rules and regulations. Sometimes replication is a bit overdone. The contrast with clininal research on non‐drug interventions (like medical devices, surgical procedures) and diagnostic interventions is striking.
30
Only when replication is attempted repeatedly in a number of different ways and uniformly unsuccesful, QRP or RM in the original study becomes a likely explanation.
31
Primary interest being truth‐finding according to the norms of science.
Conflicts of Interest are potential threats to independance and inpartiality.
Also selective reporting and selective citation may be driven by conflict of interest.
Financial CoI are quite clear and straightforward, but intellectual CoI are less
clear and more fuzzy.
32
Bero – What is in a name? Nonfinancial influences on the outcomes of systematic
reviews and guidelines – J Clin Epidemiol 2014; 1239‐41
Some CoI may be a mix of financial and non‐financial aspects.
33
Freemantle ‐ Trials of autologous bone marrow stem cells for heart disease ‐ BMJ 2014; 348 g2750
Nowbar et al ‐ Discrepancies in autologue bone marrow stem cell trials and enhancement of ejection fraction BMJ 2014; 348 g2688
Stem cells given by infusion or injection in the heart with a view to improve the pump function among patients with heart failure.
34
Mean ejection fraction effect size (change in ejection fraction in active arm minus change in ejection fraction in control arm)
The heterogeneity in effect size could to a large extent be explained by the number of discrepancies: larger ES for larger counts of discrepancies
Study limitations of this systematic review:
‐ Reviewers not blinded for effect size
‐ No standardized and validated methods for identifying and counting discrepancies
‐ No control for number of publications per RCT
35
One institution that published 4 RCTs is under investigation for research misconduct.
One RCT coming from another institution is already retracted.
36
37
38
This book explains all about selective reporting and its consequences. Impressively
documented and well‐written.
I strongly recommend reading this book.
39
Novels are a valuable source of insight in what scientists do and why.
Please consider reading a few of these novels.
40
41