Here at Brunel I run an MSc module on evidence based practice. In the first session of the module I run an honesty test. Here it is (answer it yourself and, well, be honest).
“What sections of a research paper do you routinely read. Honestly.”
Almost without exception the whole group will admit to reading the abstract and small proportion will make a claim to reading the introduction or discussion. But on closer scrutiny we manage to whittle that down further. A few only read the conclusion of the discussion and the rest only the conclusion of the abstract. A whole paper boiled down to one or two sentences. Almost nobody claims with any conviction that they read the methods or results in detail. Many writers have highlighted the problems with this selective approach to viewing the literature. A cynic’s view of the sections of a paper goes something like this:
Introduction: In which the authors seek to justify the importance of their research by cherry picking any supporting evidence and ignoring the rest.
Discussion: In which the authors seek to interpret the results in a way that does not challenge their pre-existing worldview by jumping through a selection logistical hoops.
Conclusion: In which the authors try to give you a take home message consistent with their pre-existing worldview.
Methods: One of the important bits.
Results: The other important bit (often with parts strangely missing).
This view is over the top but the problem with research papers is that they are a human endeavour, and impartiality is not one of our greater virtues. Richard Feynmann’s classic quote comes to mind: “The first principle is that you must not fool yourself, and you are the easiest person to fool.” This has become much more apparent to me since I became involved in conducting Cochrane reviews. The scale and commonality of erroneous, incomplete or selective reporting came as a big surprise.
Two new papers in the musculoskeletal field have just been published that speak loudly to this problem. The first is a fantastic cautionary tale. A French research group led by Sylvain Mathieu reviewed all RCTs in osteoarthritis, rheumatoid arthritis and the spondylarthropathies published between 2006 to 2008. They went looking for the incidence of “misleading abstract conclusions”. Specifically they looked for a selection of naughties: not reporting the results of the primary outcome, basing conclusions on secondary outcomes or the results of a sub-group analysis, presenting conclusions that are at odds with the data, claiming equivalence of efficacy in a trial not designed to test for it, and finally not considering the risk-benefit trade-off. Like a game of clinical trial Bullshit Bingo. They found evidence of misleading conclusions in 23% of reports. The only predictor of misleading conclusions was genuinely negative results. In trials with negative results the rate of misleading conclusions was, brace yourself, 45%. “WHOAH!”, I hear you exclaim.
It is well known that negative results present a unique challenge to science. Editors don’t like to publish them and researchers don’t like to submit them for publication but it seems from this evidence that they also don’t like to accept them in the first place. This represents a unique failure of the scientific process. Why ask the question if you are only prepared to hear one answer? Clinical researchers need to remember that they are not in the business of trying to validate clinical practice, they are in the business of trying to test it. The two things are not the same.
Another recent review led by Sidney Rubinstein has looked at whether methodological quality is improving in clinical trials of spinal manual therapy by reviewing trials over 5 year periods from the 1970’s to 2011. Happily they do find a trend towards improving quality but still most of the risk of bias criteria were met by less than half of included trials published in the last decade.
What does that mean for those of us trying to make sense of research and help it guide our practice? When you see a nice positive conclusion you need to look closer to be sure it is not being embellished. Abstract conclusions cannot be taken at face value, even if they are appealing. There is a florid expression in East London: “You can’t polish a turd…. but you can roll it in glitter”. Having the skills to detect said glitter in clinical trials is a must for anyone interested in evidence based practice.
If you can scrutinise a paper and make sense of the methods and data, then you can judge it and make an informed decision on how it should (or shouldn’t) affect your practice. Its not that hard to learn and its actually fun when you get into it. Embrace your inner geek. A good place to start for clinical trials would be to read the section of the Cochrane Handbook on assessing risk of bias – it’s free to read here (check out chapter 8).
But we shouldn’t fall into the trap of thinking that since research is flawed, we’re better off relying on clinical experience. These two studies are a good example of how science, unlike opinion, is self correcting. If these studies tell us anything it’s that those poor, neglected methods and results sections is where you’ll find the real gold.
As well as writing for Body in Mind, Neil O’Connell is a researcher in the Centre for Research in Rehabilitation, Brunel University, West London, UK. He divides his time between research and training new physiotherapists and previously worked extensively as a musculoskeletal physiotherapist. He also tweets! @NeilOConnell
He is currently fighting his way through a PhD investigating chronic low back pain and cortically directed treatment approaches. He is particularly interested in low back pain, pain generally and the rigorous testing of treatments. Link to Neil’s published research here. Downloadable PDFs here.
Mathieu, S., Giraudeau, B., Soubrier, M., & Ravaud, P. (2012). Misleading abstract conclusions in randomized controlled trials in rheumatology: Comparison of the abstract conclusions and the results section Joint Bone Spine, 79 (3), 262-267 DOI: 10.1016/j.jbspin.2011.05.008
Rubinstein, S., Terwee, C., de Boer, M., & van Tulder, M. (2012). Is the methodological quality of trials on spinal manipulative therapy for low-back pain improving? International Journal of Osteopathic Medicine DOI: 10.1016/j.ijosm.2012.02.001
Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org.