Why that study about alternative therapies relieving pain is worthless

    6 September 2016

    The press release by the National Institutes of Health (NIH) was unashamedly positive:

    A review of evidence from clinical trials shows that a variety of complementary health approaches — including acupuncture, yoga, tai chi, massage therapy, and relaxation techniques — hold promise for helping to manage pain.

    The review, conducted by the National Center for Complementary and Integrative Health, was published in the journal Mayo Clinic Proceedings

    The researchers sought evidence on the efficacy, effectiveness, and safety of seven widely used complementary approaches or groups of approaches: acupuncture; spinal manipulation or osteopathic manipulation; massage therapy; tai chi; yoga; relaxation techniques including meditation; and selected natural product supplements, including chondroitin, glucosamine, methylsulfonylmethane (MSM), S-adenosyl-L-methionine (SAMe), and omega-3 fatty acids.

    These approaches were examined in trials of five pain conditions often seen and treated in primary care settings: back pain, osteoarthritis, neck pain, severe headaches and migraine, and fibromyalgia.

    A trial result on efficacy or effectiveness was termed positive if the complementary approach led to statistically significant improvements in pain severity, pain-related disability, and/or function, compared to the control group. A negative result meant that there was no difference between the intervention and control groups.

    The researchers found that the following complementary approaches had more positive than negative results and thus may help some patients manage certain painful health conditions:

    Acupuncture and yoga for back pain

    Acupuncture and tai chi for osteoarthritis of the knee

    Massage therapy for neck pain — with adequate doses and for short-term benefit

    Relaxation techniques for severe headaches and migraine.

    The fact that this press release comes from the well-respected NIH gives it weight and reliability. Journalists around the world are now eagerly picking up the message. Here are just two examples:

    Data from a review of US-based clinical trials published in Mayo Clinic Proceedings suggest that some of the most popular complementary health approaches — such as yoga, tai chi, and acupuncture — appear to be effective tools for helping to manage common pain conditions…

    ‘For many Americans who suffer from chronic pain, medications may not completely relieve pain and can produce unwanted side effects. As a result, many people may turn to nondrug approaches to help manage their pain,’ said Richard L Nahin, PhD, NCCIH’s lead epidemiologist and lead author of the analysis.

    ‘Our goal for this study was to provide relevant, high-quality information for primary care providers and for patients who suffer from chronic pain.’

    After reviewing more than 50 years of studies, researchers at the National Institutes of Health say several non-drug approaches may be significantly effective against symptoms of common pain conditions…

    ‘These data can equip providers and patients with the information they need to have informed conversations regarding non-drug approaches for treatment of specific pain conditions,’ Dr David Shurtleff, deputy director of the National Center for Complementary and Integrative Health at the NIH, said.

    It goes without saying that millions of pain sufferers will gather new hope from such reports. But how reliable is the information really? Let’s look a bit closer at the actual evidence.

    According to its authors, this new article ‘examines the clinical trial evidence for the efficacy and safety of several specific approaches — acupuncture, manipulation, massage therapy, relaxation techniques including meditation, selected natural product supplements (chondroitin, glucosamine, methylsulfonylmethane, S-adenosylmethionine), tai chi, and yoga — as used to manage chronic pain and related disability associated with back pain, fibromyalgia, osteoarthritis, neck pain, and severe headaches or migraines.’

    To do this, the researchers located all relevant randomised clinical trials (RCTs), 105 in total, and extracted their findings in order to inform their conclusions. This sounds about right. However, reading the article carefully, I find it impossible not to get deeply troubled. Here are the points that worry me most:

    — The safety of a therapy cannot be evaluated on the basis of data from RCTs (particularly as it has been shown repeatedly that trials of alternative therapies often fail to report adverse effects); much larger patient samples are needed for that; any statements about safety in the aims of the paper are therefore misplaced.

    — The authors talk about efficacy but they seem to mean effectiveness; the difference might seem trivial yet it is important.

    — The authors only included RCTs from the US which must result in a skewed and incomplete overall picture.

    — The article is from the National Center for Complementary and Integrative Health, a part of the NIH which has been criticised repeatedly for being biased in favour of alternative medicine.

    — Not all of the authors are NIH staff, and I cannot find a declaration of conflicts of interest.

    — The discussion of the paper lacks critical thinking.

    — There is no assessment of the quality of the trials included in this review.

    The last point is by far the most important. A review which fails to take into account the numerous and well-known limitations of the primary studies of alternative therapies is as good as worthless. Knowing most of the RCTs included in the analyses, it is easy to predict that the overall picture generated by this review would change dramatically if the often serious flaws of the primary studies had been adequately accounted for.

    I do wonder: if a leading research institution like the NIH had actively misled the public about the ‘efficacy and safety’ of a conventional medicine, it would have caused a major scandal. But here we are dealing ‘only’ with alternative medicine — and that means exaggerations, sloppy research and misleading conclusions are permissible, and nobody will make a fuss?

    Edzard Ernst, emeritus professor at the University of Exeter, is the author of A Scientist in Wonderland and the awardee of the John Maddox Prize 2015 for standing up for science. He blogs at