Lax standards at PLOS One for peer review of CAM research papers?

Serious question: has the peer review system at the PLOS journals been doing a less-than-stellar job when it comes to evaluating complementary and alternative medicine (CAM) research for publication? If the answer is ‘yes’, why? Or if ‘no’, how does a paper like this go through PLOS ONE without some serious revisions? I refer to the systematic review and meta-analysis on effectiveness of acupuncture for essential hypertension, done by a group of researchers from the Tianjin University of Traditional Chinese Medicine (TCM) in China, led by Xiao-Feng Zhao, published on July 24, 2015, on PLOS ONE. The authors conclude that there is acceptable evidence for use of acupuncture as adjunctive therapy along with medication for treating hypertension. My perusal of the paper led to some major reservations about that assumption, as well as indicated some instances of sloppy writing which should have been corrected at the stage of review – but, strangely, wasn’t.

The Text…

For instance, let’s take this statement in the Introduction – a statement that has been made without including any supporting citation:

Antihypertensive medication serves as a major therapy for treating hypertension. However, due to various side effects or safety concerns, such as drug resistance which could affect therapeutic efficacy, this therapy is far from satisfactory. Thus, seeking for an effective and less harmful treatment becomes an important goal for treating EH. [Note: EH = essential hypertension]

Alarmed at the presumed lack of satisfaction towards a widely established therapeutic option, as well as the lack of citation in support thereof, I decided to check PubMed. Needless to say, I found – at the first glance – over a thousand articles, many of them from China, too, which refuted this odd notion. Antihypertensive medications currently in use are highly satisfactory and beneficial for most situations. Let me quote the conclusion of a recent (2015) study from Michigan which looked at safety and efficacy of antihypertensive medications prescribed at discharge from Emergency Rooms; they conclude: (I quote)

“Prescription of antihypertensive medication from the ED is associated with significantly lower sBP at short-term outpatient follow-up. Antihypertensive therapy was not associated with an increased incidence of adverse events, and BP reduction did not exceed potentially harmful levels. Initiation of chronic antihypertensive therapy in the ED is safe and effective and may be a reasonable consideration for at-risk populations.“

Again, in the next paragraph, an uncritically written sentence offers no citation in support of two concepts – both of which are rather problematic:

… acupuncture is an ancient Chinese therapy by inserting needles to the acu-points on the body surface along meridians to treat a wide range of diseases. [Bold scripts for emphasis by me]

The ancientness of acupuncture as it is practised today is in doubt. Ben Kavoussi, a trained acupuncturist, studied the historical texts of Chinese medicine, and found that many of the ancient Chinese acupuncture manuscripts, considered definitive guides to the practice, were actually manuals of medical astrology, fraught with myths and imagination.

In addition, the so-called “acupuncture meridians” themselves have not been unequivocally established to have any anatomical existence. In 2002, Helene Langevin (Harvard) and Jason Yandow (University of Vermont) claimed evidence in favor of these meridians aligning with physical anatomical structures in the body. Kavoussi’s study revealed that the original anatomical localizations in TCM texts followed the distribution of blood vessels, since the practice was essentially similar to bloodletting. An early 20th century Chinese physician decided to move the acupuncture loci away from the blood vessels, and thereby developed in the 1930s the modern meridian map popularized by Langevin and Yandow. So much for the presumed ancientness.

Why belabor this seemingly small point? Because the so-called ancientness of TCM has long been used in lieu of solid experimental evidence to claim efficacy, and that is ultimately detrimental to the ethical practice of medicine.

That is not all. As an occasional PLOS journal peer-reviewer, I found it quite disturbing that none of the reviewers, nor the editor, commented upon the rather breathless quality of the introductory narrative.

… two previous systematic reviews claimed that the evidence of acupuncture for lowering blood pressure (BP) was inconclusive, mainly due to the paucity of rigorous trials [4, 5]. However, both of them failed to include all relevant RCTs published in China. And their literature searches were performed up to 2010. Several new RCTs have since been published. And though in the past two years, another 2 systematic reviews have been published. One of them [6] only included 4 randomized sham-controlled clinical trials and the other one [7] included sufficient trials but not all of them were rigorously randomized controlled trials (RCTs).And there were some flaws in this review in term of methodology, data extraction and eligibility of included RCTs as well [8]. And there was another systematic review to evaluate the long-term efficacy of acupuncture on hypertension but it just included Chinese RCTs [9]. [Note: Bold scripts for emphasis by me]

I wouldn’t blame the authors, who likely may not be native English speakers; but it is the duty of the peer reviewers to help their fellow scientists avoid such gaffes.

… And the Analysis

Credit goes to the researchers who made an effort to incorporate an extensive and comprehensive source at the foundation of their analysis, using PubMed, EMBASE, Cochrane Central Register, ICTR, as well as various Chinese scientific and medical databases. ‘Acupuncture’ was naturally one of their search keywords, amongst which they also included ‘electroacupuncture’ (a modern-day modality about which I have expressed some serious reservations earlier). Regardless, I have no further comments about the methodology they employed. Based on their inclusion/exclusion criteria, eventually the authors identified for final analysis 23 randomized controlled trials (RCT) involving acupuncture, encompassing a good 1788 patients. Because the key topic was the modality of acupuncture, it comes as no surprise that close to 80% of the included RCTs were conducted in Chinese institutions. For the purpose of the review, however, even if there were a regional bias here (the authors acknowledge this in the discussion), I don’t know whether this factor could be adequately controlled for in this context.

However, one important revelation in the paper bears mentioning here. Scientific evaluation of hypotheses are only as good as the controls, which is why it is imperative to have good controls in the studies. Sham acupuncture is an important control for acupuncture studies for various reasons (on which I have earlier commented on here and here, and especially here), and I was rather taken aback to find that out of the 23 included RCTs, only 5 considered sham acupuncture as a control. To me, this raises doubts about the suitability of the other 18 studies. I would have readily excluded the 2003 study by Dongjie and Qunli, because with their extremely inadequate control (“improving life pattern”) the hypothesis about benefit of acupuncture on insulin resistance could not have been tested adequately, regardless of their pro-acupuncture conclusion. From this point on, I had expected the discussion to center around these 23 RCTs, and yet in the very next paragraph, ’24 trials’ is mentioned – which should have been noticed by the peer reviewers.

Having perused their results section, I understand why their various subgroup analyses seem to have been done on only a fraction of the total included RCTs; it depends on the criteria defined for analysis, which determined whether a study would be selected for a particular subgroup analysis, or not. I’d have expected at least the 18 studies done in China to be included in the subgroup analyses; yet, there seems to be a total of 11 studies each in the analyses of blood pressure outcomes, and 14 in the analysis of efficacy rate. I worry about selection/publication bias (a concern echoed by authors in the discussion), including whether the 23 RCTs included were truly representative of the quantum of observations on the outcome of acupuncture therapy; inadequate representation has proven to be a significant source of variability of the conclusions of other meta-analyses done on acupuncture modalities.

Included in the results section are several observations of significance related to the hypothesis of acupuncture’s efficacy as a therapy; for instance,

• Significant disagreement (measured as ‘heterogeneity’ in systolic blood pressure via a subgroup analysis in figure 2 of the paper under discussion) exists as to the role of acupuncture in the outcome of those studies.
• 10 RCTs with 963 subjects categorized the outcome as proportions of total subjects benefitting from either acupuncture or western medicine (defined as “efficacy rate”), and found no benefit of acupuncture.
• 7 RCTs with 510 subjects receiving either acupuncture or western medicine for their hypertension found no benefit of acupuncture on both systolic or diastolic blood pressure. In fact, overall effect on diastolic blood pressure appears negligible, regardless of treatment.
• 3 RCTs with 170 subjects examined acupuncture plus medication versus medication alone, and found no significant difference in favor of acupuncture.
• 2 RCTs with 216 subjects looked at the magnitude of blood pressure changes, and found no difference between acupuncture and sham acupuncture.
• Even in the single trial with 60 patients where acupuncture was compared to lifestyle changes as a means to reduce insulin resistance (a trial that I’d have excluded from this whole analysis), acupuncture was found to have no superior effectiveness in reduction of blood pressure.
• Statistically highly significant amount of heterogeneity exists in RCT outcomes with ‘manual’ (that is, regular, garden variety, poke-a-needle-on-presumed-acupoints) acupuncture (which, to my mind, casts a doubt on the claims of therapeutic efficacy of acupuncture made by isolated, individual studies, in which unexamined biases are likely).

Two additional interesting things to note in this study are two RCTs which were considered amongst the 23 in this meta-analysis. The Stop Hypertension with the Acupuncture Research Program (SHARP) RCT of 2006 conducted in the US with close to two hundred participants provided extraordinary control to the acupuncturists to run the study, using individualized traditional Chinese acupuncture, or standardized acupuncture at preselected points as treatments, with invasive sham acupuncture as control; and yet, it found no benefit whatsoever of acupuncture in controlling hypertension or its comorbidities, such as dyslipidemia, renal function or glucose control.

Secondly, in a 2007 German RCT with about a hundred-fifty subjects, acupuncture following TCM rules reduced hypertension during the treatment period of 6 weeks (following which the blood pressure returned to pre-treatment levels) in contrast to sham acupuncture. And yet, after the results were published, a team of Italian acupuncturists found crucial inaccuracies in the descriptions of the methodology. This, combined with the 2001 observations by Middlekauff et al. that even insertion of needles at non-acupoints (in other words, ‘sham acupuncture’) seemed to reduce blood pressure during mental stress, casts a serious doubt on the claims of specificity of acupuncture modalities and the TCM hypothesis for the treatment of any medical condition.

Despite the above, the authors conclude – on the basis of 2 and 4 RCTs with 170 and 230 subjects respectively – that acupuncture in combination with Western medicine is beneficial against hypertension compared to Western medicine used alone. Is this a tenable conclusion regarding the overall hypothesis of acupuncture’s benefit in hypertension? I think the jury’s still out, and will remain out for the foreseeable future. It brings to mind what Harriet Hall, physician/science communicator, once termed research into alternative medicine modalities with low or non-existent prior plausibility: “Tooth Fairy Science“. A therapy that wholly bases itself on some mystical energy or Qi, which has no anatomical, physiological and mechanistic correlate, may be considered magic, but certainly not medicine.

Then, the question remains…

Why did the PLOS ONE reviewers and editor not raise any of these questions and concerns? Curious minds want to know. Is this yet another instance of the slow infiltration of quackademic medicine – that plagues the world of healthcare and patient management – into scientific publications?