Evidence-Based Complementary and Alternative Medicine

Volume 2013 (2013), Article ID 708251, 12 pages

http://dx.doi.org/10.1155/2013/708251

Review Article

Blinding Measured: A Systematic Review of Randomized Controlled Trials of Acupuncture

Alex Moroz,¹ Brian Freed,¹ Laura Tiedemann,² Heejung Bang,³ Melanie Howell,¹ and Jongbae J. Park4,5

¹The Center for Musculoskeletal Care and Department of Rehabilitation Medicine, New York University School of Medicine, 333 East 38th Street, 5th Floor, Room 5-103, New York, NY 10016, USA

²Department of Neuroscience, Brown University, Providence, RI 02912, USA

³Division of Biostatistics, Department of Public Health Sciences, University of California-Davis, Davis, CA 95616, USA

⁴Asian Medicine and Acupuncture Research, Department of Physical Medicine and Rehabilitation, School of Medicine, University of North Carolina-Chapel Hill, Chapel Hill, NC 27514, USA

⁵Orofacial Pain Program, Regional Center for Neurosensory Disorders, School of Dentistry, University of North Carolina-Chapel Hill, Chapel Hill, NC 27599, USA

Received 17 October 2012; Revised 6 January 2013; Accepted 6 January 2013

Academic Editor: Gerhard Litscher

Copyright © 2013 Alex Moroz et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

Background. There is no agreement among researchers on viable controls for acupuncture treatment, and the assessment of the effectiveness of blinding and its interpretation is rare. Purpose. To systematically assess the effectiveness of blinding (EOB) in reported acupuncture trials; to explore results of RCTs using a quantitative measure of EOB. Data Sources. A systematic review of published sham RCTs that assessed blinding. Study Selection. Five hundred and ninety studies were reviewed, and 54 studies (4783 subjects) were included. Data Extraction. The number of patients who guessed their treatment identity was extracted from each study. Variables with possible influence on blinding were identified. Data Synthesis. The blinding index was calculated for each study. Based on blinding indexes, studies were congregated into one of the nine blinding scenarios. Individual study characteristics were explored for potential association with EOB. Limitations. There is a possibility of publication or reporting bias. Conclusions. The most common scenario was that the subjects believed they received verum acupuncture regardless of the actual treatment received, and overall the subject blinding in the acupuncture studies was satisfactory, with 61% of study participants maintaining ideal blinding. Objectively calculated blinding data may offer meaningful and systematic ways to further interpret the findings of RCTs.

1. Introduction

The presence of a viable control is important for any research study, allowing for valid comparison to the condition of interest. Randomized controlled trials (RCTs) are the primary way that many areas of research examine the comparative effectiveness of verum and control conditions. Patient blinding in these studies is essential for gathering reliable results that can be expanded upon. However, blinding success in nonpharmacologic interventions, such as acupuncture, can be difficult to assess, and results may not be applicable from one study to the next, due to differences in control methods and in how blinding success is determined, if it is considered at all. The CONSORT 2010 Statement on blinding suggests that it is important to include how blinding was attempted, but there is no mention in the statement of how to assess whether blinding was successful [1]. Without knowing if the specific nonpharmacologic blinding techniques used are valid overall, the information collected may or may not be reliable. We believe it is imperative that blinding success be assessed toward better understanding of the reliability of results.

With respect to acupuncture research specifically, developing a control procedure that is physiologically inert and indistinguishable from true treatment has proven to be a challenge due to the very nature of traditional acupuncture. It has resulted in a variety of control methods that up until now have not been compared in terms of blinding success [2–5]. The traditional techniques of acupuncture (penetrating needles at acupoints for different organ systems and deqi) are the verum conditions of most acupuncture studies. Control conditions include penetrating needles at “nonpoints” and “wrong points,” commercially-developed nonpenetrating devices, homemade nonpenetrating needles, and toothpicks or cocktail sticks [6–9]. “Nonpoints” are points not used for any purpose in traditional acupuncture. For the “wrong points,” sham acupuncture is done at points thought to affect a different body system than the one targeted by the verum condition. However, the “wrong point” method may cause a physiological effect similar to that of acupuncture and, therefore, may be more appropriately considered to be verum than sham acupuncture [10].

To date, there is no standard or universally accepted sham procedure for acupuncture research and no quantitative comparison of blinding between the above sham methods. This may contribute to why there is a discrepancy between the clinically recognized effectiveness of acupuncture and the relative lack of research supporting it [11]. Methodological progress for blinding characteristics, including the amount of disclosure to study participants, the variables to be collected, the analytic design, and the interpretation strategy, is needed in validation studies of sham control procedures [12].

The present meta-analysis systematically examines the status of blinding in sham acupuncture RCTs via a numerical measure of blinding index. Our primary aim is to empirically evaluate the validity (via effectiveness of blinding) of sham control techniques in order to quantitatively assess blinding across available studies. We hope to determine the reliability of the results of studies that used different sham techniques, so that we may learn which sham methods are most useful for future acupuncture research. We also believe that our systematic review is just the beginning of increasing the validity of the assessment of quantitatively assessed blinding practices in acupuncture research. Our methods can be further extended as a model to assess other nonpharmacologic treatments’ blinding techniques.

2. Methods

2.1. Data Sources and Searches

PubMed, Embase, and Web of Knowledge databases were searched for scientific articles using the keywords “acupuncture,” “sham acupuncture,” or “sham procedure.” A revised search was also performed with Ovid Medline using the keywords “acupuncture,” “sham acupuncture,” or “placebo acupuncture.” Eligible studies were those that were randomized controlled in humans and were published in English between 1985 and 2011. Our search was not limited by patient diagnosis or by the part of the body where acupuncture was administered.

2.2. Study Selection

A study was considered eligible for initial screening if the authors stated that they evaluated blinding. One of the authors (Moroz, Freed, or Tiedemann) determined if the study reported data on effectiveness of blinding (EOB) by specifically asking participants if they thought the treatment used verum (V, real needle used or needle penetrated the skin) or sham (S) needling techniques, with or without an option to say they did not know (DK). If the blinding evaluation data was not included or was unclear, the authors were contacted twice by e-mail and/or telephone and asked for additional blinding evaluation data or clarification. Studies whose authors did not respond or responded but no longer had access to the original data were excluded. Studies that used a credibility questionnaire, asking patients to choose whether or not they had received treatment based on the principles of Chinese medicine or another type of acupuncture, were excluded as well. In these studies, the patients’ ability to distinguish between verum and sham needling was not directly addressed and the questionnaire could be misinterpreted. If EOB was measured multiple times within the same study, the data collected at the end of the study or the data collected from the insertion (as opposed to sensation of deqi or needle withdrawal) arm of the study would be used. In one work that reported results of two separate studies using different acupuncture points, one on the patients’ back and the other on the upper extremity, the blinding index calculations were done independently [58].

2.3. Data Extraction

For the studies that were included in this meta-analysis, the number of patients who responded as V, S, or DK was extracted from each trial. Additionally, the following variables were extracted from each of the included studies (we hypothesized a priori that these may be associated with EOB): year of publication, subject only or staff and subject blinding, time of blinding assessment, assessment of deqi, type of sham control device used, use of penetrating or nonpenetrating sham control, patient diagnosis, and number of days without acupuncture experience prior to participation. Research staff members that were blinded were those involved in data analysis and interpretation, not those administering acupuncture.

2.4. Data Synthesis and Analysis

Statistical analyses were carried out using the blinding index (BI) in order to objectively assess EOB [63, 64]. Blinding index estimates the degree of potential unblinding beyond chance for each arm in a given study by counting the excessive numbers of correct guesses. Blinding index values are always between −1 and 1, where 1 corresponds to all correct guesses, whereas −1 corresponds to all incorrect or opposite guesses. If 50% of patient responses are correct and 50% are incorrect, then ; this is indicative of random guessing and thus is an ideal blinding scenario. Another plausible scenario indicating effective blinding is that patients tend to believe they received active treatment regardless of actual treatment received, which may reflect patients’ wish to receive active intervention. In this case, blinding index will have a positive value in the active treatment arm and a negative value in the sham treatment arm, where this scenario is denoted later as unblinded/opposite.

Verum and sham acupuncture groups were each assigned a separate blinding index value. Based on the calculated blinding index value combinations for the two treatment arms, nine possible blinding scenarios were proposed (Table 1). For classification purposes, we decided to consider unblinded; random guesses; opposite guesses [13, 64, 65]. (Remark: this cutoff value was based on authors’ consensus and used as a general tool for classification and explanation; it should not be interpreted as an absolute indication of blinding effectiveness.)

Table 1: Blinding scenarios [13].

Experimental arm (verum)
Control arm (sham)
Possible blinding and clinical effectiveness interpretations
Trials number (%)

Random guess
Random guess
Ideal, possibly most ideal from the scientific or statistical perspective
8 (15)

Random guess
Opposite guess
Rare
2 (4)

Random guess
Unblinded
Possibly little treatment effect and completely no effect in control arm
3 (6)

Unblinded
Unblinded
Possibly problematic. Treatment effect in experimental arm and no treatment effect in control arm (e.g., patients tend to know what to expect)
4 (7)

Unblinded
Opposite guess
Ideal (e.g., patients tend to have wishful thinking, strong placebo effect, and any treatment administered is perceived as real treatment)
25 (46)

Unblinded
Random guess
Possibly problematic. Treatment effect in experimental arm and no treatment effect in control arm (e.g., patients do not know what to expect in the absence of treatment)
12 (22)

Opposite guess
Opposite guess
Rare
0 (0)

Opposite guess
Random guess
Rare
0 (0)

Opposite guess
Unblinded
No treatment effect at all; patients may have low expectations
0 (0)

Individual variables hypothesized to potentially impact EOB were compared by their average VBI and SBI scores, weighted by sample size. The weighted averages were used to determine the overall blinding index scenario for each variable. Blinding scenarios were also compared to the overall outcome of each study. Finally, we looked for patterns of possible association of study design factors with EOB based on blinding index values and scenarios. The factors included were based on data extraction criteria, sample size, timing of blinding assessment, blinded parties, sensation assessed, subject’s status, subject’s experience, and sham device used.

3. Results

3.1. Data Search

Using our search inclusion criteria, 590 peer reviewed journal articles were found, with 186 of these reporting blinding data in RCTs. 133 studies were excluded from the review, most often due to a lack of patient guess of treatment allocation. One article reported two distinct studies that were included separately [58]. Fifty-four studies were included in our final analysis, with a total of 4783 patients (Figure 1).

Figure 1: Systematic review search and selection. *One article contained two separate studies.

3.2. Blinding Index Calculations

The blinding index values (point and interval estimates) computed from all 54 studies can be found in Figure 2. The average weighted blinding index values for the entire review were 0.34 for verum and −0.20 for sham groups, respectively. Overall, a correct guess is quite common in the verum arm, and opposite guess is not uncommon in the sham arm.

(a)

(b)

Figure 2: (a) Verum and (b) sham. Blinding index values with 95% confidence intervals. *Individual blinding index estimate and confidence intervals raw data are provided in Table 3. Confidence intervals are unadjusted for multiple comparisons.

After grouping studies into the nine possible blinding scenarios based on the blinding index (Table 1), 33 out of 54 (61%) of the studies might be adequately blinded. Of these, 70% (23/33) had a positive outcome reported overall; similarly, 62% (13/21) with less ideal blinding had an overall positive outcome (Table 3). Unblinded/opposite for V, S is most common, with 46% of the studies belonging to this scenario, followed by unblinded/random with 22%.

3.3. Design Characteristics and Effectiveness of Blinding

The variables hypothesized to affect blinding were compared by their average VBI and SBI values, and blinding scenarios in Table 2.

Table 2: Design characteristics that may be associated with blinding, compared by BI values and scenarios.

Study characteristics
Number of studies
BI ()
BI ()
BI

()
Blinding scenario

Sample size

43
0.44
−0.19
0.25
Unblinded/random

11
0.28
−0.20
0.08
Unblinded/opposite

Blinding assessed

    Immediately
19
0.50
−0.19
0.31
Unblinded/random

    Later
35
0.29
−0.20
0.09
Unblinded/opposite

Blinded parties

    Subjects
22
0.52
−0.27
0.25
Unblinded/opposite

    Subjects + research staff
32
0.26
−0.17
0.09
Unblinded/random

Assessed for sensation of

    Deqi
22
0.33
−0.21
0.12
Unblinded/opposite

    Puncture
32
0.35
−0.19
0.16
Unblinded/random

Subject’s status

    Healthy
12
0.43
−0.06
0.37
Unblinded/random

    Symptomatic
42
0.33
−0.22
0.11
Unblinded/opposite

Subject’s acupuncture experience

    Naïve
24
0.27
−0.13
0.14
Unblinded/random

    Experienced
19
0.32
−0.19
0.13
Unblinded/random

Sham device used

    Commercial
22
0.47
−0.32
0.15
Unblinded/opposite

    Custom
14
0.48
−0.17
0.31
Unblinded/random

    Penetrating
12
0.16
−0.08
0.08
random/random

    Toothpick or cocktail stick
6
0.55
−0.33
0.22
Unblinded/opposite

 *Raw data available in Table 4.

Table 3: Blinding index values computed from 54 studies.

Study

VBI
VBI 95% CI
SBI
SBI 95% CI
Scenario
Study outcome

Assefi et al. 2005 [14]
75
−0.17
−0.49 to 0.15
0.3
0.10 to 0.49
Random/unblinded
Negative

Kaptchuk et al. 2008 [15]
148
0.12
−0.07 to 0.31
−0.52
−0.69 to −0.35
Random/opposite
Positive

Kennedy et al. 2008 [9]
45
0.18
−0.13 to 0.49
−0.38
−0.70 to −0.05
Random/opposite
Positive

Barlas et al. 2006 [6]
48
−0.09
NA
−0.06
NA
Random/random
Positive

Deng et al. 2008 [16]
53
0.08
−0.30 to 0.46
0.19
−0.18 to 0.55
Random/random
Negative

Enblom et al. 2008 [7]
80
0.2
0.00 to 0.40
0.1
−0.13 to 0.33
Random/random
Positive

Endres et al. 2007 [17]
403
−0.16
−0.35 to 0.03
−0.04
−0.24 to 0.16
Random/random
Positive

Haake et al. 2007 [8]
692
−0.01
−0.09 to 0.07
0.11
0.03 to 0.19
Random/random
Positive

Harris et al. 2009 [18]
20
0.2
−0.26 to 0.66
−0.2
−0.57 to 0.17
Random/random
Positive

Shin et al. 2010 [19]
42
−0.05
−0.35 to 0.18
0.05
−0.23 to 0.33
Random/random
Negative

Zaslawski et al. 1997 [20]
64
0.12
−0.37 to 0.13
0.03
−0.28 to 0.21
Random/random
Positive

Deng et al. 2007 [21]
67
0
−0.29 to 0.29
0.46
0.17 to 0.75
Random/unblinded
Negative

Jubb et al. 2008 [22]
51
0.18
−0.14 to 0.51
0.3
0.03 to 0.56
Random/unblinded
Positive

Alecrim-Andrade et al. 2006 [23]
24
0.21
0.11 to 0.33
−0.29
−0.37 to −0.20
Unblinded/opposite
Negative

Berman et al. 2004 [24]
283
0.73
0.63 to 0.82
−0.48
−0.61 to −0.35
Unblinded/opposite
Positive

Brinkhaus et al. 2006 [25]
205
0.44
0.30 to 0.57
−0.24
−0.42 to −0.05
Unblinded/opposite
Positive

Enblom et al. 2011 [26]
190
0.72
0.60 to 0.83
−0.6
−0.74 to −0.46
Unblinded/opposite
Positive

Fink et al. 2001 [27]
64
1
NA
−0.75
−0.98 to −0.52
Unblinded/opposite
Positive

Goddard et al. 2005 [28]
49
0.58
0.26 to 0.91
−0.76
−1.01 to −0.50
Unblinded/opposite
Positive

Goldman et al. 2008 [29]
118
0.42
0.24 to 0.60
−0.62
−0.77 to −0.47
Unblinded/opposite
Negative

Itoh et al. 2006 [30]
19
0.48
0.14 to 0.81
−0.32
−0.70 to 0.07
Unblinded/opposite
Positive

Itoh et al. 2008 [31]
24
0.56
0.01 to 1.10
−0.5
−1.10 to 0.10
Unblinded/opposite
Positive

Lee et al. 2008 [32]
89
0.91
0.79 to 1.03
−0.68
−0.90 to −0.47
Unblinded/opposite
Positive

Lee et al. 2011 [33]
35
0.23
−0.06 to 0.51
−0.56
−0.80 to −0.31
Unblinded/opposite
Positive

Park et al. 2002 [34]
58
0.38
0.20 to 0.56
−0.31
−0.48 to −0.14
Unblinded/opposite
Positive

Park et al. 2005 [35]
94
0.47
0.32 to 0.61
−0.31
−0.48 to −0.13
Unblinded/opposite
Negative

Shen et al. 2009 [36]
12
0.63
0.24 to 1.01
−0.5
−0.99 to −0.01
Unblinded/opposite
Positive

Sherman et al. 2002 [37]
52
0.65
0.34 to 0.96
−0.52
−0.80 to −0.24
Unblinded/opposite
Positive

Smith et al. 2007 [38]
27
1
NA
−1
NA
Unblinded/opposite
Positive

So et al. 2009 [39]
370
0.52
0.43 to 0.61
−0.36
−0.46 to −0.25
Unblinded/opposite
Negative

Streitberger and Kleinhenz 1998 [40]
60
0.8
0.65 to 0.95
−0.57
−0.77 to −0.36
Unblinded/opposite
Positive

Streitberger et al. 2004 [41]
212
0.22
0.06 to 0.37
−0.25
−0.38 to −0.11
Unblinded/opposite
Negative

Tong et al. 2010 [42]
63
1
NA
−0.81
−1.06 to −0.56
Unblinded/opposite
Positive

Tough et al. 2009 [43]
37
0.57
0.30 to 0.75
−0.67
−0.93 to −0.40
Unblinded/opposite
Positive

Venzke et al. 2010 [44]
51
0.48
0.15 to 0.81
−0.42
−0.78 to −0.05
Unblinded/opposite
Negative

White et al. 1996 [45]
9
1
NA
−0.6
−1.30 to 0.10
Unblinded/opposite
Negative

White et al. 2000 [46]
44
0.61
0.35 to 0.87
−0.26
−0.56 to 0.04
Unblinded/opposite
Negative

White et al. 2007 [47]
37
0.77
0.55 to 0.98
−0.29
−0.61 to 0.03
Unblinded/opposite
Positive

Alecrim-Andrade et al. 2008 [48]
36
0.26
0.14 to 0.39
−0.12
−0.23 to −0.00
Unblinded/random
Negative

Harris et al. 2005 [49]
76
0.25
−0.09 to 0.59
−0.13
−0.50 to 0.23
Unblinded/random
Negative

Lao et al. 1999 [50]
39
0.47
0.17 to 0.78
0.05
−0.21 to 0.31
Unblinded/random
Positive

Linde et al. 2005 [51]
201
0.5
0.38 to 0.63
−0.06
−0.26 to 0.15
Unblinded/random
Negative

Nabeta and Kawakita 2002 [52]
34
0.41
0.01 to 0.81
−0.18
−0.61 to 0.26
Unblinded/random
Positive

Shen and Goddard 2007 [53]
15
0.78
0.37 to 1.19
0
−0.80 to 0.80
Unblinded/random
Positive

Streitberger et al. 2008 [54]
20
0.55
0.29 to 0.81
−0.15
0.46 to 0.16
Unblinded/random
Positive

Takakura and Yajima 2007 [55]
60
0.6
0.40 to 0.80
0.17
−0.08 to 0.42
Unblinded/random
Positive

Takakura and Yajima 2008 [56]
114
0.37
0.20 to 0.54
−0.12
−0.30 to 0.06
Unblinded/random
Positive

Tan et al. 2009 [57]
20
0.25
0.04 to 0.46
0.1
−0.12 to 0.32
Unblinded/random
Positive

Tsukayama et al. 2006 BL23 [58]
20
0.4
−0.00 to 0.80
0
−0.44 to 0.44
Unblinded/random
Negative

Wasan et al. 2010 [59]
40
0.8
0.61 to 0.99
0
−0.31 to 0.31
Unblinded/random
Negative

Chae et al. 2011 [60]
28
0.57
0.14 to 1.00
0.71
0.35 to 1.08
Unblinded/unblinded
Positive

Kreiner et al. 2010 [61]
32
0.34
0.11 to 0.57
0.44
0.22 to 0.66
Unblinded/unblinded
Positive

Tsukayama et al. 2006 L14 [58]
20
1
NA
0.3
−0.12 to 0.72
Unblinded/unblinded
Negative

Wayne et al. 2008 [62]
14
0.22
−0.19 to 0.63
0.4
−0.30 to 1.10
Unblinded/unblinded
Positive

Table 4: Study characteristics extracted from 54 studies.

Study
Total
Party blinded
Time of blinding assessment (days from beginning of study)
Deqi assessment (yES/nO)
Sham device
Penetration of sham
Study participant diagnosis
Days since last acupuncture experience

Shen and Goddard 2007 [53]
15
Staff + subject
0
NO
C
NP
Chronic TM pain
Unknown

Itoh et al. 2006 [30]
19
Staff + subject
21
NO
C
NP
Chronic LBP
Unknown

Nabeta and Kawakita

2002 [52]
34
Subject
28
NO
C
NP
Chronic neck and shoulder pain
Unknown

Tough et al. 2009 [43]
37
Subject
14
NO
C
NP
Whiplash
Unknown

Fink et al. 2001 [27]
64
Staff + subject
14
NO
C
NP
Tension headache
Unknown

Shen et al. 2009 [36]
12
Staff + subject
0
YES
C
NP
Chronic myofacial pain of jaw muscles
>365

Kreiner et al. 2010 [61]
32
Staff + subject
0
NO
C
NP
Normal
>365

Lao et al. 1999 [50]
39
Staff + subject
0.2
YES
C
NP
After oral surgery Pain
>365

Goddard et al. 2005 [28]
49
Subject
0
YES
C
NP
Normal
>365

Jubb et al. 2008 [22]
51
Subject
63
YES
C
NP
Knee OA
>365

Itoh et al. 2008 [31]
24
Staff + subject
35
NO
C
NP
Knee OA
180

Deng et al. 2008 [16]
53
Subject
30
NO
C
NP
After thoracotomy Pain
42

Takakura and Yajima

2007 [55]
60
Staff + subject
0
NO
C
NP
Normal
0

Takakura and Yajima

2008 [56]
114
Staff + subject
0
YES
C
NP
Normal
0

Smith et al. 2007 [38]
27
Staff + subject
28
NO
P
NP
TMJ
Unknown

Tan et al. 2009 [57]
20
Subject
0
NO
P
NP
Normal
>365

Kennedy et al. 2008 [9]
45
Subject
42
NO
P
NP
Acute LBP
>365

Park et al. 2002 [34]
58
Staff + subject
84
NO
P
NP
Acute stroke
>365

Park et al. 2005 [35]
94
Staff + subject
14
NO
P
NP
Stroke
>365

Enblom et al. 2011 [26]
190
Staff + subject
35
NO
P
NP
Radiation-induced nausea
365

Tsukayama et al. 2006

BL23 [58]
20
Staff + subject
0
NO
P
NP
Normal
0

Tsukayama et al. 2006

L14 [58]
20
Staff + subject
0
NO
P
NP
Normal
0

Chae et al. 2011 [60]
28
Subject
0
NO
P
NP
Normal
0

Shin et al. 2010 [19]
42
Staff + subject
21
YES
PEN
P
Dry eye
Unknown

Tong et al. 2010 [42]
63
Subject
15
YES
PEN
P
DPN
~50%

Zaslawski et al. 1997 [20]
64
Subject
28
YES
PEN
P
Stress
~50%

Harris et al. 2005 [49]
76
Staff + subject
21
NO
PEN
P
Fibromyaligia
>365

Haake et al. 2007 [8]
692
Staff + subject
168
NO
PEN
P
Chronic LBP
>365

Linde et al. 2005 [51]
201
Staff + subject
168
YES
PEN
P
Migraine HA
365

Brinkhaus et al. 2006 [25]
205
Subject
56
YES
PEN
P
Chronic LBP
365

Endres et al. 2007 [17]
403
Staff + subject
180
YES
PEN
P
Tension headache
365

Alecrim-Andrade et al. 2006 [23]
24
Staff + subject
84
YES
PEN
P
Migraine HA
84

Alecrim-Andrade et al. 2008 [48]
36
Subject
84
YES
PEN
P
Migraine HA
84

Lee et al. 2011 [33]
35
Staff + subject
70
NO
PEN
P
Chronic pelvic pain
42

Lee et al. 2008 [32]
89
Subject
70
NO
PEN
P
Chronic prostatitis
42

White et al. 2007 [47]
37
Subject
0
YES
S
NP
Chronic Pain/Normal
Unknown

So et al. 2009 [39]
370
Staff + subject
0
NO
S
NP
Infertility
Unknown

Wayne et al. 2008 [62]
14
Staff + subject
28
NO
S
NP
Endometriosis
>365

Streitberger et al. 2008 [54]
20
Subject
0
NO
S
NP
Normal
>365

Wasan et al. 2010 [59]
40
Subject
0
YES
S
NP
Chronic LBP
>365

Barlas et al. 2006 [6]
48
Staff + subject
0
YES
S
NP
Normal
>365

Streitberger and Kleinhenz 1998 [40]
60
Subject
0
NO
S
NP
Normal
>365

Enblom et al. 2008 [7]
80
Subject
0
NO
S
NP
Normal
>365

Kaptchuk et al. 2008 [15]
148
Staff + subject
42
NO
S
NP
IBS
>365

Goldman et al. 2008 [29]
118
Staff + subject
28
YES
S
NP
Arm pain
365

Venzke et al. 2010 [44]
51
Subject
168
YES
S
NP
Hot flashes
180

Streitberger et al. 2004 [41]
212
Staff + subject
1
YES
S
NP
postoperative nausea
180

Deng et al. 2007 [21]
67
Subject
42
YES
S
NP
Hot flashes
42

White et al. 1996 [45]
9
Staff + subject
42
NO
T
NP
Tension headache
>365

Harris et al. 2009 [18]
20
Staff + subject
0
NO
T
NP
Fibromyalgia
>365

White et al. 2000 [46]
44
Staff + subject
7
YES
T
NP
Tension headache
>365

Sherman et al. 2002 [37]
52
Subject
0
NO
T
NP
Chronic LBP
>365

Assefi et al. 2005 [14]
75
Staff + subject
84
NO
T
NP, P
Fibromyalgia
>365

Berman et al. 2004 [24]
283
Subject
182
YES
T
NP, P
Knee OA
>365

Sham devices: C: custom sham, P: park sham, PEN: penetrating sham, S: streitberger sham, and T: toothpick.

Of the 54 studies, 22 studies used a commercially developed sham control device, 14 studies used a custom-made sham control device, 12 used penetrating sham control, and 6 used a toothpick or cocktail stick. According to their averaged blinding index scenarios, all of the sham control devices with the exception of custom devices seemed to be effective in blinding the subjects, with the penetrating sham controls providing relatively more effective blinding.

In looking at the penetrating versus nonpenetrating dichotomy in a more direct way, the nonpenetrating group was unblinded/opposite, the mixed penetrating/nonpenetrating group was unblinded/opposite, and the penetrating group was random/random. All of these scenarios indicated effective blinding.

Studies with a greater number of subjects had a greater unblinding in the verum group. Measurements assessed later tended to have more ideal effectiveness of blinding than measurements assessed immediately. Interestingly, there was a higher tendency in both arms, when only the subjects—not staff—were blinded, to believe they received the verum treatment. The deqi group had slightly better EOB than the insertion/puncture group, and the symptomatic group had more ideal EOB than the healthy group.

Twenty-four studies used acupuncture-naïve subjects, and 19 used subjects with prior acupuncture experience. Eleven of the studies were excluded from this section of the review due to unknown prior experience or mixed experience of the subjects. According to their averaged blinding index scenarios, both groups were unblinded/random.

4. Discussion

This systematic review of 54 randomized controlled acupuncture studies showed that overall 61% of the studies (as a conservative estimate) meeting our inclusion/exclusion criteria were effectively blinded. The most common scenario encountered was unblinded in the acupuncture group and opposite guess in the sham acupuncture group, which could be indeed interpreted as “well-blinded.” In this scenario there may be a psychological phenomenon of “wishful thinking.” A majority of people, in both the verum and sham groups, guessed that they received real acupuncture. Thus, guesses are inflated towards real acupuncture in both study arms. It is also possible that once a needle is administered, a subject believes it is real acupuncture, or subjects may not know what to expect, creating a similar trend, or there is a strong placebo effect.

A similar pattern emerges when looking at the V and S groups individually; the average VBI was “unblinded” and the average SBI was “opposite.” Is it possible for one not to know when a needle is penetrating his or her skin? Perhaps the answer is “no,” given that unblinded V may mean that subjects know when their skin is being penetrated by a needle and thus increases the chance a subject chooses the V acupuncture group over the S acupuncture group upon questioning. Is it possible for one to know when a needle is not penetrating their skin? The answer seems to be “no” again; opposite guesses in S may indicate that subjects are not able to tell if they are not being penetrated by a needle, and thus are truly guessing.

Most sham control devices with the exception of custom devices were effective in blinding the subjects (Table 2). Since there was a great diversity within the custom sham group a more in depth case by case analysis of each device could be performed, but at this time there does not seem to be compelling evidence supporting the use of custom sham devices. Even though commercial sham devices and toothpick/cocktail stick devices appear to provide effective blinding, the penetrating sham controls provided relatively more effective blinding.

By comparing study blinding and study outcomes, the majority of studies reported positive outcomes, regardless of the degree of guess correctness. This leads us to believe that there is no obvious association between EOB and reported study outcomes. The current literature provides conflicting evidence so the direction of bias may be specific to context or treatment [6–9, 14–49, 51–62] or random.

Exploration of individual variables and their possible effect on EOB indicated that some design characteristics such as larger sample size, symptomatic subjects, and later assessment were associated with more effective blinding, and these may be encouraged to be further evaluated and considered in designing future acupuncture trials.

4.1. Recommendations for Future Acupuncture Research

The effect size was not a part of the extracted information from the reviewed studies. This poses an interesting idea for future investigation. Additional future research should include a sufficiently powered, prospective randomized trial comparing the EOB of different methods and sham devices by direct comparison, as well as investigating the influence of practitioner behavior, the patient’s expectations and beliefs, and the location of treatment points on blinding effectiveness. A good treatment should have a greater treatment effect than placebo effect. It is important that we collect more data in this field, especially qualitative data (e.g., reasons of guessing in a particular way). It is possible that the reasons for correct guesses in individual trials may be more revealing than our numbers. Individual trialists should be willing to share their experiences with others, as individual trialists and patients must have greater insight and more stories to tell in specific conditions within the trial than meta-analysis and readers. This is particularly important, because any analysis of available numeric data on blinding is destined to be prone to some biases.

4.2. Systematic Review Limitations

There are several potential weaknesses of this review that we recognize. Language bias may be a possibility given that we included only English language publications while acupuncture is a popular treatment modality in Asia and Europe. There is also a possibility of publication bias, which is an inherent problem in virtually all systematic reviews or meta-analyses. Moreover, it is possible that some investigators collected blinding data, saw the data, and decided not to report it in the paper, particularly when blinding was shown to be unsatisfactory. Also, for classification and decision making purposes, we used categorization/dichotomization (e.g., numerical BI cutoff of 0.2) and summary statistics (e.g., average BI). Alternative ways of analyzing the data could yield different results. Finally, in this systematic review, we used the conventional terms commonly used in the literature and equated “unblinded” with “correctly guessed,” whereas there could be other reasons for correct guesses. With its limitations, to our knowledge, this is the first systematic review based on empirical blinding data.

In conclusion, based on the status of blinding, the most common scenario encountered was a more correct guess in the real acupuncture group and an opposite guess in the sham acupuncture group, and the overall subject blinding of the evaluated acupuncture studies was satisfactory. In addition, quantitatively calculated blinding data, ideally together with more qualitative data for individual trials, may offer meaningful means to further interpret the findings of RCTs and improve the practice in the direction of a higher validity.

Disclosure

J. J. Park developed the Park Sham Device via AcuPrime in UK and was supported by the National Institute of Dental and Craniofacial Research of the National Institute of Health under award number K12DE022793. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute of Health.

Acknowledgments

The authors acknowledge the assistance of Hannah McFadden during the preparation of the paper.

References

Yeung and C. Albert, “New guidelines for trial reporting—CONSORT, 2010,” British Medical Journal, vol. 340, Article ID c332, 2010.

A. J. Vickers, “Placebo controls in randomized trials of acupuncture,” Evaluation & the Health Professions, vol. 25, pp. 421–435, 2002.

C. Vincent, “Credibility assessment in trials of acupuncture,” Complementary Medicine Research, vol. 4, pp. 8–11, 1990.

R. Wood and G. Lewith, “The credibility of placebo controls in acupuncture studies,” Complementary Therapies in Medicine, vol. 6, no. 2, pp. 79–82, 1998.

I. Boutron, D. Moher, D. G. Altman, K. F. Schulz, and P. Ravaud, “Extending the CONSORT statement to randomized trials of nonpharmacologic treatment: explanation and elaboration,” Annals of Internal Medicine, vol. 148, no. 4, pp. 295–309, 2008.

P. Barlas, S. L. H. Ting, L. S. Chesterton, P. W. Jones, and J. Sim, “Effects of intensity of electroacupuncture upon experimental pain in healthy human volunteers: a randomized, double-blind, placebo-controlled study,” Pain, vol. 122, no. 1-2, pp. 81–89, 2006.

A. Enblom, M. Hammar, G. Steineck, and S. Börjeson, “Can individuals identify if needling was performed with an acupuncture needle or a non-penetrating sham needle?” Complementary Therapies in Medicine, vol. 16, no. 5, pp. 288–294, 2008.

M. Haake, H. H. Müller, C. Schade-Brittinger et al., “German Acupuncture Trials (GERAC) for chronic low back pain: randomized, multicenter, blinded, parallel-group trial with 3 groups,” Archives of Internal Medicine, vol. 167, no. 17, pp. 1892–1898, 2007.

S. Kennedy, G. D. Baxter, D. P. Kerr, I. Bradbury, J. Park, and S. M. McDonough, “Acupuncture for acute non-specific low back pain: a pilot randomised non-penetrating sham controlled trial,” Complementary Therapies in Medicine, vol. 16, no. 3, pp. 139–146, 2008.

T. Lundeberg, I. Lund, J. Näslund, and M. Thomas, “The Emperor's sham—wrong assumption that sham needling is sham,” Acupuncture in Medicine, vol. 26, no. 4, pp. 239–242, 2008.

A. Moroz, “Issues in acupuncture research: the failure of quantitative methodologies and the possibilities for viable, alternative solutions,” American Journal of Acupuncture, vol. 27, no. 1-2, pp. 95–103, 1999.

J. Park and H. Bang, “Methodological advances needed in analysis and interpretation of sham acupuncture validation studies,” Acupuncture in Medicine, vol. 29, no. 3, pp. 168–169, 2011.

J. Park, H. Bang, and I. Cañette, “Blinding in clinical trials, time to do it better,” Complementary Therapies in Medicine, vol. 16, no. 3, pp. 121–123, 2008.

N. P. Assefi, K. J. Sherman, C. Jacobsen, J. Goldberg, W. R. Smith, and D. Buchwald, “A randomized clinical trial of acupuncture compared with sham acupuncture in fibromyalgia,” Annals of Internal Medicine, vol. 143, no. 1, pp. 10–19, 2005.

T. J. Kaptchuk, J. M. Kelley, L. A. Conboy et al., “Components of placebo effect: randomised controlled trial in patients with irritable bowel syndrome,” British Medical Journal, vol. 336, no. 7651, pp. 999–1003, 2008.

G. Deng, V. Rusch, A. Vickers et al., “Randomized controlled trial of a special acupuncture technique for pain after thoracotomy,” Journal of Thoracic and Cardiovascular Surgery, vol. 136, no. 6, pp. 1464–1469, 2008.

H. G. Endres, G. Böwing, H. C. Diener et al., “Acupuncture for tension-type headache: a multicentre, sham-controlled, patient-and observer-blinded, randomised trial,” Journal of Headache and Pain, vol. 8, no. 5, pp. 306–314, 2007.

R. E. Harris, J. K. Zubieta, D. J. Scott, V. Napadow, R. H. Gracely, and D. J. Clauw, “Traditional Chinese acupuncture and placebo (sham) acupuncture are differentiated by their effects on μ-opioid receptors (MORs),” NeuroImage, vol. 47, no. 3, pp. 1077–1085, 2009.

M. S. Shin, J. I. Kim, M. S. Lee et al., “Acupuncture for treating dry eye: a randomized placebo-controlled trial,” Acta Ophthalmologica, vol. 88, no. 8, pp. e328–e333, 2010.

C. Zaslawski, C. Rogers, M. Garvey, D. Ryan, C. X. Yang, and S. P. Zhang, “Strategies to maintain the credibility of sham acupuncture used as a control treatment in clinical trials,” Journal of Alternative and Complementary Medicine, vol. 3, no. 3, pp. 257–266, 1997.

G. Deng, A. Vickers, S. Yeung, and B. Cassileth, “Randomized, controlled trial of acupuncture for the treatment of hot flashes in breast cancer patients,” Journal of Clinical Oncology, vol. 25, no. 35, pp. 5584–5590, 2007.

R. W. Jubb, E. S. Tukmachi, P. W. Jones, E. Dempsey, L. Waterhouse, and S. Brailsford, “A blinded randomised trial of acupuncture (manual and electroacupuncture) compared with a non-penetrating sham for the symptoms of osteoarthritis of the knee,” Acupuncture in Medicine, vol. 26, no. 2, pp. 69–78, 2008.

J. Alecrim-Andrade, J. A. Maciel, X. C. Cladellas, H. R. Correa-Filho, and H. C. Machado, “Acupuncture in migraine prophylaxis: a randomized sham-controlled trial,” Cephalalgia, vol. 26, no. 5, pp. 520–529, 2006.

B. M. Berman, L. Lao, P. Langenberg, W. L. Lee, A. M. K. Gilpin, and M. C. Hochberg, “Effectiveness of acupuncture as adjunctive therapy in osteoarthritis of the knee. A randomized, controlled trial,” Annals of Internal Medicine, vol. 141, no. 12, pp. 901–910, 2004.

B. Brinkhaus, C. M. Witt, S. Jena et al., “Acupuncture in patients with chronic low back pain: a randomized controlled trial,” Archives of Internal Medicine, vol. 166, no. 4, pp. 450–457, 2006.

A. Enblom, A. Johnsson, M. Hammar, G. Steineck, and S. Borjeson, “The nonpenetrating telescopic sham needle may blind patients with diferrent characteristics and experiences when treated by several therapists,” Evidence-Based Complementary and Alternative Medicine, vol. 2011, Article ID 185034, pp. 1–12, 2011.

M. Fink, C. Gutenbrunner, J. Rollnik, and M. Karst, “Credibility of a newly designed placebo needle for clinical trials in acupuncture research,” Forschende Komplementarmedizin und Klassische Naturheilkunde, vol. 8, no. 6, pp. 368–372, 2001.

G. Goddard, Y. Shen, B. Steele, and N. Springer, “A controlled trial of placebo versus real acupuncture,” Journal of Pain, vol. 6, no. 4, pp. 237–242, 2005.

R. H. Goldman, W. B. Stason, S. K. Park et al., “Acupuncture for treatment of persistent arm pain due to repetitive use: a randomized controlled clinical trial,” Clinical Journal of Pain, vol. 24, no. 3, pp. 211–218, 2008.

K. Itoh, Y. Katsumi, S. Hirota, and H. Kitakoji, “Effects of trigger point acupuncture on chronic low back pain in elderly patients—a sham-controlled randomised trial,” Acupuncture in Medicine, vol. 24, no. 1, pp. 5–12, 2006.

K. Itoh, S. Hirota, Y. Katsumi, H. Ochi, and H. Kitakoji, “Trigger point acupuncture for treatment of knee osteoarthritis—a preliminary RCT for a pragmatic trial,” Acupuncture in Medicine, vol. 26, no. 1, pp. 17–26, 2008.

S. W. H. Lee, M. L. Liong, K. H. Yuen et al., “Acupuncture versus sham acupuncture for chronic prostatitis/chronic pelvic pain,” American Journal of Medicine, vol. 121, no. 1, pp. 79.e1–79.e7, 2008.

S. W. H. Lee, M. L. Liong, K. H. Yuen, W. S. Leong, N. K. Khan, and J. N. Krieger, “Validation of a sham acupuncture procedure in a randomised, controlled clinical trial of chronic pelvic pain treatment,” Acupuncture in Medicine, vol. 29, no. 1, pp. 40–46, 2011.

J. Park, A. White, C. Stevinson, E. Ernst, and M. James, “Validating a new non-penetrating sham acupuncture device: two randomised controlled trials,” Acupuncture in Medicine, vol. 20, no. 4, pp. 168–174, 2002.

J. Park, A. R. White, M. A. James et al., “Acupuncture for subacute stroke rehabilitation: a sham-controlled, subject- and assessor-blind, randomized trial,” Archives of Internal Medicine, vol. 165, no. 17, pp. 2026–2031, 2005.

Y. F. Shen, J. Younger, G. Goddard, and S. Mackey, “Randomized clinical trial of acupuncture for myofascial pain of the jaw muscles,” Journal of Orofacial Pain, vol. 23, no. 4, pp. 353–359, 2009.

K. J. Sherman, C. J. Hogeboom, D. C. Cherkin, and R. A. Deyo, “Description and validation of a noninvasive placebo acupuncture procedure,” Journal of Alternative and Complementary Medicine, vol. 8, no. 1, pp. 11–19, 2002.

P. Smith, D. Mosscrop, S. Davies, P. Sloan, and Z. Al-Ani, “The efficacy of acupuncture in the treatment of temporomandibular joint myofascial pain: a randomised controlled trial,” Journal of Dentistry, vol. 35, no. 3, pp. 259–267, 2007.

E. So, E. Ng, Y. Wong, W. Yeung, and P. Ho, “A randomized double blind comparison of real and placebo acupuncture in IVF treatment,” Human Reproduction, vol. 24, pp. 341–348, 2009.

K. Streitberger and J. Kleinhenz, “Introducing a placebo needle into acupuncture research,” The Lancet, vol. 352, no. 9125, pp. 364–365, 1998.

K. Streitberger, M. Diefenbacher, A. Bauer et al., “Acupuncture compared to placebo-acupuncture for postoperative nausea and vomiting prophylaxis: a randomised placebo-controlled patient and observer blind trial,” Anaesthesia, vol. 59, no. 2, pp. 142–149, 2004.

Y. Tong, H. Guo, and B. Han, “Fifteen-day acupuncture treatment relieves diabetic peripheral neuropathy,” Journal of Acupuncture and Meridian Studies, vol. 3, no. 2, pp. 95–103, 2010.

E. A. Tough, A. R. White, S. H. Richards, B. Lord, and J. L. Campbell, “Developing and validating a sham acupuncture needle,” Acupuncture in Medicine, vol. 27, no. 3, pp. 118–122, 2009.

L. Venzke, J. F. Calvert, and B. Gilbertson, “A randomized trial of acupuncture for vasomotor symptoms in post-menopausal women,” Complementary Therapies in Medicine, vol. 18, no. 2, pp. 59–66, 2010.

A. R. White, C. Eddleston, R. Hardie, K. L. Resch, and E. Ernst, “A pilot study of acupuncture for tension headache, using a novel placebo,” Acupuncture in Medicine, vol. 14, no. 1, pp. 11–15, 1996.

A. R. White, K. L. Resch, J. C. K. Chan et al., “Acupuncture for episodic tension-type headache: a multicentre randomized controlled trial,” Cephalalgia, vol. 20, no. 7, pp. 632–637, 2000.

P. White, G. Lewith, and P. Prescott, “Should we recruit patients or healthy volunteers for acupuncture studies of chronic pain?” Clinical Journal of Pain, vol. 23, no. 8, pp. 714–719, 2007.

J. Alecrim-Andrade, J. A. Maciel-Júnior, X. Carnè, G. M. Severino Vasconcelos, and H. R. Correa-Filho, “Acupuncture in migraine prevention: a randomized sham controlled study with 6-months posttreatment follow-up,” Clinical Journal of Pain, vol. 24, no. 2, pp. 98–105, 2008.

R. E. Harris, X. Tian, D. A. Williams et al., “Treatment of fibromyalgia with formula acupuncture: investigation of needle placement, needle stimulation, and treatment frequency,” Journal of Alternative and Complementary Medicine, vol. 11, no. 4, pp. 663–671, 2005.

L. Lao, S. Bergman, G. R. Hamilton, P. Langenberg, and B. Berman, “Evaluation of acupuncture for pain control after oral surgery: a placebo-controlled trial,” Archives of Otolaryngology, vol. 125, no. 5, pp. 567–572, 1999.

K. Linde, A. Streng, S. Jürgens et al., “Acupuncture for patients with migraine: a randomized controlled trial,” Journal of the American Medical Association, vol. 293, no. 17, pp. 2118–2125, 2005.

T. Nabeta and K. Kawakita, “Relief of chronic neck and shoulder pain by manual acupuncture to tender points—a sham-controlled randomized trial,” Complementary Therapies in Medicine, vol. 10, no. 4, pp. 217–222, 2002.

Y. F. Shen and G. Goddard, “The short-term effects of acupuncture on myofascial pain patients after clenching,” Pain Practice, vol. 7, no. 3, pp. 256–264, 2007.

K. Streitberger, J. Steppan, C. Maier, H. Hill, J. Backs, and K. Plaschke, “Effects of verum acupuncture compared to placebo acupuncture on quantitative EEG and heart rate variability in healthy volunteers,” Journal of Alternative and Complementary Medicine, vol. 14, no. 5, pp. 505–513, 2008.

N. Takakura and H. Yajima, “A double-blind placebo needle for acupuncture research,” BMC Complementary and Alternative Medicine, vol. 7, article 31, 2007.

N. Takakura and H. Yajima, “A placebo acupuncture needle with potential for double blinding—a validation study,” Acupuncture in Medicine, vol. 26, no. 4, pp. 224–230, 2008.

C. W. Tan, L. Christie, V. St-Georges, and N. Telford, “Discrimination of real and sham acupuncture needles using the park sham device: a Preliminary Study,” Archives of Physical Medicine and Rehabilitation, vol. 90, no. 12, pp. 2141–2145, 2009.

H. Tsukayama, H. Yamashita, T. Kimura, and K. Otsuki, “Factors that influence the applicability of sham needle in acupuncture trials: two randomized, single-blind, crossover trials with acupuncture—experienced subjects,” Clinical Journal of Pain, vol. 22, no. 4, pp. 346–349, 2006.

A. D. Wasan, J. Kong, L. D. Pham, T. J. Kaptchuk, R. Edwards, and R. L. Gollub, “The impact of placebo, psychopathology, and expectations on the response to acupuncture needling in patients with chronic low back pain,” Journal of Pain, vol. 11, no. 6, pp. 555–563, 2010.

Y. Chae, S. I. Um, S. H. Yi et al., “Comparison of biomechanical properties between acupuncture and non-penetrating sham needle,” Complementary Therapies in Medicine, vol. 19, no. 1, pp. S8–S12, 2011.

M. Kreiner, A. Zaffaroni, R. Alvarez, and G. Clark, “Validation of a simplified sham acupuncture technique for its use in clinical research: a randomised, single blind, crossover study,” Acupuncture in Medicine, vol. 28, no. 1, pp. 33–36, 2010.

P. M. Wayne, C. E. Kerr, R. N. Schnyer et al., “Japanese-style acupuncture for endometriosis-related pelvic pain in adolescents and young women: results of a randomized sham-controlled trial,” Journal of Pediatric and Adolescent Gynecology, vol. 21, no. 5, pp. 247–257, 2008.

H. Bang, L. Ni, and C. E. Davis, “Assessment of blinding in clinical trials,” Controlled Clinical Trials, vol. 25, no. 2, pp. 143–156, 2004.

H. Bang, S. P. Flaherty, J. Kolahi, and J. Park, “Blinding assessment in clinical trials: a review of statistical methods and a proposal of blinding assessment protocol,” Clinical Research and Regulatory Affairs, vol. 27, no. 2, pp. 42–51, 2010.

J. Kolahi, H. Bang, and J. Park, “Towards a proposal for assessment of blinding success in clinical trials: up-to-date review,” Community Dentistry and Oral Epidemiology, vol. 37, no. 6, pp. 477–484, 2009.