Nursing Research and Practice

Volume 2014 (2014), Article ID 896587, 28 pages

http://dx.doi.org/10.1155/2014/896587

Review Article

A Systematic Review of the Cost-Effectiveness of Nurse Practitioners and Clinical Nurse Specialists: What Is the Quality of the Evidence?

Faith Donald,¹ Kelley Kilpatrick,² Kim Reid,³ Nancy Carter,⁴ Ruth Martin-Misener,⁵ Denise Bryant-Lukosius,4,6 Patricia Harbman,4,7 Sharon Kaasalainen,⁴ Deborah A. Marshall,⁸ Renee Charbonneau-Smith,⁴ Erin E. Donald,⁹ Monique Lloyd,¹⁰ Abigail Wickson-Griffiths,⁴ Jennifer Yost,⁴ Pamela Baxter,⁴ Esther Sangster-Gormley,¹¹ Pamela Hubley,¹² Célyne Laflamme,¹³ Marsha Campbell–Yeo,5,14 Sheri Price,⁵ Jennifer Boyko,¹⁵ and Alba DiCenso4,16

¹Daphne Cockwell School of Nursing, Ryerson University, 350 Victoria Street, Toronto, ON, Canada, M5B 2K3

²Faculty of Nursing, Université de Montreal and Research Centre of Hôpital Maisonneuve-Rosemont, CSA-RC-Aile Bleue-Room F121, 5415 boulevard l’Assomption, Montréal, QC, Canada, H1T 2M4

³KJ Research, Rosemere, QC, Canada, J7A 4N8

⁴School of Nursing, McMaster University, 1280 Main Street West, Hamilton, ON, Canada, L8S 4L8

⁵School of Nursing, Dalhousie University, Box 15000, 5869 University Avenue, Halifax, NS, Canada, B3H 4R2

⁶Department of Oncology, McMaster University, 1280 Main Street West, HSC-3N28G, Hamilton, ON, Canada, L8S 4L8

⁷Health Interventions Research Centre, Ryerson University, 350 Victoria Street, Toronto, ON, Canada, M5B 2K3

⁸Department of Community Health Sciences, Faculty of Medicine, University of Calgary, Health Research Innovation Centre, Room 3C56, 3280 Hospital Drive NW, Calgary, AB, Canada, T2N 4Z6

⁹Fraser Health Authority, Suite 400-13450 102nd Avenue, Surrey, BC, Canada, V3T 0H1

¹⁰International Affairs and Best Practice Guidelines Centre, Registered Nurses’ Association of Ontario, 158 Pearl Street, Toronto, ON, Canada, M5H 1L3

¹¹School of Nursing, University of Victoria, P.O. Box 1700 STN CSC, Victoria, BC, Canada, V8W 2Y2

¹²The Hospital for Sick Children, Lawrence S. Bloomberg Faculty of Nursing, University of Toronto, 555 University Avenue, Toronto, ON, Canada, M5G 1X8

¹³Primary Health Care Nurse Practitioner Program, School of Nursing, University of Ottawa, 600 Peter Morand Crescent, Suite 101, Ottawa, ON, Canada, K1G 5Z3

¹⁴Departments of Pediatrics and Psychology and Neurosciences, Dalhousie University, P.O. Box 15000, 5869 University Avenue, Halifax, NS, Canada, B3H 4R2

¹⁵School of Health Studies, Western University, Health Sciences Building, Room 403, London, ON, Canada, N6A 5B9

¹⁶Department of Clinical Epidemiology & Biostatistics, McMaster University, 1280 Main Street West, Hamilton, ON, Canada, L8S 4L8

Received 2 March 2014; Revised 26 June 2014; Accepted 27 June 2014; Published 1 September 2014

Academic Editor: Patrick Callaghan

Copyright © 2014 Faith Donald et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

Background. Improved quality of care and control of healthcare costs are important factors influencing decisions to implement nurse practitioner (NP) and clinical nurse specialist (CNS) roles. Objective. To assess the quality of randomized controlled trials (RCTs) evaluating NP and CNS cost-effectiveness (defined broadly to also include studies measuring health resource utilization). Design. Systematic review of RCTs of NP and CNS cost-effectiveness reported between 1980 and July 2012. Results. 4,397 unique records were reviewed. We included 43 RCTs in six groupings, NP-outpatient (), NP-transition (), NP-inpatient (), CNS-outpatient (), CNS-transition (), and CNS-inpatient (). Internal validity was assessed using the Cochrane risk of bias tool; 18 (42%) studies were at low, 17 (39%) were at moderate, and eight (19%) at high risk of bias. Few studies included detailed descriptions of the education, experience, or role of the NPs or CNSs, affecting external validity. Conclusions. We identified 43 RCTs evaluating the cost-effectiveness of NPs and CNSs using criteria that meet current definitions of the roles. Almost half the RCTs were at low risk of bias. Incomplete reporting of study methods and lack of details about NP or CNS education, experience, and role create challenges in consolidating the evidence of the cost-effectiveness of these roles.

1. Introduction

Nurse practitioners (NPs) and clinical nurse specialists (CNSs) have practiced for over 50 years in the United States, followed closely by Canada and the United Kingdom, and the roles are increasingly being implemented in other countries [1]. The quest for improved quality of care and control of healthcare costs are important drivers in the decision to implement these roles. We conducted a systematic review to assess the evidence of cost-effectiveness of NP and CNS roles.

2. Background

Both NPs and CNSs are considered advanced practice nurses [2]. NPs are defined as RNs who have additional education in recognized programs, preferably at the graduate level. They demonstrate advanced competencies to practice autonomously and collaboratively to perform assessments, order laboratory and diagnostic tests, diagnose, prescribe medications and treatments, and perform procedures, as authorized by legislation and their regulatory scope of practice [2], as well as performing an advanced nursing role that includes consultation, collaboration, education, research, and leadership. CNSs are registered nurses (RNs) with a graduate degree in nursing who have expertise in a clinical specialty and perform an advanced nursing role that includes practice, consultation, collaboration, education, research, and leadership [3].

NPs and CNSs function in alternative or complementary provider roles. Those working in alternative roles provide similar services to those for whom they are substituting, usually physicians [4]. Those working in complementary roles provide additional services that are intended to complement or extend existing services. The intention of the alternative role is typically to reduce cost or workload or to address workforce shortages while maintaining or improving the quality of care; in contrast, the intention of the complementary role is to improve the quality of care [5].

During the 1970s, the first randomized controlled trials (RCTs) of NPs demonstrated their safety and effectiveness, as well as patient satisfaction with the NP role [6–17]. NPs improved resource utilization and access to care [14, 18–20], increased primary care services in the community [7], and reduced costs [15]. Over the past 30 years, a number of literature reviews and systematic reviews have summarized the findings of studies evaluating NPs [21–25]. The reviews have consistently shown no difference in the health outcomes of patients receiving NP care when compared to patients receiving physician care, but often both quality of care and patient satisfaction are higher with NP care.

Most RCTs of CNS roles have been published since 1980 except one. In 1977, Pozen and colleagues [26] found that the CNS increased the knowledge of heart disease in patients with myocardial infarction resulting in an increased rate of return to work and a reduction in smoking. Literature reviews and systematic reviews of CNSs [25, 27] reveal that CNSs are associated with reductions in hospital length of stay, readmissions, emergency room visits, and costs, as well as improvements in staff nurse knowledge, functional performance, mood state, quality of life, and patient satisfaction.

Study findings are consistent that NPs and CNSs, either in alternative or complementary provider roles, deliver high quality patient care that results in high patient satisfaction. To address a question that often surfaces, “are NPs and CNSs cost-effective?,” we conducted a systematic review of RCTs of NP and CNS cost-effectiveness (defined broadly to also include studies measuring health resource utilization) entitled “A systematic review of the cost-effectiveness of nurse practitioners and clinical nurse specialists: 1980 to July 2012.” The purpose of this paper is to report on the methodological strengths and threats to internal and external validity of these RCTs.

3. Methods

3.1. Eligibility Criteria

We sought RCTs of NP and CNS cost-effectiveness between January 1980 and July 2012. Due to inconsistencies in the use of titles and lack of role clarity for these two roles [28], we developed specific criteria to decide if the role was an NP, a CNS, or an RN in an expanded role. To be deemed an NP, the nurse had to have completed a formal postbaccalaureate or graduate NP education program or be licensed as an NP. To be deemed a CNS, the nurse had to have completed a graduate degree and the role had to be reflective of the CNS role definition. If necessary, we contacted the lead author and/or experts in advanced practice nursing from the country where the study was conducted to determine eligibility.

The principal outcomes of interest in this review were objective measures of health system utilization. These included length of stay, rehospitalization, costs of healthcare (e.g., hospital, professional, and family costs), and health resource use (e.g., diagnostic tests and prescriptions). Because it is important to examine health system utilization in the context of patient and provider outcomes, we also extracted data on all patient (e.g., mortality, morbidity, quality of life, and satisfaction with care) and provider (e.g., quality of care and job satisfaction) outcomes.

Participants were patients of any age receiving care in all types (e.g., teaching and nonteaching, public and private), sizes (e.g., small, medium, and large), and locations (e.g., rural and urban) of hospitals or community settings (e.g., long-term care, primary care, and home care).

Substantive developments since 1980 (e.g., training, payment models, and scope of practice of NPs) have reduced the relevance of pre-1980 studies to modern-day policy. In consultation with a policy advisor, we chose to exclude pre-1980 studies from this review. Studies were also excluded if (1) the NP or CNS education failed to meet our criteria or if we could not contact the author for clarification despite repeated attempts; (2) the NP or CNS was part of a multicomponent or multidisciplinary intervention in which the impact of their contribution could not be isolated from other healthcare providers on the team; (3) the study evaluated a very specific intervention (e.g., cognitive behavioural therapy) that was delivered by an NP or CNS but could be delivered by other clinicians, such as an RN; (4) the control group was also exposed to an NP or CNS during the study; (5) a measure of health system utilization was not included; (6) true randomization was not used (randomization was predictable, for example, assignment by day of hospital admission and alternating assignment).

3.2. Search Strategy

A search was conducted to identify all relevant published and unpublished RCTs reported from January 1980 to July 2012. No restrictions were imposed on jurisdiction or language. Medical librarians conducted a comprehensive search of the literature using CINAHL, EMBASE, Global Health, HealthStar, Medline, Allied and Complementary Medicine Database (AMED), Cochrane Library Database of Systematic Reviews and Controlled Trials Register, Database of Abstracts of Reviews of Effects (DARE), Health Economics Evaluation Database (HEED), and Web of Science. Relevant Medical Subject Headings (MeSH) keywords, inclusive suffixes, and search strings formed the search strategy (appendix). In addition, the following methods were used to identify primary studies: handsearching of 16 high-yield journals, checking reference lists of all relevant papers and reviews, contacting authors of an early list of relevant studies, searching personal files, reviewing bibliographies, and searching websites of nursing research and professional organizations and national, provincial/state, and territorial governments.

3.3. Study Selection

We uploaded all identified citations to a web-based reference management program (RefWorks) and removed duplicate entries. Two-member teams independently screened titles and abstracts of these citations for relevance using prespecified criteria. Translators assisted with the review of all citations in languages other than French or English. The full-text of a published paper and/or study report was obtained if it appeared to meet the inclusion criteria, if an abstract was unavailable, or if it was not possible to determine relevance from the title and abstract review. In instances where a study was reported in more than one paper, we grouped the study’s papers in a constellation and collectively reviewed them. Two-member teams independently screened these full-text papers for eligibility based on the inclusion criteria. Discrepancies were discussed and resolved by consensus. We catalogued all excluded studies and the reason for exclusion. Studies that met eligibility criteria advanced to the quality assessment phase of the review.

3.4. Quality Assessment

Two team members (AD and KR) independently assessed the methodological quality of the studies for internal validity and disagreements were resolved through discussion and consensus. The internal validity of each study was assessed using a slightly modified version of the Cochrane risk of bias criteria [29]; modifications to the criteria were three-fold. First, we did not assess for blinding of participants and personnel because the nature of NP and CNS interventions precludes this possibility. Second, outcome assessment and completeness of outcome data were evaluated separately for objective and subjective outcomes within a study. We looked for evidence of key outcomes that would typically be measured for each study’s research question [29]. Third, if outcomes had more than 20% missing data, we judged the study to be at high risk of bias for “incomplete outcome data.”

We assessed studies, assigning a high, low, or unclear risk of bias for each of the following eight questions: (1) To avoid selection bias, was the strategy used for random sequence generation likely to produce comparable groups (e.g., random number table, computer random number generator)? (2) To avoid selection bias, was a method used to conceal the allocation sequence so that group allocation could not be foreseen in advance (e.g., sequentially numbered, opaque, sealed envelopes; central allocation office)? (3) To avoid detection bias, was an appropriate method/source used to collect objective (e.g., mortality) measures (e.g., death records, blinding of outcome assessor, trained chart abstracter)? (4) To avoid detection bias, was an appropriate method used to collect subjective (e.g., quality of life) measures (e.g., blinding of outcome assessor; use of reliable, valid, established self-administered questionnaires)? (5) To avoid attrition bias, was outcome data complete for the objective measures (i.e., complete for ≥80% of sample; missing data balanced between groups; missing data imputed using appropriate methods)? (6) To avoid attrition bias, was outcome data complete for the subjective measures (i.e., complete for ≥80% of sample; missing data balanced between groups; missing data imputed using appropriate methods)? (7) To avoid reporting bias, were all outcomes described in the methods section of the study reported in the results and were all key outcomes reported? (8) Were “other” biases detected in the study (e.g., contamination bias in which the control group had exposure to the intervention)?

We sought clarification from 40 of the 43 study authors when there were insufficient details in the paper to determine the risk of bias and we received 28 (70%) responses. An overall risk of bias was assigned to each study as follows: low risk of bias (at risk in 0-1 category), moderate risk of bias (at risk in 2-3 categories), high risk of bias (at risk in 4–6 categories), and very high risk of bias (at risk in 7-8 categories).

External validity refers to the generalization or applicability of the study to other circumstances [30]. To assess external validity, two team members independently assessed the generalizability of the study population, intervention, control, and outcomes (PICO). Disagreements were resolved through discussion and consensus. Historically, RCTs of NPs and CNSs have been criticized because the number evaluated in any study has been small (e.g., one or two NPs) causing concern that those willing to be evaluated may be atypical in training, experience, knowledge, skills, or practice characteristics. We consulted with our policy advisor and together decided that 10 NPs or CNSs either within a single study or across studies combined in meta-analyses would be a reasonable minimum sample necessary to generalize results to similar NP or CNS roles.

As reported in a separate paper, we applied the Quality of Health Economic Studies (QHES) instrument [31–33] to evaluate the economic analyses in each study. The quality of the body of evidence for individual outcomes was evaluated using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system [34, 35] and GRADEpro software. The results of the GRADE assessments are reported elsewhere [36].

3.5. Data Extraction

A trained research assistant (KR) extracted data from each study into a summary table regarding general information (i.e., author, country, setting, language of publication, and publication status), characteristics of the study (design and group allocation), characteristics of the participants (number per group, sex, ages, and health conditions), characteristics of the intervention (number and type of NPs or CNSs, education and training, specific role, and comparison intervention), outcomes (health system, patient, and provider), length of follow-up, proportion followed to study completion, and study findings. If the findings of a single study were reported in two or more papers, they were extracted as one study. Team members checked the accuracy of extractions and discrepancies were resolved through discussion and consensus.

3.6. Analysis

Studies were categorized into the following six groupings: NP-outpatient, NP-transition, NP-inpatient, CNS-outpatient, CNS-transition, and CNS-inpatient. In a transition role, the NP or CNS could provide “time-limited services designed to ensure healthcare continuity, avoid preventable poor outcomes among at-risk populations, and promote the safe and timely transfer of patients from one level of care to another or from one type of setting to another” [37, page 747]. Within these groupings, studies were further categorized into alternative or complementary NP or CNS role function.

The strengths and threats to internal and external validity of the included RCTs are summarized narratively, organized by the six groupings identified above.

4. Results

4.1. Results of the Search

The searches yielded 4,397 unique records of which 3,981 were excluded during title and abstract review. Based on full-text review of the remaining 416 papers, 351 were excluded based on reasons listed in Figure 1. The remaining 65 papers described 43 relevant RCTs (28 studies reported in single papers and 15 studies reported in 37 papers). All studies were published in English. In general, the control intervention was “usual care.”

Figure 1: Identification and screening of relevant studies. Flow diagram adapted from Moher et al. [109].

The distribution of the 43 RCTs across groupings was NP-outpatient (), NP-transition (), NP-inpatient (), CNS-outpatient (), CNS-transition (), and CNS-inpatient (). We summarize the results by grouping beginning first with a brief overview of the study characteristics (Tables 1 and 2) followed by a description of threats to internal validity (Figures 2 and 3). Finally, threats to internal and external validity across studies will be described.

Table 1: Summary of NP study characteristics.

Author, year, and country (additional publications)
Study objective (number analyzed)
Participants
Intervention(NP role)
Number of sites
Number of NPs

experience and training

NP in outpatient setting ()

Allen, 2002, US [38] (Paez and Allen, 2006) [87]
Compare NP plus usual care () to usual care () in the management of blood lipids in patients with CHD
228 adults with hypercholesterolemia and CHD who were hospitalized for CABG or PCI
NP counseled on lipid management and lifestyle changes and had permission to prescribe (complementary role)
1
1 NP

Education and experience were not reported

Dierick-van Daele, 2009, NL [39] (Dierick-van Daele et al., 2010) [88]
Compare NP () and GP () in the provision of primary care
1501 patients attending a primary care appointment for common complaints
NP saw patients at first point of contact; a GP was required to sign off all prescriptions (alternative role)
15
12 NPs

All completed a 2-year ANP MSc in previous 2 months

Kinnersley, 2000, UK [40]
Compare NP () and GP () in the provision of primary care
1465 (1368 analyzed) patients seeking a same-day appointment
NP saw patients at first point of contact; a GP was required to sign off all prescriptions (alternative role)
10
10 NPs

All completed an NP diploma at least 1 year previously

Krein, 2004, US [41]
Compare NP plus usual care () and usual care () in the management of type 2 diabetes
246 adults with type 2 diabetes and poor glycemic control
NP followed the Chronic Care Model in helping patients to manage glucose levels; PCPs were required to approve medication changes (complementary role)
2
2 NPs

2-day training session; other education and experience were not reported

Limoges-Gonzalez, 2011, US [42]
Compare gastroenterology NP () to gastroenterologists () in screening colonoscopies
150 average risk patients ≥50 yrs who were referred for a screening colonoscopy
NP performed the colonoscopy under the same conditions as medical doctors and polypectomies were performed by the NP independently (alternative role)
1
1 NP

Intensive training followed by 2 years of practice

Litaker, 2003, US [43]
Compare NP plus usual care () and usual PCP care () in the management of patients with hypertension and diabetes
157 adult patients with mild-moderate hypertension and NIDDM without end-organ complications
NP saw patients at first point of contact and provided telephonic and in-office management; permission to prescribe was not reported (complementary role)
1
1 NP* 

Specific training for the management of diabetes and hypertension

Mundinger, 2000, US [44] (Lenz et al., 2002; Lenz et al., 2004) [89, 90]
Compare NP () and physician () ongoing primary care
1981 ED or urgent care adult patients with no regular source of care
NP saw patients at first point of contact and had authority to prescribe (alternative role)
5
7 PTE NPs 

School of nursing faculty members with specialties in adult primary care

Nelson, 1991, US [45]
Compare pediatric NP telephone support plus usual care () to usual care () of parents after an ED visit for their child’s acute illness
190 (184 analyzed) outpatient children (<8 yrs) who attended the ED for an acute infectious or allergic condition
NP made telephone contact with parent(s) after discharge, provided education and treatment review, answered questions, and facilitated communication between family and PCP; permission to prescribe was not reported (complementary role)
2
2 NPs 

Education and experience were not reported

Schuttelaar, 2010, NL [46] (Schuttelaar et al., 2011) [91]
Compare NP () and dermatologist () care of children with eczema
160 children with atopic dermatitis who were newly referred by their GP or paediatrician
NP provided the same services as the dermatologist and was able to prescribe independently (alternative role)
1
1 NP 

ANP masters prepared with 3-year experience in dermatology

Smith, 2006, US [47] (Lyles et al., 2003; Luo et al., 2007) [92, 93]
Compare NP plus usual care () and usual care () in the management of patients with medically unexplained symptoms
206 patients (18–65 years) with medically unexplained symptoms and high utilization of primary care services
NP coordinated and managed care over a minimum of 12 scheduled visits over a year and telephone contact between visits (complementary role)
3
4 NPs 

Certified with 84 hours of special training, no prior experience in mental health

Venning, 2000, UK [48]
Compare NP () and GP () primary care of patients seeking same day consultations
1316 patients of all ages
NP saw patients at first point of contact; a GP was required to sign off all prescriptions (alternative role)
20
20 NPs 

Diploma, BSc, or MSc prepared with 1–5 years as an NP

NP in transition role ()

Coleman, 2006, US [49] (Parry et al., 2003) [94]
Compare geriatric NP plus usual care () and usual care () of older patients with complex care needs
750 chronically ill, community-dwelling, local, older adults (≥65 yrs) admitted to hospital for 1 of 11 nonpsychiatric conditions
NP met with patient in hospital and made a home visit and telephone calls after discharge; patients transferred to a skilled nursing facility were telephoned or visited at least weekly (complementary role)
10
2 NPs* 

Experienced geriatric NPs who were skilled in patient education and advocacy

Hollingsworth 2000, US [50]
Compare NP-facilitated early discharge, follow-up care plus usual care () and usual care () of women who have had an abdominal hysterectomy
113 women (≥21 yrs) undergoing abdominal hysterectomy for nononcologic indications
NPs had contact with patient in hospital, encouraged early discharge, made home visits and telephone calls, and were available for patients and families by telephone (complementary role)
1
2 NPs (1 FTE and 1 PTE)

Masters prepared

Kotowycz, 2010, CAN [51]
Compare NP-facilitated early discharge, follow-up care plus usual care () and usual care () of patients with low-risk STEMI
54 low-risk (Zwolle Primary PCI Index ≤3) STEMI patients treated with primary or rescue PCI.
NP saw patients before and after discharge and provided education and appointment reminders; permission to prescribe was not reported (complementary role)
1
1 NP* 

Education and experience were not reported

Nathan, 2006, UK [52]
Compare respiratory specialist NP () and respiratory doctor () in the provision of follow-up care to acute asthma patients
154 acute asthma patients (>16 yrs) discharged from hospital. Those with COPD were excluded.
NP saw outpatients after discharge and for follow-up appointments; NP prescribed independently according to a patient group directive (alternative role)
1
1 NP 

Masters prepared with specialist training in acute asthma management

Rawl, 1998, US [53] (Easton et al., 1995) [95]
Compare NP postdischarge follow-up plus usual care () and usual care () of rehabilitation patients with long-term disabilities
100 rehabilitation patients (≥18 yrs), who were not confined to their home
NP contacted patients before discharge and in the rehabilitation clinic, in their home, and by telephone after discharge (complementary role)
1
1 NP 

Certified in rehabilitation, 10-year experience

NP in inpatient setting ()

Mitchell-DiCenso, 1996, CAN [54]
Compare NP () and pediatric resident teams () in neonatal intensive care
821 neonates admitted to the neonatal intensive care unit.
NP team assumed primary responsibility for neonates (alternative role)
1
4.5 FTE NPs 

All graduates of a 16-month Masters program

Pioro, 2001, US [55]
Compare NP () and house staff () in care of general medical patients
381 adult general medical patients
NPs provided many of the same services delivered by traditional house staff (alternative role)
1
2.5 FTE NPs 

Experience and training were not described

ANP: advanced nurse practitioner; BSc: Bachelor of Science; CABG: coronary artery bypass surgery; CAN: Canada; CHD: coronary heart disease; COPD: chronic obstructive pulmonary disease; GP: general practitioner; ED: emergency department; FTE: full-time equivalent; MSc: Master of Science; NIDDM: non-insulin dependent diabetes mellitus; NL: The Netherlands; NP: nurse practitioner; PCI: percutaneous coronary intervention; PCP: primary care provider; PTE: part time equivalent; STEMI: ST-elevation myocardial infarction; UK: United Kingdom; US: United States.

*Data provided by author.

Table 2: Summary of CNS study characteristics.

Author, year, and country (additional publications)
Study objective (number analyzed)
Participants
Intervention (CNS role)
Number of sites
Number of CNSs experience and training

CNS in outpatient setting ()

Alexander, 1988, US
Compare CNS () and usual primary continuity care () of poorly controlled noncompliant asthmatic children
21 asthmatic children (15 months to 13 years) from low-income families who used the ED as their primary care source
CNS promoted self-care based on the Orem Self-Care Nursing Model; permission to prescribe was not described (alternative role)
1
1 CNS 

Education and experience were not reported

Arts, 2012, NL
Compare CNS () and physician () care and cost-effectiveness in the treatment of patients with diabetes
337 patients with diabetes treated in a hospital-based setting. All required insulin treatment or oral blood-glucose medication and had inadequate regulation of blood glucose, blood pressure, or lipids
CNS managed diabetes patients in same way as the physicians, including diabetes-related clinical admissions; referrals to specialist care required a physician (alternative role)
1
4 CNSs 

Doctoral or Masters prepared with extensive experience in diabetes care

Brandon, 2009, US
Compare CNS () and usual care () of patients with HF
20 adult patients living with HF for >6 months who were capable of self-care
CNS provided education, care management and medication adherence advice, and patient support; permission to prescribe was not reported (complementary role)
1
1 CNS 

Masters prepared* 

Student practicum under cardiologist supervision plus 10-year experience in intensive and coronary care

Brooten, 2001, US
Compare CNS ( mother; 94 infants) and usual care ( mothers; 100 infants) of high-risk pregnant women
173 pregnant women at high risk due to gestational or pregestational diabetes mellitus, chronic hypertension, or preterm labour with 194 infants
CNS provided prenatal monitoring, assessment, education, counseling, and community referrals; medication regimens were adjusted after physician consultation (complementary role)
1
3 CNS 

Masters prepared specializing in high-risk pregnancies and infants 

(experience not reported)

Chien, 2012, China
Compare psychiatric CNS () and usual care () of patients with psychiatric symptoms
79 referred adult (18–49 yrs) patients with first-episode, moderately severe psychiatric symptoms who were at low risk of self-harm or violence
CNS provided 6 sessions of assessment, support system design, coordination of care, and education in symptom management; permission to prescribe was not reported (complementary role)
1
1 CNS 

Masters prepared* with training in psychosocial interventions for patients with mental health problems 

(experience not reported)

Evans, 1997, US (Strumpf et al., 1992; Patterson et al., 1995; Siegler et al., 1997; Capezuti et al., 1998) [96–99]
Compare gerontologic CNS education (), CNS education plus consultation () and neither education nor consultation () in the use of physical restraints in nursing homes
643 (463 analyzed) residents (>60 yrs) from 3 nursing homes
CNS education involved ten 30-minute sessions addressing issues surrounding restraint use; CNS consultation involved 12 hours/week of unit-based consultation for residents with clinically challenging behaviour (complementary role)
3
1 CNS 

Masters prepared 

(experience not reported)

Faithfull, 2001, UK
Compare CNS () and usual care () of men treated with radical radiotherapy for prostate and bladder cancer
115 men undergoing radical (>60 Gy) radiotherapy for prostate or bladder cancer
CNS made initial assessments, had open access clinics during therapy, and made posttherapy telephone contacts; permission to prescribe was not reported (alternative role)
1
1 CNS 

Masters prepared with expertise in radiotherapy toxicity management* 

(experience not reported)

Ritz, 2000, US
Compare CNS () and usual care () of breast cancer patients
210 women with newly diagnosed breast cancer (30–85 years) who were referred by their physician and were cared for within the system
CNS provided assessments, information, support, and coordination of care; permission to prescribe was not described (complementary role)
1
2 CNSs 

Masters prepared* 

(experience not reported)

Ryan, 2006, UK
Compare rheumatologic CNS plus usual care () and usual care () of patients with rheumatoid arthritis
71 patients with diagnosed rheumatoid arthritis who were beginning new disease modifying antirheumatic drugs
CNS provided the same service as the outpatient clinic nurse with addition of assessment and referral responsibilities; permission to prescribe was not reported (complementary role)
1
1 CNS 

Doctoral preparation with 16-year experience in rheumatology*

Swindle, 2003, US
Compare mental health CNS () and physician care () of veterans with depression
268 new patients with PRIME-MD depression diagnosis
CNS contacted patients by telephone or visits, while the CNS recommended antidepressant medication and changes to type and dose; permission to prescribe was not reported (complementary role)
2
9 CNSs 

5 had cognitive behavioral treatment training; 9–23-year experience treating depression

Tijhuis, 2002, NL (Tijhuis et al., 2003; Tijhuis et al., 2003; van Den Hout et al., 2003)[100–102]
Compare CNS outpatient care (), inpatient care (), and day-patient care () of patients with rheumatoid arthritis
210 rheumatoid arthritis patients with increasing functional limitations
CNS provided information, referrals, and hardware prescriptions; CNS did not have permission to prescribe or change drugs (alternative role)
6
6 CNSs 

Education and experience were not reported

CNS in transition role ()

Brooten, 1986, US
Compare perinatal CNS-care ( mothers; 39 infants) and usual care ( mothers; 40 infants) of very-low-birth weight infants
72 mothers and 79 very-low-birth weight infants (≤1500 g)
CNS contacted parent(s) during infant hospitalization and made home visits and telephone contact; permission to prescribe was not reported (complementary role)
1
3 CNSs (1 FTE; 2 PTE)  

Masters prepared in perinatal and neonatal nursing

Brooten, 1994, US
Compare CNS plus usual care () and usual care () of high risk postpartum women
122 postpartum women who had received an unplanned caesarean delivery
CNS provided comprehensive in hospital and follow-up care with postdischarge home visits and telephone calls (complementary role)
1
3 CNSs* 

Education and experience were not reported

Dellasega, 2000, US (Dellasega and Zerbe, 2002) [103]
Compare CNS plus usual care ( patients; 34 caregivers) and usual care ( patients; 31 caregivers) of elderly frail discharged patients
140 elderly patients who were scheduled to be discharged home, were cognitively frail and/or functionally impaired, or were a complex case (plus 65 caregivers)
CNS or NP visited patient before discharge and after discharge; additional telephone calls or visits were initiated as needed (complementary role)
3*
2 CNSs and 2 NPs 

Education and experience were not reported

Kennedy, 1987, US (Neidlinger et al., 1987) [104]
Compare gerontologic CNS plus usual care () and usual care ) of elderly patients admitted to nonintensive care units
80 consecutive elderly patients (≥75 yrs) admitted to nonintensive care units who were expected to stay ≥72 hours
CNS met patients, family, and care providers in hospital and again just prior to discharge; permission to prescribe was not reported (complementary role)
1
1 CNS 

Masters prepared with additional geriatric knowledge and skills

Laramee, 2003, US
Compare CHF CNS plus usual care () and usual care () in the management of HF patients admitted to hospital
287 patients at risk of early readmission who had been admitted to hospital for primary or secondary CHF, left ventricular dysfunction <40%, or radiologic evidence of pulmonary oedema
CNS visited patients daily in hospital and made postdischarge telephone contacts (complementary role)
1
1 CNS 

Masters prepared with 18-year experience in critical care and cardiology

McCorkle, 2000, US (Jepson et al., 1999) [105]
Compare CNS plus usual care () and usual care () of older postsurgical cancer patients
375 older (60–92 yrs) newly diagnosed solid-tumor cancer patients discharged after surgery to their home
CNS contacted patients after discharge and made home visits and telephone contacts (complementary role)
1
7 CNSs* 

Completed 2-year program in oncology

McCorkle, 2009, US (McCorkle et al., 2011) [106]
Compare oncology CNS plus usual care () and usual care () of women recovering from gynecological cancer surgery
149 (123 analyzed) women (≥21 yrs) with suspected ovarian cancer recovering from gynaecological cancer surgery and undergoing chemotherapy
CNS provided tailored specialized care through 18 postdischarge patient contacts (complementary role)
2
1 CNS and 4 NPs* 

Education and experience were not reported

Naylor, 1990, US
Compare CNS plus usual care () and usual care () of elderly patients admitted to hospital
40 English speaking inpatients (≥70 years) who had been admitted to hospital from home.
CNS contacted patients in hospital, implemented the discharge plan, and contacted patients after discharge while coordinating with PCP and providing telephone outreach (complementary role)
1
2 PTE CNSs 

Masters prepared

Naylor, 1994, US
Compare gerontologic CNS plus usual care () and usual care () of elderly patients admitted to hospital
276 English speaking inpatients (≥70 years) admitted from their homes: medical (CHF and angina/MI) and surgical (CABG and CVR) patients
CNS contacted patient in hospital, made postdischarge visits, and was available 7 days/week during hospitalization and after discharge (complementary role)
1
2 PTE CNSs 

Masters prepared with at least one year experience as a specialist

Naylor, 1999, US (Naylor and McCauley, 1999) [107]
Compare gerontologic CNS plus usual care () and usual care () of elderly patients admitted to hospital
363 hospitalized elderly patients (≥65 yrs) admitted to hospital from home who were at risk of readmission
CNS contacted patient in hospital, made home visits and weekly telephone contacts, and individualized patient management; permission to prescribe was not reported (complementary role)
2
5 PTE CNSs 

Masters prepared with a mean of 6.5 years postdegree experience

Naylor, 2004, US (McCauley et al., 2006) [108]
Compare CNS plus usual care () and usual care () of elderly patients hospitalized with HF
239 HF patients ( ≥65 years) admitted to study hospitals from their homes
CNS contacted patients in hospital and after discharge and provided discharge planning, assessments, education, and development and implementation of care goals (complementary role)
6
3 CNSs 

Masters prepared with specialized training in managing elderly HF patients

Thompson, 2005, UK
Compare CNS plus usual care () and usual care () of patients admitted to hospital for HF
106 patients with acute admissions to hospital for CHF and left ventricular ejection fraction ≤45%, who were discharged home
CNS provided clinic and home-based care within 10 days of discharge; permission to prescribe was not reported (complementary role)
2
2 CNSs 

Postgraduate education with HF management experience

York, 1997, US
Compare perinatal CNS-facilitated early discharge plus usual care ( mothers; 42 infants) and usual care ( mothers; 51 infants) of high-risk pregnant women
96 high-risk pregnant women with either diabetes or hypertension during pregnancy
CNS provided in hospital and postdischarge follow-up care; permission to prescribe was not reported (complementary role)
1
1 CNS 

Masters prepared

CNS in inpatient setting ()

Talley, 1990, US
Compare psychiatric liaison CNS consultation () and no consultation () for nursing care and the use of sitters
107 acute care patients who had been assigned lay sitters primarily because of a danger of “harm to self” or “generally unpredictable” behaviour
CNS provided individualized consultations to patients, nursing staff, and sitters sometimes on multiple occasions; permission to prescribe was not reported (complementary role)
1
2 CNSs 

Education and experience were not reported

ANP: advanced nurse practitioner; CABG: coronary artery bypass graft; CHF: congestive heart failure; CNS: clinical nurse specialist; CVR: cardiovascular recovery; GP: general practitioner; ED: emergency department; HF: heart failure; FTE: full-time equivalent; Gy: gray (unit of absorbed radiation); MI: myocardial infarction; MSc: Master of Science; NL: The Netherlands; NP: nurse practitioner; PCP: primary care provider; PRIME-MD: primary care evaluation of mental disorders; PTE: part time equivalent; UK: United Kingdom; US: United States.

*Data provided by author.

Figure 2: Risk of bias assessment of NP studies ().

Figure 3: Risk of bias assessment of CNS studies ().

4.2. Study Characteristics and Internal Threats to Validity

4.2.1. NP-Outpatient Care

Eleven RCTs of NPs in outpatient care [38–48] met our inclusion criteria (Table 1). All but one were published in the year 2000 or later. They were conducted in the United States (), United Kingdom (), or the Netherlands (). Six studies evaluated NPs in alternative provider roles and five in complementary provider roles. The number of NPs ranged from one to 20 in NP alternative provider studies and from one to four in NP complementary provider studies. Some of the trials were quite large with over 1000 patients, while most of the trials examining specific patient populations tended to be much smaller. The studies were conducted at between one and 20 sites.

Threats to Internal Validity. Overall, six of the 11 RCTs were judged to be at low risk of bias (in other words, the methods were of high quality), four at moderate, and one at high risk of bias (Figure 2). With regard to selection bias, nine studies used a random sequence generation process that was likely to produce comparable groups; for two studies, we had insufficient information about the sequence generation process to permit judgement, despite contact with one of the authors. Seven trials used an adequate process to conceal allocation so that participants and those enrolling participants could not foresee the group to which the next patient would be assigned. We judged one study as unclear because there was insufficient information and three at high risk of selection bias.

All the RCTs were judged to be at low risk of detection bias with respect to objective outcome measures (e.g., blood levels and medical records abstraction) and all but two trials were assessed at low risk of detection bias for subjective measures because most used established validated self-report instruments (e.g., SF-12 and SF-36). Three trials used blinded assessors for some data collection. Two studies were judged as unclear, one because they used self-reported dietary intake and physical activity which can be subject to recall and social desirability bias and the other because clinicians self-recorded the length of time they spent with each patient.

Seven RCTs were judged to be at low risk of attrition bias for the objective measures; one study reported a follow-up rate less than 80% for a blood cholesterol measure, and two did not report all follow-up rates. The risk of attrition bias for subjective measures was high or unclear for six studies due to failure to report follow-up rates or poor response rates for at least one self- or interviewer-administered questionnaire by last follow-up.

One study was judged at high risk of reporting bias because they did not report any patient outcomes such as child’s health status, quality of life, or parent satisfaction in a study of the appropriateness of follow-up care after attendance at an emergency department. We rated one study at high risk of “other” bias because there was substantial baseline imbalance which was not adjusted for in the analyses.

4.2.2. NP-Transition Care

Five RCTs evaluated NPs in a transition role [49–53] (Table 1). Three studies were conducted in the US, one in Canada, and one in the UK. Four of the studies were published in the year 2000 or later. One study evaluated NPs in an alternative provider role and four in complementary provider roles. One or two NPs were evaluated in each study. The number of patients included in the trials ranged from 54 to 750 and they were conducted at between one and 10 sites.

Threats to Internal Validity. Overall, two studies were judged to be at low risk of bias and three at high risk of bias (Figure 2). The trials assessed to be at low risk of selection bias used a random number generator that revealed the intervention assignment when a patient was ready for allocation and a computer generated sequence concealed in sequentially numbered, opaque, sealed envelopes. The other three studies provided insufficient information to fully judge random sequence generation and allocation concealment.

All five RCTs were judged to be at low risk of detection bias for objective measures as they used abstraction of hospital administrative records or blinded outcome assessment. With respect to subjective measures, two trials were at high risk of detection bias because patients self-reported their smoking cessation success and the NP, who delivered the intervention, also collected baseline and outcome data from the comparison groups during a guided interview.

All but one trial were at low risk of attrition bias for objective measures as they followed over 80% of participants and this was balanced across comparison groups within each study. With respect to subjective data, three trials scored high for risk of attrition bias due to poor response rates to self- or interviewer-administered questionnaires.

Two studies identified outcomes that they planned to measure but did not report, placing them at risk of reporting bias.

4.2.3. NP-Inpatient Care

Two RCTs of NPs in inpatient settings met our inclusion criteria, both of which evaluated the NP in an alternative provider role [54, 55] (Table 1). One study was conducted in the US and one in Canada. One study was conducted before and one after the year 2000. The number of NPs in the trials ranged from 2.5 to 4.5 full-time equivalent NPs. The number of patients included in the trials ranged from 381 to 821 and each study was conducted at one site.

Threats to Internal Validity. Overall, the two studies were judged to be at low risk of bias (Figure 2). Both studies were at low risk of selection bias having used acceptable random sequence generation processes (table of random numbers; computer random number generator) and having concealed allocation through the use of sequentially numbered, sealed, opaque envelopes.

Both were at low risk of detection bias as they relied on medical record and hospital database extraction of objective data such as mortality, medical complications, and length of hospital stay. In cases where study participants completed questionnaires, there were reliable, valid measures such as the SF-36 and the Minnesota Infant Development Inventory (MIDI).

Both studies were judged to be at low risk of attrition bias for the objective measures but at high risk of attrition bias for the subjective measures. While many of the primary objective outcome data were available for all study participants (e.g., mortality, complications, and length of stay), subjective self-report measures often had response rates less than 80%. We judged both studies to be at low risk of reporting bias and other biases.

4.2.4. CNS-Outpatient Care

Eleven RCTs [56–66] addressed the CNS role in delivering outpatient care (Table 2). Six studies were conducted in the US, two in the UK, two in the Netherlands, and one in China. Nine studies were published in the year 2000 or later. Four trials evaluated one to six CNSs in the alternative provider role, while seven trials evaluated one to nine CNSs in the complementary provider role. The number of patients included in the trials ranged from 20 to 643 and the studies were conducted at between one and six sites.

Threats to Internal Validity. Overall, five of the eleven studies were assessed at low risk, four at moderate risk, and two at high risk of bias (Figure 3). While seven studies used valid methods to generate the random sequence and were at low risk of selection bias, we judged the remaining four to be at unclear risk of bias because the authors did not include this information in their papers and we did not receive responses to our request for further details. With respect to allocation concealment, five trials were assessed at low risk of selection bias (e.g., central allocation and sealed envelopes) and three at unclear risk of bias because methods were not described. We judged one at high risk of bias because the patients were randomly assigned by the CNS to one of the study groups by drawing the next allocation from an envelope; using this method, it is possible that the drawn assignment could be returned to the envelope and redrawn if allocation was deemed unsuitable. Two studies used cluster randomization and allocation concealment was not applicable as the clusters were all randomized at one time.

All the studies were rated as low in risk of detection bias for objective outcome measures (e.g., mortality and rehospitalization). Of the nine studies that included subjective outcomes, eight were judged at low risk of bias as they used established, validated instruments, or blinded outcome assessment and one was at high risk of bias because the CNS who delivered the intervention also collected data from both groups before and after the intervention via telephone interviews.

With respect to attrition bias, all but three studies were assessed at low risk of bias for objective measures. Two of the studies judged at unclear risk of bias provided insufficient information to assess the completeness of all objective outcome measures and one, judged at high risk of bias, did not have cost data for at least 80% of the study participants.

While seven trials were judged to be at low risk of reporting bias, four were judged at high risk because they did not fully report all the outcomes they collected or did not collect all patient-important outcomes that would have been expected (e.g., patient/parent satisfaction with care and quality of life). Finally, one trial was judged at high risk of “other” bias because they did not adjust for cluster randomization.

4.2.5. CNS-Transition Care

Thirteen RCTs [67–79] evaluated the CNS in the delivery of transition care in the US () and in the UK () (Table 2). Seven studies were conducted before the year 2000. All trials evaluated the CNS in a complementary provider role. The studies included between one and seven CNSs. The number of patients included in the trials ranged from 40 to 375 and the studies were conducted at between one and six sites.

Threats to Internal Validity. Overall, three of the thirteen trials were at low risk, eight at moderate risk, and two at a high risk of bias (Figure 3). Most trials were not at risk of selection bias. All but one trial used valid methods to generate the random sequence and all but one trial concealed allocation.

All trials were rated at low risk of detection bias for objective measures, except one. In this study, the risk of bias was unclear because healthcare utilization outcomes were based on self-report rather than medical record review data. For subjective measures, two trials were judged to be at unclear risk of detection bias because the validity of their scales was not described, and, in one trial, treatment adherence was based on self-report rather than objective measures, such as pill counts and was assessed at high risk of bias.

For objective measures, six trials had a low risk of attrition bias but, for five trials, the risk was unclear and, for two, it was high. For subjective measures, four trials had a low risk of attrition bias but, for six trials, the risk was unclear and, for three, it was high. For those studies in which it was unclear, the response rates were not specified or data were imputed; for those at high risk of bias, the follow-up rate was less than 80%.

Of the 13 trials, four were at high risk of reporting bias, three of which did not report on all outcomes measured and one of which did not include a measure of health status. One study was at unclear risk of bias because it was unclear if measures reported at baseline should have been reported as outcomes. Finally, seven trials were assessed at high risk of “other” bias because there were baseline differences between the groups for which adjustments were not made.

4.2.6. CNS-Inpatient Care

Only one study, conducted in the US in 1990, evaluated the CNS delivering inpatient care [80] (Table 2). The study examined CNSs in a complementary role. Two CNSs participated in the study, which included 107 patients and was conducted at one site.

Summary of Threats to Internal Validity. Overall, the risk of bias for this study was judged as moderate (Figure 3). We judged the study at low risk of selection bias and detection bias. The study, however, was judged to be at high risk of attrition bias because over 20% of patients were dropped from the study after randomization as the intervention they received was changed (e.g., sitters discontinued and control group receiving CNS consultation) resulting in unequal distribution of patients in the two groups.

The study was also at high risk of reporting bias because they did not report whether the CNS and staff nurse intervention influenced patient risk behaviours as intended. Contamination bias was possible because the same staff nurses who received coaching from the CNS for intervention group patient management and for charting nursing observations cared for the control group and might have provided the same patient management and charting strategies for them. Because the associated risk of bias was unknown, we judged this as unclear “other” bias.

4.2.7. Summary

Overall, we assessed that 18 of the 43 trials (42%) were at low risk, 17 (39%) at moderate risk, and 8 (19%) at high risk of overall bias (Figures 2 and 3). No study was judged to be at very high risk of overall bias. Figures 4 and 5 summarize the studies by type of bias. With respect to the NP trials, many studies were at high risk of detection bias with incomplete (<80%) follow-up for subjective outcomes (e.g., self-administered scales). In CNS trials, a number of studies were at high risk of reporting bias because they either did not report on all outcomes measured or did not include a key outcome that we would have expected. A number of studies (especially smaller studies) had baseline differences with no mention of adjusting the analyses to account for these differences.

Figure 4: Risk of bias horizontal graph of NP studies ().

Figure 5: Risk of bias horizontal graph of CNS studies ().

Some of the potential threats to validity may not in reality be threats, but rather it may be an issue of lack of reporting. There were many instances that we rated categories as “unclear risk of bias” because there was insufficient information in the paper or from the author to permit judgment of low or high risk of bias.

4.3. Summary of Threats to External Validity

Of the 43 RCTs, 70% of the studies were conducted in the United States () and the remainder in four other countries: the United Kingdom (; 14%), The Netherlands (; 9%), Canada (; 5%), and China (; 2%). Given that healthcare systems and NP and CNS education, role implementation, and scope of practice vary internationally, applicability of study findings from one country to another may be compromised.

Some RCTs evaluating NP and CNS roles were conducted across many sites which may enhance generalizability. However, many trials were conducted in single sites, which likely limits the generalizability of study findings.

Of the 43 RCTs, 13 (30%) studies were published prior to the year 2000. Given the substantive progress that has occurred in the development of NP and CNS roles and dynamic changes in healthcare systems internationally, the results of these studies may be less relevant to current-day policy. Although we found a substantial number of eligible RCTs, when broken down by grouping, we identified only one dated RCT of CNSs in the nontransitional care role for inpatient settings. This RCT evaluated two CNSs providing consultation for a small very particular population of medical-surgical patients requiring sitters due to the risk of self-harm or unpredictable behaviour. Similarly, we identified only two RCTs of NPs in the nontransitional care role in inpatient settings, both of which were published over 10 years ago. One study evaluated NPs caring for a homogeneous population of critically ill infants in a Canadian hospital and the other evaluated NPs caring for a heterogeneous population of adults admitted to general medical wards in a US-based hospital. Given the existence of only three fairly dated RCTs of NPs or CNSs in inpatient settings and somewhat specific populations, caution is needed in generalizing these results to NPs and CNSs in other inpatient settings.

Nine (21%) trials were conducted with small numbers of patients () with specific health conditions. Four trials of NPs in outpatient settings were large with over 1,000 patients. The larger studies with patients experiencing common conditions are more readily generalizable to the general population than smaller trials with patients experiencing a specific condition. However, one of the larger trials [44] limited study entry to poor, non-English speaking Hispanic people which may limit the generalizability of the findings to other patients seeking primary healthcare.

Twenty-seven (63%) of the RCTs evaluated one or two NPs or CNSs, 9 (21%) evaluated three to five, four (9%) evaluated six to nine, and three (7%) evaluated 10 or more all of which were NP-outpatient studies. The small number of NPs and CNSs evaluated in any study raises concern that the results may not be generalizable to colleagues in similar roles. In some cases when study outcomes were similar we were able to combine study findings which increased the number of NPs or CNSs evaluated for that outcome.

About two-thirds of the studies (; 67%) specified that they evaluated experienced NPs or CNSs (i.e., NPs or CNSs who had completed their training at least one year before the evaluation and/or had graduate degrees). Many of the studies did not include information about training and experience. One study posed concern, as it compared novice NPs who had completed a two-year advanced nursing practice graduate degree in the previous two months with general practitioners who had an average of 16 years work experience [39].

Most studies used reliable and valid outcome measures to evaluate patient-important outcomes such as health status, quality of life, and satisfaction with care which strengthens the generalizability of the findings; however, some studies had very short-term follow-up periods (e.g., two weeks after the patient appointment) which may compromise generalizability of study findings over the long term [39, 48, 60].

5. Discussion

The purpose of this paper was to report on the methodological strengths and threats to internal and external validity of RCTs of NP and CNS cost-effectiveness. Based on a comprehensive search of the international literature, we identified 43 RCTs, evaluating NPs () and CNSs (). While 43 RCTs sound like a large number of evaluations of NPs and CNSs, categorizing the studies by NP or CNS role (i.e., alternative or complementary) and by setting (i.e., outpatient, transition, or inpatient) reveals the areas where further research is still required. For example, we found only one RCT of the CNS in a nontransitional role in the inpatient setting and only two RCTs of the NP in a nontransitional role in the inpatient settings, both of which were alternative provider roles.

Of the 43 RCTs, 70% () were conducted in the United States with far fewer conducted in four other countries (Canada, China, The Netherlands, and United Kingdom). In 2011, Newhouse et al. conducted a systematic review of the effectiveness of NPs and CNSs [25]. They chose to restrict the review to studies conducted in the United States to enhance the applicability of study findings to the United States healthcare system. A recent systematic review that also includes studies conducted outside the United States has not been conducted, to our knowledge. Therefore, we chose to broaden our search to include international studies in order to learn more about where NP and CNS role evaluations have been conducted and how the roles are being enacted globally.

5.1. Internal Validity

Our assessment of the risk of bias revealed that about two-fifths (; 42%) of the 43 studies were at low risk of bias, close to the same number (; 39%) were at moderate risk of bias, and about one-fifth (; 19%) at high risk of bias. When examined by date, 31% of the 13 RCTs published before the year 2000 were at high risk of bias compared to 13% of the 30 RCTs published in or after the year 2000 that were at high risk of bias which may mean that study validity is improving over time.

In many cases it was unclear if the authors met the risk of bias criteria because the required information was not reported in the paper. Consequently, we rated a large number of categories as “unclear risk of bias.” To permit complete and accurate assessment of risk of bias, researchers are encouraged to use a guide such as the Cochrane risk of bias criteria [29] when planning and reporting future studies.

Researchers reporting RCTs may also find the following recommendations helpful. A clear brief description of the sequence generation (e.g., random number table; computer random number generator) is needed to allow the reader to determine if the process should provide comparable groups [29]. A description of allocation concealment (e.g., sequentially numbered, opaque, and sealed envelope) is important for the reader to determine if allocation to groups could be manipulated. While blinding of participants is not possible in a study incorporating NPs or CNSs, a description of procedures used to blind outcome assessors and/or the description of valid outcome measures is needed to assess the quality of the study. Completeness of outcome data for each outcome measure and group, including the description of missing data and details of all participants excluded, lost to follow-up (e.g., dropped out of study or died), or reincluded at each stage, also needs to be reported. If researchers do not report outcomes that were measured or key outcomes that would be expected, a clear description is needed of the reasons for failing to report the outcome. A description of how any “other” biases were managed that threaten the quality of the study should also be reported. More detailed recommendations for reporting RCTs can be found in the Consolidated Standards of Reporting Trials (CONSORT) 2010 Statement [81, 82]. When authors are faced with cutting back on the number of words in a publication, a suggestion is to reduce the introductory sections to provide sufficient space to describe in detail the strategies used to prevent or minimize threats to internal validity.

5.2. External Validity

As others have found [5, 24], a challenge in conducting this systematic review was determining the fidelity of the intervention. The definition of the role and the education, training, and experience of the NPs or CNSs were often inadequately described or missing. When we contacted authors for this information, we found that some studies were conducted with RNs who had received as little as a few weeks of training or one course and were then called “NPs.”

Over half (; 63%) of the 43 studies evaluated only one or two NPs or CNSs and only three trials, all of NPs in outpatient settings, evaluated 10 or more. Approximately two-thirds of the studies (; 67%) evaluated experienced NPs or CNSs. Researchers are encouraged to include a detailed description of the NPs or CNSs being evaluated in their study (role in the context of an internationally accepted definition [2], education, experience in the role, and training for the specific intervention if applicable). Similar information should be provided for comparison providers. Furthermore, evaluations of these roles should not be initiated while the NPs or CNSs are still novices but rather when they have had sufficient experience in their role (i.e., at least 12 months). Challenging as it is, researchers are encouraged to plan multisite studies, to increase the number of NPs or CNSs evaluated, to increase the number of patients enrolled in the study, and to account for variations in practice to enhance the generalizability of study findings.

5.3. Strengths and Limitations

Restricting this review to RCTs may be viewed as a strength or limitation, depending on the perspective of the reader. Health service settings are complex and research is confounded by multiple variables that challenge the ability to evaluate the effectiveness of an intervention, such as NP and CNS roles. When feasible, randomization of participants to intervention and control groups is considered the optimal design to control known and unknown complexities and confounding variables [83–86]. Therefore, we chose to limit our review to RCTs. The quality of evidence in this review demonstrates that it is feasible to conduct well-designed RCTs to evaluate the effectiveness of NP and CNS roles in a variety of settings, remuneration mechanisms, and patient populations.

Strengths of our review include use of numerous strategies to identify all RCTs in any language (published or unpublished) that met our inclusion criteria, contact with authors and international expert advisors when it was unclear whether a study met our inclusion criteria, use of current education and credentialing criteria to verify that the trial was indeed evaluating an NP or CNS, use of duplicate assessment by independent reviewers and a consensus process for every stage of the review, use of an internationally recognized and established tool to assess the overall risk of bias of each trial and contact with authors when additional information was required to make our assessment, use of an established tool (Quality of Health Economic Studies) to evaluate the health economic analysis in each study, use of GRADE to evaluate outcome-specific quality of evidence, consideration of external as well as internal validity, grouping of trials by type (NP or CNS), setting (inpatient, transition, or outpatient), and role (alternative or complementary), and conducting meta-analyses whenever possible.

In future publications, we will summarize our assessment of the quality of the economic analyses of each RCT and outcome-specific quality of evidence using GRADE for each of the six groupings.

With respect to limitations, despite our attempts to identify all relevant RCTs, we may have missed some relevant studies or included some that do not meet our criteria based on author responses, advisor advice, or our interpretation of the description of the education or role. With respect to generalizability, we did not use a specific tool to assess threats to external validity but did consider the country and year of publication, number of NPs or CNSs in the study, the number of settings, and characteristics of the population, setting, intervention, and outcomes. We do not know how the exclusion of observational studies that investigate the effectiveness of NP and CNS roles may have influenced our findings [86].

6. Conclusions

This paper builds on the body of knowledge regarding quality of RCTs of NP and CNS cost-effectiveness (defined broadly to also include studies measuring health resource utilization). We have used an international lens and inclusion criteria that meet today’s definitions of the NP and CNS roles. While almost half the RCTs were found to be at low risk of bias, incomplete reporting of study methods and lack of details about NP and CNS education, experience, and roles make it difficult to fully evaluate the internal and external validity of studies of these roles. Future studies that adhere to current standards for internal validity,such as Cochrane risk of bias [29], CONSORT [81, 82], and GRADE [34, 35], will contribute to a stronger body of evidence to address policy makers’ questions regarding the cost-effectiveness of NP and CNS roles.

Appendix

Electronic Database Search Strategies

 Database: CINAHL Data range: 1981 to July 31, 2012 Results: 713 # Query S56 S53 and S55 S55 S20 or S21 or S25 S54 S26 and S53 S53 S50 or S51 or S52 S52 S47 or S48 or S49 S51 S37 or S38 or S39 or S40 or S41 or S42 or S43 or S44 or S45 or S46 S50 S27 or S28 or S29 or S30 or S31 or S32 or S33 or S34 or S35 or S36 S49 placebo N4 control S48 placebo N4 trial S47 placebo S46 (MH “Placebos”) S45 tripl N25 mask S44 tripl N25 blind S43 doubl N25 mask S42 doubl N25 blind S41 singl N25 mask S40 singl N25 blind S39 PT clinical trial S38 (MH “Double-Blind Studies”) S37 double-blind method S36 single-blind method S35 (MH “Single-Blind Studies”) S34 experimental trial S33 (MH “Clinical Trials+”) S32 controlled clinical trial S31 randomi?ed experimental trial S30 random allocat S29 (MH “Random Assignment”) S28 rct S27 randomi?ed controlled trial S26 S20 or S24 or S25 S25 nurs led S24 S11 or S12 or S13 or S16 or S17 or S18 or S19 S23 S1 or S2 or S3 or S4 or S5 or S6 or S7 or S8 or S9 or S10 or S11 or S12 or S13 or S14or S15 or S16 or S17 or S18 or S19 or S20 or S21 S22 S1 or S2 or S3 or S4 or S5 or S6 or S7 or S8 or S9 or S10 or S11 or S12 or S13 or S14 or S15 or S16 or S17 or S18 or S19 or S20 or S21 S21 S11 or S12 or S16 or S17 or S18 or S19 S20 S1 or S2 or S3 or S4 or S5 or S6 or S7 or S8 or S9 or S10 S19 (MH “Nurse Anesthetists”) S18 nurse anesthetist S17 nurse anaesthetist S16 nurse clinician S15 NP S14 np S13 “np” and nurse S12 CNS and nurse S11 specialist nurse S10 nurse specialist S9 nurse practitioner S8 (MH “Nurse Practitioners+”) S7 clinical nurse specialist S6 (MH “Clinical Nurse Specialists”) S5 apn S4 advanced practice nurs S3 (MH “Advanced Practice Nurses+”) S2 advanced nursing practice S1 (MH “Advanced Nursing Practice+”)

 Database: EMBASE Date: 1980 to July 31, 2012 Results: 1552 1. advanced nursing practice.mp. 2. advanced practice nurs.mp. 3. apn.mp. 4. clinical nurse specialist.mp. 5. (cns and nurs).mp. 6. exp advanced practice nurse/ 7. nurse practitioner.mp. 8. (np and nurse).mp. 9. nurse specialist.mp. 10. specialist nurse.mp. 11. nurse clinician.mp. 12. nurse an?esthetist.mp. 13. nurs led.mp. 14. or/1-13 15. randomized controlled trial/ 16. randomi?ed controlled trial.mp. 17. rct.mp. 18. randomi?ed controlled trial.pt. 19. randomized controlled trial.pt. 20. RANDOMIZATION/ 21. random allocat.mp. 22. randomi?ed experimental trial.mp. 23. controlled clinical trial/ 24. controlled clinical trial.mp. 25. clinical trial/ 26. clinical trial.mp. 27. clinical trial.pt. 28. experimental trial.mp. 29. single blind procedure/ 30. double blind procedure/ 31. triple blind procedure/ 32. ((singl or doubl or tripl or trebl) adj25 (blind or mask)).mp. 33. (placebo or (placebo adj4 (trial or control))).mp. 34. or/15-33 35. 14 and 34

 Database: Global Health Date: 1973 to July 31, 2012 Results: 38 1. advanced nursing practice.mp. 2. advanced practice nurs.mp. 3. apn.mp. 4. clinical nurse specialist.mp. 5. (cns and nurs).mp. 6. advanced practice nurse/ 7. nurse practitioner.mp. 8. (np and nurse).mp. 9. nurse specialist.mp. 10. specialist nurse.mp. 11. nurse clinician.mp. 12. nurse an?esthetist.mp. 13. nurs led.mp. 14. or/1-13 15. randomized controlled trial/ 16. randomi?ed controlled trial.mp. 17. rct.mp. 18. randomi?ed controlled trial.pt. 19. randomized controlled trial.pt. 20. RANDOMIZATION/ 21. random allocat.mp. 22. randomi?ed experimental trial.mp. 23. controlled clinical trial/ 24. controlled clinical trial.mp. 25. clinical trial/ 26. clinical trial.mp. 27. clinical trial.pt. 28. experimental trial.mp. 29. single blind procedure/ 30. double blind procedure/ 31. triple blind procedure/ 32. ((singl or doubl or tripl or trebl) adj25 (blind or mask)).mp. 33. (placebo or (placebo adj4 (trial or control))).mp. 34. or/15-33 35. 14 and 34

 Database: HealthStar Date range: 1966 to July 31, 2012 Results: 1170 1. advanced nursing practice.mp. 2. advanced practice nurs.mp. 3. APN.mp. 4. clinical nurse specialist.mp. 5. (CNS and nurs).mp. 6. nurse practitioners/ 7. nurse practitioner.mp. 8. (NP and nurs).mp. 9. nurse specialist.mp. 10. specialist nurse.mp. 11. nurse clinicians/ 12. nurse clinician.mp. 13. nurse anesthetists/ 14. nurse anesthetist.mp. 15. nurse anaesthetist.mp. 16. nurs led.mp. 17. or/1-16 18. randomized controlled trials/ 19. randomi?ed controlled trial.mp. 20. randomized controlled trial.pt. 21. RCT.mp. 22. random allocation/ 23. random allocat.mp. 24. randomi?ed experimental trial.mp. 25. controlled clinical trial.mp. 26. controlled clinical trial.pt. 27. randomized controlled trial.pt. 28. controlled clinical trial.mp. 29. experimental trial.mp. 30. single-blind method/ 31. double-blind method/ 32. ((singl or doubl or tripl or trebl) adj25 (blind or mask)).mp. 33. (placebo or (placebo adj4 (trial or control))).mp. 34. or/18-33 35. 17 and 34

 Database: Medline Date range: 1950 to July 31, 2012 Results: 1349 1. advanced nursing practice.mp. 2. advanced practice nurs.mp. 3. APN.mp. 4. clinical nurse specialist.mp. 5. (CNS and nurs).mp. 6. nurse practitioners/ 7. nurse practitioner.mp. 8. (NP and nurs).mp. 9. nurse specialist.mp. 10. specialist nurse.mp. 11. nurse clinicians/ 12. nurse clinician.mp. 13. nurse anesthetists/ 14. nurse anaesthetist.mp. 15. nurse anesthetist.mp. 16. nurs led.mp. 17. or/1-16 18. randomized controlled trial/ 19. randomi?ed controlled trial.mp. 20. RCT.mp. 21. randomized controlled trials as topic/ 22. randomized controlled trial.pt. 23. random allocation/ 24. random allocat.mp. 25. randomi?ed experimental trial.mp. 26. controlled clinical trial/ 27. controlled clinical trial.mp. 28. clinical trial.pt. 29. clinical trial.mp. 30. experimental trial.mp. 31. single-blind method/ 32. double-blind method/ 33. ((singl or doubl or tripl or trebl) adj25 (blind or mask)).mp. 34. (placebo or (placebo adj4 (trial or control)).mp. 35. or/18-34 36. 17 and 35

 Database: AMED Date range: all years to July 31, 2012 Results: 30 1. advanced nursing practice.mp. 2. advanced practice nurs.mp. 3. APN.mp. 4. clinical nurse specialist.mp. 5. (CNS and nurse).mp. 6. nurse practitioner.mp. 7. (np and nurse).mp. 8. nurse specialist.mp. 9. specialist nurse.mp. 10. nurse clinician.mp. 11. nurse anesthetist.mp. 12. nurse anaesthetist.mp. 13. or/1-12 14. randomized controlled trials/ 15. randomi?ed controlled trial.mp. 16. RCT.mp. 17. randomized controlled trial.pt. 18. Random allocation/ 19. random allocat.mp. 20. randomi?ation.mp. 21. randomi?ed experimental trial.mp. 22. controlled clinical trial.mp. 23. Clinical trials/ 24. clinical trial.mp. 25. clinical trial.pt. 26. experimental trial.mp. 27. ((singl or doubl or tripl or trebl) adj25 (blind or mask)).mp. 28. (placebo or (placebo adj4 (trial or control))).mp. 29. or/14-28 30. 13 and 29 31. nurs led.mp. 32. 31 or 13 33. 32 and 29

 Database: Cochrane Date range: all years to July 31, 2012 Results: 145 1. advanced nursing practice.mp. 2. advanced practice nurs.mp. 3. apn.mp. 4. clinical nurse specialist.mp. 5. (CNS and nurse).mp. 6. nurse practitioner.mp. 7. (np and nurse).mp. 8. nurse specialist.mp. 9. specialist nurse.mp. 10. nurse clinician.mp. 11. nurse anaesthetist.mp. 12. nurse anesthetist.mp. 13. or/1-12 14. randomi?ed controlled trial.mp. 15. RCT.mp. 16. random allocat.mp. 17. randomi?ed experimental trial.mp. 18. controlled clinical trial.mp. 19. clinical trial.mp. 20. experimental trial.mp. 21. ((singl or doubl or tripl or trebl) adj25 (blind or mask)).mp. 22. (placebo or (placebo adj4 (trial or control))).mp. 23. or/14-22 24. 13 and 23 25. nurs led.mp. 26. (13 or 25) and 23

 Database: Cochrane Central Date range: all years to July 31, 2012 Results: 474 1. advanced nursing practice.mp. 2. advanced practice nurs.mp. 3. apn.mp. 4. clinical nurse specialist.mp. 5. (CNS and nurse).mp. 6. nurse practitioner.mp. 7. (np and nurse).mp. 8. nurse specialist.mp. 9. specialist nurse.mp. 10. nurse clinician.mp. 11. nurse anaesthetist.mp. 12. nurse anesthetist.mp. 13. or/1-12 14. randomi?ed controlled trial.mp. 15. RCT.mp. 16. random allocat.mp. 17. randomi?ed experimental trial.mp. 18. controlled clinical trial.mp. 19. clinical trial.mp. 20. experimental trial.mp. 21. ((singl or doubl or tripl or trebl) adj25 (blind or mask)).mp. 22. (placebo or (placebo adj4 (trial or control))).mp. 23. or/14-22 24. 13 and 23 25. nurs led.mp. 26. (13 or 25) and 23

 Database: DARE Data range: all years to July 31, 2012 Results: 85 1. advanced nursing practice.mp. 2. advanced practice nurs.mp. 3. apn.mp. 4. clinical nurse specialist.mp. 5. (CNS and nurse).mp. 6. nurse practitioner.mp. 7. (np and nurse).mp. 8. nurse specialist.mp. 9. specialist nurse.mp. 10. nurse clinician.mp. 11. nurse anaesthetist.mp. 12. nurse anesthetist.mp. 13. or/1-12 14. randomi?ed controlled trial.mp. 15. RCT.mp. 16. random allocat.mp. 17. randomi?ed experimental trial.mp. 18. controlled clinical trial.mp. 19. clinical trial.mp. 20. experimental trial.mp. 21. ((singl or doubl or tripl or trebl) adj25 (blind or mask)).mp. 22. (placebo or (placebo adj4 (trial or control))).mp. 23. or/14-22 24. 13 and 23 25. nurs led.mp. 26. (13 or 25) and 23

 Database: HEED Data range: all years to July 31, 2012 Results: 108 1. advanced nursing practice.mp. 2. advanced practice nurs.mp. 3. apn.mp. 4. clinical nurse specialist.mp. 5. (CNS and nurse).mp. 6. nurse practitioner.mp. 7. (np and nurse).mp. 8. nurse specialist.mp. 9. specialist nurse.mp. 10. nurse clinician.mp. 11. nurse anaesthetist.mp. 12. nurse anesthetist.mp. 13. or/1-12 14. randomi?ed controlled trial.mp. 15. RCT.mp. 16. random allocat.mp. 17. randomi?ed experimental trial.mp. 18. controlled clinical trial.mp. 19. clinical trial.mp. 20. experimental trial.mp. 21. ((singl or doubl or tripl or trebl) adj25 (blind or mask)).mp. 22. (placebo or (placebo adj4 (trial or control))).mp. 23. or/14-22 24. 13 and 23 25. nurs led.mp. 26. (13 or 25) and 23

 Database: Web of Science Includes: Science Citation Index, Social Sciences Index, Arts & Humanities Index, Conference Proceedings Citation Index (Science), Conference Proceedings Citation Index (Social Sciences and Humanities) Data range: all years Results: 1333 #16  #15 AND #8 #15  #14 OR #13 OR #12 OR #11 OR #9 #14 TS = (placebo OR (placebo SAME (trial OR control))) #13  TS = ((singl or doubl or tripl or trebl) AND (blind OR mask)) #12  TS = ((single or double or triple or trebl) SAME blind procedure) #11  TS = (clinical trial OR experimental trial) #10  TS = ((randomied controlled trial) OR rct) #9  TS = ((randomied controlled trial) OR rct OR randomiation OR random allocation OR randomied experimental trial) #8  #7 OR #6 #7  TS =  (nurs lead) #6  #5 OR #4 OR #3 OR #2 OR #1 #5  TS = (nurse anesthetist) #4  TS = (nurse practitioner) #3  TS = (nurse SAME specialist) #2  TS = ((apn or cns or np) AND nurse) #1  TS = (advanced SAME nurs SAME practice).

Conflict of Interests

The authors declare that there are no conflict of interests regarding the publication of this paper.

Acknowledgments

This systematic review was made possible through funding by the Ontario Ministry of Health and Long-Term Care, the Office of Nursing Policy in Health Canada, and the Canadian Foundation for Healthcare Improvement. The authors thank librarians, Laura Banfield and Jo-Anne Petropoulos, and staff, Julie Vohra, R. James McKinlay, Sara Kaffashian, and Rose Vonau. An international panel was very helpful in determining whether studies conducted outside of North America were evaluating advanced practice nursing roles. This panel included Melanie Rogers (UK), Petrie Roodbol (The Netherlands), Madrean Schober (Singapore), Barbara Sheer (USA), Helen Ward (UK), and Frances Wong (Hong Kong). The authors thank Brian Hutchison who provided input from a health policy perspective and Gordon Guyatt who provided methodological advice regarding the risk of bias. Special thanks go to our Advisory Board for their thoughtful feedback and suggestions: Theresa Agnew, Rachel Bard, Donna Denney, Barbara Foster, Elsabeth Jensen, Sandra MacDonald-Rencz, Lynne McVey, Robin Newhouse, and Linda Sawchenko. A special thank you goes to the many authors of papers included in our review for their research and for providing additional information. The views expressed in this paper are those of the authors and do not necessarily reflect those of the Ontario Ministry of Health and Long-Term Care, Office of Nursing Policy, Health Canada, or the Canadian Foundation for Healthcare Improvement.

References

M. L. Delamaire and G. Lafortune, “Nurses in advanced roles: a description and evaluation of experiences in 12 developed countries,” OECD Health Working Paper 54, Organization for Economic Cooperation and Development, 2010, http://www.oecd-ilibrary.org/social-issues-migration-health/nurses-in-advanced-roles_5kmbrcfms5g7-en;jsessionid=20j38qefphtcs.x-oecd-live-01-01.

International Council of Nurses, Nurse Practitioner/Advanced Practice Nurse: Definition and Characteristics, 2009, http://acnp.org.au/sites/default/files/33/definition_of_apn-np.pdf.

F. Affara, ICN Framework of Competencies for the Nurse Specialist, International Council of Nurses, European Parkinson’s Disease Association, 2009, http://www.epda.eu.com/EasySiteWeb/GatewayLink.aspx?alId=7465.

M. Laurant, M. Harmsen, H. Wollersheim, R. Grol, M. Faber, and B. Sibbald, “The impact of nonphysician clinicians: do they improve the quality and cost-effectiveness of health care services?” Medical Care Research and Review, vol. 66, no. 6, pp. 36S–89S, 2009.

M. Laurant, D. Reeves, R. Hermens, J. Braspenning, R. Grol, and B. Sibbald, “Substitution of doctors by nurses in primary care,” Cochrane Database of Systematic Reviews, vol. 18, no. 2, Article ID CD001271, 2005.

G. M. Batchelor, W. O. Spitzer, A. E. Comley, and G. D. Anderson, “Nurse practitioners in primary care. IV. Impact of an interdisciplinary team on attitudes of a rural population,” Canadian Medical Association Journal, vol. 112, no. 12, pp. 1415–1420, 1975.

L. W. Chambers, P. Bruce-Lockhart, D. P. Black, E. Sampson, and M. Burke, “A controlled trial of the impact of the family practice nurse on volume, quality, and cost of rural health services,” Medical Care, vol. 15, no. 12, pp. 971–981, 1977.

L. W. Chambers and A. E. West, “Assessment of the role of the family practice nurse in urban medical practices,” Canadian Journal of Public Health, vol. 69, no. 6, pp. 459–468, 1978.

L. W. Chambers and A. E. West, “The St John’s randomized trial of the family practice nurse: health outcomes of patients,” International Journal of Epidemiology, vol. 7, no. 2, pp. 153–161, 1978.

D. L. Sackett, W. O. Spitzer, M. Gent, and R. S. Roberts, “The Burlington randomized trial of the nurse practitioner: health outcomes of patients,” Annals of Internal Medicine, vol. 80, no. 2, pp. 137–142, 1974.

W. O. Spitzer, D. L. Sackett, J. C. Sibley et al., “The Burlington randomized trial of the nurse practitioner,” The New England Journal of Medicine, vol. 290, no. 5, pp. 251–256, 1974.

W. O. Spitzer, D. J. Kergin, M. A. Yoshida, W. A. Russell, B. C. Hackett, and C. H. Goldsmith, “Nurse practitioners in primary care. III. The southern Ontario randomized trial,” Canadian Medical Association Journal, vol. 108, no. 8, p. 1005, 1973.

W. O. Spitzer, D. J. Kergin, M. A. Yoshida, W. A. Russell, B. C. Hackett, and C. H. Goldsmith, “Nurse practitioners in primary care: the southern Ontario randomized trial,” Health Care Dimensions, pp. 105–119, 1975.

W. O. Spitzer, R. S. Roberts, and T. Delmore, “Nurse practitioners in primary care. VI. Assessment of their deployment with the utilization and financial index,” Canadian Medical Association Journal, vol. 114, no. 12, pp. 1103–1108, 1976.

R. Burnip, R. Erickson, G. D. Barr, H. Shinefield, and E. J. Schoen, “Well-child care by pediatric nurse practitioners in a large group practice. A controlled study in 1,152 preschool children,” American Journal of Diseases of Children, vol. 130, no. 1, pp. 51–55, 1976.

R. A. Hoekelman, “What constitutes adequate well baby care?” Pediatrics, vol. 55, no. 3, pp. 313–326, 1975.

K. Soghikian, “The role of nurse practitioners in hypertension care,” Clinical Science and Molecular Medicine Supplement, vol. 4, pp. 345s–358s, 1978.

L. W. Chambers, “Financial impact of family practice nurses on medical practice in Canada,” Inquiry, vol. 16, no. 4, pp. 339–349, 1979.

R. E. Lees, “Physician time-saving by employment of expanded-role nurses in family practice,” Canadian Medical Association Journal, vol. 108, no. 7, pp. 871–875, 1973.

W. O. Spitzer, W. A. Russell, and B. C. Hackett, “Financial consequences of employing a nurse practitioner,” Ontario Medical Review, vol. 29, pp. 96–100, 1973.

U.S. Congress Office of Technology Assessment, Nurse Practitioners, Physician Assistants, and Certified Nurse-Midwives: A Policy Analysis, (Health Technology Case Study 37), OTA-HCS-37, Office of Technology Assessment, 1986, http://ota-cdn.fas.org/reports/8615.pdf.

S. A. Brown and D. E. Grimes, “A meta-analysis of nurse practitioners and nurse midwives in primary care,” Nursing Research, vol. 44, no. 6, pp. 332–339, 1995.

M. Fry, “Literature review of the impact of nurse practitioners in critical care services,” Nursing in Critical Care, vol. 16, no. 2, pp. 58–66, 2011.

S. Horrocks, E. Anderson, and C. Salisbury, “Systematic review of whether nurse practitioners working in primary care can provide equivalent care to doctors,” British Medical Journal, vol. 324, no. 7341, pp. 819–823, 2002.

R. P. Newhouse, J. Stanik-Hutt, K. M. White et al., “Advanced practice nurse outcomes 1990–2008: a systematic review,” Nursing Economics, vol. 29, no. 5, pp. 230–250, 2011.

M. W. Pozen, J. A. Stechmiller, W. Harris, S. Smith, D. D. Fried, and G. C. Voigt, “A nurse rehabilitator’s impact on patients with myocardial infarction,” Medical Care, vol. 15, no. 10, pp. 830–837, 1977.

J. S. Fulton and K. Baldwin, “An annotated bibliography reflecting CNS practice and outcomes,” Clinical Nurse Specialist CNS, vol. 18, no. 1, pp. 21–39, 2004.

F. Donald, D. Bryant-Lukosius, R. Martin-Misener et al., “Clinical nurse specialists and nurse practitioners: title confusion and lack of role clarity,” Nursing Leadership, vol. 23, special issue, pp. 189–201, 2010.

J. P. T. Higgins and S. Green, Cochrane Handbook for Systematic Reviews of Interventions: Version 5.1.0. [Updated], Cochrane, 2011, http://handbook.cochrane.org/.

P. Jüni, D. G. Altman, and M. Egger, “Systematic reviews in health care: assessing the quality of controlled clinical trials,” The British Medical Journal, vol. 323, no. 7303, pp. 42–46, 2001.

C. F. Chiou, J. W. Hay, J. F. Wallace et al., “Development and validation of a grading system for the quality of cost-effectiveness studies,” Medical Care, vol. 41, no. 1, pp. 32–44, 2003.

J. J. Ofman, S. D. Sullivan, P. J. Neumann et al., “Examining the value and quality of health economic analyses: implications of utilizing the QHES,” Journal of Managed Care Pharmacy, vol. 9, no. 1, pp. 53–61, 2003.

L. E. Peterson, C. Goodman, E. K. Karnes, C. J. Chen, and J. A. Schwartz, “Assessment of the quality of cost analysis literature in physical therapy,” Physical Therapy, vol. 89, no. 8, pp. 733–755, 2009.

G. Guyatt, A. D. Oxman, E. A. Akl et al., “GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings table,” Journal of Clinical Epidemiology, vol. 64, no. 4, pp. 383–394, 2011.

G. H. Guyatt, A. D. Oxman, R. Kunz et al., “GRADE guidelines: 6. Rating the quality of evidence—imprecision,” Journal of Clinical Epidemiology, vol. 64, no. 12, pp. 1283–1293, 2011.

F. Donald, A. DiCenso, D. Bryant-Lukosius, et al., “A systematic review of the effectiveness and cost-effectiveness of nurse practitioners and clinical nurse specialists,” Final Report Grant 06514, Ontario Ministry of Health and Long-Term Care; The Office of Nursing Policy, Health Canada; and the Canadian Foundation for Healthcare Improvement, 2014.

M. D. Naylor, L. H. Aiken, E. T. Kurtzman, D. M. Olds, and K. B. Hirschman, “The care span: the importance of transitional care in achieving health reform,” Health Affairs, vol. 30, no. 4, pp. 746–754, 2011.

J. K. Allen, R. S. Blumenthal, S. Margolis, D. R. Young, E. R. Miller III, and K. Kelly, “Nurse case management of hypercholesterolemia in patients with coronary heart disease: Results of a randomized clinical trial,” American Heart Journal, vol. 144, no. 4, pp. 678–686, 2002.

A. T. Dierick-van Daele, J. F. Metsemakers, E. W. Derckx, C. Spreeuwenberg, and H. J. M. Vrijhoef, “Nurse practitioners substituting for general practitioners: randomized controlled trial,” Journal of Advanced Nursing, vol. 65, no. 2, pp. 391–401, 2009.

P. Kinnersley, E. Anderson, K. Parry et al., “Randomised controlled trial of nurse practitioner versus general practitioner care for patients requesting “same day” consultations in primary care,” The British Medical Journal, vol. 320, no. 7241, pp. 1043–1048, 2000.

S. L. Krein, M. L. Klamerus, S. Vijan et al., “Case management for patients with poorly controlled diabetes: a randomized trial,” The American Journal of Medicine, vol. 116, no. 11, pp. 732–739, 2004.

M. Limoges-Gonzalez, N. S. Mann, A. Al-Juburi, D. Tseng, J. Inadomi, and L. Rossaro, “Comparisons of screening colonoscopy performed by a nurse practitioner and gastroenterologists: a single-center randomized controlled trial,” Gastroenterology Nursing, vol. 34, no. 3, pp. 210–216, 2011.

D. Litaker, L. C. Mion, L. Planavsky, C. Kippes, N. Mehta, and J. Frolkis, “Physician-nurse practitioner teams in chronic disease management: the impact on costs, clinical effectiveness, and patients’ perception of care,” Journal of Interprofessional Care, vol. 17, no. 3, pp. 223–237, 2003.

M. O. Mundinger, R. L. Kane, E. R. Lenz et al., “Primary care outcomes in patients treated by nurse practitioners or physicians: a randomized trial,” The Journal of the American Medical Association, vol. 283, no. 1, pp. 59–68, 2000.

E. W. Nelson, S. Van Cleve, M. K. Swartz, W. Kessen, and P. L. McCarthy, “Improving the use of early follow-up care after emergency department visits: a randomized trial,” American Journal of Diseases of Children, vol. 145, no. 4, pp. 440–444, 1991.

M. L. A. Schuttelaar, K. M. Vermeulen, N. Drukker, and P. J. Coenraads, “A randomized controlled trial in children with eczema: nurse practitioner vs. dermatologist,” British Journal of Dermatology, vol. 162, no. 1, pp. 162–170, 2010.

R. C. Smith, J. S. Lyles, J. C. Gardiner et al., “Primary care clinicians treat patients with medically unexplained symptoms: a randomized controlled trial,” Journal of General Internal Medicine, vol. 21, no. 7, pp. 671–677, 2006.

P. Venning, A. Durie, M. Roland, C. Roberts, and B. Leese, “Randomised controlled trial comparing cost effectiveness of general practitioners and nurse practitioners in primary care,” British Medical Journal, vol. 320, no. 7241, pp. 1048–1053, 2000.

E. A. Coleman, C. Parry, S. Chalmers, and S. J. Min, “The care transitions intervention: results of a randomized controlled trial,” Archives of Internal Medicine, vol. 166, no. 17, pp. 1822–1828, 2006.

A. Hollingsworth and S. Cohen, “Outcomes of early hospital discharge of women undergoing abdominal hysterectomy,” in Measuring Patient Outcomes, M. T. Nolan and V. Mock, Eds., pp. 155–167, Sage, London, UK, 2000.

M. A. Kotowycz, T. L. Cosman, C. Tartaglia, R. Afzal, R. P. Syal, and M. K. Natarajan, “Safety and feasibility of early hospital discharge in ST-segment elevation myocardial infarction—a prospective and randomized trial in low-risk primary percutaneous coronary intervention patients (the Safe-Depart Trial),” American Heart Journal, vol. 159, no. 1, pp. 117.e1–117.e6, 2010.

J. A. Nathan, L. Pearce, C. Field et al., “A randomized controlled trial of follow-up of patients discharged from the hospital following acute asthma: best performed by specialist nurse or doctor?” Chest, vol. 130, no. 1, pp. 51–57, 2006.

S. M. Rawl, K. L. Easton, S. Kwiatkowski, D. Zemen, and B. Burczyk, “Effectiveness of a nurse-managed follow-up program for rehabilitation patients after discharge,” Rehabilitation Nursing Journal, vol. 23, no. 4, pp. 204–209, 1998.

A. Mitchell-DiCenso, G. Guyatt, M. Marrin et al., “A controlled trial of nurse practitioners in neonatal intensive care,” Pediatrics, vol. 98, no. 6, part 1, pp. 1143–1148, 1996.

M. H. Pioro, C. S. Landefeld, P. F. Brennan et al., “Outcomes-based trial of an inpatient nurse practitioner service for general medical patients,” Journal of Evaluation in Clinical Practice, vol. 7, no. 1, pp. 21–33, 2001.

J. S. Alexander, R. E. Younger, R. M. Cohen, and L. V. Crawford, “Effectiveness of a nurse-managed program for children with chronic asthma,” Journal of Pediatric Nursing, vol. 3, no. 5, pp. 312–317, 1988.

E. E. Arts, S. A. Landewe-Cleuren, N. C. Schaper, and H. J. Vrijhoef, “The cost-effectiveness of substituting physicians with diabetes nurse specialists: a randomized controlled trial with 2-year follow-up,” Journal of Advanced Nursing, vol. 68, no. 6, pp. 1224–1234, 2012.

A. F. Brandon, J. B. Schuessler, K. J. Ellison, and R. B. Lazenby, “The effects of an advanced practice nurse led telephone intervention on outcomes of patients with heart failure,” Applied Nursing Research, vol. 22, no. 4, pp. e1–e7, 2009.

D. Brooten, J. M. Youngblut, L. Brown, S. A. Finkler, D. F. Neff, and E. Madigan, “A randomized trial of nurse specialist home care for women with high-risk pregnancies: outcomes and costs. [Erratum appears in American Journal of Managed Care 2001 Sep;7(9):855],” The American Journal of Managed Care, vol. 7, no. 8, pp. 793–803, 2001.

W. T. Chien, S. F. Leung, and C. Chu, “A nurse-led, needs-based psycho-education intervention for Chinese patients with first-onset mental illness,” Contemporary Nurse, vol. 40, no. 2, pp. 194–209, 2012.

L. K. Evans, N. E. Strumpf, S. L. Allen-Taylor, E. Capezuti, G. Maislin, and B. Jacobsen, “A clinical trial to reduce restraints in nursing homes,” Journal of the American Geriatrics Society, vol. 45, no. 6, pp. 675–681, 1997.

S. Faithfull, J. Corner, L. Meyer, R. Huddart, and D. Dearnaley, “Evaluation of nurse-led follow up for patients undergoing pelvic radiotherapy,” British Journal of Cancer, vol. 85, no. 12, pp. 1853–1864, 2001.

L. J. Ritz, M. J. Nissen, K. K. Swenson et al., “Effects of advanced nursing care on quality of life and cost outcomes of women diagnosed with breast cancer,” Oncology Nursing Forum, vol. 27, no. 6, pp. 923–932, 2000.

S. Ryan, A. B. Hassell, M. Lewis, and A. Farrell, “Impact of a rheumatology expert nurse on the wellbeing of patients attending a drug monitoring clinic,” Journal of Advanced Nursing, vol. 53, no. 3, pp. 277–286, 2006.

R. W. Swindle, J. K. Rao, A. Helmy et al., “Integrating clinical nurse specialists into the treatment of primary care patients with depression,” International Journal of Psychiatry in Medicine, vol. 33, no. 1, pp. 17–37, 2003.

G. J. Tijhuis, A. H. Zwinderman, J. M. W. Hazes, W. B. Van Den Hout, F. C. Breedveld, and T. P. M. Vliet Vlieland, “A randomized comparison of care provided by a clinical nurse specialist, an inpatient team, and a day patient team in rheumatoid arthritis,” Arthritis & Rheumatism, vol. 47, no. 5, pp. 525–531, 2002.

D. Brooten, S. Kumar, L. P. Brown et al., “A randomized clinical trial of early hospital discharge and home follow-up of very-low-birth-weight infants,” The New England Journal of Medicine, vol. 315, no. 15, pp. 934–939, 1986.

D. Brooten, M. Roncoli, S. Finkler, L. Arnold, A. Cohen, and M. Mennuti, “A randomized trial of early hospital discharge and home follow-up of women having cesarean birth,” Obstetrics and Gynecology, vol. 84, no. 5, pp. 832–838, 1994.

C. A. Dellasega and T. M. Zerbe, “A multimethod study of advanced practice nurse postdischarge care,” Clinical Excellence for Nurse Practitioners, vol. 4, no. 5, pp. 286–293, 2000.

L. Kennedy, S. Neidlinger, and K. Scroggins, “Effective comprehensive discharge planning for hospitalized elderly,” The Gerontologist, vol. 27, no. 5, pp. 577–580, 1987.

A. S. Laramee, S. K. Levinsky, J. Sargent, R. Ross, and P. Callas, “Case management in a heterogeneous congestive heart failure population: a randomized controlled trial,” Archives of Internal Medicine, vol. 163, no. 7, pp. 809–817, 2003.

R. McCorkle, N. E. Strumpf, I. F. Nuamah et al., “A specialized home care intervention improves survival among older post-surgical cancer patients,” Journal of the American Geriatrics Society, vol. 48, no. 12, pp. 1707–1713, 2000.

R. McCorkle, M. Dowd, E. Ercolano et al., “Effects of a nursing intervention on quality of life outcomes in post-surgical women with gynecological cancers,” Psycho-Oncology, vol. 18, no. 1, pp. 62–70, 2009.

M. D. Naylor, “Comprehensive discharge planning for hospitalized elderly: a pilot study,” Nursing Research, vol. 39, no. 3, pp. 156–161, 1990.

M. Naylor, D. Brooten, R. Jones, R. Lavizzo-Mourey, M. Mezey, and M. Pauly, “Comprehensive discharge planning for the hospitalized elderly: a randomized clinical trial,” Annals of Internal Medicine, vol. 120, no. 12, pp. 999–1006, 1994.

M. D. Naylor, D. Brooten, R. Campbell et al., “Comprehensive discharge planning and home follow-up of hospitalized elders: a randomized clinical trial,” Journal of the American Medical Association, vol. 281, no. 7, pp. 613–620, 1999.

M. D. Naylor, D. A. Brooten, R. L. Campbell, G. Maislin, K. M. McCauley, and J. S. Schwartz, “Transitional care of older adults hospitalized with heart failure: a randomized, controlled trial,” Journal of the American Geriatrics Society, vol. 52, no. 5, pp. 675–684, 2004.

D. R. Thompson, A. Roebuck, and S. Stewart, “Effects of a nurse-led, clinic and home-based intervention on recurrent hospital use in chronic heart failure,” European Journal of Heart Failure, vol. 7, no. 3, pp. 377–384, 2005.

R. York, L. P. Brown, P. Samuels et al., “A randomized trial of early discharge and nurse specialist transitional follow-up care of high-risk childbearing women,” Nursing Research, vol. 46, no. 5, pp. 254–261, 1997.

S. Talley, D. S. Davis, N. Goicoechea, L. Brown, and L. L. Barber, “Effect of psychiatric liaison nurse specialist consultation on the care of medical-surgical patients with sitters,” Archives of Psychiatric Nursing, vol. 4, no. 2, pp. 114–123, 1990.

K. F. Schulz, D. G. Altman, and D. Moher, “CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials,” BMC Medicine, vol. 8, article 18, 2010.

D. Moher, S. Hopewell, K. F. Schulz, et al., “CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials,” Journal of Clinical Epidemiology, vol. 63, no. 8, pp. e1–e37, 2010.

D. A. Richards and J. P. H. Hamers, “RCTs in complex nursing interventions and laboratory experimental studies,” International Journal of Nursing Studies, vol. 46, no. 4, pp. 588–592, 2009.

J. J. Deeks, J. Dinnes, R. D’Amico et al., “Evaluating non-randomised intervention studies,” Health Technology Assessment, vol. 7, no. 27, pp. 1–173, 2003.

C. Y. Lu, “Observational studies: a review of study designs, challenges and strategies to reduce confounding,” International Journal of Clinical Practice, vol. 63, no. 5, pp. 691–697, 2009.

J. Odgaard-Jensen, G. E. Vist, A. Timmer, et al., Randomisation to Protect against Selection Bias in Healthcare Trials (Review), Cochrane Database of Systematic Reviews, 2011, http://apps.who.int/rhl/reviews/MR000012_standard.pdf.

K. A. Paez and J. K. Allen, “Cost-effectiveness of nurse practitioner management of hypercholesterolemia following coronary revascularization,” Journal of the American Academy of Nurse Practitioners, vol. 18, no. 9, pp. 436–444, 2006.

A. T. M. Dierick-van Daele, L. M. G. Steuten, J. F. M. Metsemakers, E. W. C. C. Derckx, C. Spreeuwenberg, and H. J. M. Vrijhoef, “Economic evaluation of nurse practitioners versus GPs in treating common conditions,” The British Journal of General Practice, vol. 60, no. 570, pp. e28–e35, 2010.

E. R. Lenz, M. O. Mundinger, S. C. Hopkins, S. X. Lin, and J. L. Smolowitz, “Diabetes care processes and outcomes in patients treated by nurse practitioners or physicians,” Diabetes Educator, vol. 28, no. 4, pp. 590–598, 2002.

E. R. Lenz, M. O. Mundinger, R. L. Kane, S. C. Hopkins, and S. X. Lin, “Primary care outcomes in patients treated by nurse practitioners or physicians: two-year follow-up,” Medical Care Research and Review, vol. 61, no. 3, pp. 332–351, 2004.

M. L. Schuttelaar, K. M. Vermeulen, and P. J. Coenraads, “Costs and cost-effectiveness analysis of treatment in children with eczema by nurse practitioner vs. dermatologist: results of a randomized, controlled trial and a review of international costs,” British Journal of Dermatology, vol. 165, no. 3, pp. 600–611, 2011.

J. S. Lyles, A. Hodges, C. Collins et al., “Using nurse practitioners to implement an intervention in primary care for high-utilizing patients with medically unexplained symptoms,” General Hospital Psychiatry, vol. 25, no. 2, pp. 63–73, 2003.

Z. Luo, J. Goddeeris, J. C. Gardiner, and R. C. Smith, “Costs of an intervention for primary care patients with medically unexplained symptoms: a randomized controlled trial,” Psychiatric Services, vol. 58, no. 8, pp. 1079–1086, 2007.

C. Parry, E. A. Coleman, J. D. Smith, J. Frank, and A. M. Kramer, “The care transitions intervention: a patient-centered approach to ensuring effective transfers between sites of geriatric care,” Home Health Care Services Quarterly, vol. 22, no. 3, pp. 1–17, 2003.

K. L. Easton, S. M. Rawl, and D. Zemen, “The effects of nursing follow-up on the coping strategies used by rehabilitation patients after discharge,” Rehabilitation Nursing Research, vol. 4, no. 4, pp. 119–127, 1995.

N. E. Strumpf, L. K. Evans, J. Wagner, and J. Patterson, “Reducing physical restraints: developing an educational program,” Journal of Gerontological Nursing, vol. 18, no. 11, pp. 21–27, 1992.

J. E. Patterson, N. E. Strumpf, and L. K. Evans, “Nursing consultation to reduce restraints in a nursing home,” Clinical Nurse Specialist, vol. 9, no. 4, pp. 231–235, 1995.

E. L. Siegler, E. Capezuti, G. Maislin, M. Baumgarten, L. Evans, and N. Strumpf, “Effects of a restraint reduction intervention and OBRA ’87 regulations on psychoactive drug use in nursing homes,” Journal of the American Geriatrics Society, vol. 45, no. 7, pp. 791–796, 1997.

E. Capezuti, N. E. Strumpf, L. K. Evans, J. A. Grisso, and G. Maislin, “The relationship between physical restraint removal and falls and injuries among nursing home residents,” Journals of Gerontology Series A: Biological Sciences and Medical Sciences, vol. 53, no. 1, pp. M47–M52, 1998.

G. J. Tijhuis, K. G. Kooiman, A. H. Zwinderman, J. M. W. Hazes, F. C. Breedveld, and T. P. M. Vliet Vlieland, “Validation of a novel satisfaction questionnaire for patients with rheumatoid arthritis receiving outpatient clinical nurse specialist care, inpatient care, or day patient team care,” Arthritis & Rheumatism, vol. 49, no. 2, pp. 193–199, 2003.

G. J. Tijhuis, A. H. Zwinderman, J. M. W. Hazes, F. C. Breedveld, and P. M. T. Vliet Vlieland, “Two-year follow-up of a randomized controlled trial of a clinical nurse specialist intervention, inpatient, and day patient team care in rheumatoid arthritis,” Journal of Advanced Nursing, vol. 41, no. 1, pp. 34–43, 2003.

W. B. van den Hout, G. J. Tijhuis, J. M. Hazes, F. C. Breedveld, and T. P. M. Vliet Vlieland, “Cost effectiveness and cost utility analysis of multidisciplinary care in patients with rheumatoid arthritis: a randomised comparison of clinical nurse specialist care, inpatient team care, and day patient team care,” Annals of the Rheumatic Diseases, vol. 62, no. 4, pp. 308–315, 2003.

C. Dellasega and T. M. Zerbe, “Caregivers of frail rural older adults. Effects of an advanced practice nursing intervention,” Journal of Gerontological Nursing, vol. 28, no. 10, pp. 40–49, 2002.

S. H. Neidlinger, K. Scroggins, and L. M. Kennedy, “Cost evaluation of discharge planning for hospitalized elderly,” Nursing Economics, vol. 5, no. 5, pp. 225–230, 1987.

C. Jepson, R. McCorkle, D. Adler, I. Nuamah, and E. Lusk, “Effects of home care on caregivers’ psychosocial status,” Journal of Nursing Scholarship, vol. 31, no. 2, pp. 115–120, 1999.

R. McCorkle, S. Jeon, E. Ercolano, and P. Schwartz, “Healthcare utilization in women after abdominal surgery for ovarian cancer,” Nursing Research, vol. 60, no. 1, pp. 47–57, 2011.

M. D. Naylor and K. M. McCauley, “The effects of a discharge planning and home follow-up intervention on elders hospitalized with common medical and surgical cardiac conditions,” The Journal of Cardiovascular Nursing, vol. 14, no. 1, pp. 44–54, 1999.

K. M. McCauley, M. B. Bixby, and M. D. Naylor, “Advanced practice nurse strategies to improve outcomes and reduce cost in elders with heart failure,” Disease Management, vol. 9, no. 5, pp. 302–310, 2006.

D. Moher, A. Liberati, J. Tetzlaff, D. G. Altman, and PRISMA group, “Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement,” British Medical Journal, vol. 339, p. b2535, 2009.