Rationale for Intervention and Dose Is Lacking in Stroke Recovery Trials: A Systematic Review

Background The ineffectiveness of most complex stroke recovery trials may be explained by inadequate intervention design. The primary aim of this review was to explore the rationales given for interventions and dose in stroke rehabilitation randomised controlled trials (RCTs). Methods We searched the Cochrane Stroke Group library for RCTs that met the following criteria: (1) training based intervention; (2) >50% participants who were stroke survivors; (3) full peer-reviewed text; (4) English language. We extracted data on 16 quality items covering intervention dose (n= 3), trial design (n= 10), and risk of bias (n= 3) and 18 items related to trial method. Logistic regression analyses were performed to determine whether (1) reporting of trial quality items changed over time; (2) reporting of quality items was associated with the likelihood of a positive trial, adjusted for sample size and number of outcomes. Results 27 Cochrane reviews were included, containing 9,044 participants from 194 trials. Publication dates were 1979 to 2013, sample size was median 32 (IQR 20,58), and primary outcome was reported in 49 trials (25%). The median total quality score was 4 (IQR 3,6) and improved significantly each year (OR 1.12, 95% CI 1.07, 1.16, p<0.001). Total quality score was not associated with likelihood of a positive trial, but trials containing a biological rationale for the intervention were more likely to find a difference in patient outcome (OR 2.18, 95% CI 1.14, 4.19, p=0.02). Conclusion To develop breakthrough treatments we need to build the rationale for research interventions and testing of intervention dosage. This will be achieved through a collective research agenda to understand the mechanistic principles that drive recovery and identification of clearer targets for clinical trials.


Introduction
Understanding acute stroke biology, critical time windows, and careful patient selection has transformed acute stroke treatment and improved patient outcomes. This did not happen overnight [1] but progress is now rapid. While the methodological quality of randomised controlled trials (RCTs) in postacute stroke rehabilitation has improved over the past 40 years [2,3] our intervention toolkit remains bereft of treatments that markedly alter expected recovery trajectories. The relationship between trial quality and outcomes is not self-evident, since a clear, positive relationship between lower trial quality and positive outcomes has been shown in both preclinical [4] and clinical research [5]. As our attention increasingly turns to the field of stroke recovery, the international community is focused on the development of breakthrough recovery treatments [6,7]. Beyond the logistical challenges and costs inherent in conducting complex rehabilitation and recovery trials [8] developing suitable methods and improved reporting are critical steps in this process [7][8][9] with specific recommendations recently published in several key areas [10][11][12][13][14].
Over the last few years and in response to a number of meta-analyses suggesting higher dose (amount) of motor interventions is associated with better recovery [16][17][18][19] it has been tempting to blame neutral results in large and small trials on "insufficient dose" [20]. While authors of a recent 2 Stroke Research and Treatment  [13]. †If authors did not report these details, this was recorded as "no" in data extraction. ‡Based on best practice methods but not stated in Medical Research Council guidelines. §TIDieR guidelines [12]. ||Reported in Cochrane review as "Blinding all outcomes", "Blinding: performance and detection bias", or "Blinding: outcome assessors". #When no primary outcome was specified by authors, the outcome most closely matching the purpose of the trial was nominated by data extractors. Comparison group categories: usual care, other active intervention, waitlist, sham intervention, or attention control. Dose intensity: prespecified treatment target (i.e., the amount of physical or mental work) that participants attempted to reach in a given session (either uniform for all participants or individually tailored) [15]. Dose schedule: session length (minutes), number of sessions per day, number of sessions per week, duration of intervention (weeks), and total number of sessions. Positive trial: a significant difference between groups on the primary outcome after intervention, in favour of the intervention group [1]. Sites: single site= hospital inpatient-based intervention conducted at a single site. Multi-site= inpatient based intervention conducted at 2 or more sites. Multi-country= studies conducted in more than one country. Trial phase: research phase of randomised controlled trial-development, feasibility or piloting, and evaluation-based on the Medical Research Council Guidelines for developing and evaluating complex interventions [13]. review suggest that no amount of training changes outcomes [21], we know that high doses of motor interventions early after stroke can lead to poorer outcomes [22]. Within this context, we felt it was timely to review the attention that rehabilitation trialists have given to developing the rationale for interventions in clinical trials, to put a line in the sand so to speak, before new recommendations for intervention development and reporting hopefully take effect [23]. Therefore, the primary aim of this review was to explore the rationale given and reporting of interventions and dose in stroke rehabilitation RCTs over time. A secondary aim was to examine the association between trial quality and positive trial results. Given the volume and exponential growth in rehabilitation research over the past 10 years (13,477 Medline 2012-2016, for search strategy please see Supplementary Table  1), a systematic review with data extraction from all RCTs was simply unfeasible. We elected to limit this review to RCTs that had already met inclusion for Cochrane Systematic Reviews of training based rehabilitation (motor, speech, and cognitive) interventions where researchers studied two or more intervention doses.

Methods
This review follows the PRISMA reporting guidelines [24]. We searched the Cochrane library (http://stroke.cochrane .org, September 2013) for reviews with RCTs that met the following criteria: (1) evaluation of a training based intervention, (2) at least 50% of participants who were stroke survivors, (3) full text article published in a peer-reviewed journal, and (4) available in English. We excluded trials of pharmacological agents, nutritional supplementation, surgical interventions, or therapeutic devices that were not embedded in physical training programs.
Two researchers (KB; AR) screened titles or abstracts of the Cochrane reviews and their included RCTs to determine whether individual RCTs met inclusion criteria, with a third adjudicator (JB) consulted if necessary. If any selected Cochrane review had been updated prior to data extraction, the most recent review was included. Data were extracted by one of six researchers (AR, KB, SK, KH, HW, and JB). We examined interrater reliability by double rating 40 trials (20%).
Because no existing tool met our objective of exploring the rationale for intervention design and dose, we developed a composite data extraction tool with 16 items that included: intervention dose (n= 3, based on the TIDIER guidelines [12]), trial design (n= 10, based on Medical Research Council recommendations [13]), risk of bias (n= 3, taken from the Cochrane reviews' risk of bias tool), and 18 trial methods items that are included in Table 1. "Positive trial" was defined as a significant difference between groups on the primary outcome after intervention, in favour of the intervention group. Definitions of key terms came from existing literature or group consensus when no definition was available.

Data Analysis
Descriptive statistics were used to summarise trial characteristics. Collinearity was tested between items related to trial design (e.g., hypothesis stated; reporting of adverse events) and intervention dose (e.g., justification of dose, reporting of dose intensity). Values were within acceptable limits. "Total quality score" was calculated as count (without hierarchy) of the number of quality items addressed in each paper (i.e., total score was out of 16).
Two logistic regression analyses were performed. The first was to investigate whether reporting of trial quality items changed over time. In these analyses, individual quality items were dependent variables and year of publication was the independent variable.
In the second analysis, under the caveat that the primary outcome was prespecified at data extraction by our researchers if it was not reported in the manuscript, we performed an exploratory analysis of the whether trial design factors were associated with the likelihood of a positive trial. In these models, "positive trial" was the dependent variable with each of the 16 quality items input individually as independent variables to predict associations. Models were adjusted for study sample size and number of outcomes assessed, given that these items will influence effect size and therefore the likelihood of finding a positive effect.
Analyses were performed using STATA v 13 IC statistical software (StataCorp LP, College Station, Texas, USA). A significance level of p= 0.05 was set for all statistical tests and no correction for multiplicity was undertaken due to the exploratory nature of this analysis [25].

Included Trials.
Fifty-eight Cochrane stroke rehabilitation reviews were identified and 27 met inclusion criteria, Figure 1. Included Cochrane reviews contained 292 trials, of which 28 were included in more than 1 review, and 70 did not meet inclusion criteria. The 194 included trials comprised 9,044 participants. Details of included Cochrane reviews are shown in Supplementary Table 2, and characteristics of trials included in this review are displayed in Table 2. Most trials (n= 183, 94%) were published after the 1996 release of the Consolidated Standards of Reporting Trials (CONSORT) guidelines for reporting of RCTs [10]. Half of the trials (n= 101, 52%) were published after July 1 2005, the date that compulsory trial registration was mandated for International Committee of Medical Journal Editors (ICMJE) publication [27].

Interrater
Reliability. The kappa agreement for extracted quality items was >60% (ranging from substantial to almost perfect agreement), except for "dose intensity reported", which showed moderate agreement (55%). The primary outcome measure was specified in only 49 trials (25%), and of these, the timing of the primary outcome assessment was stated (or there was only one assessment after intervention) in 27 trials (55%). Over time, per year, it was increasingly likely that study authors specified a primary outcome (OR 1.21, 95% CI 1.11, 1.33, p<0.001). The median number of outcomes assessed per trial was 4 (IQR 3 to 8). There was a single outcome measured in 10 trials (5%), and the maximum number was 28. There were 139 different outcome measures specified, and of these, only 45 (33%) were used in more than one trial. Walking speed was the most commonly assessed outcome (n= 15, 7%).

Trial Development
The median total quality score was 4 (IQR 3, 6), and trial quality improved significantly each year (OR 1.12, 95% CI 1.07, 1.16, p <0.001). Total trial quality was not associated with likelihood of a positive trial (OR 1.04, 95% CI 0.92, 1.19, p= 0.51). Associations between positive trial and the 16 individual characteristics of trial quality are reported in Table 3. The biological rationale for the intervention was described in only 60 trials (31%), but the inclusion of a biological rationale was associated with increased odds of a positive trial (OR= 2.18, 95% CI 1.14, 4.19, p=0.02).

Discussion
The paper by Kidwell, Saver, and colleagues in 2001 [1] showcased the significant changes in the quality, precision and focus of acute stroke trials in the 1980s and 90s that heralded the discovery of life changing acute stroke treatments. This review of stroke rehabilitation research echoes the earlier findings of Kidwell et al., suggesting we are around 15 years behind our acute stroke colleagues.
Remarkably few trials prior to 2010 were international, many were small and trial quality was generally low. Encouragingly we found small improvements in overall methodological quality over time, potentially aided by adherence to reporting guidelines and mandated trial registration, but the quality score of trials was not associated with likelihood of a positive trial. It is perhaps not surprising that we have little breakthrough in recovery trials, as we considered that the majority of trials were actually feasibility or pilot work (despite often being described as efficacy trials) rather than staged development of phase 3 studies underpinned Stroke Research and Treatment 5  Table 3 for breakdown by country. † includes 31 home-based interventions, where participants were located in a single geographical region. ‡ includes 5 home-based interventions, where the study was undertaken in several regions within the same country. by a strong biological rationale. This paucity of trials with well defined, justified, piloted and biologically informed interventions is a critical consideration in future trial designs.
The idea of a critical, early window in which recovery can be optimised has gained considerable ground in recent years [28][29][30][31] yet in only 22% of trials did training start within a month of stroke. Similarly, while achieving an "optimal" treatment dose (generally higher than current usual care) is believed by many, but not all [32,33] to be the key to driving recovery, in relatively few trials was the dose of intervention, its schedule and progression well defined. The median experimental dose schedule across the trials in our review was 50 minutes per session, 5 days per week, for 6 weeks. We would argue that this closely resembled, or was even lower than, current clinical practice (i.e., it was pragmatic) rather than testing mechanistically derived hypotheses about what is needed to drive recovery [3,7]. Interestingly, we observed that trials in which a biological rationale for the intervention was presented, no matter how simple, were more likely to find a difference in patient outcome.
Although we focused significant attention on describing intervention timing and dose in this review, we recognise that what is being tested in rehabilitation trials may be just as important as questions of how, how much and 6 Stroke Research and Treatment  when interventions are delivered. We currently have few biological targets in stroke recovery; our challenge is to understand which biological targets are relevant to augment natural neural repair. Algorithms based on infarct volume and initial level of impairment have recently been developed in animal models, to predict the minimum threshold of rehabilitation required to "activate" recovery [29,30]. This model demonstrates an exciting opportunity for clinical translation.
Patient selection, stratification, and tailoring of dose were seldom reported in the trials included in this review but are critical considerations in the quest for effective recoverypromoting treatments [7,34]. Biomarkers of recovery will increasingly play an important role to guide patient selection and targeting of interventions. New guidelines for stroke recovery trials have been published that cover development, reporting, and measurement to align preclinical and clinical stroke trials [7,23,[34][35][36]. Key recommendations include imaging biomarkers in preclinical and clinical studies and standardised assessment time points and measures. We encourage the stroke community to take up these recommendations.
Our choice to perform a detailed review of 194 trials from Cochrane reviews rather than more broadly review the thousands of published trials from just the past 5 years while being pragmatic is a limitation. However, we consider Cochrane reviews to be an important source of evidence synthesis in topic areas of interest and which are likely to have informed current clinical practice. Further, we could utilise independent assessments of risk of bias from these reviews. We recognise that we have more to learn from recent high quality trials [22,[37][38][39] that are yet to be incorporated into Cochrane reviews. To capture other items that reflect trial quality [13] and intervention reporting standards [12] we defined, standardised, and extracted additional data from the trials. While errors in data extraction are possible with use of multiple raters, interrater reliability on most items was substantial to almost perfect.
To revolutionise stroke recovery we need to understand the mechanistic principles that drive recovery and identify clearer targets [1]. We must clearly define interventions, provide clear and relevant primary outcomes [3], and consider the biological plausibility of interventions and dose. Collaborative multidisciplinary research programs need to be developed that span the complex stroke continuum and include stroke consumers and clinicians, clinical researchers and basic scientists, editors, and funding bodies [7]. Similar to our acute stroke colleagues, we can expect to have more failures before we succeed, but with clearer understandings, research approaches, and a collective agenda we will make progress. There has never been a better time to be doing stroke recovery research.

Conflicts of Interest
The authors declare no other conflicts of interest.