Poor Reporting of Outcomes Beyond Accuracy in Point-of-Care Tests for Syphilis: A Call for a Framework

Background. Point-of-care (POC) diagnostics for syphilis can contribute to epidemic control by offering a timely knowledge of serostatus. Although accuracy data on POC syphilis tests have been widely published, few studies have evaluated broader outcomes beyond accuracy that impact patients and health systems. We comprehensively reviewed evidence and reporting of these implementation research outcomes (IROs), and proposed a framework to improve their quality. Methods. Three reviewers systematically searched 6 electronic databases from 1980 to 2014 for syphilis POC studies reporting IROs. Data were abstracted and findings synthesised narratively. Results. Of 71 studies identified, 38 documented IROs. IROs were subclassified into preference (7), acceptability (15), feasibility (15), barriers and challenges (15), impact (13), and prevalence (23). Using our framework and definitions, a pattern of incomplete documentation, inconsistent definitions, and lack of clarity was identified across all IROs. Conclusion. Although POC screening tests for syphilis were generally favourably evaluated across a range of outcomes, the quality of evidence was compromised by inconsistent definitions, poor methodology, and documentation of outcomes. A framework for standardized reporting of outcomes beyond accuracy was proposed and considered a necessary first step towards an effective implementation of these metrics in POC diagnostics research.


Introduction
Syphilis is an important public health issue; the latest available worldwide estimates conducted by the World Health Organization in 1999 suggested approximately 12 million cases of syphilis [1]. With 90% of infected individuals unaware of their serostatus, a lack of timely diagnosis is a driving force of the syphilis epidemic, particularly in resource limited settings where use of standard diagnostics presents important challenges [2]. Standard syphilis screening involves initial testing with a nontreponemic specific assay and confirmation with a treponemal specific assay [3][4][5][6]. These tests are usually expensive, must be conducted by skilled, laboratory-based personnel, and have a long turnaround time resulting in loss of patients to followup [7,8]. Point-of-care (POC) tests therefore offer a novel, low cost, easy to use solution that provides results in a single visit, expediting linkages to confirmatory testing, treatment, and referral. Designed for settings with limited infrastructure, POC technologies are potentially transformative tools for global syphilis control [3].
Systematic reviews demonstrated the accuracy of syphilis POC tests [9,10]. However, as advisory bodies on diagnostics have emphasized, accuracy is a necessary but not a sufficient condition for widespread uptake of syphilis POC tests [11]. Our objective is to facilitate decision-making related to introduction of syphilis POC tests for health planners and policymakers in diverse geographical settings, by synthesising evidence on outcomes beyond accuracy, referred to 2 AIDS Research and Treatment as implementation research outcomes (IRO). These IROs are relevant to patients, caregivers, and health systems. This systematic review presents the first comprehensive portrait of available evidence on IROs for syphilis POC tests.

Information Sources.
We searched the MEDLINE, EMBASE, GLOBAL HEALTH, CINAHL, Web of Science, and SCOPUS electronic databases over the period from January 1, 1980, to January 31, 2014. All languages were considered and non-English articles were translated.

Search String.
We used the following terms: (syphilis OR Treponema pallidum) AND (point-of-care OR rapid test OR rapid assays). Keywords rather than MeSH terms were used to capture recent papers not yet indexed and to be able to search multiple databases at once. No filters for diagnostic studies were applied as these have been shown not to capture all relevant studies [12].

Study Selection.
Three reviewers (Yalda Jafari [YJ], Sushmita Shivkumar [SS], and Rohit Vijh [RV]), independently conducted database searches and reviewed the titles and abstracts of articles retrieved. Articles satisfying the eligibility criteria were retained for full-text screening. Study inclusion was determined by discussion between the initial reviewers who, in cases of discordance, appealed to a fourth reviewer (Nitika Pant Pai [NPP]).

Eligibility Criteria.
We considered studies satisfying the following criteria.
Study Design. All studies conducted on humans or human samples reporting IROs.
Participants. Live participants of any age and risk group using whole blood or serum specimens, either from participants or serum panels.  as a proportion, where numerator consisted of individuals who tested seropositive during the study period, and denominator consisted of the total study sample [14]. If available, 95% CI was expressed. For the purposes of the review, studies that used POC tests solely to determine prevalence of syphilis in the study sample were included in this category.

Data Extraction.
Reviewers extracted relevant information independently using a standardized data abstraction form pretested on a subset of the sample. One reviewer (YJ) extracted data on 100% of papers, and second reviewers (SS & RJ) abstracted data on 50% of papers.

Summary Measures.
A narrative approach was taken to synthesize outcome data. This is standard practice for systematic reviews with considerable methodological heterogeneity such as IROs.

Results
Of 71 full-text articles, 38 (53.5%) documented IROs. Following PRISMA guidelines [15], our study selection process has been presented in Figure 1. Of 38 articles, three studies (7.9%) were in languages other than English (2 in Spanish, 1 in Portuguese) and were translated. Thirty-four (89.5%) articles referred to studies conducted in low-and middleincome country settings. Thirty-four studies employed a cross-sectional design (89.5%), one was a case-control (2.6%), and one was a clustered randomized trial (2.6%) design. Study descriptions for IROs are available in Table 1.
(D) Barriers and challenges: barriers and challenges hindered an ideal implementation of POC testing strategies and impeded feasibility. Fifteen studies provided qualitative discussions of these challenges [16-20, 23, 26, 27, 29, 34, 35, 37-40]. From the patient perspective, assumed pain from finger prick was a hindrance to testing [19,23,29], as were reports of pain or discomfort [16,20,40]. Patients also expressed a distrust of POC test results [19]. From the providers' perspective, challenges included a lack of confidence in POC test results due to the inability to differentiate between past and present infections leading to overtreatment [16,17,19,35]. In addition, disagreements between test readers [37], suboptimal field conditions [34], unreadable [36] or difficulty reading results [39], concerns over procurement of POC tests [38], and the time-consuming nature of offering one-stop services with POC [19,38] posed challenges in delivery of these services. Furthermore, not all patients agreed to undergo confirmation testing [27], hampering ideal execution of testing process. In two articles evaluating POC tests in Mongolia, inadequacies of partner notification strategies for women were brought to light, raising issues such as risk of reinfection of women if partners are not treated [40] and risk of partner violence [40]. A lack of political leadership, required to ensure the success of antenatal care (ANC) programs to prevent congenital syphilis, was also highlighted as a major barrier to uptake of POC testing [26].
In comparison to clinics that used conventional testing, clinics with POC testing showed a significantly increased number of women who were tested and a significantly increased number of women and their partners who were treated [40]. In a multicountry study, the increase in numbers screened compared to preimplementation of POC testing ranged from 1.0% in China to 9.2% in Zambia [22].

Discussion
Using our framework, all the studies could be evaluated for quality of their reporting. Although evidence was primarily obtained from cross-sectional studies, outcomes were incompletely defined and reported in many studies. IROs such as preference, acceptability, feasibility, barriers and challenges, and impact outcomes were not well defined. Furthermore, the quality of evidence was impaired by unclear definitions/framework for IROs for diagnostic studies. Documentation was unclear and IROs were often expressed as percentages often without confidence intervals or without a denominator. Heterogeneity of definitions, lack of reporting of study methodology, and poor quality of reporting hampered drawing of meaningful results, although the bulk of evidence was reported in favour of POC strategies. A determined effort needs to be made to improve the quality of assessment and reporting of IROs beyond accuracy. Impact outcomes have been clearly defined by the International Initiative for Impact Evaluation [53]. Often a vast majority of studies are cross-sectional and conducted to evaluate the feasibility of a POC strategy. Patient centered outcomes of acceptability, preference, and patient experience become equally important to justify the benefits of an alternative POC strategy to stakeholders or to scale up a project once feasibility is proven. A framework for reporting outcomes that has been proposed and utilized in this review could then be used. Clear definitions and transparency of IROs will not only help improve reporting of studies and quality of research with standardized reporting, but also facilitate comparability of findings for evaluation, across all settings. Standardized reporting will allow policymakers and providers to draw meaningful conclusions for their practice and policy. Reporting results with 95% CIs should also be emphasized. These help understand and appreciate the range of variability possible for an outcome. With these improvements, research on POC diagnostics beyond accuracy will not only be more meaningful but will also positively impact public health and clinical practice [54,55].

Conclusion
In view of the growing interest in the elimination of mother to child transmission of HIV and syphilis promoted by WHO and UNICEF and an ever growing need for simultaneous rapid screening tests for HIV and related coinfections for at risk populations, POC screening strategies stand to become more complex in the near future. To report outcomes and metrics, we need to standardize and emphasize precise reporting, documentation, and measurement in publications. In this paper, we have presented a framework that defines a few of the commonly reported IROs beyond accuracy in research that involves POCs in syphilis. To improve evidence on metrics, a methodological framework that standardizes measures and parameters and ensures standardized collection, reporting, and documentation of data beyond accuracy must be emphasized. This move will greatly improve the quality of outcomes reporting in POC diagnostics and allow for enhanced documentation beyond traditional "accuracy outcomes. "

Summary
Implementation research outcomes documented in pointof-care (POC) diagnostic evaluations for syphilis are poorly defined, evaluated, and reported across studies. We defined operational definitions and proposed a framework to help evaluate the reporting of these outcomes. This framework for syphilis POC diagnostics research will facilitate standardized collection, documentation, synthesis of outcomes, and their inclusion in policy initiatives.