The evidence underpinning public health policy is often of low quality, leading to inconsistencies in recommended interventions. One example is the divergence in national policies across Europe for managing contacts of invasive meningococcal disease. Aiming to develop consistent guidance at the European level, a group of experts reviewed the literature and formulated recommendations. The group defined eight priority research questions, searched the literature, and formulated recommendations using GRADE methodology. Five of the research questions are discussed in this paper. After taking into account quality of evidence, benefit, harm, value, preference, burden on patient of the intervention, and resource implications, we made four strong recommendations and five weak recommendations for intervention. Strong recommendations related not only to one question with very low quality of evidence as well as to two questions with moderate to high quality of evidence. The weak recommendations related to two questions with low and very low quality of evidence but also to one question with moderate quality of evidence. GRADE methodology ensures a transparent process and explicit recognition of additional factors that should be considered when making recommendations for policy. This approach can be usefully applied to many areas of public health policy where evidence quality is often low.
The incidence of invasive meningococcal disease (IMD), caused by
Divergences in national policies are particularly problematic in cross-border settings, as they lead to differences in disease management among population groups with the same exposure. For instance, passengers sharing an aeroplane flight with a case of IMD might or might not receive chemoprophylaxis depending on their country of residence [
It was clear from the outset that high quality evidence in this area would be limited as in many areas of public health [
Here we share our experience and lessons learned in using different types and quality of evidence to develop guidance on the public health management of IMD for European countries within a short time period using GRADE methodology. The aim of this guidance, available on the ECDC website, was to assist countries across Europe in making decisions about appropriate measures to control and prevent IMD in contacts of cases at national and subnational levels [
We adapted existing methods for producing evidence-based guidelines to deal with the short time frame and the scarcity of direct evidence (see Section
We set up a consortium of national experts: four in the area of epidemiology and public health surveillance and one in the area of microbiology of meningococcal disease. The consortium members represented five EU countries, previously involved in the assessment of national practices for IMD management across the European Union [
The consortium identified other national epidemiologists and microbiologists working with meningococcal disease from all EU countries through two established European networks (the European Meningococcal Disease Society and the ECDC European Invasive Bacterial Diseases Surveillance Network). These EU experts were asked to provide any related grey literature and technical advice during the process. Additionally, the consortium consulted two patient group networks, both based in the United Kingdom, on patient-related values and preferences.
The consortium defined research questions for guidance focused on the prevention of subsequent cases following sporadic cases of IMD and based on the needs identified through two previous surveys among public health representatives of EU countries [
What is the effectiveness of chemoprophylaxis to a case of IMD before discharge from hospital in preventing further cases of IMD? What is the effectiveness of chemoprophylaxis to household contacts of an IMD case in preventing further cases? What is the effectiveness of chemoprophylaxis to contacts of an IMD case in pre-school and school settings in preventing further cases? What is the effectiveness of chemoprophylaxis to those sharing the same transport vehicle as an IMD case in preventing further cases? Which antibiotic regimes are most effective in eradicating carriage among adults, children and pregnant women?
We opted for GRADE (Grading of Recommendations Assessment, Development, and Evaluation) methodology to assess evidence and produce guidance; we used the GRADE guidance available at the time of this study (2008-09) [
The consortium developed protocols, templates, and checklists for screening abstracts/papers retrieved in the literature searches to ensure a homogenous process across the research questions and across reviewers. The process was also reviewed against the criteria for guideline development as defined by the AGREE collaboration [
When defining the most suitable terms for the population, intervention, comparison, and outcome (PICO) to define our research questions for the search strategy (see examples in Table
Summary of findings and recommendations for three research questions.
Assessment of evidence quality | Assessment for recommendations | |||||||||
---|---|---|---|---|---|---|---|---|---|---|
Number of studies selected/number of studies reviewed | Design and quality | Inconsistency | Indirectness | Other modifying factors |
Grade of evidence | Benefits | Harm | Costs and burdens | Values and preferences | Grade of recommendation |
| ||||||||||
| ||||||||||
| ||||||||||
| ||||||||||
| ||||||||||
|
||||||||||
4/349 | All observational. No studies addressed the intervention | Different antibiotic regimens, times of swabbing, sampling, and study populations | Proxy outcome: prevalence of carriage following discharge from hospital | Small sample sizes. Study results are statistically homogenous | Very low | Potential reduction of the disease burden among close contacts of discharged cases | Low risk of treatment side effects | Very low cost and low burden for the patient (oral single dose) | Treatment widely accepted | Strong |
|
||||||||||
| ||||||||||
|
||||||||||
| ||||||||||
|
||||||||||
| ||||||||||
| ||||||||||
| ||||||||||
| ||||||||||
| ||||||||||
|
||||||||||
7/103 | Only reports on sporadic cases and 3 clusters linked to travel. No studies addressed the intervention | No consistency across case reports | Proxy outcome: risk of subsequent cases among fellow passengers whether prophylaxis was given or not | No studies clearly established evidence of transmission in transport vehicles | Very low | No evidence of reduction of subsequent cases among contacts sharing the same transport and taking prophylaxis | Low risk of treatment side effects but potential anxiety among those not receiving prophylaxis if targeted | Low cost of the intervention. However, contact tracing can lead to considerable cost and may not be feasible | Treatment likely to be accepted even if objective risk is low. Possible public pressure to give prophylaxis | Weak |
|
||||||||||
| ||||||||||
|
||||||||||
| ||||||||||
|
||||||||||
| ||||||||||
|
||||||||||
| ||||||||||
| ||||||||||
| ||||||||||
|
||||||||||
28 |
17 RCTs and 3 observational studies; no serious limitations | High consistency of results across studies | Proxy outcome: eradication of carriage. Only assessed in students/army for azithromycin and cefixime | High associations | High: rifampicin, ciprofloxacin, and ceftriaxone |
The 5 antibiotics are highly effective (eradication in 79–100%) | Limited harm of antibiotics and mild side effects. |
Low cost. |
High acceptability of intervention. A single oral dose is likely to be preferred | Strong for the 5 antibiotics. |
|
||||||||||
| ||||||||||
|
||||||||||
|
RCT: randomized clinical trial; IMD: invasive meningococcal disease.
We defined inclusion and exclusion criteria for selecting studies, applied to each research question. All European languages were included to avoid publication biases. As most of our studied interventions either were standard clinical practice or involved rare outcomes, clinical trials had not been conducted for ethical or logistical reasons. We thus did not limit inclusion to experimental studies but also included observational studies that involved comparison groups and case series with at least 10 cases.
We searched MEDLINE, Embase, Global Health, the Cochrane database of systematic reviews, and the Cochrane central register of controlled trials. The search terms for each question were agreed by at least two members of the consortium. Due to the short time frame, we applied some of the strategies previously used in rapid reviews by limiting the search to the period from January 1990 to the date of the literature search (December 2008) and giving priority to systematic reviews [
We examined reference lists of the selected papers for other relevant publications and searched Google Scholar for citations of identified key papers. For instance, for the research question on effectiveness of antibiotic regimens (Question E), we found a nonindexed but peer reviewed trial on antibiotic prophylaxis by using Google Scholar, though this study had not been retrieved by a previous Cochrane systematic review [
As studies measuring direct evidence on outcomes could not be found in four of the five research questions discussed in this paper, we defined and searched for indirect evidence on outcomes. For example, the relevant direct outcome for the question on the effectiveness of chemoprophylaxis before hospital discharge in preventing further cases among contacts (Question A) would be the incidence of subsequent cases in household contacts of the IMD patients who received antibiotics prior to discharge from hospital. A prior systematic review did not identify relevant studies measuring this outcome but showed that eradicating nasopharyngeal carriage in household contacts reduced the risk of further cases [
We also did not find direct evidence on the research question regarding whether chemoprophylaxis of contacts in school settings would prevent further cases (Question C). However, we obtained indirect evidence by comparing the risk of subsequent cases in school contacts (not receiving chemoprophylaxis) with the background incidence rates of IMD in the relevant population [
Even when the literature search provided direct evidence on the benefits of an intervention (e.g., effectiveness in preventing secondary cases), the evidence was often insufficient on its harm. In particular, direct evidence on the adverse events of antibiotics administered as chemoprophylaxis (Question E) was insufficient, but we found and reviewed indirect evidence on adverse events of these antibiotics when administered for indications other than chemoprophylaxis (e.g., ciprofloxacin used in cystic fibrosis).
We extracted and summarized the evidence on benefits and harm and prepared evidence profiles. When possible, we pooled estimates retrieved from selected studies. For instance, for Question B on chemoprophylaxis for household contacts, we extracted results from a recent study published after a systematic review and analysed these together with the three former studies from the review [
Estimate of effect of chemoprophylaxis to household contacts following a sporadic IMD case.
If the retrieved systematic reviews did not provide the level of detail needed to calculate pooled estimates of effectiveness or to fully answer the research questions, we extracted the necessary data from primary studies when appropriate. For instance, in the research question on antibiotic regimes for different subgroups (Question E), we identified Cochrane systematic review on antibiotics for preventing meningococcal infections, but this review did not present detailed analyses by antibiotic dosage and duration of therapy and did not stratify estimates by subgroups such as children and pregnant and lactating women [
Evidence was classified for all questions as either direct or indirect. We graded bodies of evidence according to GRADE guidelines and classified them as high, moderate, low, or very low, based on study design and quality, inconsistency, indirectness, imprecision, and strength of the association [
We found high or moderate quality evidence for only two of the five research questions, that is, regarding chemoprophylaxis to household contacts (Question B) and antibiotic regimes (Question E). Only low or very low quality of evidence was found for the remaining three research questions (Questions A, C, and D). Table
Because these factors varied across settings [
We found little information in the literature on burden of intervention perceived by patients and on their values and preferences. We searched for alternative data sources: for instance, information on perceived burden and values was requested from EU experts and national IMD representatives as well as from two meningitis patient associations. This confirmed that IMD is perceived as a severe disease that generates a high level of anxiety, and thus prevention measures are widely accepted, even if associated with some level of discomfort.
The consortium met face-to-face to develop recommendations according to GRADE, based on the quality of evidence and the balance between the benefits and harm, taking into account burden, values, preferences, and costs (see examples in Table
Recommendations were classified as strong or weak as recommended by GRADE [
The strong recommendation for which the quality of evidence on the benefit was very low was related to the research question on chemoprophylaxis of IMD cases before hospital discharge (Question A). Despite the very low quality evidence on the benefits, the consortium considered that harm, cost, burden, and values were strongly in favour of the intervention: the low cost of the intervention, the low number of patients not treated with an eradicating antibiotic regimen prior to discharge, and the potential benefit in reducing risk from a life-threatening disease were balanced against limited harm from antibiotics (Table
Consensus on the recommendations and grading of their strength was difficult to reach regarding chemoprophylaxis in day care settings (Question C). Here, the quality of evidence was low, and divergent recommendations were in place in the consortium members’ native countries. Thus each expert was probably influenced by his/her existing national policy. This highlighted that recommendations are built not only on rigorous scientific reviews but also on expert interpretation and judgment of the evidence. An advantage of the GRADE approach is to promote useful dialogue and ensure transparency by making these value judgments explicit [
We involved stakeholders and potential users of the guidance in the final steps. As our aim was to produce guidance that could be adapted to the needs of different EU countries, the draft document was circulated through EU experts and patient groups and reviewed by representatives of EU countries in the ECDC Advisory Forum. The feedback from ECDC and EU experts on the draft report allowed useful additions to the guidance [
We described how the guidance would potentially change current practice in EU countries. For instance, for the research question (E) on effectiveness of antibiotic regimens in IMD prophylaxis, we described what policy changes would be required and potential obstacles to the implementation of this guidance in a EU setting, based on whether the intervention or the specific drug is available and whether the recommended regimen differs from those currently recommended. In particular, some effective dosages did not correspond to recommendations and formulations available in EU countries and would require a change in current guidance. For instance, high quality evidence was available for the effectiveness of a single dose of 750 mg ciprofloxacin for the eradication of meningococcal carriage. However, in many countries, ciprofloxacin is recommended as a 500 mg single dose, although the effectiveness of this lower dosage has not been assessed in a controlled trial.
The guidance was successfully completed within six months and was approved and endorsed by ECDC in 2010 [
An advantage of GRADE process in developing public health recommendations is the integrated appraisal of related values, preferences, burden to the patient, and resource implications in addition to quality of evidence and the balance between benefits and harm. Based on GRADE 2004–08 guidance, we made strong recommendations for some areas in which the quality of evidence was low or very low. The long deliberations often required to arrive at final agreement of recommendations were facilitated by frequent communication, mainly by e-mail. It should be noted that GRADE work published later provides a systematic approach by describing circumstances in which a strong recommendation is warranted despite low or very low confidence in effect estimates, but these were not available at the time of developing our guidance [
The influence of national policies on the judgment of each consortium expert to formulate recommendations (described above) was dealt with by explicitly discussing each recommendation in the entire group. One advantage of having experts from five EU countries in the consortium was also that they had knowledge of current practices and health systems when considering implications for practice of the guidance.
The development of this guidance led to the identification of areas of uncertainty and research gaps, and we identified priorities for further research in each area. It was also a unique opportunity to progress towards common European health policy. Divergent health policies may cause confusion among the public and the media. The most objective argument for common health policies consists of a systematic and transparent search for and evaluation of available evidence. In 2013, we evaluated the impact of this guidance on the recommendations for public health management of IMD in European countries and found out that 90% of the 31 EU countries or regions found it useful at the national level and that 50% used it to update their national guidelines within the three years following the publication of the guidance [
WHO adopted a very similar process for developing evidence-based immunization recommendations, published after we initiated this work [
The consortium included mostly experts in epidemiology and microbiology. It could have benefited from including clinical experts and members of patient organization groups, but the short time frame was already challenging for finalizing the project.
The limited time (6 months) and available resources imply that our literature reviews could not meet the standards of a full systematic review. In addition, we could not cover all aspects of IMD public health management.
Indeed, the comprehensive application of the GRADE methodology including exhaustive systematic reviews may require substantial resources and more time is often required for rigorous development of guidelines [
We relied mostly on systematic reviews (including one Cochrane review) for the quality appraisal of individual studies for Questions B and E [
As explained above, one of the challenges was that we only found a low quality of evidence (according to GRADE) in most areas, as evidence from RCTs was only available on the effectiveness of antibiotic regimes in eradicating carriage.
The GRADE guidance required defining the burden of the intervention to the patient as well as patients’ values, preferences, and resource implications to aid in the development of recommendations. However, the GRADE guidance available at the time of developing these recommendations did not provide a methodology to collect and appraise the evidence in these areas. We found scarce information in the literature on the burden, values, and preferences surrounding interventions and limited data on cost in a few countries, and these may differ across countries. Although we questioned EU representatives and two UK-based meningococcal patient organizations, a representative survey of patients across Europe would be required for obtaining sound and representative evidence. However, we did not have the resources to initiate a multinational public survey on these issues. Furthermore, the GRADE guidance did not standardize how the data on burden and values should affect the recommendation; this is left to deliberation on the part of the decision-making group and has been criticized as a weakness of the GRADE process [
We developed evidence-based guidance on the public health management of meningococcal disease for EU countries in a short time frame and with limited resources. A number of recommendations in this guidance were based on a low quality of sometimes indirect evidence due to the impracticability of conducting clinical trials on interventions for outcomes that are rare or that have become standard practice. However, the recommendations were generated systematically and transparently, following GRADE and AGREE standards. This approach, that explicitly integrates additional criteria with the quality of evidence, can be usefully applied to the many areas of public health policy in which quality of evidence is often low or indirect. A recent survey of European countries showed that the majority found the guidance based on this process useful, about half had used the guidance to update their national recommendations, and a higher proportion of countries since 2013 compared to that in 2007 recommended evidence-based measures for IMD public health management [
The funder had no role in the design, collection, analysis, and interpretation of data, in the preparation of the paper, and in the decision to submit it for publication.
The authors declare that they have no conflict of interests regarding the publication of this paper.
James M. Stuart, Germaine Hanquet, Pawel Stefanoff, Wiebke Hellenbrand, and Sigrid Heuberger designed the study, performed the literature review, and developed the guidance. Germaine Hanquet, Pawel Stefanoff, and James M. Stuart wrote the paper with substantial contributions from Wiebke Hellenbrand, Sigrid Heuberger, and Pierluigi Lopalco. All authors approved the final version. Germaine Hanquet and Pawel Stefanoff contributed equally to this paper.
The authors wish to thank the European meningococcal experts who commented on the ECDC report and provided information. The European Centre for Disease Prevention and Control (ECDC) commissioned and partly funded the development of the guidance and covered the paper-processing charges.