How to Make Feedback More Effective? Qualitative Findings from Pilot Testing of an Audit and Feedback Report for Endoscopists

Background. Audit and feedback (A/F) reports are one of the few knowledge translation activities that can effect change in physician behavior. In this study, we pilot-tested an endoscopist A/F report to elicit opinions about the proposed report's usability, acceptability and usefulness, and implications for knowledge translation. Methods. Semi-structured qualitative interviews were conducted with eleven endoscopists in Ontario, Canada. We tested an A/F report template comprising 9 validated, accepted colonoscopy quality indicators populated with simulated data. Interview transcripts were coded using techniques such as constant comparison and themes were identified inductively over several team meetings. Results. Four interrelated themes were identified: (1) overall perceptions of the A/F report; (2) accountability and consequences for poor performance; (3) motivation to change/improve skills; and (4) training for performance enhancement and available resources. The A/F report was well received; however, participants cited some possible threats to the report's effectiveness including the perceived threat of loss of privileges or licensing and the potential for the data to be dismissed. Conclusions. Participants agreed that A/F has the potential to improve colonoscopy performance. However, in order to be effective in changing physician behavior, A/F must be thoughtfully implemented with attention to the potential concerns of its recipients.


Introduction
Audit and feedback (A/F) is widely used as a strategy to improve professional practice and is one of the more effective knowledge translation strategies that can effect a change in physician behavior [1]. A/F involves providing a physician with a summary of their clinical performance over a specified period of time with the goal of promoting improvement in practice [2]. It is intended to provide an objective measure of a physicians' performance compared to their peers and jurisdictional targets [3]. The ultimate goal is to prompt physicians to take action and modify their performance if their clinical practice is identified as suboptimal [2].
Improving physician performance can be challenging and may be influenced by the method of feedback and the characteristics of the targeted behavior [4,5]. A recent systematic review of 140 A/F trials [2] reported that multifaceted and complementary interventions such as the addition of reminders [6] and incentives [7] were more beneficial than A/F alone. Furthermore, the literature suggests that feedback may be more effective when baseline performance is low, the source is a supervisor or colleague, it is provided more than once, it is delivered in both verbal and written formats, and it includes both explicit goals and an action plan [2]. Changing complex physician behavior, such as a procedural skill, may present greater challenges than changing simpler behaviors such as prescribing or test-ordering [2]. Perhaps because of the complex nature of colonoscopy practice, there are relatively few trials of A/F reports for colonoscopy.
High quality colonoscopy is integral to the prevention and diagnosis of colorectal cancer (CRC). Globally, CRC is the 3rd leading cause of cancer-related death among men and women [8]. Unfortunately, colonoscopy quality is highly variable [9,10] and poor quality colonoscopy adversely affects patients as it is associated with increased morbidity and mortality. Variation in the quality of colonoscopy is likely multifactorial; however, variation in endoscopist performance is clearly an important contributor.
In prior research, we determined the most accurate definitions of key colonoscopy quality indicators from Ontario health administrative data. Cancer Care Ontario (CCO) recently updated its Guideline for Colonoscopy Quality Assurance in Ontario [11] which identifies evidence-based endoscopist quality indicators for colonoscopy. In the current study, we developed an endoscopist A/F report template to measure colonoscopy performance based on these quality indicators. The objective of this study was to pilot-test the endoscopist A/F report to elicit opinions about the proposed report's usability, acceptability, and potential usefulness for knowledge translation.

Participants and Setting.
We undertook a qualitative research study [12]. Participants were purposively sampled using a maximum variation strategy to ensure that they were at different stages in their careers, employed in a range of health care settings across Ontario, and diverse in terms of age and sex. A key recruitment resource was an expert panel on colonoscopy struck by CCO in order to advise on the development of a colonoscopy quality management program. These endoscopists were approached to participate because of their considerable expertise and leadership in this field.
Recruitment, development of interview materials, data collection, and analysis were collaboratively carried out by an interdisciplinary research team which consisted of a gastroenterologist with an academic affiliation (J. T.), a medical sociologist (F. W.), a medical anthropologist (K. R.), and a research coordinator (J. P.). The study was approved by the Research Ethics Board at Sunnybrook Health Sciences Centre in Toronto, Ontario.

Audit and Feedback
Report. An endoscopist A/F report template was developed that incorporated nine colonoscopy quality indicators: annual colonoscopy volume, polypectomy rate (by sex), cecal intubation rate (by sex), polypectomyassociated bleeding rate, number of perforations, CRC detection rate, poor bowel preparation rate, postcolonoscopy CRC rate, and early repeat colonoscopy rate. CCO's Guideline for Colonoscopy Quality Assurance [11] was the primary resource for these indicators. An information sheet with definitions and methodology used to derive the indicators was included as supplementary material and was provided to participants during the interview. For the purposes of the current study, one sample report was populated with simulated data; however, participants were informed that these reports could be generated centrally using health administrative data (i.e., populated with real data) in the future for all endoscopists in the province of Ontario. All participants provided feedback on the same sample report.

Data Collection.
Data were collected through semistructured, one-on-one interviews that took place either in person or by telephone (with a web conferencing component) between April and July 2013. All participants signed an informed consent form prior to the interviews and received an honorarium for their time.
An interview guide was developed and pilot-tested prior to commencing the study. All interviews were approximately one hour in length and were led by a moderator experienced in qualitative research (K. R.). Feedback was obtained on the content and usability of the A/F report, preliminary impressions of the acceptability of the report, and how recipients might translate knowledge from the report into improving their endoscopy practice. Sample size was determined by saturation, when the team collaboratively decided that no new information or themes were being generated, at which point participant recruitment and interviewing stopped [13].

Data Analysis.
Interviews were audio-recorded, transcribed verbatim, and analyzed using standard qualitative strategies including those from constructivist grounded theory approaches. Constructivist grounded theory refers to an approach to qualitative research that recognizes the researcher's role in the co-construction of meaning during interviews and in the multiplicity of meanings that events may have for participants [14]. Therefore, reflexivity is an important part of the research process, from design to analysis. Transcripts were imported into a qualitative software program (NVivo 10, QSR International Pty Ltd.). The constant comparison method was utilized; that is, each new transcript was compared with previous interview data. Using this approach, the verbatim text was coded line by line by two qualitative analysts (K. R. and J. P.). The analysts independently coded the first two transcripts, met to compare their interpretations, and developed a coding framework. The remaining transcripts were divided and each analyst coded half. The analysts categorically merged the codes; codes were then organized into categories and themes were constructed. Several team meetings were held over the duration of the data analysis phase, to discuss the implications and possible interpretations of the data. The regular meetings provided an opportunity for reflexive sharing and ensured rigor. Research validity was enhanced as the analytic approach incorporated the different disciplinary perspectives of the research team.

Results
In total, 11 endoscopists in Ontario were interviewed. Three (27%) were women, 82% had been in practice for over 15 years, 64% were gastroenterologists, 36% were general surgeons, and 55% had community-based practices. Seven (64%) participants were members of the Ontario colonoscopy expert panel struck by CCO (Table 1). Our original questions probed the concepts of usability, acceptability, and usefulness but our interpretation of the data suggests that other factors may be important in terms of how the A/F report might be used. Four main themes emerged from the interviews: (1) overall perceptions of the A/F report; (2) accountability and consequences for poor performance; (3) motivation to change or improve skills; and (4)  Despite the consensus around the importance of receiving feedback, some participants raised concerns that lowperforming endoscopists might challenge the data's accuracy.
. . .So one of the reactions if somebody tells you your measurements are off is you usually say, "Well, look, the measurement tool's not very good, is it, really?" (Gastroenterologist, Interviewee #11) Some participants also expressed their preferences for data presentation that is meaningful and relevant. In particular, peer comparison to those with a similar practice and patient mix was emphasized.
I' d like to compare myself to other people with a similar practice, not the people who see 90% screening colonoscopies in outpatient clinic. . .So some bit of demographics on the physicians I'm being compared to, because I would expect that I might get more complications or I might get less cecal intubation rates or something. . .. (Gastroenterologist, Interviewee #9) Accountability and Consequences for Poor Performance. There was considerable disagreement as to how the A/F reports should be used and to what extent the information should be made public. Opinions on the extent of disclosure ranged from keeping the data highly confidential to making it fully public, seemingly reflecting a tension between the threat to the individual endoscopist's professional standing and transparency. Participants endorsing the former opinion were concerned about the impact the report may have on endoscopists' hospital privileges and licensing and suggested that the report should at least initially be confidential with the goal of enhancing performance only. Participants began to differentiate between individuals having information to assess their own performance "compared to peers" versus data that could be used by administrators or regulators, with the attendant possible threats, such as loss of privileges. Not all participants were apprehensive about sharing information with the public despite the potential negative impact described above. It was suggested that individual endoscopists were accountable for their own clinical performance. These participants focused on the possible consequences of poor quality colonoscopy for patients who they felt should be "more informed." Motivation to Change or Improve Skills. Participants were asked if they believed most endoscopists would be motivated to change/improve their skills if they received a poor report. Responses varied and this variation seemed to be linked to participants' perceptions of how self-motivated clinicians were. Participants were mainly optimistic that the motivation to change would be high because they perceived that, as a group, endoscopists tend to be high performing and selfmotivated. They hypothesized that they would be positively motivated to change/improve their skills. Receiving a low score would therefore induce a desire to improve.
. . .Now, remember, most people that are in surgery and medicine. . .they're used to be considered highperforming individuals, right. . .So, you know, most of these folks kind of went through school and they were always at the top tier, they don't want to be at the bottom tier. . .. (Gastroenterologist, Interviewee #1) Given the perception that there is a natural tendency for endoscopists to be high performers, many believed that the process of improving skills should be voluntary rather than mandatory and that at least initially performance change should rely on endoscopists' personal motivation to improve. It was suggested that a certain subgroup of endoscopists, consisting of older and end-of-career endoscopists, might be the most challenging group with respect to improving their skills. Because of the stage in their career, several participants assumed that this subgroup would be less motivated to improve.
You know, the problem is it's going to end up being, for the most part, a certain group of endoscopists in a certain demographic -they're going to be surgically trained, and they're going to be older. . .and often retired from hospital practice. . .That is going to be the biggest group that's going to have the worst numbers, and to date there's not been a way to deal with those people. (Gastroenterologist, Interviewee #8)

Training for Performance Enhancement and Available
Resources. Several participants identified a gap in terms of the resources currently available for additional training/skill enhancement for those identified as low performing. Most participants agreed that they were not aware of existing educational opportunities and would not know where to begin if they wanted to improve their colonoscopy performance.

Yeah, that's a good question, because I think the average [endoscopist] ha[s] no idea where to go.
(Gastroenterologist, Interviewee #2) Some felt that it was therefore imperative to provide resources to assist with improvement along with the A/F report. They suggested that information about interventions for skills enhancement could be provided with the report in order to facilitate the process of translating knowledge from the report into improving practice. Participants underscored the importance of having networks and colleagues with whom one could share poor scores and suggested that access to networks and/or training might vary by jurisdiction. One participant noted that being "recently out of training" enabled them to still access those who trained them in seeking help, suggesting that trainees or recent graduates would find it easier to seek and to get help.

Discussion
The A/F report was well received and the participants in our study confirmed that A/F has the potential to be an effective quality improvement strategy [4]. However, participants also cited some possible threats to the report's effectiveness including the perceived threat of loss of privileges or licensing, the potential for the data to be perceived as inaccurate, and the general lack of awareness of available resources for skill enhancement.
Participants suggested that underperforming physicians would likely challenge the data, either by rejecting the data or by suggesting that their practice differs importantly from their peers, rendering the reports' comparisons unfair and meaningless. This finding highlights one of the challenges of providing feedback to physicians-the concept of cognitive dissonance [15], which refers to the situation where an individual struggles with holding two conflicting beliefs simultaneously. Using the A/F reports as an example, for some physicians, acceptance of the data in the report would mean acknowledging that they are underperforming. However, this acceptance would likely conflict with the physician's self-perception that they are capable and competent practitioners. In order to resolve the conflict, physicians will likely find it easier to dismiss the data than question their own competency. A growing body of evidence suggests physicians are generally poor at self-assessment and tend to overestimate rather than underestimate their own performance [16][17][18]. Physicians' poor self-assessment skills may further push them to resolve cognitive dissonance by dismissing the data. It has been suggested that if coaching accompanies the feedback of "objective" data such as that found in our report, it may help the recipient interpret the data and set goals while maintaining their sense of self as competent practitioners [15].
An important consequence of dismissing the data is that the recipient's attention will be shifted away from the "task," in this case colonoscopy performance. According to Kluger and DeNisi's feedback intervention theory [19], a key characteristic of effective feedback is that it directs the recipient's attention towards the task at hand; conversely, a feedback tool that shifts the recipient's attention away from the task will be less effective. The variation of the effectiveness of A/F reports reported in the literature [2] may be partially explained by variation in this characteristic. Characteristics of feedback that direct the attention towards the task which have been shown to be associated with more effective feedback include individualized, frequent, and nonpunitive feedback [5,19]. In addition, provision of correct solution information (feedback that helps recipients understand what needs to change in order to improve performance) [20] and goal setting [2,21] are also associated with effectiveness.
Participants had a clear idea of who would likely be underperformers: older, surgically trained physicians who perform a low volume of colonoscopies relative to their younger and/or more specialized peers. They also implied that the solutions for low-performing physicians could differ depending on their career stage. For example, further training could be provided to young, underperforming physicians, while older physicians could be offered incentives to retire.
However, it should be noted that our findings pertain to endoscopists' perceptions of poor performers but do not necessarily indicate that older, surgically trained physicians are indeed underperforming relative to their younger peers. It has been reported that one professional group will tend to see another in narrow terms and often attribute tensions and even conflicts about goals and values to these perceived differences [22]. Many of the participants in our study made statements that suggested they imagined that poor performers as being "other" in some ways to themselves.
Our study had some limitations. The participants were experts in their field and were identified through the professional networks of the investigators. Most were well established in their careers, considered opinion leaders, with years of professional experience. Physicians from rural and remote jurisdictions were underrepresented among the interviewed cohort. Therefore, the views of our participants may not be broadly representative of all physicians who perform colonoscopies in Ontario. Additionally, the reactions that we elicited were based on a report populated with simulated data; therefore, the responses/quotes elicited are hypothetical and are removed from the participants' personal experience and professional practice. Reports populated with real data may elicit different reactions. In the future, we plan on eliciting reactions to the endoscopist A/F report from "everyday endoscopists" in Ontario using real data.

Conclusions
Findings from our study highlight the tension between the exceptional autonomy that physicians have historically had in their practice and more recent concerns with public accountability in the health care system [23]. This is an ongoing tension that has important implications for both policy and practice. Future research should explore from both provider and patient perspectives how discourses and practices of accountability are taken up across various settings. In our study there was agreement that the use of A/F has the potential to improve colonoscopy performance. However, our findings indicate that this intervention must be implemented thoughtfully such as by addressing cognitive dissonance, directing attention towards the task at hand, and ensuring access to resources to help improve colonoscopy performance.