Gastroenterology fellowship training : Approaches to curriculum assessment and evaluation

University of Calgary, Calgary, Alberta Correspondence: Dr Maitreyi Raman, University of Calgary, 3330 Hospital Drive, Calgary, Alberta T2N 4N1. Telephone 403-220-5767, fax 403-210-9345, e-mail mkothand@ucalgary.ca Received for publication October 30, 2007. Accepted March 25, 2008 M Raman, E Shaffer, J Lockyear. Gastroenterology fellowship training: Approaches to curriculum assessment and evaluation. Can J Gastroenterol 2008;22(6):559-564.

M edical education requires ongoing curriculum develop- ment and evaluation to incorporate new knowledge and competencies (1).Despite demands for curricular change, many faculty members in academic medical centres do not possess the necessary skills and background in formal educational instruction or curriculum development (2).Consequently, curricula may be poorly designed and function inefficiently for both learner and teacher.(3) There are two approaches to developing and assessing curriculum viability -the Kern model of curricular development (4) and the approach advocated by the Royal College of Physicians and Surgeons of Canada (RCPSC) through its accreditation structures (5).

KERN MODEL OF CURRICULAR DEVELOPMENT
The Kern model ( 4) is a generic model for curricular development designed specifically for use in medical education.The Kern model shows robustness and has been used in other assessments of medical curricula (6)(7)(8).It proposes a six-step guide to curriculum design.The steps necessary to ensure a successful curriculum include: • Problem identification and general needs assessment.Problem identification refers to the identification and critical analysis of a health care or educational need.The problems identified via the Kern model may range from individual learner problems with the stated goals and objectives of the curriculum existing in the current state to institutional or societal problems on a global scale.Therefore, regardless of the scope of problems identified via this method, potential solutions may be considered at an individual or global level.For the purposes of the present study, we will focus on individual learner-related problems.
Complete problem identification requires an analysis of the current approach of existing practitioners, followed by the identification of the ideal approach reflecting how health care providers should be addressing the need.
• Needs assessment of targeted learners.This involves assessing the needs of one's targeted group of learners, which may differ from the needs of a larger learner group in general.
• Goals and specific measurable objectives.Goals and objectives are paramount to any curriculum.Goals are generally broad and nonspecific, in contrast to objectives, which are specific and measurable.Objectives may include cognitive, affective or psychomotor skills for the learner.Objectives are crucial because they help shape curricular content and help focus the learner.
• Educational strategies.Educational strategies such as small or large group lectures, case-based interactive sessions and bedside teaching, among other techniques, should reflect the predetermined goals and objectives.
• Implementation.Implementation of a curriculum involves gathering political support and resources, identifying possible barriers to implementation, and constructing an appropriate administrative structure that is conducive to a successful curriculum.
• Evaluation and feedback of the student and program.Evaluation and feedback is essential for performance assessment of both individuals and a curriculum.Evaluation can be formative such that learners have the opportunity to improve based on suggestions from content experts, and summative, to provide a final assessment.Curriculum development does not proceed in sequence, one step at a time, but in fact, is a dynamic, interactive process, whereby progress on one step will affect and require adjustments to other steps (Figure 1).

THE RCPSC ACCREDITATION MODEL
The RCPSC has a model for curricular design, similar to the Kern model, but adapted to Canadian residency training programs.The RCPSC model is described within its "General Standards for Accreditation".This model includes six steps.
• Establishing an administrative structure for the program.This standard requires that each program has a director and a committee structure, and attends to curriculum development and monitoring, the supervision and evaluation of trainees, teacher evaluation, and the establishment of structures and support for the evaluation of the program, resident appeals and support for trainees.
• Goals and objectives.The RCPSC requires that there be overall goals for the program and objectives that reflect the Canadian Medical Education Directives for Specialists (CanMEDS) competencies.Each rotation is required to have rotation-specific objectives based on CanMEDS criteria that are precirculated to the trainees and regularly updated.
• Structure and organization of the program.Accredited programs must be organized so that rotations and other educational experiences enable the trainee to achieve competence in the specialty.
• Resources.The program must have sufficient resources including teaching faculty, patients, physical and technical resources, and supporting facilities to provide an opportunity for all trainees to achieve the objectives for the program.
• Clinical, academic and scholarly content of the program.The clinical, academic and scholarly content of the program must be appropriate, so that the resident can fulfil the CanMEDS competencies.
• Evaluation of resident performance.The evaluation must be based on the goals and objectives of the program and identify the methods to be used for evaluation and the level of performance expected of residents in the achievement of the objectives.The approach required for all residency programs is complementary to the generic Kern model and ensures that all Canadian residency programs meet the standards for accreditation.The Kern model, however, differs because it emphasizes and includes problem identification and the assessment of learner needs in its The purpose of the present study was to apply the Kern model to an assessment of a residency program in Gastroenterology.We describe our approach as a case study.A case study is a method of qualitative research that is designed to help researchers understand people and the societal contexts within which they live (9).Case studies emphasize detailed contextual analysis of a limited number of events or conditions, and their relationships.It is an empirical inquiry that investigates a contemporary phenomenon within its reallife context when the boundaries between phenomena and context are not clearly evident (10).

THE MODEL STUDIED: UNIVERSITY OF CALGARY GASTROENTEROLOGY FELLOWSHIP PROGRAM
The University of Calgary (Calgary, Alberta) has a gastroenterology (GI) residency training program that is accredited by the RCPSC.There are a total of 14 adult GI residency training programs in Canada, with two programs in Alberta.The University of Calgary GI residency consists of a mandatory two years of clinical training.Further training may include clinical or research fellowships.There are currently 10 GI fellows with five in each year, although the number of fellows each year varies.All GI fellows have completed their core three years of internal medicine residency in Canada, although not necessarily at the University of Calgary.After the completion of the first two years of fellowship, the GI fellow is expected to take the RCPSC GI certification examinations.Some fellows may subsequently opt for additional clinical or research training.

INTERNAL REVIEW PROCESS
The internal review is an integral component of the RCPSC accreditation process and is generally conducted two years before the regular RCPSC site survey.The purpose of the internal review is to assist the university in maintaining the quality of its residency programs and to provide the postgraduate medical education committee with valuable information regarding the strengths and weaknesses of their program.The internal review then enables each residency program to take corrective measures regarding any weaknesses identified before the next RCPSC survey.Each university is responsible for conducting an internal review of all residency training programs.At the University of Calgary, the internal review occurs every three years.The internal review team consists of two program directors, or ex-program directors of other residency programs, along with a resident member -none of whom have any involvement in the program under scrutiny.They are responsible for meeting with the GI program director, the division head, all resident members and, finally, the resident training committee.The review team has access to all documentation regarding the program, including the RCPSC presurvey questionnaire.

Needs assessment
Kern advocates that needs assessments be conducted for individuals who are actually in the program.Such needs assessments help to focus the learning objectives and learning strategies for the learners.GI fellows inevitably have had different experiences and exposure to GI rotations before entering their fellowship.Some fellows may have had prior advanced degree training such as an MSc or PhD.Others may be considering future advanced degrees upon completion of their GI fellowship.A subset of fellows may have definite areas of subspecialty interest upon completing their core GI training.
There may be variations in terms of those wishing to have an academic versus a community-based career.Hence, marked variability likely exists between GI fellows, and all are not equal.Hence, an intake questionnaire systematically assessing these differences and surveying the needs of these fellows would serve to better tailor specific rotations in advance.Additionally, a follow-up questionnaire upon completion of the first year of core training re-evaluating some of these issues may highlight deficiencies noted after one year of training, and serve to correct these perceived deficiencies before taking the RCPSC GI examinations.If career goals have subsequently changed in this one year period of training, rotations might be modified to reflect this.Hence, this process can be iterative.A targeted needs assessment of the GI fellows entering the University of Calgary GI training program is not routinely performed.

Objectives
The RCPSC states that there should be a clearly worded statement outlining the goals of the residency program and the educational objectives for the residents (5).All residents should receive a copy of the goals and objectives at the beginning of the program, as should all faculty members involved in supervising and educating residents.These goals and objectives should reflect the structure of the training program, and ultimately guide evaluation.Such objectives should represent the 'ends' toward which learning efforts should be directed.Not only should objectives be specific and measurable, they should relate to learning in the cognitive, affective and psychomotor domains.At the time of the internal review, the GI fellowship program at the University of Calgary had a set of 'pseudoobjectives' that provided a list of broad topics designated for discussion purposes at the mandatory academic one-half day or weekly morning GI rounds.These objectives did not provide terminal expectations nor measurable outcomes.They did not focus on the CanMEDS roles, another expectation of RCPSC specialty programs.Similarly, rotation-specific objectives had not been written.

Educational strategies
The Kern model recommends using diverse approaches to facilitate learning.These approaches, referred to as educational strategies, are the methods that will allow learners to achieve their curricular objectives.Educational strategies involve both content and methods.Content refers to the specific material to be included in the curriculum, while methods refer to the ways in which the content is presented.Similarly, the RCPSC internal review curricular model has a section termed clinical, academic and scholarly content of the program.This model states that the academic and scholarly content of the program must be appropriate for university postgraduate education and adequately prepare residents to fulfil all the roles of the specialist.The quality of the scholarship in the program will in part be demonstrated by a spirit of enquiry during clinical discussions, at the bedside and in clinics, in addition to seminars, rounds and conferences.
The formal educational program also includes a mandatory academic one-half day consisting of formal presentations and independent study during the academic one-half day.There is a weekly pathophysiology course, weekly GI rounds, a biweekly journal club and monthly pathology rounds.The GI division sponsors four annual weekend retreats for the GI fellows, which focus on a variety of topics related to gastroenterology.Residents attend annual national and international GI meetings including the annual GI fellows' course that is appended to the annual Canadian Digestive Diseases Week.
In addition to the formal academic events outlined above, the GI fellows receive a complete clinical experience encompassing inpatient consultation service, outpatient clinics, diagnostic and therapeutic endoscopy, hepatology, inflammatory bowel disease and electives.Each resident is assigned his or her own 'Fellows Clinic' that is designed to provide an opportunity for more longitudinal and independent patient care.Additionally, the fellows are encouraged to develop a research project in the first year of their training and execute this project in their second year.
The GI curriculum for fellows uses a mix of educational strategies ranging from small group learning sessions for fellows alone, to large group divisional lectures occurring on a regular basis.These sessions are both resident-driven and facultydriven.Didactic sessions are of benefit because they impart a large volume of information in a relatively short period of time.The residents are required to present weekly didactic GI rounds, in addition to weekly pathophysiology sessions.

Implementation
Kern espouses that for a curriculum to achieve its potential, careful attention must be paid to issues of implementation.This includes ensuring that sufficient resources, political and financial support, and administrative strategies have been developed to successfully implement the curriculum.Here, the RCPSC is in accord: there must be sufficient resources including teaching faculty, number and variety of patients, and physical and technical resources, as well as the supporting facilities and services necessary to provide the opportunity for all residents in the program to achieve the educational objectives and receive full training.
During the recent internal review of the University of Calgary GI program there were 46 faculty members in pediatric and adult GI/hepatology, in addition to colorectal surgery, among three teaching sites that had a major role to play in educating GI fellows.Approximately 60% of the faculty had been practising in GI for less than 15 years, while 20% of the faculty had 15 to 25 years of clinical experience, and 20% had more than 25 years of clinical experience.There is a broad range of consultations in adequate volume in patients with gastrointestinal illness at all three teaching sites.The attending staff assumes part of this load when the consultation volume is high.The more interesting cases, in which the resident has less experience, are directed to the trainee to facilitate their education of rare problems, or uncommon presentations of common problems.Similarly, the volume of endoscopic diagnostic and therapeutic procedures is sufficiently high, allowing for the development of adequate procedural skills.The service at all sites does not have assigned beds or a ward.It is purely a consulting service, with the exception being chronic inflammatory bowel disease or chronic liver diseases.These generally account for no more than three to five patients at any given time.The typical assignment on the GI service at the Foothills Hospital (Calgary) is a two-team approach, in which each team is assigned one faculty preceptor and GI fellows attending for a two-week block of time rotate with a second complete team for the same duration.The GI fellow completes a minimum of four consecutive weeks on the consultative service.The GI fellow supervises, teaches and advises rotating residents or clerks.
There is a rich source of patient material in the ambulatory clinics at all three teaching sites.The resident is the first contact for such consultations.Similarly, consultations also come from the intensive care unit, emergency room departments, surgical services and subspecialists of internal medicine.The most common source of outpatient consultations are community family physicians.
The resources available for resident training in ambulatory care are broad.They include fully equipped offices that are well supported by nursing and clerical staff.In general, the rooms are of sufficient size that the fellow, preceptor and one to two other learners can review findings on each patient.There is a private area such that the trainee and preceptor can review the case, and there are adequate facilities for dictation.

Evaluation and feedback
Kern recommends that an evaluation process should include both evaluation of the trainees and evaluation of the program or curriculum.Regarding learner evaluation, the RCPSC guidelines suggest that there must be mechanisms in place to ensure the systematic collection and interpretation of evaluation data on each resident enrolled in the program.Concurrent with evaluation are guidelines regarding feedback.The RCPSC guidelines state that there must be honest, helpful and timely feedback provided to the resident, and that these sessions must occur regularly, at least at the end of every rotation (5).
The internal review identified that evaluations were not consistently performed according to RCPSC guidelines.They did not always receive formative or summative evaluations in a timely manner, although the GI program director does have six-month reviews with each trainee.These reviews occur more frequently when necessary.These evaluations provide a collated collective summary of the comments previous preceptors have documented.
The internal review process -which, for University of Calgary residency programs, takes place every three yearsdoes provide a robust structure for program evaluation and ensures that all programs are functioning in consistent ways.

DISCUSSION
The Kern model for curricular development adds two important domains that are not covered within the RCPSC structure: the identification of individual problems to be met by the program and the needs of learners in the program.Learner problem identification may uncover such issues as gaps between learner career goals and structured rotations, as well as perceived specific weaknesses yet program obligations that need to be met.Needs assessments conducted among individual trainees can provide information that will critically guide a trainee's program in the early stages.Trainee's past experiences and future career goals are subject to variability.Potentially, a curriculum that evaluates learner needs at the start of fellowship training and periodically thereafter, may serve to enrich the learning experience.Considering learner needs may serve to develop objectives, select rotations, identify a research mentor and ultimately guide program evaluation.
Both models include the writing of objectives, learning strategies, implementation and evaluation.Both the internal review and an examination of the curriculum vis-a-vis the Kern model would have identified that the objectives were suboptimal.Hence, both models overlap and are complementary.
As a result of the review, a new set of educational objectives have been developed for the program using the CanMEDS criteria.These objectives were presented to the University of Calgary graduate education committee (GEC) for review.Suggestions regarding modifications, revisions and additions were provided by the GEC, resulting in a second draft of objectives that subsequently received approval from the GEC.Work is now in progress to develop rotation-specific objectives for trainees of all levels.These will be written by local content experts in the specific subspecialized area of training using the CanMEDS criteria.
As the RCPSC presurvey showed, the current educational strategies used are multiple and thorough.However, based on the Kern model, the program has adopted a number of other changes.These include the use of interactive case-based discussions to replace some of the didactic lectures.Indeed, the RCPSC certifying GI examinations require that GI fellows use the higher orders of Bloom's Taxonomy, particularly application, analysis and synthesis.These will be structured so that a fellow will be required to discuss a case with a faculty member, with importance given to development of consultancy skills and content expertise, facilitating critical thinking.
Similarly, the program has addressed the challenges surrounding incomplete evaluations and inconsistent feedback on the fellows' performance.We have implemented an electronic Web-based evaluation process.Under the direction of the GI program director and division chief, all faculty have been instructed to provide interim two-week evaluations, in addition to a summative evaluation after four weeks of each rotation.Identified areas of weakness are highlighted at the two-week point in a formative fashion, with an adequate period of time provided for the fellow to institute the changes recommended.The formative evaluation and feedback process is not formally documented, but is in the form of an oral communication.Evaluations now occur monthly and are discussed with each GI fellow, with suggestions for improvement where necessary before submission.If preceptors are unable to provide feedback at the designated time, they must make an appointment with the resident within a one-week period of rotation completion to discuss the evaluation and provide feedback.We have appointed a site director at all sites used by the training program.The site director will be responsible for ensuring that all GI fellows receive evaluations following each completed rotation, and that these evaluation reports are fed back constructively to the fellows.This will ensure that evaluations are communicated in a timely fashion.
Although the Kern model has advantages to curriculum design in considering problem identification and learner needs, it is not as specific as the RCPSC standards and what the presurvey questionnaire expect for the development of a GI or subspecialty curriculum.It is, after all, a generic model that can apply to any curriculum.The RCPSC general standards provide expectations for specialty education through the material required for the presurvey, which we use as a template for the internal reviews.

CONCLUSIONS
The RCPSC presurvey and the Kern model for curricular development are complementary and address the pertinent issues necessary to developing a curriculum.Most of the issues outlined would have been identified using either model.The Kern model of curricular development allows for problem identification at a national, provincial and individual level.GI fellowship programs may modify their respective curriculum to address problems identified in a certain geographic area.Additionally, learner identification of curricular problems provides an objective impetus for instituting change to enrich the learning experience of the trainee.Additionally, the needs assessment of the learners at the beginning of the GI fellowship program, using the Kern model, allows for consideration of learner background and future career goals when designing specific rotations and educational strategies in providing the best possible learning experiences for the fellows.Consideration by the RCPSC should be provided to add the missing elements of curricular design, namely problem identification and needs assessment of the learners to their accreditation structure in providing a complete model for curriculum development.

Figure 1 )
Figure 1) Kern's six-step guide to curriculum design.Reprinted with permission from the Johns Hopkins University Press (4) questionnaire, the results of the internal review, minutes of meetings and other documents in the GI program, and discussions with key individuals in the program.We identify what the Kern model would add to an examination of curriculum, the limitations of the Kern model for RCPSC residency programs, and suggest how the Kern model can enhance the RCPSC structure.Although having a basic skill set in research methods is desirable, it may not be essential to this individual's career path that he or she spend months of dedicated time to fill this objective.Conversely, an individual that has career aspirations in the field of basic laboratory research may have program requirements of rotating through months of community gastroenterology that may not be in line with his or her long-term career goals.Therefore, a potential problem may exist with the curriculum in the current format.Conducting a tailored individual needs assessment may allow for a more streamlined approach to satisfy all involved, yet meet accreditation standards.
We now apply the Kern model for curricular design to the GI fellowship program at the University of Calgary.In examining the GI program vis-a-vis the Kern model, we have drawn upon the RCPSC presurvey