Introduction. Web-based learning (WBL) is increasingly used in medical education; however, residency training programs often lack guidance on its implementation. We describe how the use of feasibility studies can guide the use of WBL in anesthesia residency training. Methods. Two case-based WBL emergency airway management modules were developed for self-directed use by anesthesia residents. The feasibility of using this educational modality was assessed using a single cohort pretest/posttest design. Outcome measures included user recruitment and retention rate, perceptions of educational value, and knowledge improvement. The differences between pre- and postmodule test scores and survey Likert scores were analysed using the paired t test. Results. Recruitment and retention rates were 90% and 65%, respectively. User-friendliness of the modules was rated highly. There was a significant improvement in perceptions of the value of WBL in the postsurvey. There was a significant knowledge improvement of 29% in the postmodule test. Conclusions. Feasibility studies can help guide appropriate use of WBL in curricula. While our study supported the potential feasibility of emergency airway management modules for training, collaboration with other anesthesia residency programs may enable more efficient development, implementation, and evaluation of this resource-intensive modality in anesthesia education and practice.
1. Introduction
Web-based learning (WBL) can be defined as the “usage of computers and networks in education,” including learning management systems, online tutorials, discussion forums, and simulation [1, 2]. WBL has become increasingly used in medical education; however, it is important to understand how to best design and implement it as an educational modality [3–20].
Difficult and emergency airway management is an essential skill to acquire during anesthesia residency training. WBL may complement preexisting curricula and address gaps in clinical training in this area. Advantages of WBL include flexibility and access, enhancement of other educational modalities, ease for content updating, and appeal for the current “millennial learners” [21, 22].
While learning theories can guide the pedagogical use of WBL, major barriers to implementation in residency include time, cost, and technical expertise requirements as well as inadequate learner and faculty uptake [20, 21]. Therefore, WBL is effective only if successfully integrated in curricula and consistently used by learners [23]. Given the comparable educational outcomes of WBL and traditional methods [18, 19, 21], it is crucial to assess the feasibility [24] of WBL implementation in order to balance expense and educational benefits, especially for departments with limited resources.
The WBL literature provides little guidance on how to assess the feasibility of implementing a new teaching modality. Therefore, the purpose of this paper is to describe how we developed WBL modules for emergency airway training and explored their feasibility for implementation into our anesthesia residency curriculum at McMaster University, Canada. Feasibility is assessed by examining the recruitment and retention rate, user perceptions, and knowledge improvement. We hope our experience will inform other training programs that are considering incorporating WBL into their curricula.
2. Materials and Methods2.1. Module Design and Development
We conducted a literature review and consulted an educational technology instructional designer in order to develop the modules according to principles of effective instructional web design [21–23]. Based on active learning theories, aspects of WBL design that improve its educational efficacy include interactivity, practice exercises, feedback, and repetition [2, 18, 21, 23, 25]. Embedded case-based, self-assessment questions have also been shown to improve learning outcomes [26].
We selected two common emergency airway topics (burn and facial trauma) and developed them into two separate case-based airway modules for self-directed online use. Background information was provided in each patient scenario along with relevant images and diagrams. Embedded multiple choice questions allowed the participant to interactively make clinical management decisions throughout the modules. After selecting an answer to a question, the user received immediate feedback with a detailed explanation.
We developed the study objectives and content based on information from the Airway Evaluation and Management Guidelines of the National Curriculum for Canadian Anesthesia Residency [27], anesthesia textbooks [28–30], and expert opinion papers [31–35]. The content of the modules was peer-reviewed by several faculty anesthesiologists and revised for clarity and appropriateness.
The study instruments included pre- and postmodule survey and knowledge test. The pre- and postmodule surveys (supplemental content in Supplementary Material available online at http://dx.doi.org/10.1155/2015/971406) assessed the participants’ perceptions of the value and user-friendliness of WBL and other educational modalities. The pre- and postmodule knowledge tests assessed the module content. The modules, surveys, and tests together formed the Training for Emergency Airway Management (TEAM) course, which was launched on Avenue to Learn (a computer-based McMaster learning management system).
After logging into Avenue to Learn, residents were directed to complete TEAM in the following order: premodule knowledge test and survey, the two airway modules, and finally the postmodule knowledge test and survey. Residents had five weeks to complete TEAM. Further details of how the modules and study instruments were created are outlined in the supplemental content. The total time for the development, design, testing, and launch of TEAM was 165 hours.
2.2. Feasibility Study Procedures
This study was approved by the McMaster University Research Ethics Board. We used a single cohort pretest/posttest design where participants served as their own control [36]. Because TEAM was developed for junior residents, only the current year’s cohort of anesthesia residents in postgraduate years (PGY) 1–4 (total study population = 29) were invited to participate.
2.3. Data Collection
Data from the participant responses on Avenue to Learn were exported to Excel data files, deidentified, and assigned individual codes by our Research Coordinator before analysis by the investigators.
2.4. Outcome Measures
The feasibility of incorporating WBL into the curriculum was assessed by the recruitment rate (% of participants/total study population) and retention rate (% of participants who successfully completed entire TEAM/total participants), user perceptions (reported as mean Likert scores), and knowledge improvement (measured by comparing pre- and postmodule knowledge test scores).
This WBL program would be considered feasible [24] if the following predefined criteria were met: (1) “definitely feasible” if recruitment rate and retention rate were at least 80%, (2) “possibly feasible” if recruitment rate and retention rate were between 60 and 79%, (3) mean user perception ratings of at least 5 on a 7-point Likert scale, (4) increased postmodule Likert scores of perceptions of the educational value of WBL, and (5) significant knowledge improvement between the pre-and postmodule test scores.
2.5. Statistical Analysis
Pre- and postmodule test scores and survey Likert scores were summarized as mean and standard deviation (SD). The differences between post- and premodule test scores and survey Likert scores were assessed using two-sided paired t tests. The results were reported as mean differences, with corresponding 95% confidence interval and associated p values. We set the level of significance at alpha = 0.05 and did not adjust this for multiple comparisons since these were primarily exploratory. Further exploratory analyses using regression were performed to assess associations between residency year and outcomes. All analyses were conducted using STATA 10.1 (College Station, TX).
3. Results3.1. Recruitment and Retention
Twenty-six out of a total of 29 eligible anesthesia residents consented to the study, for a recruitment rate of 90% (Table 1). Eighteen of the 26 residents completed both the pre- and posttests, but one of these residents failed to complete the postsurvey (Figure 1). Therefore, the retention rate was 65% based on those 17 residents who completed all the pre- and postmodule tests and surveys. Incomplete data from the other participants were not included in the analysis.
Participants’ demographics (n = 17).
Characteristics
n (%)
Gender
Male
8 (47)
Female
9 (53)
Residency year
PGY 1
6 (35)
PGY 2
6 (35)
PGY 3
2 (12)
PGY 4
3 (18)
PGY, postgraduate training year; n, sample size of participants that completed all components of TEAM.
Flow diagram of residents’ participation.
3.2. User Perceptions
User perceptions were assessed by questions relating to user-friendliness and educational value of WBL, using a 7-point Likert scale. User-friendliness was perceived to be high, as measured by “ease of use” (mean score: 6.76, SD: 1.15), “interpretability” (mean score: 6.5, SD: 1.26), and “visual aid” (mean score: 5.35, SD: 2).
With respect to perceptions of educational value, in the pre-and postmodule survey, residents ranked “simulation” and “on call experience with faculty member” as the first and second preferred learning methods, respectively, for learning to manage emergency airways (Figure 2). WBL was ranked in the middle with a mean value of 4.82 (SD: 1.55). Self-directed learning was ranked the lowest at 3.59 (SD: 1.37). After modules, both of these modalities increased significantly in perceived value with mean of 4.24 (SD: 1.25) (95% CI of 0.14 to 1.16, p=0.017) and 5.76 (SD: 1.39) (95% CI of 0.41 to 1.47, p=0.002), respectively, suggesting a positive impact on perceptions from using WBL.
Participant perception of teaching modality value in premodule versus postmodule survey comparison.
3.3. Knowledge Improvement
Knowledge improvement was measured by comparing the pre- and postmodule knowledge scores. The mean knowledge test scores were calculated out of 14 questions, instead of the original 15, due to a reported broken image link on one of the questions. A comparison analysis revealed no statistical difference between pre- and postmodule test scores of 14 questions compared to 15. On average, residents scored a mean of 7.39 (SD: 1.97) out of 14 on the premodule test (Table 2). The postmodule test score was significantly higher, with a mean of 11.44 (SD: 1.72) (p<0.001). This represented 29% improvement in the postknowledge test.
Pre- and postmodule comparisons of tests and surveys.
Premodule
Postmodule
Difference
Mean (SD)
Mean (SD)
Post-pre
(95% CI)
p value
Module survey
Teaching sessions
4.65 (1.62)
4.88 (1.22)
0.24
(−0.63, 1.10)
0.512
Day list experience with staff
5 (1.94)
5.59 (1.37)
0.59
(−0.44, −1.62)
0.243
On call experience with staff
6.13 (2.19)
6.63 (1.15)
0.50
(−0.58, 1.58)
0.341
Self-directed learning
3.59 (1.37)
4.24 (1.25)
0.65
(0.14, 1.16)
0.017
Simulation training
6.24 (1.82)
6.76 (1.20)
0.53
(−0.05, 1.11)
0.070
Online teaching modules
4.82 (1.55)
5.76 (1.39)
0.94
(0.41, 1.47)
0.002
Test
Test score (out of 14)
7.39 (1.97)
11.44 (1.72)
4.06
(3.15, 4.97)
<0.001
Test percentage
52.78 (13.93)
81.94 (12.35)
29.17
(22.73, 35.61)
<0.001
95% CI, 95% confidence interval. Results are expressed as means with standard deviation (SD).
Two-sided paired t tests for differences.
Regarding each individual test question, a predictably higher percentage of participants answered the posttest version correctly compared to the pretest version, with the exception of two questions, possibly due to ambiguous wording or content (Figure 3). Overall, premodule test scores tended to increase slightly with the level of training, although there was no statistically significant difference in scores between junior and more senior residents.
Individual question breakdown in pretest versus posttest comparison for percentage of participants who answered correctly.
4. Discussion
WBL is increasingly being used in anesthesia training. Bello et al. [9] demonstrated that the combination of online lecture slides and videos of difficult airway procedures resulted in an increase in postknowledge test scores and positive satisfaction scores. Soto et al. [17] paired didactic online lectures on anesthesia drug costs with “in-person” presentations to improve knowledge scores. Kopp and Smith [14] showed improved test scores in regional anesthesia with both case-based and traditional textbook style modules.
In their meta-analysis and systematic review, Cook et al. [18, 19] concluded that WBL is effective compared with no intervention and is at least as effective as traditional educational interventions. Therefore, given these findings, research should focus on examining the appropriate conditions for effective WBL implementation and use [18, 19]. In order to examine the usefulness of WBL in enhancing emergency airway management training in our curriculum and guide further development, we assessed the feasibility of implementing WBL modules into the anesthesia curriculum with respect to user uptake, perceptions, and educational value.
According to our feasibility criteria, the findings suggest that implementing interactive WBL emergency airway management modules in our anesthesia curriculum is “possibly feasible” as demonstrated by the recruitment and retention rates, user satisfaction, and evidence of educational value. The high recruitment rate of 90% is promising, suggesting that almost all of the residents were interested in WBL. However, the lower retention rate of 65% is problematic, pointing to a need to further examine contributing factors, such as technical issues, complexity or length of the modules, and/or the study measurement tools.
The user-friendliness of the modules in terms of ease of use and interpretability were highly rated. The lower rating on the use of visual aids might have been improved with more images or other multimedia. With respect to perceptions of educational value, resident “buy-in” for the use of WBL is further affirmed by a significantly increased mean Likert score and ranking of the postmodule survey compared with the premodule survey (4.8, SD: 1.55, versus 5.76, SD: 1.39). The educational value and construct validity of these modules are also supported by a significant overall improvement in postknowledge test scores.
Of all the educational modalities, “self-directed learning” received the lowest rating of 3.59 (SD: 1.37) and although it did significantly improve to 4.24 (SD: 1.25), it still remained low. The low ratings could have resulted from varying interpretations of what the term meant (as we did not clearly define it). Furthermore, it was asked separately from the WBL question; therefore residents may not have made the connection between self-directed learning and WBL. It is also possible that, in the critical area of emergency airway management, residents preferred to receive more instructor guidance than would be provided by self-directed learning. This finding warrants further investigation.
On call experience with faculty and simulation training were valued the highest both before and after survey. This finding is consistent with studies which show that learners often seek experiential knowledge around the context of patient encounters [37, 38]. Similar to other studies [39–42], our findings suggest that WBL is regarded as a valued supplement but not replacement for current experiential educational modalities.
The results of our feasibility study as well as our systematic evidence-informed approach [43–45] to the development of these modules help inform our decision to implement WBL as a potentially efficacious modality that will enhance anesthesia residency training. Based on these outcomes, future development would include improving the technical aspects of WBL design, addressing the limitations of the learning management system, examining issues associated with retention, and incorporating facilitated online discussions.
There were several outcomes that we were unable to report due to limitations in the learning management system. The software program did not allow comments to be collected from participants, reporting on the length of time spent on the different components or identifying whether “incorrect” questions were truly incorrect or simply not attempted at all. While knowledge improved after the modules, longer term knowledge retention needs to be assessed. We did not enable any online forums in order to maintain the participant’s anonymity; however, this should be considered as a venue for feedback in order to improve the modules. We cannot generalize our specific results to other programs; nevertheless, our approach to developing WBL modules as well as assessing feasibility of incorporating WBL is helpful to inform other programs that are considering incorporating WBL in their curricula.
5. Conclusion
In conclusion, our study shows how we can assess whether WBL is a feasible and potentially efficacious educational modality for implementation into an anesthesia training curriculum. Although the initial time and resource commitments were substantial, our findings suggest the possible feasibility of using WBL modules to enhance the clinical training in difficult and emergency airway management. It would be important to consider collaborating with other anesthesia training programs in order to pool resources and more efficiently develop, implement, and further evaluate this educational modality for anesthesia education and practice.
AbbreviationsWBL:
Web-Based Learning
TEAM:
Training for Emergency Airway Management
PGY:
Postgraduate year
SD:
Standard deviation.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Disclosure
Ada Hindle is the first author and Principal Investigator. Ada has recently completed her anesthesia residency in the Department of Anesthesia, McMaster University, Hamilton, Ontario, Canada. Ji Cheng is a coauthor. She is a Research Associate with the Department of Anesthesia, McMaster University, and the Biostatistics Unit at St. Joseph’s Healthcare Hamilton. Lehana Thabane is a coauthor. He is a Research Methodologist, Professor, and Associate Chair, Department of Clinical Epidemiology and Biostatistics, and Director of the Biostatistics Unit, St. Joseph’s Healthcare Hamilton. Anne Wong is a coinvestigator, corresponding author, and supervisor of the resident research project. She is a Professor in the Department of Anesthesia, McMaster University.
Acknowledgments
Thanks are due to Toni Tidy for research coordination, Devon Mordell for technology support, webpage design supervision, and development, and Drs. G. Peachey, L. Olivieri, M. Hollidge, and G. Rosenblood for reviewing content of modules.
SajevaM.E-learning: web-based education200619664564910.1097/aco.0b013e328010bec82-s2.0-33750892943CookD. A.LevinsonA. J.GarsideS.DuprasD. M.ErwinP. J.MontoriV. M.Instructional design variations in internet-based learning for health professions education: a systematic review and meta-analysis201085590992210.1097/acm.0b013e3181d6c3192-s2.0-77953585957GutmarkR.HalstedM. J.PerryL.GoldG.Use of computer databases to reduce radiograph reading errors200741656810.1016/j.jacr.2006.08.0162-s2.0-33845754881FungK.-M.HassellL. A.TalbertM. L.WiechmannA. F.ChaserB. E.RameyJ.Whole slide images and digital media in pathology education, testing, and practice: the Oklahoma experience2012351374010.3233/acp-2011-00282-s2.0-84855572963FriedlR.HöpplerH.EcardK.ScholzW.HannekumA.StrackeS.Development and prospective evaluation of a multimedia teaching course on aortic valve replacement20065411910.1055/s-2005-8658712-s2.0-32544438926FearingN.BachmanS.HolzmanM.ScottD.BruntM.Evaluation of a video-based curriculum for laparoscopic biliary surgery: a pilot study from the SAGES MIS Web Learning Center201024123141314310.1007/s00464-010-1105-z2-s2.0-78649446134SatterwhiteT.SonJ.CareyJ.ZeidlerK.BariS.GurtnerG.ChangJ.LeeG. K.Microsurgery education in residency training: validating an online curriculum201268441041410.1097/sap.0b013e31823b6a1a2-s2.0-84858826199KulierR.HadleyJ.WeinbrennerS.MeyerroseB.DecsiT.HorvathA. R.NagyE.EmparanzaJ. I.CoppusS. F. P. J.ArvanitisT. N.BurlsA.CabelloJ. B.KaczorM.ZanreiG.PiererK.StawiarzK.KunzR.MolB. W. J.KhanK. S.Harmonising evidence-based medicine teaching: a study of the outcomes of e-learning in five European countries20088, article 271010.1186/1472-6920-8-272-s2.0-43749107982BelloG.PennisiM. A.MavigliaR.MaggioreS. M.BocciM. G.MontiniL.AntonelliM.Online vs live methods for teaching difficult airway management to anesthesiology residents200531454755210.1007/s00134-005-2561-02-s2.0-17444382324ChuL. F.YoungC.ZamoraA.KurupV.MacarioA.Anesthesia 2.0: internet-based information resources and web 2.0 applications in anesthesia education201023221822710.1097/aco.0b013e328337339c2-s2.0-77952196271GiglioliS.BoetS.De GaudioA. R.LindenM.SchaefferR.BouldM. D.DiemunschP.Self-directed deliberate practice with virtual fiberoptic intubation improves initial skills for anesthesia residents20127844564612-s2.0-84859396780GarfieldJ. M.PaskinS.PhilipJ. H.An evaluation of the effectiveness of a computer simulation of anaesthetic uptake and distribution as a teaching tool198923545746210.1111/j.1365-2923.1989.tb00902.x2-s2.0-0024453143JerathA.VegasA.MeineriM.SilversidesC.FeindelC.BeattieS.CorrinM.TaitG.An interactive online 3D model of the heart assists in learning standard transesophageal echocardiography views201158114212-s2.0-7875159802410.1007/s12630-010-9410-5KoppS. L.SmithH. M.Developing effective web-based regional anesthesia education: a randomized study evaluating case-based versus non-case-based module design201136433634210.1097/aap.0b013e3182204d8c2-s2.0-79959983161LambdenS.MartinB.The use of computers for perioperative simulation in anesthesia, critical care, and pain medicine201129352153110.1016/j.anclin.2011.05.0062-s2.0-80051966389SchwidH. A.RookeG. A.MichalowskiP.RossB. K.Screen-based anesthesia simulation with debriefing improves performance in a mannequin-based anesthesia simulator2001132929610.1207/s15328015tlm1302_42-s2.0-0035291313SotoR. G.CormicanD. S.GallagherC. J.SeidmanP. A.Teaching systems-based competency in anesthesiology residency: development of an education and assessment tool20102225025910.4300/jgme-d-09-00078.1CookD. A.LevinsonA. J.GarsideS.DuprasD. M.ErwinP. J.MontoriV. M.Internet-based learning in the health professions: a meta-analysis2008300101181119610.1001/jama.300.10.11812-s2.0-51449111015CookD. A.GarsideS.LevinsonA. J.DuprasD. M.MontoriV. M.What do we mean by web-based learning? A systematic review of the variability of interventions20104487657742-s2.0-7795471808910.1111/j.1365-2923.2010.03723.xWongA. K.Educational technologies in anesthesia training2013711013RuizJ. G.MintzerM. J.LeipzigR. M.The impact of e-learning in medical education200681320721210.1097/00001888-200603000-000022-s2.0-33645747704ChuL. F.ErlendsonM. J.SunJ. S.ClemensonA. M. P.MartinP.EngR. L.Information technology and its role in anaesthesia training and continuing medical education2012261335310.1016/j.bpa.2012.02.0022-s2.0-84860605362CookD. A.DuprasD. M.A practical guide to developing effective web-based learning200419669870710.1111/j.1525-1497.2004.30029.x2-s2.0-3042527033ThabaneL.MaJ.ChuR.ChengJ.IsmailaA.RiosL. P.RobsonR.ThabaneM.GiangregorioL.GoldsmithC. H.A tutorial on pilot studies: the what, why and how201010, article 11010.1186/1471-2288-10-12-s2.0-77249142355US Department of Health and Human ServicesResearch- Based Web Design and Usability Guidelines2006, http://guidelines.usability.gov/CookD. A.ThompsonW. G.ThomasK. G.ThomasM. R.PankratzV. S.Impact of self-assessment questions and learning styles in web-based learning: a randomized, controlled, crossover trial200681323123810.1097/00001888-200603000-000052-s2.0-33645739134LevineM.MurphyP.20131sthttp://www.med.uottawa.ca/anesthesia/assets/documents/National_Curriculum_for_Canadian_Anesthesia_Residency.pdfBarashP. G.20096thPhiladelphia, Pa, USALippincott Williams & WilkinsHungO.MurphyM. F.20122ndNew York, NY, USAMcGraw-HillMillerR. D.20107thPhiladelphia, Pa, USAChurchhill Livingston ElsevierMcCallJ. E.CahillT. J.Respiratory care of the burn patient200526320020610.1097/01.bcr.0000162150.32733.232-s2.0-19544383596RehbergS.MaybauerM. O.EnkhbaatarP.MaybauerD. M.YamamotoY.TraberD. L.Pathophysiology, management and treatment of smoke inhalation injury20093328329710.1586/ers.09.212-s2.0-70049096501CrosbyE. T.Airway management in adults after cervical spine trauma200610461293131810.1097/00000542-200606000-000262-s2.0-33744959276MercerS. J.LewisS. E.WilsonS. J.GroomP.MahoneyP. F.Creating airway management guidelines for casualties with penetrating airway injuries20101564, supplement 135536010.1136/jramc-156-04s-152-s2.0-79952640502MohanR.IyerR.ThallerS.Airway management in patients with facial trauma2009201212310.1097/scs.0b013e318190327a2-s2.0-67650257423NormanG. R.van der VleutenC. P. M.NewbieD. L.2002Dordrecht, The NetherlandsKluwer Academic PublishersEdsonR. S.BeckmanT. J.WestC. P.AronowitzP. B.BadgettR. G.FeldsteinD. A.HendersonM. C.KolarsJ. C.McDonaldF. S.A multi-institutional survey of internal medicine residents' learning habits201032977377510.3109/014215910036926982-s2.0-77956076131MonaghanS. F.BlakelyA. M.RichardsonP. J.MinerT. J.CioffiW. G.HarringtonD. T.The reflective statement: a new tool to assess resident learning2012178261862210.1016/j.jss.2012.07.0342-s2.0-84869087235BraeckmanL. A.FieuwA. M.Van BogaertH. J.A web- and case-based learning program for postgraduate students in occupational medicine2008141515610.1179/oeh.2008.14.1.512-s2.0-39749140955FergusonC. M.WarshawA. L.Failure of a Web-based educational tool to improve residents' scores on the American Board of Surgery in-training examination2006141441441610.1001/archsurg.141.4.4142-s2.0-33645876173DyrbyeL.CumynA.DayH.HeflinM.A qualitative study of physicians' experiences with online learning in a masters degree program: benefits, challenges, and proposed solutions2009312e40e4610.1080/014215908023661292-s2.0-67649921023SmithH. L.MenonD. K.Teaching difficult airway management: is virtual reality real enough?200531450450510.1007/s00134-005-2576-62-s2.0-17444411500WongG.GreenhalghT.PawsonR.Internet-based medical education: a realist review of what works, for whom and in what circumstances20101011010.1186/1472-6920-10-122-s2.0-77249155667EricssonK. A.Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains20047910, supplementS70S812-s2.0-4644318381McGaghieW. C.IssenbergS. B.PetrusaE. R.ScaleseR. J.A critical review of simulation-based medical education research: 2003–20092010441506310.1111/j.1365-2923.2009.03547.x2-s2.0-74049154816