Using Kern’s Method to Develop Asynchronous E-learning for the Clinical Curriculum: One Facility’s Experience

Introduction

Critical thinking is a desirable skill in optometric clinical learners. The 2019 ASCO Optometric Education Global Summit identified recruitment of students who exhibit critical-thinking skills as one of the opportunities in future development of optometric education.¹ Critical thinking is often identified as a component of clinical reasoning,² and clinical reasoning is a goal of most, if not all clinical curricula throughout the world.

The National Council for Excellence on Critical Thinking and Education Reform defines critical thinking as “…conceptualizing, applying, analyzing, synthesizing and/or evaluating information…as a guide to belief and action.”³ These action words are consistent with Bloom’s Taxonomy of Learning, with cognitive skills like application, analysis, synthesis and evaluation positioned higher on Bloom’s hierarchy. These more advanced abilities are built on top of the foundational skills of knowing, remembering and comprehending.⁴

Generally, this framework is layered so that the higher on the taxonomy a learner climbs, the greater the mastery of the topic of regard, while attaining the lower levels often helps in attaining the upper levels.⁵ An optometric curriculum should be designed with this layering in mind, first calling for knowing, remembering and comprehending foundational concepts before moving on to the higher cognitive skills which allow for critical thinking and clinical reasoning.

A natural conceptual division of pedagogy in optometric programs is classroom learning vs experiential learning in the clinic. Ideally, all taxonomic levels of regard should be taught in the classroom learning phase of the curriculum, while the higher taxonomic functions are reinforced during clinical rotations. But there is a caveat that the learning or relearning of foundational knowledge will be facilitated as needed before moving on to guided and intentional practice the learner needs to master those higher functions.

By the clinical phase of the education process, the lower taxonomic strata tend to become known simply as “background knowledge,” and two things may be evident to clinical preceptors about background knowledge. First, background knowledge can and should be itself layered with simpler concepts preceding higher concepts. Though intuitive enough, this principle of building complexity over time decreases the intrinsic cognitive load which is thought to enhance learning.⁶ Second, clinical preceptors are often faced with handling knowledge gaps as some learners may not always present with adequate foundational knowledge. In that event, the preceptor’s responsibilities include helping the learner identify any gaps in foundational knowledge that interfere with clinical reasoning, assisting in correcting those knowledge gaps, and facilitating the higher levels of clinical reasoning that follow.

Figure 1: Kern’s Six Step Approach to Medical Curriculum Development. Click to enlarge

From this perspective, the clinical educators at three optometric clinics within the VA Illiana Healthcare System set out to ensure their students begin the clinical rotation prepared to maximize development of critical thinking as it relates to a single clinical skill – interpretation of retinal nerve fiber layer thickness (RNFLT) optical coherence tomography (OCT) – by utilizing Kern’s Six Step Approach for medical curriculum development.⁷ Kern’s approach is presented in Figure 1. The three involved clinics are in different geographic areas within Central Illinois and host clinical learners from five different schools of optometry, each school with a different start date for clinical rotations and two of the three clinics hosting students from multiple schools simultaneously. Depending on the school, rotations lasted 3 to 4 months.

Step 1. Informal Needs Assessment

Some, but certainly not all, students send a letter of introduction to their assigned optometry clinic prior to a clinical rotation. Unfortunately, this is often the only information the preceptor receives regarding the learner’s educational progress, strengths and weaknesses.

Anecdotally, within these introductory letters, learners sometimes indicate OCT interpretation as a self-assessed target area of growth. There may be three reasons for this. First, the presence of OCT in optometry clinics in the US has become ubiquitous and OCT sales world-wide are increasing,⁸ therefore there is a need for clinical trainees to have mastery of this technology. Secondly, critical analysis of OCT, particularly RNFLT OCT, requires a high degree of training and critical thinking to discriminate against a host of potential confounding factors.^9,10,11,12 Lastly, teaching experience in these clinics suggests that learners are unable to interpret these scans competently. Improvement in foundational understanding would be helpful, with the ultimate goal of overall skill mastery.

Critical analysis of OCT requires higher level taxonomic functions that are likewise reliant on lower levels of background knowledge, with mastery often depending on clinical experience. One aspect of the informal needs assessment included observation of and conversation with clinical learners. This led the authors to devise, by open conversation, a list of four concepts that could be considered foundational to the higher-level skills of critical OCT analysis. These four foundational concepts are represented in Table 1.

Table 1: Four concepts identified to be foundational to mastery of critical analysis of RNFLT OCT. Click to enlarge

One concept that must be understood for much of this critical analysis is scan location. It was observed that clinical learners are apt to think the RNFLT scan is a scan of the nerve head itself rather than a circular, peripapillary scan. After all, the scan and resulting graph are presented in a linear fashion and so a circular RNFL scan is not as intuitive as a macular scan, for example. This discrepancy may be due to the clinical learner’s understanding that a macular scan is typically acquired as a line raster vs the RNFL scan which is acquired in a circular fashion. Therefore, this knowledge gap could be an unperceived need for clinical learners, as they may have never questioned if these scans are acquired differently.

Another foundational skill that can be challenging for clinical learners, but vital for critical analysis, is the identification of retinal blood vessels in the RNFLT scan. This is an important skill as vessel branching pattern may play a role in RNFLT^13,11,14 and many small artifacts will be found adjacent to blood vessels.^15,16,17,18

Similar to being able to identify retinal blood vessels and adjacent artifacts, it is crucially important for clinical learners to correlate a potential artifact from the scan with the location on the corresponding graph. This skill was included in the list of necessary foundational concepts.

Lastly, the authors identified RNFL identification, or differentiation of RNFL from non-RNFL, as a foundational skill often underdeveloped in early clinical learners. This is important because segmentation errors are a significant source of artifact in RNFLT results and need to be examined critically.¹⁸ This can only be done if the clinician can identify the RNFL without relying exclusively on automated segmentation.

Step 2. Formal Needs Assessment of Targeted Learners

Figure 2.Questions adapted from the formal needs assessment. The pretest questions and posttest questions were the same. The answer key was not shared with the learners. RNFL=Retinal Nerve Fiber Layer. OCT=Optical Coherence Tomography. Click to enlarge

As part of the formal needs assessment prescribed by Kern et al.,⁷ the authors developed a test to evaluate the clinical learners’ understanding of these foundational concepts. This test has been adapted for publication and is presented in Figure 2. Early versions of the test were paper-based, but a computer-based version was subsequently developed and administered using standardized images. The earlier paper test presented questions as text and allowed respondents to circle labels and draw arrows to structures. This modality was abandoned in favor of the more standardized computer version which presented textual questions provided as multiple choice or multiple response multiple choice. To determine if current teaching strategies were effectively remediating these concepts, the test was given to a cohort of eight students as a post-test at the completion of rotation through the VA Illiana clinics. In this way, if a learner failed a question or indicated low confidence with a correct answer, it showed that not only did the learner likely enter with a knowledge gap, but that knowledge gap was not effectively remediated during the learner’s rotation. If a learner correctly answered the question and noted high confidence with the answer, that student was identified as having passed that question. Participation was voluntary and students were made aware that the results would not impact their summative evaluation for the rotation. The results of this phase of the curriculum development are shown as rates of those learners who passed each question in Figure 3 and show some knowledge gaps persisted throughout the rotation. These results prompted the development of a teaching strategy to target these knowledge gaps.

With the understanding that participation was voluntary and the results would not impact summative assessment for the rotation, the test was then administered as a pretest at the start of the rotation for subsequent cohorts totaling 14 clinical learners. As seen again in Figure 3, a lower percentage of learners passed each of the tested concepts than for the previous cohort who took the test at the end of the rotation. These results indicated two important findings: 1) students were starting with knowledge gaps in these foundational concepts, and 2) despite some knowledge gaps persisting for some students, completing a rotation through a VA Illiana optometric clinic was overall helpful in bridging these gaps.

Figure 3: Passed Pretest vs Posttest with and without Targeted Intervention Grouped by Learning Objective. Click to enlarge

It is important to highlight that this test assessed foundational knowledge, the knowledge needed to gain the skills necessary to critically analyze the RNFLT OCT, rather than assessing the skill of critical analysis itself. Assessing critical-analysis skills was outside of the scope of this curriculum.

Step 3. Goals and Specific Measurable Objectives

Table 2 Learning objectives developed for the learning module on RNFLT foundational concepts. Click to enlarge

With the goals of shifting the learning curve and helping clinical learners conceptualize data presented in the RNFLT OCT report, learning objectives were developed. They were written in the SMART Goals format, being Specific, Measurable, Achievable, Relevant and Timebound.^19,20 These learning objectives, based on the targeted foundational concepts, are listed in Table 2.

Step 4. Educational Strategy

The formal needs assessment underscored the need for remediation at the start of a learner’s clinical rotation. However, the authors found it unrealistic to personally remediate each student at the beginning of each rotation because each clinic has multiple students from different schools, and students from different schools have different rotation start dates yielding an unreasonable teaching workload in a busy clinic.

As a solution, an asynchronous e-learning module was chosen as the best educational strategy. The video portion of the module was created by the lead author, using Heidelberg Spectralis OCT images obtained through informed consent and processing those images through Procreate on iPad Pro. Animations were also added using Procreate and a second-generation Apple Pencil. Both testing and the learning module were delivered to participating learners using web-based teaching.

Both the pretest and the learning module have been edited for a wider audience and are available for review by readers. The pretest/post-test questions and answer key are shown in Figure 2, while the video portion of the learning module is presented as Figure 4.

Figure 4. Video portion of learning module. RNFLT=Retinal Nerve Fiber Layer Thickness. Click to play.

Step 5. Implementation

To minimize the impact of guessing correctly on multiple choice questions, success was defined by simultaneously answering a question correctly and also indicating high confidence with that answer (questions one and two). Students who failed any question of the pretest were invited within the orientation period of clinical rotation to participate in the e-learning module.

Step 6. Evaluation and Feedback

The learning module included knowledge checks on all targeted concepts. These knowledge checks included the same questions as before and acted as a post-test at completion of the learning module. Following module completion, all students successfully identified the fundus location of the RNFL scan. Improvement from pretest scores was seen for all other questions. Figure 2 also shows these results compared to results of pretest and post-test without intervention.

Figure 5: Results from seven participants who evaluated the learning module. Click to enlarge

Learners who took the pretest, completed the module, and took the post-test were also invited to share their perceptions of the learning module. Seven of eight students provided feedback, and the results of this feedback (Figure 5) were positive. These evaluations demonstrated that: the learning goals were clearly stated, there was strong agreement the module was helpful, the skills covered were those that other skills could be built on, and the targeted concepts are important to clinical practice.

Kern’s Six Step Approach advocates for curriculum evaluation to promote continuous improvement. Despite the positive feedback from students, evaluation of test results shows a trend that will need to be addressed in future iterations of this module as it is clear the module targets two of the four concepts disproportionately. While the concepts of fundus location of the RNFL scan and blood vessel identification showed marked reductions in knowledge gaps following participation in the learning modules, gains made for the other two concepts – relationship between scan and graph as well as RNFL identification – were less robust. Because the pretest, learning module and post-test were all administered within the first 2 weeks of the rotation, no doubt, the failure rate would continue to drop by the end of the rotation. However, the goal here is to minimize knowledge gaps at the rotation’s start, so revising this learning module further would be advisable.

Discussion

Speculating on why knowledge gaps persist past the classroom phase of education is difficult. There are many possible reasons a student may move into clinic with deficiencies in background knowledge. One possible source of these knowledge gaps is prioritization. With so many different modes of practice awaiting the clinical optometric learner, including primary care, disease, contact lens, low vision, pediatrics and binocular vision, it would be difficult for any educational program or any learner to effectively prioritize the information that will be most needed for an individual clinical rotation from the vast breadth of topics taught. Other reasons may include individual learner inefficiency or a lack of review prior to rotation start, though further study would be warranted. It should also be considered that the learners evaluated here are part of the COVID generation of learners, so there is a possibility these students have knowledge gaps due to disruptions in education caused by the pandemic.

During a patient encounter, when a preceptor probes for prior knowledge and finds the student has a knowledge gap, the preceptor has three choices. The first is to teach only the prior knowledge. This means the higher-level concept that should have been taught must wait until the next time it is encountered in the clinic by that student and there is a chance it will not be. At a minimum, this tactic delays mastery. The second option is to reteach the concept and then teach the broader concept that is built on that knowledge, effectively teaching two concepts rather than one. This is not only time consuming, and often prohibitively so, but also serves to increase the intrinsic cognitive load of the concept at hand, reducing the chances it will be understood, retained or utilized.²¹ Concepts taught in this manner may need to be retaught in the future. For this reason, it is widely recommended that clinical educators resist the urge to teach multiple concepts per patient encounter, instead focusing only on a general rule.^22,23 The third choice is for the preceptor to assume the role of facilitator rather than teacher and to assign the foundational concept as homework for the clinical learner to review on their own. However, assigned learning needs to be assessed to ensure the learning occurred, so this option requires the preceptor revisit the topic later with the learner. Should this occur in a clinic with multiple learners, it may only serve to postpone the discussion. The broader topic could then be taught, but at a time when it is less relevant to the learner as the patient encounter has passed. Again, with this choice, mastery is delayed. A more efficient solution than any of these might be for the clinical learner to eliminate the knowledge gap before ever seeing the patient.

Strengths

This pedagogical approach works for several reasons. Targeting these knowledge gaps is important as they hinder a learner’s journey to mastery of critical analysis of RNFLT OCT results. Classically, this process of gaining mastery is distilled down to an S-shaped curve.^29,30 This idealized curve is presented in Figure 6a, with the early latent period representing little to no progress towards mastery, building to an upward slope representing gains in mastery, before plateauing as mastery is achieved.

Figures 6a & 6b. Click to enlarge

This is a conceptual simplification, but one that helps to illustrate a few simple points. To begin with, everything that happens before the plateau has the potential to negatively impact the patient as well as the preceptor. For the patient, the risk of medical error is higher, while the preceptor, whose responsibilities include patient safety and educating the student, will see decreased efficiency in both of those tasks. At a minimum, this results in increased chair time and with patient harm resulting in the worst case. Similarly, a clinical rotation is of a specified duration, and if it ends before mastery is achieved, then the consequences may be felt by the learner or by society at large, with the learner being less prepared to reach full professional potential while future patients bear the risk of being cared for by a provider who is less than maximally prepared. Lastly, as represented in Figure 6b, these scenarios can be mitigated by shortening the latent period and shifting the curve left.

Shifting the learning curve was a goal of this learning module. Rather than simply “teaching to the test” by providing the answers to the test questions, this e-learning module teaches concepts. It uses animations, which have been found in previous studies to be more engaging when teaching medical concepts.^24,25 Further, it otherwise complies with Mayer’s Multimedia Principles to reduce extraneous cognitive load and facilitate understanding.^26,27,28

Limitations

One deficiency in this pedagogic approach is the formats used to test the learners were not consistent throughout. Testing and video delivery evolved over the span of curriculum development with the goal of maximizing automation. A consistently high level of automation would be needed if implemented in a larger test population.

Device type is also a limiting factor as some devices may not display graphics as clearly as others.  However, as part of the curriculum assessment, students were invited to share open-response feedback regarding what could be done to make the module better, and the issues of image size or image quality were not identified as an improvement opportunity by any participating learner.

Additionally, the video portion of the module is a first draft and needs refining. The current iteration is geared heavily toward users of the Heidelberg Spectralis OCT, which is employed in all three teaching clinics. As proof of concept, however, this module succeeds in teaching the targeted topics and is functional enough even for interested outside users and clinics using OCT platforms other than Heidelberg.

Some of the pretest questions employed here are, unfortunately, necessarily complex. Earlier versions involved the learner drawing arrows and circling structures, but this approach was abandoned due to possible ambiguity in responses. Multi-select multiple choice questions were ultimately designed as they enabled less variability of answers. The concepts covered here are conceptually complex, which resulted in multiple choice prompts that were more difficult to understand.

It has been noted that decreasing extraneous cognitive demand improves understanding of the questions and concepts. Expecting students to complete this learning while balancing a heavy patient load is impractical and experience shows this can lead to poor comprehension of the questions and the learning module. If given after the learner begins seeing patients in the clinic, protected time should be given to complete the task, or it should be assigned to do after clinic hours. Even so, it should be expected that there will still be students who require face to face remediation, though the goal is to minimize this need.

This learning module is designed for learners beginning a rotation in a clinic with a high proportion of ocular disease as opposed to a contact lens clinic or a pediatric/binocular vision clinic. The participating VA clinics are heavy in ocular disease. However, only VA clinics were involved and other clinic modalities, such as private practice clinics or other government administered clinics, may have different outcomes if implementing such a learning strategy. Similarly, learner outcomes could be impacted by the students’ rotation experiences preceding the clinical rotation that utilizes this strategy. It is not clear if all students in all clinics would have the same benefit from this educational strategy.

There are some unanswered questions. First, does starting a rotation with a better grasp on foundational concepts translate into more efficient mastery of the critical thinking that builds on foundational knowledge? This is presumed to be the case, but this question was not directly addressed in any of the steps of curriculum development here. Second, would the lessons learned in this scenario benefit clinical optometry trainees at large? The authors see evidence for this, but more reporting may be needed. And third, are there other common gaps in foundational knowledge holding learners back in their progress towards critical thinking as applied to other concepts and disorders? Here again, the authors believe the answer is yes, but there is certainly room for further study.

In summary, the authors and students have found this asynchronous e-learning module useful in remediating the four targeted concepts, setting the stage for more efficient mastery of critical RNFLT OCT analysis. This pedagogical method shows promise and warrants exploration of a community-based solution. Utilizing existing online teaching platforms and building a broader curriculum to help remediate common knowledge gaps, this solution could help clinical learners throughout the world gain mastery more efficiently. This would help not only the learners but would also improve the experience of both current and future patients. It might also decrease some of the educational burden contributing to preceptor burn-out.

Acknowledgments

This project was supported in part by the Veterans Health Administration’s (VHA’s) Rural Interprofessional Faculty Development Initiative and will be transitioning to the VHA’s Innovation Network, or iNet, program as a Spark Idea project aimed at exploring the use of a web-based application to improve mastery by filling foundational knowledge gaps for clinical learners.

The authors offer a special thank you to Jane Lakayil, OD, FAAO; Aimee Beluch OD, FAAO; and all optometry providers at VA Illiana for assisting with curriculum implementation at their respective clinics.

Disclaimer

The authors have no financial interest in Edpuzzle.com. The service is free on a limited basis and does not generate advertising revenue. The authors have no interest in Heidelberg Engineering or any of their products. The views expressed in this discourse do not necessarily represent the views of the VA or the United States.