Background

Experiential learning is key in the education of healthcare professionals.¹ It is a significant and valued component in the education of third- and fourth-year optometry learners at the University of Waterloo School of Optometry and Vision Science (WOVS). As part of the clinical program, external geriatric and pediatric services are provided by optometrists and students to underserved populations at several facilities including nursing and retirement homes, elementary schools and public health facilities. The populations served include the elderly, children, people with special needs and Mennonite communities throughout the region. The intended purpose of this program is to improve the quality of life of those populations, while providing optometry students with a rich clinical experience to develop the competencies needed to continue this care within their own community after graduation. Evidence shows that the introduction of a clinical outreach program does produce this intended result.²

Curricular planning based on predetermined outcomes is standard in healthcare education.³ An important component of curriculum development is program evaluation, which determines whether intended (or unintended) outcomes are achieved and provides feedback for improvement and enhancement.⁴

Several different models of program evaluation are used to assess outcomes or process depending on the need. For this study, we chose to follow the Kirkpatrick Model, which is a common method for evaluating the outcomes of a training program.⁴ It is comprised of four levels: reaction, learning, behavior and results. In the reaction level, students subjectively report whether they like or are satisfied with a specific educational program. The second level, learning, determines whether the students have acquired the intended knowledge, skill or judgement. Level three, behavior, sets out to determine whether the program has influenced or changed a student’s future method of practice by applying what was learned. Level four, results, evaluates whether the program has achieved the desired outcomes.⁵ The anticipated result of implementing the outreach experience is improved quality of life for underserved populations through better healthcare provision.

The Outreach Experiential Program at WOVS

Optometry students at WOVS are first introduced to direct patient care one day per week in the third year of the educational program. Students are scheduled in external clinics (geriatric and pediatric) three to four days in the third year clinical curriculum and one to three days (geriatric only) in the fourth year. The eye examinations are conducted by the students under the direct supervision of a registered (licensed) optometrist. The external geriatric service provides routine and focused optometric care that includes therapeutic management of disease and low vision rehabilitation to 20 long-term-care and retirement facilities in the Kitchener-Waterloo and surrounding region. The external pediatric service provides full eye examinations with an emphasis on binocular vision to approximately 12 facilities and schools. The eye examinations are conducted for both clinics utilizing portable and handheld equipment.

The outreach clinics have continually grown since their inception in 2004. This research is intended to evaluate these services at the level of learner reaction in order to obtain useful information to improve the program.

Methods

Click to enlarge

The study received clearance from the University of Waterloo Office of Research Ethics, which follows the principles of the Declaration of Helsinki. A questionnaire was derived from a validated survey developed by the Veterans Affairs Learners’ Perceptions Survey Steering Committee from 1999-2001.⁶ This particular survey was chosen because it was created specifically to evaluate the satisfaction of students with their medical education, which is similar to the intent of this study. Demographic information, including the gender of the respondents and the number of external clinic days experienced, was collected. The survey was divided into specific questions related to satisfaction with four domains: faculty/preceptors, learning environment, working environment, and physical environment. The responses were obtained using a five-point Likert scale (very satisfied, somewhat satisfied, neither, dissatisfied, very dissatisfied). The survey concluded with a global measurement of overall satisfaction with the training program graded on a scale from 0-100.

The survey was adapted for the outreach program study. Two demographic-type questions were added regarding student gender and total days assigned to external clinics. A sixth point (not applicable) was added to the Likert scale for the outreach program survey as there were several questions that pertained particularly to medical education and had the potential to falsely increase the numbers of those dissatisfied with the optometric program. Table 1 shows the survey.

The paper survey was given to 90 optometry students at the end of the third year of their program (April 2013) after eight months of clinical education experience. It was administered anonymously by an independent individual who was a Vision Science Master’s candidate and neither a member of the research team nor a supervisor in outreach clinics. The students were familiar with the individual through previous laboratory teaching. The survey was administered during regularly scheduled classroom activity for 10 minutes. Participation was voluntary. Students left completed or uncompleted surveys on a table as they left the classroom.

The overall satisfaction responses were tabulated and percentages were calculated. The data retrieved for overall satisfaction for each domain were calculated in the same manner. The responses to specific questions in each domain were also reviewed. Percentages of those satisfied were calculated based on the sum of “very satisfied” and “somewhat satisfied” responses. Not applicable (N/A) responses were removed from the total count.

Results

# = number satisfied
n = total respondents with ‘not applicable’ responses removed
% = percent satisfied tabulated from combined ‘very satisfied’ and ‘somewhat satisfied’ responses

Click to enlarge

The response rate for the survey was 97%. Approximately 68% of respondents were females, and 68% of those who participated were scheduled in outreach clinics three to four days within the last academic year. Overall, students were satisfied with the outreach program, which received an average rating of 76 out of 100 (range of 40-95). The remaining results of the survey are presented in Tables 1 and 2. Students were most satisfied with the clinical supervisors assigned to the outreach program with an overall satisfaction score of 92%. They were not as satisfied with the remaining domains with overall satisfaction scores of 73% for learning environment, 60% for working environment, and 68% for physical environment. Within the clinical supervisor domain, students were most appreciative of the supervisors’ clinical skills (95%) and teaching ability (90%), but were less satisfied with the evaluation received (82%). In the learning environment section, students were satisfied with the degree of supervision (93%), but less than half were satisfied with the interdisciplinary approach and the amount of scut work such as preparing the room (47% and 48% respectively). Approximately two-thirds were satisfied with the preparation for future training and clinical practice. In the working environment section, students were satisfied with faculty/preceptor morale (86%), but the vast majority of the remaining components were considered to be not applicable to the optometry experience as is evident from the lack of response to these categories. Many were not satisfied with the access to computers or the internet (13% and 20% satisfied respectively). Only 52% were satisfied with the workspace available, and 23% were satisfied with the orientation program. In the physical environment component, students were most satisfied with personal safety and parking but felt that call rooms and food on call were not applicable to the program. Only 39% of the students were satisfied with the lighting available and 51% were satisfied with maintenance of optometric equipment.

Discussion

The intended outcome of the outreach program is to positively impact the quality of life of those served through better healthcare provision achieved by graduating students with the competencies necessary to continue this service provision. Seniors residing in collective dwellings have significantly higher rates of ocular disease (3-15 times) than those living in community.⁷ They are also likely to have uncorrected refractive error.⁸ Visual impairment has been associated with increased falls, social isolation and depression.^9,10 While there is an increased demand for care due to an aging population,¹¹ providers have varying opinions about the elderly and a low interest in caring for them.¹² The solution, however, may be increased exposure to positive experiences with caring for the elderly¹² and interaction with satisfied optometrists and mentors who practice geriatric optometry.¹³ Similarly, vision disorders are common among the pediatric population in Canada, with an estimated 25% of children between the ages of 0-18 years affected. At six months of age, manifestations of strabismus, high refractive error, and anisometropia can be detected.¹⁴ While treatment varies depending on the severity of the condition, early diagnosis is preferred.¹⁵ Providing timely access to care may contribute to an improved quality of life and, in some cases, help alleviate the need to utilize the social and healthcare resources within the Canadian healthcare system.

This study demonstrated that optometry students are satisfied overall with the outreach clinic program; however, several areas needing improvement were highlighted. The response rate was quite high, possibly as a result of the convenience of completing the survey during the last 10 minutes of regularly scheduled class time. In the clinical supervisors section, students were most content with the clinical skills and teaching ability of the supervising optometrists, who was seen as role models. The overall satisfaction rate for clinical supervision was high. Satisfaction with supervisors to the extent that they are viewed as worthy role models has been demonstrated as a key component to continued provision of services.¹³

Students were least satisfied with the fairness in evaluation and timeliness of feedback. It is suspected that this lack of satisfaction may have been a significant contributor to the overall satisfaction rating of the outreach program.¹⁶ This is likely from a misalignment of the learning objectives, learning opportunities and evaluation methods. Students are evaluated with a global rating scale on five components of the oculo-visual examination. The scale range is from 40-100 with 70 set as the pass score. Students are evaluated separately on case history and counseling; technical ability; diagnosis and management; record-keeping; and efficiency. The benchmark is the examination of a normal adult patient. At the outset of the third-year optometry training, students have demonstrated the ability to assess normal adult populations, but have not yet acquired the skills needed to assess children and the elderly. They have not been taught many of the techniques required to conduct an examination with only portable equipment such as Perkins tonometry, direct ophthalmoscopy, handheld biomicroscopy, trial-frame refraction, and Mohindra retinoscopy nor have they had didactic teaching in geriatrics and pediatrics. A sufficient knowledge base is necessary for the development of clinical reasoning.¹⁷ It is necessary to provide learning opportunities situated at the level of the student’s education to have an effective program. Interaction with patients leads to further competence development and greater self-confidence, which, in turn, promotes greater competency,¹⁸ but conducting examinations without adequate preparation leads to defeat and lack of confidence in abilities. These factors make evaluation difficult and frustrating for both students and supervising clinicians. Students were scheduled into these external programs as a result of curriculum scheduling challenges rather than optimum educational experience. In addition to fairness in evaluation, just more than half of the students were satisfied with the program’s ability to assist in preparation for future training or clinical practice. This may also be due to an insufficient number of exposures to practice.

Although the care provided in an external setting often involves extensive interaction with multiple individuals (personal support workers, doctors and nurses in geriatrics; and parents, social workers and teachers in pediatrics) students reported low levels of satisfaction with the interdisciplinary approach to care.^19,20,21 Students may be unaware of the interprofessional collaborations that occur because most take place before or after the actual examination and may not directly involve the student. Attention to actively involving the student in these discussions is warranted.

Many students were not satisfied with the working environment or physical environment of the external clinical program. This is likely tied to the dissatisfaction reported (23% satisfied) with the orientation program. The workspace is often limited to what the facility (elementary school, long-term-care facility, etc.) can provide and varies greatly from what students are familiar with. The optometry program representatives have limited control over workspace provided beyond discussing ideal conditions for adequate assessment (size, lighting, access to sink, etc.). In order to address these issues, all students are provided with a handbook outlining the administrative processes, recommended techniques to review, and clinical pearls for conducting the external examination. In addition, one day at Waterloo Optometry and Vision Science at the beginning of third year is set aside to introduce students to the techniques they will be using. This does not provide sufficient preparation for the outreach clinic setting. Lack of satisfaction with maintenance of handheld optometric equipment is often a result of an inability to conduct timely maintenance due to limited supply of equipment available.

In order to address these issues, the learning objectives need to be revised, the timing and expectations of participation in the external programs within the clinical experience needs to be changed, the number of clinical encounters needs to be increased, and the student evaluation method needs to be improved to align with the actual expectations of student performance. A full discussion of student evaluation methods is beyond the scope of this paper.²² It is recommended that introduction to participation in external clinics should begin with observation (legitimate peripheral participation) and progress to full patient care by the end of the final year of training. To recognize the nature of the multidisciplinary approach within an outreach clinic, students should be encouraged to participate in the collaborations with other disciplines or asked to reflect on how these collaborations occur to help the patient. Laptop computers should be provided to allow access to electronic medical record forms as well as the internet. Additional equipment should be acquired to allow for timely maintenance.

Two drawbacks to this study have been identified. First, the survey grouped together both the external pediatric and geriatric clinical experience. Separation of the clinics for this evaluation would allow for further analysis and potentially more targeted improvements. The second drawback was the chosen program evaluation method. Educational programs are closely equated to a social system where there are interactions between the various components (including students and supervising optometrists) and with the environment.⁴ These interactions are not necessarily linear wherein satisfaction with a program leads to better learning which leads to more change in behavior.²³ The Kirkpatrick Model is ideal for evaluating educational program outcomes, which was the intent of this study.

This study has shown that students are satisfied with the outreach experience. Further evaluation is needed to fully appreciate the program’s effectiveness and to determine whether the intended outcomes are achieved. Future studies are planned with the same group of students and the second and third levels of the Kirkpatrick Model of training evaluation.

References
1. Yardley S, Teunissen P, Dornan T. Experiential learning: AMEE Guide No. 63. Med Teach. 2012; 34(2):e102-115.
2. Davis JG. In clinical optometric education, does extern outreach training produce outreach providers? Optom Ed. 2011;37(1):22-27.
3. Harden RM. Outcome-based education – the ostrich, the peacock and the beaver. Med Teach. 2007;29:666-671.
4. Frye AW, Hemmer PA. Program evaluation models and related theories: AMEE Guide No. 67. Med Teach. 2012;34:e288-299.
5. Kirkpatrick D. Great ideas revisited. Training & Development. 1996;50(1):54-59.
6. Keitz SA, Holland GJ, Melander EH, Bosworth HB, Pincus SH. The Veterans Affairs Learners’ Perceptions Survey: the foundation for education quality improvement. Acad Med. 2003;78(9):910-917.
7. Owsley C, McGwin Jr G, Scilley K, Meek C, Seker D, Dyer A. Effect of refractive error correction on health-related quality of life and depression in older nursing home residents. Arch Ophthalmol. 2007;125:1471-1477.
8. Tielsch JM, Javitt JC, Coleman A, Katz J, Sommer A. The prevalence of blindness and visual impairment among nursing home residents in Baltimore. N Engl J Med. 1995;332:1205-1209.
9. Black A, Wood J. Vision and falls. Clin Exp Optom. 2005;88:212-222.
10. Dev MK, Paudel N, Joshi ND, Shah DN, Subba S. Psycho-social impact of visual impairment on health-related quality of life among nursing home residents. BMC Health Serv Res. 2014;14:345.
11. Statistics Canada. Population projections: Canada, the provinces and territories, 2013 to 2063. 2015; [accessed July 9, 2015]. Available from: https://www.statcan.gc.ca/pub/91-520-x/91-520-x2014001-eng.htm.
12. Fitzgerald JT, Wray LA, Halter JB, Williams BC, Supiano MA. Relating medical students’ knowledge, attitudes, and experience to an interest in geriatric medicine. The Gerontologist. 2003;43(6):849-855.
13. Diachun LL, Hillier LM, Stolee P. Interest in geriatric medicine in Canada: How can we secure a next generation of geriatricians? J Am Geriatr Soc. 2006;54:512-519.
14. Scheiman MM, et al. Pediatric Eye and Vision Examination: Reference Guide for Clinicians. St. Louis, MO: American Optometric Association, 2002.
15. Williams C, Northstone K, Harrad RA, Sparrow JM, Harvey I. Amblyopia treatment outcomes after screening before or at age 3 years: follow up from randomised trial. BMJ. 2002;324:1549-51.
16. Schiekirka S, Raupach T. A systematic review of factors influencing student ratings in undergraduate medical education course evaluations. BMC Medical Education. 2015;15:30.
17. Schuwirth L. Is assessment of clinical reasoning still the Holy Grail? Med Educ. 2009;43:298-299.
18. Dornan T, Boshuizen H, King N, Scherpbier A. Experience-based learning: a model linking the processes and outcomes of medical students’ workplace learning. Med Educ. 2007;41:84-91.
19. Zwarenstein M, Goldman J, Reeves S. Interprofessional collaboration: effects of practice-based interventions on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2009 Jul 8;(3):CD000072.
20. Ontario Ministry of Health and Long Term Care. Ontario’s Action Plan for Health Care; c2012 [assessed June 19, 2014]. Available from: https://www.health.gov.on.ca/en/ms/ecfa/healthy_change/docs/rep_healthychange.pdf.
21. Goldman J, Meuser J, Rogers J, Lawrie L, Reeves S. Interprofessional collaboration in family health teams: an Ontario-based study. Can Fam Physician. 2010;56(10):368-74.
22. Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach. 2007 Nov;29(9):855-71.
23. Bates R. A critical analysis of evaluation practice: the Kirkpatrick model and the principle of beneficence. Evaluation and Program Planning. 2004;27:341-347.

 Save article as PDF