Student Perceptions of Attaining the Association of Schools and Colleges of Optometry Graduate Attributes
Raymond H. Chu, OD, MS, and Stanley Woo, OD, MS, MBA
Education assessment seeks to determine how well students are learning and is an integral part of improving education outcomes. A survey was created and administered to the graduating classes of 2015, 2016 and 2017 to assess whether graduates perceived they successfully attained the Association of Schools and Colleges of Optometry (ASCO) graduate attributes and competencies.1 By incorporating an assessment of the ASCO graduate attributes into the college’s assessment plan, a new data point has been added that incorporates students as stakeholders.
Key Words: assessment, graduate attributes, exit survey, competency
Assessment of learning is a necessary step in providing feedback to students on their learning and providing the institution valuable information for making programmatic changes. Assessment is defined as the process of gathering data to understand changes in students’ knowledge, ability or attitude.2,3 Assessment can take many forms. In the classroom, formative assessment is used to provide ongoing feedback to the instructor on how to improve teaching and to the student on how to improve learning. Likewise, summative assessments are employed in the classroom to assess student learning outcomes, and on a program level to assess program learning outcomes. For example, Standard 1.3 of the Professional Optometric Degree Standards adopted by the Accreditation Council on Optometric Education (ACOE) states: “The program must identify and use outcome measures to evaluate its effectiveness by documenting the extent to which its goals and objectives have been met and must use such assessment to improve its performance.”4 As part of the ACOE standards 1.3 and 1.4, programs are obligated to assess:
- passage rates on the National Board of Examiners in Optometry licensure exams
- graduation rates
- attrition rates
Although these are important metrics for gauging the education effectiveness of an institution, they provide little perspective about the students’ experiences. To address this gap, professions such as allopathic medicine have annually administered the Medical School Graduation Questionnaire (GQ), which serves as a tool for program evaluation and feedback on how to improve the medical student experience.5 Rickards et al. noted the limitation with most graduate medical education surveys is the lack of reliability and validity evidence on the survey tool; however, the GQ has been assumed to be valid because of its employment since 1978 and its many uses within education research.6-9
In 2000, Heath et al. published an initial report outlining attributes of a U.S.-trained optometric graduate.10 Rather than documenting education effectiveness through the number of clock hours spent or resources dedicated to learning, the report took a more contemporary view by focusing on outcomes that graduates are expected to demonstrate. In 2011, the Association of Schools and Colleges of Optometry (ASCO) revised the competencies to reflect current practices.1 Similar to learning objectives, graduate attributes are orienting statements used to describe the profession’s expectations of graduates.11 Institutions of higher learning have adopted the use of graduate attributes as a means of articulating to faculty, accrediting agencies and the workforce community alignment with expected knowledge and skills. Hughes and Barrie advocated for the use of student perceptions and longitudinal studies as part of the assessment of graduate attributes.12 The purpose of this paper is to report on how students perceived their attainment of the ASCO graduate attributes and how the data has been used in curriculum review.
ASCO graduate attributes survey development
“Attributes of Students Graduating from Schools and Colleges of Optometry” contains a series of attribute statements that broadly define entry-level competencies expected of students graduating from a U.S.-based optometry program.1 The document defines competency within three attribute areas: 1) professional values and ethics, 2) knowledge, and 3) skill. To assess students’ self-perceptions of attainment of the ASCO graduate attributes, the skills domain was used because it encompasses both the cognitive and motor skills of a new Doctor of Optometry:1
- all the skills required for the diagnosis, triage, management and/or treatment of common visual conditions, including or resulting from:
- refractive anomalies
- abnormalities of accommodation, monocular or binocular vision skills, oculomotor and sensory/perceptual dysfunctions
- ocular disease and trauma
- prior ocular surgery and/or laser intervention
- systemic disease
- environmental or occupational conditions
- the ability to order and interpret frequently needed laboratory and diagnostic procedures
- the critical-thinking skills needed to assess the patient’s visual and physical status and to interpret and process the data to formulate and execute effective management plans
- the ability to prescribe or use ophthalmic materials, contact lenses, vision therapy, low vision devices, pharmaceuticals and certain surgical procedures to treat and manage vision disorders and disease
- an understanding of nutritional influences on ocular physiology and systemic health and disease
- the ability to understand, evaluate and apply the use of contemporary imaging technologies in the provision of eye and vision care
- the ability to recognize and initiate the coordination of patient care requiring advanced medical, systemic, interprofessional or specialty care
- the ability to recognize life-threatening conditions and to initiate immediate intervention
- effective communication skills, both oral and written, as appropriate for maximizing successful patient care outcomes
- the ability to appropriately use all resources, including the use of ancillary personnel, intra- and interprofessional collaboration, co-management and referral, in ensuring the best quality patient care
- the ability to access evidence-based knowledge (including through the use of information technology) and manage information, and to apply that information in making decisions about patient care and healthcare delivery
- the ability to embrace the cultural diversity and individual differences that characterize patients, populations and the healthcare team
- the ability to work in cooperation with those who receive care, those who provide care, and others who contribute to or support the delivery of prevention and health services
Ten survey items (Table 1) were created from the skills domain list and were embedded within the Southern California College of Optometry at Marshall B. Ketchum University (SCCO) Graduating Class Exit Survey where students responded based on a four-point Likert scale ranging from Strongly Disagree, Disagree, Agree and Strongly Agree.
Survey development and deployment
The SCCO Graduating Class Exit Survey was designed to learn from the graduating students’ opinions about their didactic and clinical education, career aspirations, satisfaction with student affairs services, and treatment during optometry school. The questionnaire was based on the types of questions asked in the GQ, which include:5
- pre-clinical, clinical and elective experiences
- general medical education and readiness for residency
- student services
- experiences of negative behaviors
- financial aid and indebtedness
- career intentions
- strengths of the medical school and areas that need improvement
Beginning in the 2014-2015 academic year, students completed the SCCO Graduating Class Exit Survey (Appendix A). Students from the graduating classes of 2015, 2016 and 2017 were invited to respond to the exit survey. The class of 2015 had 96 members, the class of 2016 had 96 members, and the class of 2017 had 101 members. Students volunteered to respond to the exit survey during the week leading up to the commencement ceremony (May of each year) and were assured their responses would be anonymous. The survey was posted on the MyCoursEval (Invoke Solutions, Waltham, MA) portal for each class. The survey remained open until the day after the ceremony, with an initial invitation and two email reminders. The process was the same for all classes.
An application was submitted to the Institutional Review Board at Marshall B. Ketchum University, and the research was found to be exempt due to the anonymity of the survey.
Descriptive statistics (mean, distribution and standard deviation) for each survey item were generated by the MyCoursEval software. Questions using a four-point Likert scale were designated with “Strongly Agree” equal to a numerical score of four and “Strongly Disagree” equal to a numerical score of one. A one-way analysis of variance (ANOVA) was performed to assess statistical significance between the three graduating classes with statistical significance set at a p-value less than 0.05.
Survey responses from all three graduating classes were also aggregated to assess areas of strengths and weaknesses with the following definitions:
- 90% or above of responders agreeing or strongly agreeing with a survey statement = an area of strength
- 80-89% percent of responders agreeing or strongly agreeing with a survey statement = an area to monitor
- less than 80% of responders agreeing or strongly agreeing with a survey statement = an area to focus on change
Refer to Table 2 for the survey response rates from each class. From the class of 2015, 76 responses (79%) were recorded. From the class of 2016, 59 responses (61%) were recorded. From the
class of 2017, 29 responses (29%) were recorded.
Based on ASCO Annual Student Data Reports, the demographics of the SCCO class of 2015 were 66.7% (n=64) female and 33.3% (n=32) male.13 The ethnic distribution of the class was 67.7% (n=65) Asian, 27% (n=26) White, and 5.2% (n=5) Black/Latino/other/unknown. The demographics of the class of 2016 were 77.1% (n=74) female and 22.9% (n=22) male. The ethnic distribution of the class was 46.9% (n=45) Asian, 34.4% (n=33) White, and 18.7% (n=18) Black/Latino/other/unknown. The demographics of the class of 2017 were 68.4% (n=67) female and 31.6% (n=31) male. The ethnic distribution of the class was 51.0% (n=50) Asian, 35.7% (n=35) White, and 13.3% (n=13) Black/Latino/other/unknown.
ASCO graduate attributes
Mean responses on the attainment of the ASCO Attributes of Students Graduating from Schools and Colleges of Optometry were reported by class year in Figure 1. Each class reported the highest mean (3.4 to 3.6) with question 17 (practice in a professional and ethical manner) and the lowest mean (2.4 to 2.5) with question 12 (order and interpret laboratory and diagnostic procedures).
The distribution of student responses were reported in Table 3 along with the results from the one-way ANOVA analysis. The responses for individual questions were not statistically different among the 2015, 2016 and 2017 graduating classes (p=0.44 to 0.69).
When all responses for each survey item were combined (Figure 2), students indicated the following as areas of strength (≥90% strongly agree or agree):
- question 10 (95%): prescribe or use ophthalmic materials, contact lenses, vision therapy, low vision devices, pharmaceuticals and surgical procedures to treat and manage vision disorders and disease
- question 14 (96%): use written and oral communication that is understandable to patients, families and other healthcare team members
- question 16 (90%): engage in continuous professional and interprofessional development
- question 17 (99%): practice in a professional and ethical manner
- question 18 (95%): promote wellness and disease prevention services
The highest response came from practice in a professional and ethical manner (99%).
Students indicated the following as areas to monitor (80-89% strongly agree or agree):
- question 13 (82%): understand, evaluate and apply the use of contemporary imaging technologies in the provision of eye and vision care
- question 15 (83%): access evidence-based knowledge, manage information, and to apply that information in making decisions about patient care and healthcare delivery
Students indicated the following as areas needing improvement (<80% strongly agree or agree):
- question 11 (79%): recognize and initiate the coordination of patient care requiring advanced medical, systemic, interprofessional or specialty care
- question 12 (46%): order and interpret laboratory and diagnostic procedures
- question 19 (74%): work within an interprofessional collaborative team to improve patient outcomes
The two lowest responses came from ordering and interpreting laboratory and diagnostic procedures (46%) and working within an interprofessional collaborative team (74%).
The American Optometric Association defines an optometrist as an independent healthcare provider who examines, diagnoses treats and manages diseases, injuries and disorders of the visual system, the eye and associated structures as well as identifies related systemic conditions affecting the eye.14 Based on the results from the SCCO Graduating Class Exit Survey, students graduating from SCCO felt competent in fulfilling the core responsibilities of a Doctor of Optometry. Most students, but not all, felt competent with imaging technologies and implementing evidenced-based practice.
Working within an interprofessional collaborative team and recognizing and initiating the coordination of patient care requiring advanced medical, systemic, interprofessional or specialty care were areas identified for curriculum improvement. The mission of SCCO is to “educate caring, inspired healthcare professionals who are prepared to deliver collaborative, patient-centric health care in an interprofessional environment.”15 In fulfilling the mission, students within the three programs at Marshall B. Ketchum University ― SCCO, College of Pharmacy, and School of Physician Assistant Studies ― began attending classes together in the 2015-2016 academic year in an effort to prepare future graduates for interprofessional collaborative practice. Students were enrolled in Medical Ethics (first year), Population and Public Health (first year), Evidence-Based Practice (second year), and Interprofessional Case Conferences (third year) so they could learn with, from and about each other.16 In addition to classroom learning, complementary clinical experiences that model interprofessional collaborative practice are being developed. Because the students from the graduating classes of 2015, 2016 and 2017 did not complete the interprofessional education curriculum, the results from questions 11 and 19, coordination of care and work within an interprofessional collaborative team, serve as a baseline for evaluating the effectiveness of the interprofessional education curriculum.
Similar to the ASCO graduate attributes, the Medical School Objectives Project was developed by the Association of American Medical Colleges to describe the skills, attitudes and knowledge a graduating medical student should possess.17 Promes et al. administered a survey to first-year medical residents and found that the variability in undergraduate medical school curricula resulted in varying levels of competence.18 Sanders et al. made a similar observation when surveying medical school associate deans for academic affairs where the teaching and assessment of technical procedures had differing levels of rigor.19 Question 12, order and interpret laboratory and diagnostic procedures, was identified as a skill needing additional instruction. Learning from the lessons from medicine, collaboration within the university’s other health professions, identifying and monitoring opportunities within clinical externship, and more rigorous forms of assessment are all being considered.
Although the administration of the exit survey was exactly the same for each class year, the response rates varied. The especially low response rate for the class of 2017 may have been the result of this cohort being particularly non-responsive to survey inquiries. Despite the different response rates and diminished number of responses with each class year (2015: 79%, 2016: 61%, 2017: 29%), there was good agreement among the responders to the survey items.
The results of the survey have limited generalizability due to the response rate and the sampling from one program’s curriculum.
Chen et al. recommended that reform to medical education should be empirically based, and noted that little is known about how graduates feel regarding preparation for work and life as medical residents.20 For SCCO, assessing the ASCO graduate attributes with other assessment data has helped provide a more holistic view in triangulating program learning outcomes.
Gehlbach et al. recommended a seven-step process for education research survey development, which includes a final step of pilot testing.21 The information learned from the SCCO Graduating Class Exit Survey can serve as pilot testing for future study consideration expanding the assessment of graduate attributes across graduating students from all ASCO member institutions.
- Attributes of Students Graduating from Schools and Colleges of Optometry [Internet]. Rockville, MD: Association of Schools and Colleges of Optometry; published Oct. 11, 2011 [cited 2016 Oct 13]. Available from: http://www.opted.org/files/2011_AttributesReport.pdf.
- Huba ME, Freed JE. Learner-centered assessment on college campuses: shifting the focus from teaching to learning. Needham Heights, MA: Allyn & Bacon; 2000. 286 p.
- Harden RM. AMEE Guide No. 14: outcome-based education: part 1-an introduction to outcome-based education. Medical Teacher. 1999;21(1):7-14.
- Professional Optometric Degree Standards [Internet]. Accreditation Council on Optometric Education; [cited 2021 Oct 4]. Available from: https://www.aoa.org/education/accreditation-council-on-optometric-education/accreditation-resources-and-directories-of-programs?sso=y#Degree.
- Graduation Questionnaire (GQ) [Internet]. Washington, DC: Association of American Medical Colleges; [cited 2018 Feb 7]. Available from: https://www.aamc.org/data/gq/.
- Rickards G, Magee C, Artino AR. You can’t fix by analysis what you’ve spoiled by design: developing survey instruments and collecting validity evidence. J Grad Med Educ. 2012 Dec;4(4):407-10.
- Pugnaire MP, Purwono U, Zanetti ML, Carlin MM. Tracking the longitudinal stability of medical students’ perceptions using the AAMC graduation questionnaire and serial evaluation surveys. Acad Med. 2004;79(10 Suppl):S32-35.
- Hodgson CS, Teherani A, Guiton G, Wilkerson L. The relationship between student anonymity and responses from two medical schools on the Association of American Medical Colleges’ graduation questionnaire. Acad Med. 2002 Oct;77(10 Suppl):S48-50.
- Dial TH, Lindley DW. Predictive validity of specialty choice data from AAMC graduation questionnaire. Association of Medical Colleges. J Med Educ. 1987 Dec;62(12):955-8.
- Heath D, Daum K, DiStefano A, Haine C, Schwartz S. Attributes of students graduating from schools and colleges of optometry. Optometric Education. Fall 2000;26(1):15-18.
- Barrie S, Hughes C, Smith C. The national graduate attributes report: integration and assessment of graduate attributes in curriculum (Research Report). Australian Learning and Teaching Council; 2009.
- Hughes C, Barrie S. Influences on the assessment of graduate attributes in higher education. Assessment & Evaluation in Higher Education. 2010;35(3):325-334.
- Annual Student Data Reports [Internet]. Rockville, MD: Association of Schools and Colleges of Optometry; [cited 2020 Nov 6]. Available from: https://optometriceducation.org/student-data-reports/.
- What is a Doctor of Optometry? [Internet]. St. Louis, MO: American Optometric Association; [cited 2021 Oct 4]. Available from: https://www.aoa.org/about-the-aoa/what-is-a-doctor-of-optometry?sso=y.
- About Us [Internet]. Fullerton, CA: Marshall B. Ketchum University; [cited 2018 Sept 4]. Available from: http://www.ketchum.edu/about.
- Chu R. The effect on knowledge and attitude of an interprofessional education curriculum for optometry and physician assistant students. Optometric Education. 2016;41(3):12-17.
- Medical School Objectives Project (MSOP) [Internet]. Washington, DC: Association of American Medical Colleges; c2021 [cited 2018 June 12.] Available from: https://www.aamc.org/initiatives/msop/.
- Promes SB, Chudgar SM, Grochowski CO, Shayne P, Isenhour J, Glickman SW, Cairns CB. Gaps in procedural experience and competency in medical school graduates. Acad Emerg Med. 2009 Dec;16 Suppl 2:S58-62.
- Sanders CW, Edwards JC, Burdenski TK. A survey of basic technical skills of medical students. Acad Med. 2004 Sep;79(9):873-5.
- Chen CA, Kotliar D, Drolet BC. Medical education in the United States: do residents feel prepared? Perspect Med Educ. 2015 Aug;4(4):181-5.
- Gehlbach H, Artino AR Jr, Durning S. AM last page: survey development guidance for medical education researchers. Acad Med. 2010 May;85(5):925.