Comparing self, peer, and faculty assessments within an interprofessional hybrid course

Introduction

Effective interprofessional communication skills are critical in achieving efficient and safe care of patients.¹ Written consultation and referral requests are the most common methods by which healthcare providers communicate crucial patient information.² Poor communication can lead to increased healthcare costs and delays or even errors in diagnosis.³ Thus, it is essential for health professional students, including optometry students, to practice and receive feedback on writing consultations/referrals prior to direct involvement in the clinical care of patients. As stated by the Association of Schools and Colleges of Optometry, schools and colleges of optometry shall ensure that before graduation each student will have demonstrated “effective communication skills, both oral and written, as appropriate for maximizing successful patient care outcomes”.⁴

Providing effective and timely feedback from faculty regarding communication skills can be difficult, particularly in courses with large student enrollment. Self- and peer assessments have been shown to serve as learning tools to stimulate student participation, allow students to gain insight into their own performance, promote self-directed lifelong learning,^5-8 build reflection and self-awareness skills,⁹ and provide constructive feedback.¹⁰ The ability to understand ones’ own strengths and weaknesses through the process of self-assessment is an important component in health professions training.^11,12

While self- and peer assessments have been shown to be effective learning tools, there are mixed impressions regarding their use as an assessment tool.^6,7,13-17 The goal of this study is to assess for similarities and differences between optometry students’ self-assessments of written interprofessional communication skills compared to assessments from peers and faculty to thus evaluate the effectiveness of self- and peer assessment as a reliable assessment tool in this context.

Methods

Description of the Course

Team Training in Healthcare I (IPE 6000) is a course at Western University of Health Sciences. During the Fall 2015 delivery of IPE 6000, the course had over 850 students enrolled across eight health professions programs (dental medicine, graduate nursing, optometry, osteopathic medicine, pharmacy, physical therapy, podiatric medicine and veterinary medicine) on two campuses (Pomona, California, and Lebanon, Oregon) and was mainly delivered online. Students enrolled in IPE 6000 completed written consultations/referrals based on a case created by faculty from students’ respective professions. The process and rubric were introduced to students through a video briefing session.

Self- and Peer Assessments

Table 1: SBAR tool. Click to enlarge

After students uploaded their consultations/referrals onto a Learning Management System (Blackboard), they were asked to provide self-assessments and anonymous peer assessments from two health professions students enrolled in the course using a generalized rubric focused on interprofessional communication skills. Given the randomization process, peer assessments were completed by students from any of the enrolled eight health professions programs. Thus, the peer feedback could have been done by an optometry or a non-optometry student. The design of the rubric was based on the SBAR (Situation-Background-Assessment-Recommendation) tool from the Institute for Healthcare Improvement (Table 1)¹⁸ and the core competencies from the Interprofessional Education Collaborative.¹⁹ Consultations/referrals written by optometry students only were evaluated in this study.

Table 2: Rubric categories. Click to enlarge

The specific rubric categories were Situation, Background, Assessment, Recommendation, Roles and Responsibilities, Organization, Relationship-building and Values and Ethics (Table 2). Ideal descriptions for each category were provided. For each rubric category, students had to rate performance as Ineffective, Somewhat effective or Effective. Students were provided with the opportunity to provide qualitative feedback for each category, asked to provide an overall rating and encouraged to provide overall qualitative feedback.

Students subsequently received peer feedback on their own consultation/referral and a video debrief with overall feedback from the instructor of record (Yumori) was provided after the completion of each case. Figure 1 summarizes the student perspective of the consultation/referral process.

Figure 1: Student perspective of the consultation/referral process. Click to enlarge

Faculty Assessment

Two college of optometry faculty members (Yumori and Napier-Dovorany) completed faculty assessments on consultations/referrals written by second-year optometry students enrolled in the Fall 2015 delivery of IPE 6000. To reduce inter-rater variability, the faculty members separately evaluated three consultations/referrals using the same rubric as used by students (Figure 1) and then discussed scores. After this step, each faculty member then independently evaluated 29 consultations/referrals using the same rubric. Consultations/referrals were de-identified prior to the faculty assessment process.

Statistical Analysis

With identifying information removed, a total of 58 consultations/referrals written by optometry students were included for this study. Ten consultations were excluded from analysis due to missing self- or peer assessments, and 48 consultations/referrals were used for analysis. For each consultation/referral, optometry students’ self-assessment was given a numeric composite score, with one point for Ineffective, two points for Somewhat effective and three points for Effective across each of the eight rubric categories (Situation, Background, Assessment, Recommendation, Roles and Responsibilities, Organization, Relationship-building and Values and Ethics). This process was used to calculate the numeric composite score from each peer (two total) and from the faculty assessment. The Peer Average was calculated by averaging the numeric composite score from both peer assessments. Descriptive statistics and intra-class correlation coefficient were applied comparing scores between Self and Peer Averages, Peers (Peer 1 vs Peer 2), Self and Faculty, and Peer Average and Faculty. Scores were also categorized (23-24, 21-22, 19-20, and 18 or lower) and Cohen’s kappa statistics and 95% confidence intervals were calculated to assess for agreement between Self and Peer Average, Peer 1 and Peer 2, Self and Faculty, and Peer Average and Faculty. This study was approved by to the Institutional Review Board at Western University of Health Sciences.

Results

Overall, mean (SD) composite scores were relatively high (23.5 (0.8) for Self, 23.2 (0.7) for Peer Average, and 20.7 (2.2) for Faculty), with Self and Peers assessments tending to show higher scores compared to Faculty (Figure 2).

Figure 2: Distribution of consultation/referral scores. Click to enlarge

Intra-class correlation coefficients using composite scores were “highest” between Peers (Peer 1 and Peer 2) at 0.13, followed by Self and Peer Average at 0.12, Peer Average and Faculty at 0.06, and lowest for Self and Faculty scores at -0.01.

Table 3: Inter-observer agreement. Click to enlarge

When composite scores were categorized (23-24, 21-22, 19-20, and 18 or lower), Cohen’s kappa statistics and 95% confidence intervals were 0.11 (-0.25-0.46), -0.01 (-0.27-0.25), and 0.001 (-0.05-0.05) for Self and Peer Average, Peer 1 and Peer 2, and Peer Average and Faculty, respectively (Table 3).

Agreement differences between the Peer Average and Faculty scores varied based on rubric categories (Figure 3). Higher agreement was seen with the Values and Ethics (100%), Relationship-building (98%), and Organization (91%) categories. Lower levels of agreement were seen between the Peer Average and Faculty scores on the Assessment (30%), Background (35%), Situation (43%), Recommendation (48%), and Roles and Responsibilities (54%) categories.

Figure 3: Percentage of agreement for each domain between Peer Average and Faculty. Click to enlarge

Discussion

In this study, which compared self-, peer and optometry faculty assessments of consultations/referrals written by second-year optometry students within an interprofessional education course, overall, scores were relatively high, with generally higher scores from self-assessments, followed by peer assessments and lastly assessments from faculty. The trend of students tending to self-rate their communication abilities high is common, as seen in by Chu and Woo, where 96% of optometry students self-rated “use of written and oral communication that is understandable to patients, families and other healthcare team members” as an area of strength.²⁰ Papinczak, Young, Groves and Haynes highlight that students with greater self-efficacy tend to score their performance higher and found moderate correlations between peer assessments and tutor.⁹ This is similar to our statistical agreement, with the “highest” intra-class correlation coefficient between peers and lowest between self and faculty. However, given that stratification was done by category with good agreement in organization, Relationship-building and Values and Ethics and lower agreement in the SBAR dimensions, there is poor overall agreement. This difference in agreement is likely due to the shared perspectives peers encompass compared to the differences in level of development and experience between students and faculty. Over-estimation common in self-grading may also have been a contributing factor.²¹

Weak agreement between self-, peer and faculty assessments may also be partly based on differences across rubric categories, with more agreement seen between peers and faculty in more subjective categories (Values and Ethics, Relationship-building and Organization). Such categories may thus be appropriate to assign to peers from an assessment perspective, allowing faculty to dedicate more time in providing feedback to other sections that perhaps require more experience. While self- and peer assessments may not align closely with faculty evaluations and self-assessment accuracy can change over time,^22,23 these strategies offer benefits that should not be overlooked. Self-grading has been shown to promote self-regulation of learning,²⁴ improve metacognition,²¹ and support lifelong learning,²⁵ all of which are crucial skills for healthcare professionals, including optometrists. Peer assessment helps students develop critical evaluation skills and exposes them to diverse perspectives, which is essential for effective interprofessional communication. Both self- and peer assessments allow students to be more active and motivated throughout the learning process,²⁶ especially in lower stakes learning opportunities. Furthermore, value may lie in the learning process itself rather than in producing scores that match faculty grading. These methods can serve as powerful complementary tools to faculty assessment, particularly in large courses where timely, individualized feedback from instructors may be challenging to provide.

When composite scores were categorized little to no agreement was seen, with the no agreement seen between self and faculty, between peers and between the peer average and faculty assessments; slight agreement was seen between the peer average and self-assessments.²⁷ We believe the weak to no agreement may be due in part to ceiling effects. Trudel and colleagues shared two definitions of ceiling effects:²⁸ the first is “an intervention having limited effect because the population is already at, or near, a pinnacle point,”²⁹ which may fit with our study that involved optometry students, who must be highly qualified academically in order to enter an optometry program.³⁰

The second definition Trudel and colleagues reference regarding ceiling effects,²⁸ those caused by “the limitation of an assessment to capture the extent and variance of accomplishment because essentially the assessment is too simplistic,”²⁹ is also appropriate. It is possible that the profession-specific case the second-year optometry students based their consultation/referrals on was too simple and/or the rubric was not specific or sensitive enough to capture meaningful differences. Selecting a more complex case and using more specific assessment tools such as checklists, which provide more structured guidance to students, may correct this measurement problem that limits our ability to accurately assess.^31,32 Other course-specific characteristics, such as providing more extensive training session²³ and more detailed rubric criteria,³³ may additionally be influential in students’ accuracy in self- and peer scoring, with faculty scores as comparison.

Future directions involve enhancing the rubric design by incorporating detailed descriptors that clearly define the criteria for each performance level: Ineffective, Somewhat effective and Effective. Capturing student-specific characteristics such as overall academic performance, gender and self-efficacy and competency to see whether these affect self-, peer and faculty assessments of optometry students’ interprofessional communication skills may also be helpful. Additional future directions include assessing optometry students’ performance as they gain more clinical experience, comparing assessments from and of optometry students across different institutions, and to expand this work beyond the profession of optometry to allow comparisons between different professions.

It is important to note that based on the course and Learning Management System design, it was not possible for us to control for which health professional student evaluated which consultation/referral. It would be exciting to see whether there is a difference in assessments from optometry versus non-optometry students and furthermore between optometry faculty members (as seen in this study) compared to faculty from other health professions. Other areas to explore include featuring other types of clinical cases to see whether more complex cases demonstrate less influence from ceiling effects and any differences between written and verbal interprofessional communication.

Moving forward, educators should consider implementing a balanced approach that combines faculty, self- and peer assessments. This multi-faceted strategy can provide students with a more comprehensive learning experience, fostering critical skills such as self-reflection, peer evaluation and interprofessional communication. However, it is crucial to provide clear grading criteria and training for students to maximize the effectiveness of self- and peer assessments.

Conclusion

This study highlights the challenges of relying solely on self- and peer assessments in interprofessional communication skills training, especially in large courses. The weak correlations between faculty, self- and peer evaluations suggest students may lack the objectivity or experience to accurately assess their own and their peers’ performance in these complex skills.

Instructors need to re-evaluate how we use self- and peer assessments, perhaps shifting towards incorporating more structured assessment rubrics, providing explicit training on evaluation criteria, and increasing direct faculty feedback, even in large enrollment courses. Thoughts about breaking down communication skills into smaller, more observable behaviors that are easier for students to evaluate may be appropriate.

This experience furthermore underscores the need to design activities that specifically encourage students to calibrate their self-perception against expert feedback. If self- and peer assessment are maintained as valuable components of an IPE course, students can be advised to reflect on expert observations and feedback so that they may be more calibrated in future iterations of assessment.

Ultimately, if there is a desire for students to develop effective interprofessional communication skills, assessment methods need to be valid, reliable and provide meaningful feedback that drives improvement. This will allow students to become active, self-directed lifelong learners, build reflection and self-awareness skills and provide constructive feedback, all of which are crucial skills to develop in optometry and all health professional students.