Abstract Body

Background: Assessment is fundamental to learning yet many of us feel uncertain about the role of assessment in our daily work. Assessment theory can help educators to understand how to match tools to purpose, what tools can and cannot tell us, how to design effective programmatic assessment, and how to consider the concepts of assessment for learning, assessment as learning, and assessment of learning in designing assessment programs. The best assessment programs meet two needs: 1) support learner progress towards clinical competence; and 2) result in rigorous and accountable assessment data.

Aim: This session will help translate assessment principles and theories into practical day to day solutions for learner assessment, as well as offer guidance in how to design an overall programmatic assessment approach.

Learning Objectives: By the end of this workshop, participants will be able to:

1. Describe the basic principles of assessment.

2. Apply the principles of assessment of, assessment for, and assessment as learning to improve teaching, assessment, and learning.

3. Implement strategies to begin designing programmatic assessment for their home program.

Methods and Timetable: This workshop combines didactic components with practical activities that will allow participants to apply what they are learning through discussion, polls, and case examples. There will be three 10 minute didactic portions, alternating with three 15 minute interactive portions. The remaining 15 minutes will include introductions at the beginning, and a wrap up of key learnings and a “homework assignment” at the end.

Proposed result: Participants will leave the workshop with an assessment plan to enhance or complement assessment in their home program.

Abstract Body

Background and purpose: The shift to competency-based medical education has not always been smooth. Effective competency-based assessment (CBA) is challenging, with scant evidence about its effectiveness. In 2010, Canadian family medicine residency training programs adopted the Continuous Reflective Assessment for Training (CRAFT) model of assessment. Evaluation data about the effectiveness of CRAFT is now available, and can inform implementation of CBA in other programs.

Methods: The intervention: CRAFT, a programmatic assessment model, involves regular low stakes workplace assessments and regular high stakes performance reviews. In performance reviews, learners, guided by a continuous advisor, reflect on their progress and develop proactive learning plans for their next period of training. Participants: Canadian family medicine residency programs (N=4). Research design: Mixed methods. Primary data sources are learning analytics, questionnaires, and focus groups. Outcome measures: Differentiation between learners at different levels of training; range of assessment information beyond Medical Expert role; evidence of self-reflection by learners; evidence of learning plans across training; increase in quality of feedback. Analysis: Descriptive statistics, thematic analysis of qualitative data, ANOVAs, Chi square tests.

Results: Learners and preceptors report a significant increase in self-reflection opportunities. All programs report moderate to large increases in feedback quality and in assessment data about non-Medical Expert CanMEDS roles. One program reports significant increases in early identification and remediation of learners in difficulty. Programs report varying levels of implementation of learning plans. For all programs, faculty development is the biggest challenge in effective use of CRAFT.

Conclusion: Overall, CRAFT appears to be an effective approach to programmatic assessment.