Validity evidence for programmatic assessment in competency-based education

Introduction Competency-based education (CBE) is now pervasive in health professions education. A foundational principle of CBE is to assess and identify the progression of competency development in students over time. It has been argued that aprogrammatic approach to assessment in CBE maximizes student learning. The aim of this study is to investigate if programmatic assessment, i. e., asystem of assessment, can be used within aCBE framework to track progression of student learning within and across competencies over time. Methods Three workplace-based assessment methods were used to measure the same seven competency domains. We performed aretrospective quantitative analysis of 327,974 assessment data points from 16,575 completed assessment forms from 962 students over 124weeks using both descriptive (visualization) and modelling (inferential) analyses. This included multilevel random coefficient modelling and generalizability theory. Results Random coefficient modelling indicated that variance due to differences in inter-student performance was highest (40%). The reliability coefficients of scores from assessment methods ranged from 0.86 to 0.90. Method and competency variance components were in the small-to-moderate range. Discussion The current validation evidence provides cause for optimism regarding the explicit development and implementation of aprogram of assessment within CBE. The majority of the variance in scores appears to be student-related and reliable, supporting the psychometric properties as well as both formative and summative score applications.