Predictive validity evidence of Yes-No Angoff standard setting in a Pre-Clinical medical school curriculum

Abstract Background Standard setting plays a critical role in determining student outcomes by defining the level of biomedical knowledge required to be considered competent. This is especially important for accurately classifying medical students as ready (or not) to progress through the pre-clinical curriculum. In multiple-choice medical knowledge exams, the Yes-No Angoff method may be used for setting passing scores. This method relies on faculty experts’ judgments about whether students with borderline but adequate competence would answer each question correctly. Using a construct validity framework with the construct of academic success defined as the ability of a student to progress without obstacles, we examined the predictive validity of the passing standards set by this method. Methods We analyzed academic success for four pre-clinical semesters across three student cohorts. First, we identified passing standards for pre-clinical courses using the Yes-No Angoff method. Then, we applied binary logistic regression and receiver-operator characteristic (ROC) analyses with area under the curve (AUC) to evaluate passing standards. For binary outcomes, we defined academic success in terms of unimpeded progress through the curriculum and students’ first-attempt passage of the United States Medical Licensing Examination (USMLE) Step 1. Model predictors for ROC analyses included Yes-No Angoff passing standards, Medical College Admissions test scores, and grade point averages for math and science courses. Results ROC analyses showed a low but acceptable area under the curve for a single semester in one cohort and excellent or outstanding AUCs for the remaining 11 semesters. Rates of overall classification accuracy for the Yes-No Angoff passing scores ranged between 89% and 96% for predicting academic success for all pre-clinical semesters across all cohorts. Conclusions The Yes-No Angoff method yielded passing standards that aided in accurately predicting academic success, providing predictive validity evidence for our school’s passing standards in a pre-clinical medical curriculum.