A Comparison of Scoring Strategies for Operational Situational Judgment Tests

Abstract

The use of situational judgment tests (SJTs) has increased recently. SJTs offer many advantages compared to other assessments, including validity, flexibility in administration, objective scoring, and reduced mean group differences. While use has increased (Roth, Bobko, & Buster, 2013), there is little consensus regarding SJT scoring (Bergman et al., 2006). This study used data from operational SJTs to evaluate and compare scoring strategies in terms of convergence, reliability, mean group differences, and candidate rank order. Results demonstrate that different scoring strategies can produce wildly different rank orders and can contribute to differences in reliability values and mean group test scores.