The effects of the number of scale points and non-normality on the generalizability coefficient: A Monte Carlo study

Steven R. Shumate, James Surles, Robert L. Johnson, Jim Penny

Research output: Contribution to journalArticle

7 Scopus citations


Increasingly, assessment practitioners use generalizability coefficients to estimate the reliability of scores from performance tasks. Little research, however, examines the relation between the estimation of generalizability coefficients and the number of rubric scale points and score distributions. The purpose of the present research is to inform assessment practitioners of (a) the optimum number of scale points necessary to achieve the best estimates of generalizability coefficients and (b) the possible biases of generalizability coefficients when the distribution of scores is non-normal. Results from this study indicate that the number of scale points substantially affects the generalizability estimates. Generalizability estimates increase as scale points increase, with little bias after scales reach 12 points. Score distributions had little effect on generalizability estimates.

Original languageEnglish
Pages (from-to)357-376
Number of pages20
JournalApplied Measurement in Education
Issue number4
StatePublished - Sep 13 2007


Cite this