The effects of the number of scale points and non-normality on the generalizability coefficient: A Monte Carlo study

Steven R. Shumate, James Surles, Robert L. Johnson, Jim Penny

Research output: Contribution to journalArticlepeer-review

7 Scopus citations

Abstract

Increasingly, assessment practitioners use generalizability coefficients to estimate the reliability of scores from performance tasks. Little research, however, examines the relation between the estimation of generalizability coefficients and the number of rubric scale points and score distributions. The purpose of the present research is to inform assessment practitioners of (a) the optimum number of scale points necessary to achieve the best estimates of generalizability coefficients and (b) the possible biases of generalizability coefficients when the distribution of scores is non-normal. Results from this study indicate that the number of scale points substantially affects the generalizability estimates. Generalizability estimates increase as scale points increase, with little bias after scales reach 12 points. Score distributions had little effect on generalizability estimates.

Original languageEnglish
Pages (from-to)357-376
Number of pages20
JournalApplied Measurement in Education
Volume20
Issue number4
DOIs
StatePublished - Sep 13 2007

Fingerprint Dive into the research topics of 'The effects of the number of scale points and non-normality on the generalizability coefficient: A Monte Carlo study'. Together they form a unique fingerprint.

Cite this