Replications in Human Factors research: Implications for education

Replication research benefits the scientific community for two primary reasons. First, it demonstrates that a given finding can or cannot be repeated. This allows researchers to adjust their confidence in that finding accordingly (Hendrick, 1990). By doing so, replications allow scientific knowledge to be cumulative (Fisher, 1974; Kelly, 2006; Nickerson, 2000; Rosenthal, 1993). Second, replications sometimes demonstrate that a given finding does or does not generalize. Specifically, replications that employ different manipulations and procedures can demonstrate whether a given finding generalizes to different situations (Lindsay & Ehrenberg, 1993). This allows researchers to understand where a given finding might or might not be applicable. The current consensus suggests that replication research is uncommon outside of the "hard" sciences for a variety of reasons. Many believe that they lack the time, subjects, or funds necessary to reproduce original findings (Smith, 1970). Also, conducting a replication presents a risk that the study might produce different results (Lindsay & Ehrenberg, 1993). Lastly, by emphasizing the production of original findings, journal editors may discourage researchers from pursuing replications (Lindsay & Ehrenberg, 1993). If the consensus is correct, then there are two primary implications for the education and training of Human Factors professionals. First, Human Factors professionals should be taught to limit their confidence in a finding until they have established that it has been replicated. Second, Human Factors professionals should be taught more about the importance of replications and how to conduct them. However, we do not currently know how often replications are conducted in Human Factors. This study is the first to investigate the extent to which Human Factors professionals conduct replications. To do so, eight articles (hereafter referred to as parent articles) were selected from the 1991 issues of the journal, Human Factors. Each article that had referenced one of the eight parent articles between 1991 and September 2006 (hereafter referred to child articles) were also retrieved (N - 127). Two investigators coded and compared each child article against its 1991 parent article to determine whether the child article replicated its parent article. Our results indicated that the majority of the parent articles were replicated 1 to 10 times. In general, parent articles were more likely to be replicated when they had a large number of child articles. These replications were conducted despite the belief that social scientists rarely replicate previous research (Hendrick, 1990; Kelly, 2006; Lindsay & Ehrenberg, 1993; Schneider, 2004; Smith, 1970). This is a positive finding for two reasons. First, Human Factors professionals are replicating despite the constraints previously mentioned (Lindsay & Ehrenberg, 1993; Smith, 1970). Second, it implies that Human Factors professionals are being educated about the importance of replication research. However, there is one negative aspect of our findings. Our results revealed that not all original findings are being replicated. Therefore, Human Factors professionals should be taught to limit their confidence in previous research that has not been replicated.