An evaluation of mutation and data-flow testing: A meta-analysis

Mutation testing is a fault-based testing technique for assessing the adequacy of test cases in detecting synthetic faulty versions injected to the original program. The empirical studies report the effectiveness of mutation testing. However, the inefficiency of mutation testing has been the major drawback of this testing technique. Though a number of studies compare mutation to data flow testing, the summary statistics for measuring the magnitude order of effectiveness and efficiency of these two testing techniques has not been discussed in literature. In addition, the validity of each individual study is subject to external threats making it hard to draw any general conclusion based solely on a single study. This paper introduces a novel meta-analytical approach to quantify and compare mutation and data flow testing techniques based on findings reported in research articles. We report the results of two statistical meta-analyses performed on comparing and measuring the effectiveness as well as efficiency of mutation and data-flow testing based on relevant empirical studies. We focus on the results of three empirical research articles selected from the premier venues with their focus on comparing these two testing techniques. The results show that mutation is at least two times more effective than data-flow testing, i.e., odds ratio= 2.27. However, mutation is three times less efficient than data-flow testing, i.e., odds ratio= 2.94.