TY - GEN
T1 - An evaluation of mutation and data-flow testing
AU - Kakarla, Sahitya
AU - Momotaz, Selina
AU - Namin, Akbar Siami
PY - 2011
Y1 - 2011
N2 - Mutation testing is a fault-based testing technique for assessing the adequacy of test cases in detecting synthetic faulty versions injected to the original program. The empirical studies report the effectiveness of mutation testing. However, the inefficiency of mutation testing has been the major drawback of this testing technique. Though a number of studies compare mutation to data flow testing, the summary statistics for measuring the magnitude order of effectiveness and efficiency of these two testing techniques has not been discussed in literature. In addition, the validity of each individual study is subject to external threats making it hard to draw any general conclusion based solely on a single study. This paper introduces a novel meta-analytical approach to quantify and compare mutation and data flow testing techniques based on findings reported in research articles. We report the results of two statistical meta-analyses performed on comparing and measuring the effectiveness as well as efficiency of mutation and data-flow testing based on relevant empirical studies. We focus on the results of three empirical research articles selected from the premier venues with their focus on comparing these two testing techniques. The results show that mutation is at least two times more effective than data-flow testing, i.e., odds ratio= 2.27. However, mutation is three times less efficient than data-flow testing, i.e., odds ratio= 2.94.
AB - Mutation testing is a fault-based testing technique for assessing the adequacy of test cases in detecting synthetic faulty versions injected to the original program. The empirical studies report the effectiveness of mutation testing. However, the inefficiency of mutation testing has been the major drawback of this testing technique. Though a number of studies compare mutation to data flow testing, the summary statistics for measuring the magnitude order of effectiveness and efficiency of these two testing techniques has not been discussed in literature. In addition, the validity of each individual study is subject to external threats making it hard to draw any general conclusion based solely on a single study. This paper introduces a novel meta-analytical approach to quantify and compare mutation and data flow testing techniques based on findings reported in research articles. We report the results of two statistical meta-analyses performed on comparing and measuring the effectiveness as well as efficiency of mutation and data-flow testing based on relevant empirical studies. We focus on the results of three empirical research articles selected from the premier venues with their focus on comparing these two testing techniques. The results show that mutation is at least two times more effective than data-flow testing, i.e., odds ratio= 2.27. However, mutation is three times less efficient than data-flow testing, i.e., odds ratio= 2.94.
KW - Data flow testing
KW - Meta-analysis
KW - Mutation testing
KW - Statistical analysis
UR - http://www.scopus.com/inward/record.url?scp=80051616324&partnerID=8YFLogxK
U2 - 10.1109/ICSTW.2011.51
DO - 10.1109/ICSTW.2011.51
M3 - Conference contribution
AN - SCOPUS:80051616324
SN - 9780769543451
T3 - Proceedings - 4th IEEE International Conference on Software Testing, Verification, and Validation Workshops, ICSTW 2011
SP - 366
EP - 375
BT - Proceedings - 4th IEEE International Conference on Software Testing, Verification, and Validation Workshops, ICSTW 2011
Y2 - 21 March 2011 through 25 March 2011
ER -