Closed Testing in Pharmaceutical Research: Historical and Recent Developments

In pharmaceutical research, making multiple statistical inferences is standard practice. Unless adjustments are made for multiple testing, the probability of making erroneous determinations of significance increases with the number of inferences. Closed testing is a flexible and easily explained approach to controlling the overall error rate that has seen wide use in pharmaceutical research, particularly in clinical trials settings. In this article, we first give a general review of the uses of multiple testing in pharmaceutical research, with particular emphasis on the benefits and pitfalls of closed testing procedures. We then provide a more technical examination of a class of closed tests that use additive-combination-based and minimum-based p-value statistics, both of which are commonly used in pharmaceutical research. We show that, while the additive combination tests are generally far superior to minimum p-value tests for composite hypotheses, the reverse is true for multiple comparisons using closure-based testing. The loss of power of additive combination tests is explained in terms worst-case “hurdles” that must be cleared before significance can be determined via closed testing. We prove mathematically that this problem can result in the power of a closure-based minimum p-value test approaching 1, while the power of a closure-based additive combination test approaches 0. Finally, implications of these results to pharmaceutical researchers are given.