Multiple testing of general contrasts: Truncated closure and the extended shaffer-royen method

Powerful improvements are possible for multiple testing procedures when the hypotheses are logically related. Closed testing with a-exhaustive tests provides a unifying framework for developing such procedures, but can be computationally difficult and can be "non-monotonic in p values." Royen introduced a "truncated" closed testing method for the case of all pairwise comparisons in the analysis of variance that is monotonie in p values. Shaffer developed a similar truncated procedure for more general comparisons, but using Bonferroni tests rather than α-exhaustive tests, and Westfall extended Shaffer's method to allow α-exhaustive tests. This article extends Royen's method to general contrasts and proves that it is equivalent to the extended Shaffer procedure. For k contrasts, the method generally requires evaluation of O(2^k) critical values that correspond to subset intersection hypotheses and is computationally infeasible for large k. The set of intersections is represented using a tree structure, and a branch-and-bound algorithm is used to search the tree and reduce the O(2 ^k) complexity by obtaining conservative "covering sets" that retain control of the familywise type I error rate. The procedure becomes less conservative as the tree search deepens, but computation time increases. In some cases where hypotheses are logically restricted, even the more conservative covering sets provide much more power than standard methods. The methods described herein are general, computable, and often much more powerful than commonly used methods for multiple testing of general contrasts, as shown by applications to pairwise comparisons and response surfaces. In particular, with response surface tests, the method is computable with complete tree search, even when k is large.