Explanation of answers to Quiz #10
The first problem, dealt with an anticonservative F test.
An anticonservative test is one that rejects with too high a
probability if the null hypothesis is true. This is due to the p-values
obtained using the null sampling distribution of the test statistic, an
F distribution, being too small --- so that some of the p-values
obtained that are less that the nominal size of the test, say 0.05,
should be larger than the size of the test if the p-values corresponded
to the true null sampling distribution of the test statistic. So it's
an inflated proportion of small p-values obtained from the incorrect F
distribution that makes the test anticonservative.
The second problem dealt with the power characteristics of a test
which is robust for validity. Such a test need not have close to
optimal power. As an extreme case, one can randomly decide to reject
or not reject the null hypothesis and have a valid test (one that has
the an acceptable probability of rejecting when the null hypothesis is true),
but have a test with very low power. (If the size of the test is 0.05,
the power will also be 0.05.) While tests based on assumptions of
normality may not do this poorly when used with data from heavy-tailed
distributions, the power of a normal theory test can be quite a bit less
than the power of a nonparametric test in some cases.