Glossary

conservative and anticonservative

A conservative test is one that has a maximum type I error probability less than the nominal level or size of the test. p-values from a conservative test tend to be too large. A conservative test is considered to be a valid test. An anticonservative test is one for which the maximum type I error probability is greater than the nominal level or size of the test. Since such a test does not respect the allowable type I error rate, an anticonservative test is not a valid test. p-values from an anticonservative test tend to be smaller than they should be --- leading to too high of a type I error probability. It should be noted that a given test procedure can be conservative in some situations and anticonservative in others. A conservative confidence interval procedure is one that has associated with it a coverage probability greater than the nominal confidence level. Such a confidence interval procedure is considered to be valid, but leads to confidence intervals which are generally wider than they need to be. A confidence interval procedure is anticonservative if it has associated with it a coverage probability that is smaller than the nominal confidence level. Such a confidence interval procedure is not valid, and it generally produces confidence intervals shorter than they need to be, although the coverage probability may suffer due to the intervals not being centered correctly, as opposed to them being too short.

heavy tails and light tails

Normal distributions have neither heavy tails nor light tails --- they have neutral tailweight. A distribution that has a tendency to produce a higher proportion of extreme values in a sample than is the case for a normal distribution is referred to as a heavy-tailed distribution. The tails of it's density don't thin out (go towards zero in thickness) at as high a rate (in a relative sense) as do the tails of a normal distribution, leading to a higher proportion of values away from the clump of values due to the main body of the distribution. Heavy tails are sometimes called fat tails or long tails (since the portions of the tails having nonnegligible thickness is relatively longer than is the case with normal distributions). A distribution that has a tendency to produce a lower proportion of extreme values in a sample than is the case for a normal distribution is referred to as a light-tailed distribution. A sample from a light-tailed distribution (aka, thin-tailed distribution) will tend to have a smaller proportion of outliers than we get from sample from a normal distribution (which don't tend to have a lot of outliers anyway). Skewed distributions can have one heavy tail and one light tail. For symmetric distributions, the kurtosis is one indicator of tail weight, with heavy-tailed distributions tending to have positive kurtosis and light-tailed distributions tending to have negative kurtosis. (Normal distributions have a kurtosis of 0.) Probit plots are a useful device for learning something about the tail wieght of the parent distribution of a sample.

heteroscedasticity and homoscedasticity

Homoscedasticity is the condition of equal variances, and heteroscedasticity is the condition of unequal variances (or nonconstant variance).

size and level (of a test)

If the null hypothesis completely determines the sampling distribution of a test statistic, then the size of the test is the probability that a type I error (a false rejection of the null hypothesis) will occur, if the null hypothesis is true and the test is performed. If the null hypothesis allows for more than one possibility, then the probability of a type I error may depend upon which of the specific possibilities is true if the null hypothesis is true. In such a case, the size of the test is the maximum probability that a type I error will occur, considering all specific possibilities. (Note: Being precise, it's the supremum of the probabilities (i.e., the least upper bound of the probabilities), although in almost all practical situations, the supremum is a maximum.) So, for example, if the null hypothesis is that either a, b, or c is true, and if a is true the probability of a type I error is 0.03, if b is true the probability of a type I error is 0.02, and if c is true the probability of a type I error is 0.01, the size of the test is 0.03, since the probability of a type I error could be as large as 0.03 if the null hypothesis is true, but cannot be larger than 0.03. The level of a test can be any value greater than or equal to the size. The level specifies an upper bound on the allowable probability of a type I error, but this upper bound need not be achieved. For example, we can refer to a size 0.03 test as a level 0.05 test, since the probability of a type I error will be no larger than 0.05. It may not be immediately clear why one would refer to a test as a level 0.05 teat (which seems to suggest that the probability of a type I error might be as large as 0.05) if the actual size of the test is less than 0.05. But consider a lab that does a lot of different tests of hypotheses. They might want to state that their policy is to always do level 0.05 tests, meaning that the probability of a type I error can never be larger than 0.05. In some situations, if one rejection region is used, the size of the test may be 0.07, but if a slightly smaller rejection region is used, the size of the test drops to 0.03. Given a choice between these two tests, the size 0.03 test would be chosen, since the size 0.07 test violates the level requirement. The people doing the statistical tests would have to make sure that the size of the test is never larger than 0.05 if they want to respect the level 0.05 requirement. They would generally choose the rejection region to be such that the size is as close as possible to 0.05, without exceeding 0.05, in order to have the level 0.05 test with the greatest possible power to reject the null hypothesis when it should be rejected (i.e., when the alternative hypothesis is true). If a test statistic has a continuous sampling distribution, then it is not necessary to ever have the size of the test be less than the stated level. But if a test statistic has a discrete distribution, then none of the possible rejection regions may correspond to a size 0.05 test, and a test having a size less than 0.05 must be chosen in order to comply with the level requirement. Because a lot of books only (or primarily) deal with tests for which the sampling distributions of the test statistics are continuous, and for such tests the size need not differ from the level, some books just use the term level, and do not introduce the term size (even though the size is perhaps the more important of the two concepts). In STAT 535, some of the nonparametric tests which are used have test statistics which only take on integer values and have discrete sampling distributions. With such tests, for some sample sizes, it may not be possible to find a rejection region which gives a size 0.05 test. (Of course, in STAT 535, the emphasis is on p-values, and not just indicating whether or not a rejection occurs with a certain sized test.)