Some Comments about the Preface of Samuels & Witmer
- (p. vii) The first paragraph describes the aims of S&W, the more
elementary of the two books for STAT 535. (The other book, G&H, covers
methods for more complicated data sets than those which can be dealt
with using the material from S&W.)
- (p. vii, Emphasis on Ideas paragraph) The warning against
"confusing statistical nonsignificance with practical insignificance"
deserves comment (although what I state here will certainly be expressed
when I cover hypothesis testing in class). S&W's comment is a way of
stating that a failure to find statistically significant evidence for a
particular research hypothesis shouldn't be taken to mean that the
research hypothesis is not true. The statistical nonsignificance could
be due to the sample size(s) being too small given the amount of
experimental noise. On the other hand, the nonsignificance means that
we cannot rule out the possibility that the research hypothesis isn't
true, since the data is compatible with the null hypothesis.
S&W recommend looking at a confidence interval to learn something about
the possible magnitude of whatever effect the experiment hoped to
detect. But their comment is a bit strange, because while the
confidence interval may include values consistent with some sort of an
effect, if the test result is nonsignificant, it will also include
values consistent with the research hypothesis not being true
(i.e., the test results may indicate uncertainty, and certainly
indicates a lack of strong evidence in favor of the research
hypothesis). In summary, lack of statistical significance may occur
even if the unknown truth is of practical significance, but it may
be due to the truth being of no practical significance, and so we
certainly cannot make a strong statement in support of a practically
significant result. *** S&W should have also warned
of a related phenomenon: that
of a statistically significant test result, but an estimated effect of
practical insignificance. For example, data may suggest that a certain
medication does indeed tend to lower blood pressure, but a confidence
interval indicates that the amount of lowering tends to be rather small,
and perhaps of no practical importance. This can happen when the
true magnitude of the effect is small, but nonzero,
and the sample size is large --- the
large sample size allows us to conlude that there is strong evidence of
some effect, even though the estimated magnitude of the effect is small.