Problems with StatXact's One-Sample K-S Test for Discrete Distributions

Note: I got a reply about this from Cytel. (See bottom of this page for information.)


Page 77 of Vol. 1 of the StatXact 8 User Manual describes how p-values are obtained for the One-Sample Kolmogorov-Smirnov Goodness-of-Fit Test. The indication is that for one-sided tests, exact p-values are obtained whether the specified distribution is discrete or continuous. Then (about 2/3 the way down p. 77) it states that
"For the two-sided case we use the method of Conover (1972) to approximate the exact p-value. The value reported is conservative."
This seems to imply that for the two-sided case, the "exact" p-values reported by StatXact aren't really exact. This made me curious, and I decided to investigate.

First I used the following data to test the null hypothesis that the distribution underlying the data is a discrete uniform distribution on the set of outcomes {1, 2, 3, 4, 5, 6, 7, 8} against the general alternative (so a "two-sided" test).

Response Frequency
1 29
2 19
3 18
4 25
5 17
6 10
7 15
8 11

StatXact gave 0.0001304 as the exact p-value. This value is more than 100 times smaller than the asymptotic p-value of 0.01329 provided by StatXact (even though the sample size (n = 144) is not real small, and the expected count for each outcome under the null hypothesis is 18, which isn't thought of as being small in this setting).

Then I used StatXact's Monte Carlo option to produce an estimate of the exact p-value, using 100,000 Monte Carlo trials (instead of the default 10,000) and fixing the seed at 23456. The estimated p-value provided by StatXact is 0.00469 (and a 99% confidence interval for the exact p-value is (0.004133, 0.005247)). So the Monte Carlo estimate of the exact p-value is about 36 times larger than the exact p-value reported. I found this to be very strange since while the manual does suggest that the exact p-value may not actually be exact, it indicates that the approximated exact p-value reported should be conservative, and that is not the case if the Monte Carlo estimate of the exact p-value is close to the true exact p-value, since the Monte Carlo estimate is larger than the supposedly conservative approximation.

Because I noticed a problem with this test years ago, I decided to investigate further, working with a really small sample size example where I could easily determine the true exact null hypothesis sampling distribution of the test statistic, and thus determine what the exact p-value ought to be in various cases. (Note: Years ago (maybe as many as 10) I reported problems with this test to the people at Cytel. I got a reply that a programming error was discovered. I'm thinking that perhaps not all of the errors were corrected.)

To investigate further, I considered a discrete uniform distribution putting 1/3 probability on 1, 2, and 3. I worked out the exact null sampling distributions of the three test statistics given by (5.14), (5.16), and (5.18) on pages 76 and 77 of the StatXact 8 User Manual for the simple case of n = 2. (So there are just 9 equally-likely outcomes to consider for (X1, X2).) The following table can be used to compare the exact p-values provided by StatXact to what I think any reasonable statistician would give as the true exact p-value for the case of x1 = 2 and x2 = 3.

Statistic Value of
Statistic 		StatXact
Point
Probability 		Actual
Point
Probability 		StatXact
Exact
p-value 		Actual
Exact
p-value 		StatXact
Monte Carlo
p-value
T 0.3333 0.0000 0.5556 0.8889 0.7778 0.775
T + 0.3333 0.0000 0.3333 0.4444 0.4444 0.4411
T - 0.0000 0.2222 0.3333 0.6667 1.0000 1.0000

There are a few things worth noting.
  1. The distributions of all three test statistics are discrete, and none of the point probabilities should be 0.
  2. StatXact's Monte Carlo p-values are in close agreement with the actual exact p-values (obtained using simple probability).
  3. Only one of StatXact's "exact" p-values is correct, and two are incorrect.
  4. All three of StatXact's test statistics are correct, so getting the correct values of the statistics isn't the problem.
  5. Since 0 is the smallest possible value of all three test statistics, if the observed value of the statistic is 0, the p-value should clearly be 1 (and that's not the case for StatXact's "exact" p-value resulting from t - = 0).
To point out another problem with using the One-Sample Kolmogorov test with a discrete uniform distribution on {1, 2, 3), if one has n = 2, x1 = 2, and x2 = 2, one gets the following error message:
Error: This test is used for one sample data.

So now for my questions.
  1. Am I correct that StatXact does not always compute exact p-values correctly for the One-Sample Kolmogorov test when used with a discrete distribution (even for one-sided tests)? (If so, an argument could be made that the Monte Carlo option should always be used.)
  2. Is is really the case that StatXact does not give exact p-values for the One-Sample Kolmogorov test when used for a two-sided test with a continuous distribution, as indicated by the statement
    "For the two-sided case we use the method of Conover (1972) to approximate the exact p-value. The value reported is conservative."
    about 2/3 down page 77, or does that statement just apply to when the test is used with a discrete distribution? (Based on some examples I checked out, StatXact does seem to be obtaining exact p-values correctly when the distribution is continuous.)
Please contact me at csutton@gmu.edu with your replies.


Note: I got a reply from Cytel, and they are in agreement with me. The exact version of the test is not done correctly when the distribution is discrete (and so it's best to use the Monte Carlo option in such cases). They also indicated that when the distribution is continuous, the exact p-values are correct. (The manual is confusing about this.)