Some Notes Pertaining to Ch. 13 of E&T
The percentile interval is covered in this relatively short chapter. Since the BCA interval covered in
the next chapter can be thought of as an improvement of the percentile interval, in the end the percentile interval won't be recommended for use. Nevertheless, it'll be good to understand the percentile interval in order to better be able to understand the BCA interval of Ch. 14.
Sec. 13.2 makes the observation that the endpoints of the standard interval (which is based on an assumption of
approximate normality for the estimator) coincide with certain percentiles of the sampling distribution
of a bootstrap replicate of the estimator, given that a normal parametric model is assumed. (See (13.2) for a statement
of the equalities.)
For example, the standard interval confidence bounds for a 90% confidence interval are the 5th and 95th percentiles
of the bootstrap replicate's sampling distribution (based on parametric bootstrapping using a normal model with
estimated parameters --- the model of (13.1) on p. 168).
The percentile interval is defined by (13.3), or equivalently (13.4), at the start of Sec. 13.3 We have that
the endpoints of a 90% percentile interval are just the 5th and 95th percentiles of the bootstrap replicate's nonparametric
sampling distribution.
( This R code shows how to compute a percentile interval in a simple situation.)
If the estimator's distribution is approximately normal, and the sample size isn't too small, so that we have the empirical distribution (the distribution of the Xi*) being close to the actual
distribution underlying the Xi, then the standard interval and the percentile interval should not be very different, because nonparametric sampling distribution of the bootstrap replicates shouldn't be much different from the parametric sampling distribution of the bootstrap replicates.
(Note: E&T's reference to the central limit theorem 3 lines below (13.5) only makes sense if the estimator is a linear estimator like the sample mean or sample kth moment.)
Now let's consider the case of an estimator's sampling distribution (given the sample size at hand) not being
approximately normal. Now we don't have that the two confidence interval methods should necessarily produce similar
results (with high probability). So, if the two methods produce different results, which one should be believed?
Clearly, the standard interval, which is based on an assumption of approximate normality, is defective. But what about the
percentile interval --- is it any better?
To address the last question posed above, let's consider the special case where μ is the
distribution mean, the estimand is
exp(μ), and the estimator used is the plug-in estimator. If we start by considering the estimation of
μ
instead of
exp(μ),
then, because of the approximate normality of the sample mean, we could use the percentile method to get a decent confidence interval for
μ.
Letting L and U denote the lower and upper confidence bounds (viewing them as random variables) for the
percentile interval for
μ,
we can consider
exp(L) and exp(U) to be the confidence bounds for a decent confidence interval for
exp(μ),
because we have
P(L < μ < U) =
P(exp(L) < exp(μ) < exp(U)).
Now note that if
L and U are percentile interval confidence bounds for
μ,
then we also have that
exp(L) and exp(U) are exactly the percentile interval confidence bounds for
exp(μ).
Thus, in this case, because the percentile interval method works with the approximately normal estimator, the sample mean,
it follows that it also works with the plug-in estimator of
exp(μ),
which is not approximately normal for small sample sizes (unless the standard deviation of the underlying distribution is
sufficiently small).
It's important to note that to obtain the percentile interval for
exp(μ),
it wasn't necessary to know the specific transformation of the estimator which results in an approximately normal
estimator --- that transformation was only used to justify the percentile interval based on the nonnormal estimator.
This seems to suggest that we only have to believe that there is some transformation to approximate normality in order to
believe in the (approximate) validity of the percentile interval for any particular setting, and we don't even
have to know what the normalizing transformation is, since the endpoints of the percentile interval can be based
on the nonnormal estimator. (In Sec. 13.8
and Ch. 14 it will be pointed out that there is a slight snag in this argument. Briefly,
even if there is a transformation to normality, there is no guarantee that a percentile interval based on the transformed estimator will be valid: in order to have good performance for the percentile interval based on the transformed estimator,
we need for it to be the right kind of approximately normal estimator --- one which is unbiased or has negligible bias.)
This R code shows how to compute several different confidence intervals for
exp(μ) in the situation considered on p. 171.
Sec. 13.4 compares the interval given by (13.9) to the percentile interval. Problem 13.5 on p. 177 indicates how (13.9)
can be derived, and I'll go over the derivation in class.
( This .pdf file shows the derivation of (13.9).)
It can be noted that a 90% interval for
θ will be long on the left (meaning that the distance the lower bound is below the point estimate is greater than the distance the upper bound is above the point estimate) if the sampling distribution of a bootstrap replicate is
such that it's 95th percentile is farther above the original point estimate than it's 5th percentile is below the original point estimate. But in such a case the percentile interval will be long on the right
(as
this .pdf file shows)
... which is perhaps at least a little
bit puzzling, and makes one wonder which way is correct? (Note: If the sampling distribution of the bootstrap replicates is symmetric about the original point estimate then the two intervals will be in agreement.)
In the case of an unbiased estimator, or one having negligible bias, the justification provided above for the percentile interval seems pretty sound. Plus, if the underlying distribution of the Xi is positively skewed, nonbootstrap confidence intervals for the mean (Johnson's modified t interval) and the median (the confidence interval related to the sign test) will tend to be long to the right, as will the percentile intervals, but not the intervals obtained using (13.9).
(It's not easy to explain why the interval given by (13.9) is often faulty ... and so I won't attempt to do so in class.
There simply isn't a simple argument that handles all cases. E&T seem to suggest that it may have to do with the fact that the estimator minus the estimand isn't pivotal, but that isn't always true --- in some cases it would be pivotal. (Note:
If the distribution of the estimator minus the estimand depends on theta, then this difference isn't pivotal. An example would be the difference between the sample mean of exponential random variables and the distribution mean --- the
standard deviation of this difference is proportional to the distribution mean.))
Section 13.5 deals with the coverage probabilities of standard and percentile intervals in a sepcific setting.
I used R to obtain estimated coverage probabilites for various confidence intervals in a few different settings.
This R code can be used to obtain Monte Carlo estimates of the coverage probabilities of Student's t interval, the percentile interval, and the bootstrap-t interval for estimating the mean of a normal distribution from a sample of 25 obsevations. The table belows shows estimates of the probability of missing to the left, the probability of missing to the right, and the estimated coverage probability for each of the interval estimation methods considered.
The nominal coverage probability is 0.9 (i.e., 90% confidence intervals). All of the methods perform decently, but the percentile method appears to be a bit anticonservative. (Note: Even though 2500 Monte Carlo trials were used, the standard error of the coverage probability estimator may be as large as 0.006, and so two standard errors could be greater than 0.01.)
| |
miss left |
miss right |
coverage prob. |
| Student's t |
0.0516
|
0.0412
|
0.9072
|
| percentile |
0.0656
|
0.0528
|
0.8816
|
| bootstrap t |
0.0532
|
0.0416
|
0.9052
|
For normal samples of size 100, the behavior of the bootstrap interval estimators is improved.
(The table below gives the results for the n = 100 case.)
| |
miss left |
miss right |
coverage prob. |
| Student's t |
0.0476
|
0.0532
|
0.8992
|
| percentile |
0.0480
|
0.0544
|
0.8976
|
| bootstrap t |
0.0472
|
0.0508
|
0.9020
|
I performed a similar Monte Carlo study for interval estimators of the mean of an exponential distribution.
This R code was used to obtain the results for the case of
n = 25, and the results are shown in the table below. Note that the bootstrap t interval outperforms Student's t interval in this nonnormal setting.
| |
miss left |
miss right |
coverage prob. |
| Student's t |
0.0996
|
0.0204
|
0.8800
|
| percentile |
0.1044
|
0.0336
|
0.8620
|
| bootstrap t |
0.0552
|
0.0448
|
0.9000
|
The table below gives the results for the n = 100 case for exponential random variables.
| |
miss left |
miss right |
coverage prob. |
| Student's t |
0.0732
|
0.0292
|
0.8976
|
| percentile |
0.0708
|
0.0360
|
0.8932
|
| bootstrap t |
0.0464
|
0.0444
|
0.9092
|
The table below gives results similar to those in Table 13.3 on p. 175 of E&T. The estimand in exp(μ),
where μ is the mean of a normal distribution. I considered 90% confidence intervals (E&T consider 95% intervals), and I used samples of size 10 (as did E&T).
( This R code was used to obtain the results. It takes an extremely long time to run compared to other examples of R code that I've given you this summer.)
| |
miss left |
miss right |
coverage prob. |
| standard |
0.1000
|
0.0348
|
0.8652
|
| percentile |
0.0748
|
0.0820
|
0.8432
|
| bootstrap t |
0.0720
|
0.0816
|
0.8464
|
Finally, I used samples of size 25 and obtained (nominal) 95% confidence intervals. Increasing the sample size helped performance. It can be noted that the standard interval did the best for both cases considered.
(Note: On p. 175, E&T seem to suggest that the undercoverage of the percentile interval is a consequence of nonparametric inference. I think it may be better to state that the misbehavior of all three bootstrap intervals in this setting may be largely due to the extremely small sample size and the fact that they are nonparametric. With such a small sample size the information obtained in the bootstrap world may be rather inaccurate.)
| |
miss left |
miss right |
coverage prob. |
| standard |
0.0480
|
0.0128
|
0.9392
|
| percentile |
0.0372
|
0.0328
|
0.9300
|
| bootstrap t |
0.0328
|
0.0328
|
0.9344
|
At this point, it's hard to identify a clearcut winner amongst the various bootstrap methods. The BCA method of Ch. 14 is supposed to be an improvement on the percentile method, and so perhaps when it's included in comparisons the situation will clear up a bit.
Sections 13.6 and 13.7 deal with properties of percentile intervals.
The transformation-respecting property means that confidence intervals for
θ
and
g(θ)
are consistent with one another, provided that g is a monotone transformation. For example, if a percentile confidence interval for
θ
is (1.25, 10.00), then the percentile interval for
1/θ
will be (0.10, 0.80).
The range-preserving property means that provided the point estimator cannot take on impossible values for the
estimand , the endpoints
of the confidence interval won't be impossible values.
Not all confidence intervals are like this, an example being that a commonly used confidence interval for the probability of success for Bernoulli trials can include values less than 0 or greater than 1.