comments about Efron & Gong, STAT 789

Comments about Efron and Gong's 1983 article in The American Statistician

I'm not providing a summary of the entire article here, or even the sections that I address. Rather, I'm giving some comments about certain parts of the article, where I think some comments may be helpful to you.

Note: There is a minor error in the paper --- 2 lines below expression (3) on the 1st page, the bar is ommited from the x on the right side of the equation.

Section 1

The 3rd paragraph contains the sentence "A good parametric analysis, when appropriate, can be more efficient than its nonparametric counterpart." This means that when the assumptions of the parametric analysis are completely met, that for a given sample size, a parametric estimator can outperform a nonparametric estimation method like bootstrapping or jackknifing. (Bootstrapping and jackknifing are nonparametric because they are used the same way to obtain estimates of bias and standard error no matter what the underlying distribution of the observations is --- the estimation procedure and the general formula used to obtain the estimate doesn't depend on what the underlying distribution is.) In some cases, for example, estimating the standard error of the sample mean, bootstrapping and jackknifing do just about as good as a parametric procedure --- the jackknife estimate of the standard error of the sample mean is exactly the same as the usual parametric estimate, and the bootstrap estimate differs only slightly. But in other cases, a good parametric procedure can greatly outperform bootstrapping and jackknifing. For an example see the right portion of Table 2 in the paper --- the parametric procedure has a much smaller MSE than the bootstrapping and jackknifing procedures.

The last sentence of the 3rd paragraph suggests that bootstrapping and jackknifing have some strengths too! They can yield trustworthy estimates when one cannot confidently select a good parametric model, whereas parametric procedures can do very badly if their parametric assumptions are not met.

The middle portion of the section contains material similar to what I covered in class.

The last portion of the section pertains to cross validation. Don't bother too much this this now --- I'll discuss cross validation later this summer.

Section 2

The first example dealt with in the section is a bit different than most of the examples I discussed in class, since the sample is of bivariate data. But one can note that the bootstrapping method is carried out the same way as usual --- the only difference is that resampling is done from the empirical distribution estimate of a bivariate distribution.

Table 1 isn't explained as well as it should be. Here's what they did to get the values for the left side of the table. They generated some number, M, of samples of size 15 from the standard normal distribution. (For Table 2 they indicate that M = 200, but I don't see where they give the value of M for Table 1.) From each of the M samples, the bootstrap and jackknife estimates of the standard error of the 25% trimmed mean were computed. The sample mean of the M standard error estimates is given for each estimation procedure in the Ave column, and the sample standard deviation of the M standard error estimates is given for each estimation procedure in the Sd column. (These things tend to be a bit confusing at times. The value of .084 in the Sd column is the estimated standard deviation of the jackknife estimator of the standard error of a 25% trimmed mean. That is, .084 is the estimated standard error of an estimator of standard error.) At the bottom of the Ave column is the true standard error of the 25% trimmed mean --- the estimand for the bootstrap and jackknife estimation procedures. The value of .287 is an estimate of the expected value of the bootstrap estimator. (We can refer to the bootstrap procedure as an estimator even though we can't write the estimator as a closed form expression involving the observations.) Since the estimand is .286, it appears that the bootstrap estimator is unbiased or nearly unbiased (and I'll guess it's not exactly unbiased). Table 1 suggests that the bootstrap estimator of standard error is superior to the jackknife estimator in this setting, since it appears to be less biased and also have a smaller standard error. (The larger the value of M is, the more seriously we can take the Monte Carlo results.) The better performance of the bootstrap estimator isn't surprising in this setting since the 25% trimmed mean is not a smooth estimator. It would have been better to have also had a column giving the estimated MSE (or RMSE (the square root of the MSE)) for the two estimators, as was done in Table 2. (To estimate the MSE of the bootstrap estimator using the M Monte Carlo trials, one would just use the average of the M squared deviations of the M estimates from the known estimand.)

It can be seen from Table 2 that once again the bootstrap outperformed the jackknife when estimating the standard error. (The comparison of performances is made easier in Table 2 since estimates of the RMSE are given and so we don't have to compute the bias using the Ave value and the true value of the estimand, and then correctly combine the bias with the standard error estimate in the Sd column to obtain an estimate of the MSE.) Again, this isn't suprising for the settings considered since the standard error estimates are for estimators which aren't linear estimators.

Some of the various bootstrap estimators considered in Table 2 are smoothed bootstrap estimators. I didn't have time to discuss the smoothed bootstrap in class, so it's nice that the article provides you with some information.

Section 3

The 1st paragraph of the section states that the jackknife "performs less well than the bootstrap in Tables 1 and 2, and in most cases investigated by the author," but to be fair to the jackknife it should be pointed out that there are some circumstances in which it can perform just about as well (or even slightly better) than the bootstrap. Indeed, the result of the Theorem given in the section implies that if an estimator is a linear estimator, so that a linear approximation of the estimator would in fact be the estimator, then there can be very little difference between the jackknife and bootstrap estimates of standard error. (My web page describes the situations in which jackknife and bootstrap estimates of bias and standard error differ little.)