Information Pertaining to the Final Exam

Basics

Extra Office Hours

• Saturday, May 6, 2:00-4:00 PM;
• Sunday, May 7, 7:00-9:00 PM.

Description of the Exam

• The first problem has 12 parts to it, all pertaining to the same one-parameter exponential family distribution. Each part is worth 5 points, and I'll count your best 10 of 12 scores.
  • Part (a) requests a relatively straightforward MME.
  • Part (b) requests a relatively straightforward MLE.
  • Part (c) requests an expected value which should be helpful in answering part (d).
  • Part (d) requests a relatively straightforward UMVUE.
  • Part (e) requests a CRLB.
  • Part (f) requests a different CRLB.
  • Part (g) requests a MSE.
  • Part (h) requests a variance.
  • Part (i) requests a p-value. (Note: This need not be a one-sided UMP test. It could be a MP test (related to N-P lemma), or it could be a two-sided GLR test. I could want an exact p-value, or I may be content with an approximate p-value resulting from the use of an asymptotic method. (At this point I know exactly what is being requested, but I'm not telling --- I want you to study all of the different types of tests, and be prepared to obtain both exact and asymptotic p-values.))
  • Part (j) requests an asymptotic confidence interval.
  • Part (k) requests an exact confidence interval.
  • Part (l) requests a MLE.
• The second problem has 2 parts to it, both pertaining to the same hypothesis testing situation. (Hints: It will be a setting where the N-P lemma can be used. I will want the critical function for an exact randomized test. It will be a small sample situation with which it may be best to just consider (by listing the possibilities) the likelihood ratio for each possible outcome which can be observed.)
  • Part (a), worth 20 points, requests a critical function.
  • Part (b), worth 5 points, requests a power value.
• The third problem has 2 parts to it, both pertaining to the same hypothesis testing situation.
  • Part (a), worth 20 points, requests that you determine (and justify your reply) whether or not a null hypothesis can be rejected in favor of an alternative with an approximate (hint!) test.
  • Part (b), worth 5 points, requests an exact p-value.
• The fourth problem is worth 25 points, and requests a MLE.

What to Study

• Sufficient Statistics: Although I won't have you identify a sufficient statistics as one of the parts of the exam, knowing something about sufficient statistics may prove to be useful. Both MLEs and UMVUEs should depend on the observable random variables only through sufficient statistics (and the same is true of some test statistics and confidence interval endpoints (as is suggested by the sufficiency principle on p. C-11)). More specifically, an mle should depend on the data only through the value of a minimal sufficient statistic (if one exisits), and a umvue should depend on the data only through the value of a complete sufficient statistic. (The theorem on p. D-25 covers part of this, but it doesn't state that it should be a minimal sufficient statistic, if one exists.) While establishing minimal sufficiency and completeness directly can be very difficult (and I won't expect you to do anything like this for the exam), in the nice case of one-parameter exponential family (see p. C-15 and the top of p. C-16 for the basics of one-parameter exponential families) it is often quite easy to identify that a statistic is a complete minimal sufficient statistic (see the top part of p. C-16 and also see the important theorem on p. E-36).
• Useful Theorem on p. C-18: The theorem gives three results, but I don't expect that you'll have to do anything with mgf's on the exam, so the results pertaining to the mean and variance of the natural sufficient statistic are the ones I want you to be able to use (although you won't necessarily have to use these on the exam --- you could obtain the same information by going another route). Note that in some cases it is necessary to reparameterize to natural form by replacing c(theta) by eta. But if when you identify c(theta) T(x) you see that c(theta) is just equal to theta, then you simply apply the results with theta in the role of eta and d(theta) (a term in the usual exponential family form) in the role of d₀(eta). In such a case, getting the mean and variance of T(x) is often a lot less messy than what is the case for the example on p. C-18 and p. C-19.
• Consistent Estimators: All of the estimators based on a sample of size n that you will be asked to find on the exam will be consistent, which means that they converge to the estimand. (See the definition on p. D-4, and recall that it extends to estimators of functions of theta. For example, a consistent estimator of exp(theta) converges in probability to exp(theta).)
• Method of Moments Estimators (MMEs):
  • The example which begins on p. D-11 and ends on p. D-12 is a good one. It shows the form I like to see for a solution when I request an MME based on the first sample moment: (1) express m₁ = E(X_i) as a function of the unknown parameter; (2) solve for the parameter, putting it as a function of the distribution mean; (3) simply replace the distribution mean by the sample mean to obtain the MME.
  • Properties of MMEs are covered somewhat near the top of p. D-12. One can often use the invariance property to find the MME of some function of the unknown parameter. For example, if I have solved to put theta = g(m₁), where g is some suitable function, then I can write that 1/theta = 1/g(m₁), and obtain the MME of 1/theta by replacing m₁ by the sample mean.
  • In the usual case of a MME being consistent, the consistency follows from the law of large numbers: the sample mean converges to the true mean (and this gives us that the estimator coverges to the estimand). Note on the bottom line of p. D-12 that the law of large numbers can be applied to the X_i²: the sample mean of the X_i² converges to E(X_i²). (As an aside, it's also true that we have results such as the sample mean of the exp(X_i) converging to E[exp(X_i)].)
• Maximum Likelihood Estimators (MLEs):
  • The equation near the middle of p. D-16 defines mles. (Note that an mle must belong to the parameter space.)
  • The invariance property of mles is given near the bottom of p. D-18. The two examples on p. D-19 and p. D-20 are good ones --- note that in the part of the first example that is on p. D-20, the log-likelihood is worked with, and to me this seems easier than working with the likelihood function directly, as is done on p. D-19. Also note that in the part of the first example that is on p. D-20, the 2nd derivative is used to confirm that the solution obtained when setting the derivative of the log-likelihood to 0 is a maximizing value of the log-likelihood. (On the exam, like on the HW, I want you to confirm that what you claim is the maximizing value is in fact the maximizing value --- second derviatives can often be useful for this, but others methods can also be used.)
  • The two examples that begin on p. D-22 are good ones.
  • On the exam I won't have you find an MLE when there is more than one unknown parameter, and so for now you can not worry about the example that begins on p. D-23.
  • The example at the top of p. D-25 is a good one --- it serves as a reminder that sometimes an mle doesn't exist.
  • Some may find the theorem on p. D-25 to be useful, but one can find MLEs without using this theorem. An application of the theorem is given in the example on p. D-26.
• Mean Squared Error (MSE): The risk associated with the squared-error loss function, expressed near the middle of p. E-6, is referred to as the MSE. The last displayed equation on p. E-6 expresses the MSE of an estimator as being equal to the sum of its variance and the square of it's bias. (For unbiased estimators, the MSE is equal to the variance.)
• Uniform Minimum Variance Unbiased Estimators (UMVUEs):
  • For the exam I won't expect you to use the Rao-Blackwellization approach covered on p. E-30 through p. E-33. Also, you don't have to worry about the definition of completeness and the related examples covered on p. E-34 through p. E-36.
  • You should refer to the Lehmann-Scheffe theorem (see p. E-37) when establishing that an estimator is a UMVUE. Basically, to apply the theorem, you need to show that an estimator is an unbiased estimator having finite variance, and that the estimator depends on the observable random variables only through a complete sufficient statistic. Typically (and on the exam you can do it this way) you can use the theorem on p. E-36 to establish that a certain statistic is a complete sufficient statistic (but to do this you need to put the joint pmf or joint pdf in exponential family form, or refer back to a previous part of the exam where you did this). A key step is to establish that your candidate is in fact an unbiased estimator. Really, one should also show that the candidate has finite variance, but on the exam I'll let you skip that step (since for the Spring 2004 exam you will be asked to give the MSE of the UMVUE, and if the MSE is finite, the variance will of course be finite).
• Cramer-Rao Lower Bound (CRLB):
  • Don't worry about the regularity conditions on p. E-45, other than to note that typically if you have a one-parameter exponential family, then the Proposition on p. E-46 will give you that the conditions hold, and that the Fisher information number (see p. E-46, and also the alternative way to determine I(theta) given near the top of p. E-47) will exist.
  • Note that the lower bound given on the right-hand side of the inequality in the theorem on p. E-47 is the CRLB. When the estimand is just theta, the special case for the CRLB is given on p. E-48. Also, on p. E-48, one can see that I(theta) can be expressed as n times the information contained in one observation.
  • The example on p. E-50 is a good one.
  • The result given in the sentence that begins at the bottom of p. E-52 and concludes at the top of p. E-53 is sometimes useful. It gives us when we have the CRLB achieved by the UMVUE. (If the UMVUE is a linear function of the natural minimal sufficient T(x), or equivalently, if the estimand is a linear function of E[T(x)], the UMVUE will achieve the CRLB. Also, in such cases, the UMVUE will be the same as the MLE.) If the CRLB is achieved, then if we want the variance of the UMVUE we can determine the CRLB instead of obtaining the variance some other way.
  • As with MLEs, for UMVUEs I won't have you deal with multiparameter models on the exam (and so you can skip studying the multiparameter material on p. E-57 and p. E-58).
• Asymptotic Results for Point Estimators: For the exam, don't be concerned with the multiparameter results for MLEs, and don't be concerned with the results for frequency substitution estimators and MMEs. Just focus on the fact that MLEs obtained by setting the derivative of the likelihood (or log-likelihood) to 0 are asymptotically normal, and that the estimand can be used as the asymptotic mean, and the CRLB as the asymptotic variance. (You don't have to worry about the sketch of the proof given on p. E-77 and p. E-78.) So, for the exam, you don't have to be concerned about the vast majority of p. E-69 through p. E-80. It's just the one key result pertaining to MLEs that I want you to know. I'll refer to the use of this result in my comments below pertaining to Section F, and if you can use the asymptotic normality of MLEs to obtain asymptotic confidence intervals as described in Section F, that will be all you need to know how to do with this asymptotic material for the exam.
• Confidence Intervals:
  • If I request an exact confidence interval on the exam, it can be obtained by using either the result at the very top of p. F-14 or the related result near the top of p. F-15. A good example is given on p. F-15.
  • The use of a properly standardized MLE as an approximate pivot as is shown in the very good example on p. F-19 is important. The Wilson interval given in the example at the top of p. F-20 can be obtained by the same general method, but the deails are much messier. (On the exam I will give you something more along the lines of the example on p. F-19.) The Wald interval for a Bernoulli parameter, derived in the example on the bottom of p. F-21, makes use of the same general method, but also incorporates the use of Slutsky's theorem (see p. B-19). (You might be expected to do something similar on the exam. (I know if you will or you won't, but I'm not telling.))
• Basics of Hypothesis Testing: The size of a test (top line on p. G-6), power of a test against a particular alternative (p. G-8), and the p-value (p. G-9) are all important concepts. I recommend getting comfortable with the basic definitions, and then test your understanding by being able to determine values for size, power, and p-value in specific situations, like those covered in the supplemental handout pertaining to hypothesis testing examples. Critical functions are introduced on p. H-4, as are randomized tests. You should try to understand everything on p. H-4 through p. H-6. UMP (uniformly most powerful) tests are defined on p. H-7.
• Neyman-Pearson Lemma:
  • P. H-1 gives us that the simple likelihood ratio test (top portion of p. H-1) is the MP test when both hypotheses are simple.
  • The example on p. H-2 is a good one --- I show how to obtain the power of a size 0.05 test against a particular alternative, and you should also be able to write down the critical function of a size alpha test, and obtain the p-value given a specific sample of observed values.
  • The example on p. H-3 is also good --- I show how to determine the size corresponding to two different rejection regions, and in class I explained how to get a p-value given a particular observed sample.
  • P. H-22 and p. H-23 cover the asymptotic theory. (Note: My last lecture simplifies the presentation of this material. In particular, I stress that because of a result given about 30% of the way down on p. H-23, we can obtain the asymptotic p-value by determing the probability mass in some upper-tail region of a standard normal distribution. It should be noted that if you're dealing with a one-parameter exponential family, it's often a lot easier to base a z-score on the natural minimal sufficient statistic, T(x), than it is to base it on the likelihood ratio pieces.)
  • The more general version of the N-P lemma is given on p. H-6.
• UMP Tests Pertaining to One-parameter Exponential Families:
  • The theorem on p. H-8 is important, as are the examples on p. H-9 and p. H-10. The extension of the theorem given on the bottom portion of p. H-10 is very important. (Using it eliminates the need to do anything awkward like what is done in the middle portion of p. H-10.)
  • The key asymptotic results are given about 60% down the page on p. H-24. (Note: My last lecture simplifies the presentation of this material.)
• Other Monotone Likelihood Ratio Tests: The definition and theorem given on p. H-11 are important, as are the examples given on p. H-11 and p. H-12.
• Generalized Likelihood Ratio (GLR) Tests:
  • Two versions to the generalized likelihood ratio are given on p. H-13. The first one is often the choice if the parameter space is discrete, while the second version is almost always the choice if the parameter space is an interval.
  • The example on p. H-14 is a good one. For a relatively simple setting like that, I could expect you to simplify the ratio and determine an equivalent statistic that has a nice null sampling distribution which could be used to determine critical functions and p-values.
  • The example on p. H-16 to H-18 becomes too complicated to be a good model for an exam problem. At most I would expect you to obtain the form of lambda given about half way down on p. H-16, and then perhaps apply the important asympotic result presented on p. H-25 to it. (In fact, during the last lecture I plan to skip the example on p. H-25 and p. H-26, and revisit the example on p. H-16 to demonstate how to apply the asymptotic result for GLR tests.)
  • The example that starts on p. H-18 and ends on p. H-21 is too messy to be a good model for an exam problem. (I usually skip presenting this example in class because it takes up too much time.)

Homework Problems to Review

HW #2

1(a), 1(b), 1(d), 2(a), 2(b), 3

HW #4

1(a), 1(b), 1(c), 1(d), 1(e)

HW #5

1(a), 1(b)

HW #6

2(a), 2(b), 2(c), 2(d), 3, 4