Comments about Ch. 4 of Applied Logistic Regression, 2nd Ed.

  1. (p. 92, lines 7-9) I don't think it's necessarily true that "The more variables included a the model, the greater the estimated standard errors become" since variables that are strongly related to the dependent variable serve to reduce uncertainty.
  2. (p. 92) Although some worry about overfitting and reducing the power of tests, a good point to note is that if a risk factor is statistically significant in an overfit model, there is little question that it is associated with the response. (Even after many other variables have been adjusted for, the risk factor is found to improve the fit.)
  3. (p. 93, 1st new paragraph) Since the OR is eB, if the data suggests that the OR is either 0 or infinite, the estimate of B would have to be extreme in order to have a decent correspondence to the suggested OR.
  4. (p. 95) By eliminating large p-value variables at this step (temporarily at least ... the variables are given a 2nd chance later), we are only omitting variables that have little association with the response, and so there seems to be little to lose in tossing them (even if some were originally thought to be important).
  5. (p. 95) Doing a best subsets regression serves as a safeguard against having originally tossed a variable that only exhibits its worth in the presence of certain other variables. (So it seems as though we would want to toss some variables initially in order to have a more manageable situation --- but we should always reconsider such variables that were discarded at an early stage.)
  6. (pp. 96-97) By again eliminating some variables at this step, one can consider a smaller number variables in the next step where the model is fine-tuned by consideration of transformations and interactions.
  7. (p. 97, line 1) On p. 16, it is indicated that the likelihood ratio test is favored over the Wald test for single predictor models. Here, use of the Wald test is suggest. I wonder if it's better in multiple predictor settings, or if it's suggested use is just due to convenience (since maybe software packages tend to give all of the Wald statistics upon fitting a single model with all of the variables in it, and getting likelihood ratio test p-values would involve more work).
  8. (p. 97, lines 3-4) I'm assuming that variables should satisfy both criteria ((a) and (b) on lines 1 and 2 of p. 97) in order to avoid elimination. If this is correct, it seems way different from the choice of keeping all clinically important variables. (The last sentence of the paragraph seems to indicate that a variable could be eliminated just because it isn't statistically significant. I guess deleting a variable thought to be clinically important doesn't matter much if including the variable really doesn't have much effect.)
  9. (p. 97, lines 8-12) Couldn't a marked change in a coefficient be due to deleting a variable that was highly correlated with another one? In such a case, deleting it wouldn't adversely effect the fit of the model.
  10. (p. 97) Note that transforming the continuous variables is part of step (4), even though the details of how to do this aren't given until after step (5) is presented.
  11. (p. 98) It isn't made clear at this point, but judging by the examples which follow, one should consider any transformed versions of the variables created in the previous step when interactions are considered in this step, and not start with the linear version of the variables which were deemed necessary to transform (although after considering the transformed versions, one might then also investigate if the interaction can be included in a less complex manner ... that is, if two variables are both represented with two terms (from the fractional polynomial scheme), one might want to avoid having a model with four product terms in it for the interaction).
  12. (pp. 98-99) H&L indicate that interaction terms should be statistically significant in order to include them in the model. Given what is on p. 77, I guess significance at level 0.1 is all that is required.
  13. (p. 99, last two lines) I would have put significantly better as opposed to "significantly different."
  14. (p. 100) Note that allowing both the inverse and the square root of a variable to be in the model is something that is contrary to the common practice of regression modelers.
  15. (p. 101, line 16) Since all log likelihood values are nonpositive for logistic regression fits, the largest log likelihood will be the one having the smallest magnitude.
  16. (p. 102) According to seminar group member Ed Prokop, the paper of Royston and Altman recommends something different from the 2 step procedure at the bottom of p. 102. They suggest testing to see if best J = 2 model is better than best J = 1 model first, and if so use the J = 2 model. Otherwise, test best J = 1 model against the model with the variable included in a simple linear manner, and use the linear model unless the J = 1 model is significantly better.
  17. (p. 103, lines 1-8) I guess one updates to include the newly determined transformations in the model only at the end of a complete cycle, instead of immediately including each one as it is found, but this isn't completely clear. (What was the point of ordering the variables by the Wald statistic values as suggested on p. 102?)
  18. (P. 106) Note that AGE was only deemed to be marginally important important in its univariate model, but it is highly significant once other variables are also included in the model. Often, things are just the opposite --- a variable may appear to be strongly important in a univariate model, but once other variables are also included, this variable no longer seems highly important.
  19. (p. 107) To get the plot, does one just plug the weighted average of the yi (as given on p. 94) in for pi in the formula for the logit? (Has anyone tried to reproduce the plot?)
  20. (p. 107, 2nd to last line) Note that although the plot bends sharply upwards on the right, the right portion of the plot is (I think) based on relatively few observations. The nongraphical methods do not support going away from AGE being linear in the model --- they weight each case equally, and there aren't many cases that suggest a need to move away from linear.
  21. (p. 110) Recall that smoothing sometimes suffers from edge effects, causing a linear trend to be "rounded over" somewhat. I don't think the edge effect would by itself create a false mode, but if the sample was a bit nonrepresentative, the edge effect could contribute to the appearance of a mode that isn't real. The design variable method would eliminate the edge effect of smoothing, but I suggest using more than 4 groups in order to get a better feel for the experimental noise --- maybe there will be other evidence of nonmonotonicity, which would take away from the evidence that there is a maximum away from 0.
  22. (p. 111) Note that if J = 1 model compared to linear without also investigating the J = 2 model, one might decide to claim no nonlinearity and go with a simple linear model. So it may be a good idea to always go at least one more step farther than what may appear necessary. A nonmonotonic trend may not be modeled well with a J = 1 model, and so in such a case a linear term may do about as well as the best J = 1 model, but the best J = 2 model could do appreciably better. (Recall that Harrell favors using measures of association that are sensitive to nonmonotonic trends.)
  23. (p. 112) Note that the plot suggests a leveling off for 15 or more that is not reflected in the fitted model (although one can se that the data is relatively sparse for those values). To me it seems sensible that if treatments hadn't worked the first dozen or so times, it may not matter much if someone has three more treatments or 30 more treatments.
  24. (p. 113) It isn't clear to me how STATA can apply the fractional polynomial scheme to centered variables, which would create both positive and negative values to transform --- wouldn't applying the square root transformation lead to a problem, and the inverse transformation yield a screwy result?
  25. (p. 113) As you're reading along, you might wonder where the intercept value of -4.314 in the model equation at the botto of the page comes from. It's explained on the next page, where one also can see that the value is a bit arbitrary. (For the plot on p. 112, the arbitrariness of the intercept doesn't matter (as is explained on p. 114).)
  26. (p. 114) Note that the procedure described in this chapter won't uncover interactions unless both variables have been previously identified as "main effect" variables. CART is a good tool to use to search for variables that are mostly important because of interactions (and so one could use CART on the data set in order to identify possible interactions to explore for the logistic regression model).
  27. (p. 115) Note that the AGExNDRGFP2 term is omitted, even though the 2 df likelihood ratio test supported its inclusion. I guess overall, when one considers the Wald test p-values, and the results of fitting the model without the AGExNDRGFP2 term, a good case can be made for excluding this interaction term (since it doesn't seem to add anything to the model having just the AGExNDRGPF1 term representing the interaction between the two variables). However, a lot of modelers would keep both terms to represent this two variable interaction (doing so would correspond to their standard practice). (Also, some modelers would consider quadratic terms for the main effects as well --- looking at a full 2nd order model.) Overall, H&L seem to favor keeping the number of terms in the model to a relatively small number, even though they don't require that each variable justify its inclusion with a p-value of less than or equal to 0.05 (or even 0.1 perhaps). I like H&L's approach for the most part. Some modelers would include a lot more terms, allowing for slight adjustments due to variables which are not very important. If variables are mildly related to the response, including them doesn't add a whole lot. If some of the terms really shouldn't be in the model, then the overfitting is definitely doing more harm than good. So to omit some questionable terms isn't such a bad thing. But at the same time, H&L will allow a variable in the model (as a main effect, represented by a linear term) even though it doesn't justify itself with a highly significant p-value. I guess one way of thinking about it is to remember that there is no strong reason to give the null hypothesis of not including the variable the benefit of the doubt. For sure, H&L's practice is somewhat of a "middle of the road" approach --- they don't easily allow transformations and interactions, but aren't that strict about omitting marginal main effect terms, but tend to allow them to be in the model in a simple linear way. Note that they are lenient with linear main effect terms because they are more limited in number than interaction terms --- if there are k predictor variables, there are just k such linear terms to consider, but there are a total of k choose 2 potential interaction terms, and if a J = 2 fractional polynomial representation is used for each variable, it's somewhat like estimating 4k unknowns, since choosing the powers to use is like estimation of an unknown (it's just for the sake of simplicity one may choose to limit the search to a smallish set of candidates). So including a variable as a simple linear term isn't too costly with regard to potential overfitting, but one can wind up trying to estimate too many unknowns with too few data points if interactions and fractional polynomial representations are liberally included, and this can lead to overfitting (and also it makes it harder to easily interpret the model). Note that the larger the sample size, the less crucial these issues are --- if we exclude a term because it's p-value wasn't small enough, it won't make a lot of difference if the term is included, because with a large sample size, the estimated coefficient doesn't have to be too large in magnitude to result in a small p-value (and so excluded terms would be ones with small coefficients anyway), and if we include some terms that should be omitted then the overfitting effects shouldn't be so bad if the sample size is large. (So small sample size situations are the tough ones --- one does have to worry about overfitting, and at the same time, a variable which would make an appreciable adjustment won't necessarily produce a small p-value.)
  28. (pp. 118-119) Comments about the selection of pE and pR are given. Are these in agreement with other guidelines?
  29. (p. 120) The descriptions of the two methods given on this page aren't really clear to me. Fortunately, the examples given on pp. 122-124 clarify matters.
  30. (p. 120) Should we discuss the first sentence of the last complete paragraph?
  31. (p. 121, middle paragraph) This is one reason why some only like to use variables that are known to be clinically meaningful.
  32. (p. 125) Do you agree with adding RACE and SITE? Even though SITE is related to the design of the experiment, could it not be that it has no effect on the response? What if RACE was only significant in prior studies that did not include other varaibles which are associated with RACE and should be adjusted for --- couldn't it be that if the present study included such variables, that RACE doesn't really have an effect on the response? Note on p. 134 that the model favored that includes RACE and SITE is only the 5th best model as indicated in the table.
  33. (p. 125, middle portion of page) Why use different tests for entry and removal? (Note that on p. 126 we are reminded that the likelihood ratio test is the most accurate test, and so I guess other tests are sometimes used just as a matter of convenience --- but why use different tests for entry and removal?)
  34. (p. 125 & p. 128) I like the idea of using a stepwise procedure to help identify the set of important interaction terms.
  35. (p. 128) Before moving on to the next section, I'm wondering if it wouldn't be a good idea to consider the removal of some main effects variables after the interaction terms have been added. Couldn't it possibly be that when interactions are adjusted for, some of the other variables are no longer needed?
  36. (pp. 128-130) Best subsets is generally favored over stepwise, but it seems like with some software, stepwise may be easier to do. I would recommend doing one or both of these in addition to going through the selection routine described in Sec. 4.2 (what H&L refer to in some places as purposeful selection of variables) --- it never hurts to have a check of your work.
  37. (p. 133) Note that H&L's collection of best subsets don't all have the same number of variables. In some places best subsets in used to obtain several sets of 2 variables, several sets of 3 variables, and so on. But here the list consists of the overall best sets as measured by Cq.
  38. (p. 133, near bottom) It seems to me like replacing the Pearson chi-square statistic by its (null) mean would only make sense if one assumes that the null hypothesis is true. (I'm not going to worry too much about understanding the details of the method described on pp. 133-134.)
  39. (p. 137) With data like that given in the table, I don't know why one would want to create a logistic regression model --- basically you just have three sets of binary outcomes, and I would be tempted to just do a few simple things with the data instead of fooling with a logistic regression model. (E.g., one could test for homogeneity (are the three underlying probabilities of success all equal), or one could allow for different probabilities of success and estimate them (and recall one doesn't have to use the simple sample proportion).)
  40. (p. 141) Note that the large estimated coefficients for x1 and x2 are of opposite sign.