Optimization Methods for Applications in Statistics

by James E. Gentle

Table of Contents

---

• 1 Statistical Methods as Optimization Problems
• 2 Basic Definitions; Properties of Functions
  • 2.1 Vectors, Vector Spaces, and Matrices
    • 2.1.1 Inner Products
    • 2.1.2 Norms
    • 2.1.3 Metrics and Distances
    • 2.1.4 Orthogonal Vectors
    • 2.1.5 Vectors in a Cartesian Geometry
    • 2.1.6 Projections
    • 2.1.7 Orthogonalization Transformations
    • 2.1.8 Cones
  • 2.2 Function Spaces
    • 2.2.1 Inner Products, Norms, and Metrics
    • 2.2.2 Basis Sets in Function Spaces
    • 2.2.3 Approximation of Functions
    • 2.2.4 Hilbert Spaces
  • 2.3 Shapes of Sets and Functions
  • 2.4 Optimization of Functions
  • 2.5 Stationary Points of Differentiable Functions
  • Exercises

  ---

• 3 Finding Roots of Equations
  • 3.1 Linear Equations
    • 3.1.1 Direct Methods
    • 3.1.2 Iterative Methods
  • 3.2 Nonlinear Equations
    • 3.2.1 Basic Methods for a Single Equation
    • 3.2.2 Systems of Equations
  • Exercises

  ---

• 4 Unconstrained Descent Methods in Dense Domains
  • 4.1 Direction of Search
  • 4.2 Line Searches
  • 4.3 Steepest Descent
  • 4.4 Newton’s Method
  • 4.5 Accuracy of Optimization Using Gradient Methods
  • 4.6 Quasi-Newton Methods
  • 4.7 Fitting Models to Data Using Least Squares; Gauss-Newton Methods
  • 4.8 Iteratively Reweighted Least Squares
  • 4.9 Conjugate Gradient Methods
  • 4.10 The EM Method and Some Variations
  • 4.11 Fisher Scoring
  • 4.12 Stochastic Search Methods
  • 4.13 Derivative-Free Methods
    • 4.13.1 Nelder-Mead Simplex Method
    • 4.13.2 Price Controlled Random Search Method
    • 4.13.3 Ralston-Jennrich Dud Method for Least Squares
  • 4.14 Summary of Continuous Descent Methods
  • Exercises

  ---

• 5 Unconstrained Combinatorial Optimization; Other Direct Search Methods
  • 5.1 Simulated Annealing
  • 5.2 Evolutionary Algorithms
  • 5.3 Guided Direct Search Methods
  • 5.4 Neural Networks
  • 5.5 Other Combinatorial Search Methods
  • Exercises

  ---

• 6 Optimization under Constraints
  • 6.1 Constrained Optimization in Dense Domains
    • 6.1.1 Equality Constraints
    • 6.1.2 Linear Programming
    • 6.1.3 Duality
    • 6.1.4 General Constrained Optimization over Dense Domains
  • 6.2 Constrained Combinatorial Optimization
  • Exercises

  ---

• 7 Multiple Extrema and Multiple Objectives
  • 7.1 Multiple Extrema and Global Optimization
  • 7.2 Optimization with Multiple Criteria
  • 7.3 Optimization under Soft Constraints
  • Exercises

  ---

• 8 Applied Optimization: Numerical Methods and Software
  • 8.1 Numerical Computations
  • 8.2 Computer Storage and Manipulation of Data
    • 8.2.1 The Floating-Point Model for the Reals
    • 8.2.2 The Fixed-Point Number System
  • 8.3 Numerical Algorithms and Analysis
    • 8.3.1 Error in Numerical Computations
    • 8.3.2 Efficiency
    • 8.3.3 Formulas and Algorithms
  • 8.4 Software for Optimization
    • 8.4.1 Fortran and C Libraries
    • 8.4.2 Optimization in General-Purpose Interactive Systems
    • 8.4.3 Software for General Classes of Optimization Problems
  • 8.5 Modeling Languages and Data Formats
  • 8.6 Testbeds for Optimization Software
  • Exercises
  Exercises

  ---

• 9 Selected Applications in Statistics
  • 9.1 Fitting Models by Minimizing Residuals
    • 9.1.1 Statistical Inference Using Least Squares
    • 9.1.2 Fitting Using Other Criteria for Minimum Residuals
    • 9.1.3 Fitting by Minimizing Residuals while Controlling Influence
    • 9.1.4 Fitting with Constraints
    • 9.1.5 Subset Regression; Variable Selection
    • 9.1.6 Multiple Criteria Fitting
    • 9.1.7 Regularized Minimization
  • 9.2 Nonparametric Smoothing
  • 9.3 Maximum Likelihood Estimation
    • 9.3.1 The Maximum Likelihood Approach
    • 9.3.2 Maximum Likelihood Estimation with Constraints
    • 9.3.3 Penalized Maximum Likelihood Estimation
  • 9.4 Optimal Design and Optimal Sample Allocation . . . . .
    • 9.4.1 D-Optimal Designs . . . . . . . . . . . . . . . . . .
    • 9.4.2 Optimal Sample Allocation . . . . . . . . . . . . . .
  • 9.5 Clustering and Classification* . . . . . . . . . . . .
  • 9.6 Multidimensional Scaling . . . . . . . . . . . . . . .
  • 9.7 Time Series Forecasting* . . . . . . . . . . . . . . .
  • Exercises
  Exercises . . . . . . . . . . . . . . . . . . . . . . . . .