## Optimization Methods for Applications in Statistics

### by James E. Gentle

• 1 Statistical Methods as Optimization Problems
• 2 Basic Definitions; Properties of Functions
• 2.1 Vectors, Vector Spaces, and Matrices
• 2.1.1 Inner Products
• 2.1.2 Norms
• 2.1.3 Metrics and Distances
• 2.1.4 Orthogonal Vectors
• 2.1.5 Vectors in a Cartesian Geometry
• 2.1.6 Projections
• 2.1.7 Orthogonalization Transformations
• 2.1.8 Cones
• 2.2 Function Spaces
• 2.2.1 Inner Products, Norms, and Metrics
• 2.2.2 Basis Sets in Function Spaces
• 2.2.3 Approximation of Functions
• 2.2.4 Hilbert Spaces
• 2.3 Shapes of Sets and Functions
• 2.4 Optimization of Functions
• 2.5 Stationary Points of Differentiable Functions
• Exercises

• 3 Finding Roots of Equations
• 3.1 Linear Equations
• 3.1.1 Direct Methods
• 3.1.2 Iterative Methods
• 3.2 Nonlinear Equations
• 3.2.1 Basic Methods for a Single Equation
• 3.2.2 Systems of Equations
• Exercises

• 4 Unconstrained Descent Methods in Dense Domains
• 4.1 Direction of Search
• 4.2 Line Searches
• 4.3 Steepest Descent
• 4.4 Newton’s Method
• 4.5 Accuracy of Optimization Using Gradient Methods
• 4.6 Quasi-Newton Methods
• 4.7 Fitting Models to Data Using Least Squares; Gauss-Newton Methods
• 4.8 Iteratively Reweighted Least Squares
• 4.10 The EM Method and Some Variations
• 4.11 Fisher Scoring
• 4.12 Stochastic Search Methods
• 4.13 Derivative-Free Methods
• 4.13.2 Price Controlled Random Search Method
• 4.13.3 Ralston-Jennrich Dud Method for Least Squares
• 4.14 Summary of Continuous Descent Methods
• Exercises

• 5 Unconstrained Combinatorial Optimization; Other Direct Search Methods
• 5.1 Simulated Annealing
• 5.2 Evolutionary Algorithms
• 5.3 Guided Direct Search Methods
• 5.4 Neural Networks
• 5.5 Other Combinatorial Search Methods
• Exercises

• 6 Optimization under Constraints
• 6.1 Constrained Optimization in Dense Domains
• 6.1.1 Equality Constraints
• 6.1.2 Linear Programming
• 6.1.3 Duality
• 6.1.4 General Constrained Optimization over Dense Domains
• 6.2 Constrained Combinatorial Optimization
• Exercises

• 7 Multiple Extrema and Multiple Objectives
• 7.1 Multiple Extrema and Global Optimization
• 7.2 Optimization with Multiple Criteria
• 7.3 Optimization under Soft Constraints
• Exercises

• 8 Applied Optimization: Numerical Methods and Software
• 8.1 Numerical Computations
• 8.2 Computer Storage and Manipulation of Data
• 8.2.1 The Floating-Point Model for the Reals
• 8.2.2 The Fixed-Point Number System
• 8.3 Numerical Algorithms and Analysis
• 8.3.1 Error in Numerical Computations
• 8.3.2 Efficiency
• 8.3.3 Formulas and Algorithms
• 8.4 Software for Optimization
• 8.4.1 Fortran and C Libraries
• 8.4.2 Optimization in General-Purpose Interactive Systems
• 8.4.3 Software for General Classes of Optimization Problems
• 8.5 Modeling Languages and Data Formats
• 8.6 Testbeds for Optimization Software
• Exercises
Exercises
• 9 Selected Applications in Statistics
• 9.1 Fitting Models by Minimizing Residuals
• 9.1.1 Statistical Inference Using Least Squares
• 9.1.2 Fitting Using Other Criteria for Minimum Residuals
• 9.1.3 Fitting by Minimizing Residuals while Controlling Influence
• 9.1.4 Fitting with Constraints
• 9.1.5 Subset Regression; Variable Selection
• 9.1.6 Multiple Criteria Fitting
• 9.1.7 Regularized Minimization
• 9.2 Nonparametric Smoothing
• 9.3 Maximum Likelihood Estimation
• 9.3.1 The Maximum Likelihood Approach
• 9.3.2 Maximum Likelihood Estimation with Constraints
• 9.3.3 Penalized Maximum Likelihood Estimation
• 9.4 Optimal Design and Optimal Sample Allocation . . . . .
• 9.4.1 D-Optimal Designs . . . . . . . . . . . . . . . . . .
• 9.4.2 Optimal Sample Allocation . . . . . . . . . . . . . .
• 9.5 Clustering and Classification* . . . . . . . . . . . .
• 9.6 Multidimensional Scaling . . . . . . . . . . . . . . .
• 9.7 Time Series Forecasting* . . . . . . . . . . . . . . .
• Exercises
Exercises . . . . . . . . . . . . . . . . . . . . . . . . .

• Notation and Definitions
• Bibliography
• Index

James Gentle, jgentle@gmu.edu