UBC CPSC 406 – Scaled Descent

Scaled descent

conditioning
scaled gradient direction
Gauss Newton

Zig-zagging

Consider the quadratic function with symmetric and positive definite

level sets are ellipsoids:

gradient descent from two starting points:

eigenvectors of are principal axes
eigenvalues are the lengths of the “unit ellipse” axes

Gradient descent zig-zags

Let be the iterates generated by gradient descent with exact linesearch. Then

Proof: exact steplength satisfies

optimality of step

because and

Condition number

The condition number of an positive definite matrix is

ill-conditioned if
condition number of Hessian influences speed of convergence of gradient descent
- : gradient descent converges in one step
- : gradient descent zig-zags
if is twice continuously differentiable, define the condition number of at solution as

Scaled gradient method

make a linear change of variables: where is nonsingular to get rescaled problem
apply gradient descent to scaled problem
multiply on left by to get -update

scaled gradient method

Scaled descent

If , the scaled negative gradient is a descent direction because
Recall: a matrix is positive definite if and only if
- with diagonal and nonsingular
- with nonsingular

Choosing the scaling matrix

Observe relationship between optimizing and optimizing its scaling

condition number of governs convergence of gradient descent

choose such that is well-conditioned, ie,

Level sets of scaled and unscaled problems

Close to solution , levels sets of

are ellipsoids and

are circles for ideal because

Question

Consider the change of variables to the quadratic function

to obtain the scaled function
Which choice of the nonsingular scaling matrix will transform the level sets of into circles (i.e., result in a perfectly conditioned Hessian for )?

(the identity matrix)
(the diagonal part of )

Common scalings

Make as well conditioned as possible

Scaled Descent

Scaled descent

Zig-zagging

Gradient descent zig-zags

Condition number

Scaled gradient method

Scaled descent

scaled gradient method

Choosing the scaling matrix

Example (quadratic)

Level sets of scaled and unscaled problems

Question

Common scalings

Gauss Newton
Nonlinear Least Squares

Nonlinear least squares

Example – localication problem

data

Linearization of residual

Gauss Newton as scaled descent

Gauss Newton for NLLS

Gauss Newton for NLLS