Computational Optimization

CPSC 406 Term 2 of 2023/24

Course goals and emphasis

\[ \def\argmin{\operatorname*{argmin}} \def\Ball{\mathbf{B}} \def\bmat#1{\begin{bmatrix}#1\end{bmatrix}} \def\Diag{\mathbf{Diag}} \def\half{\tfrac12} \def\ip#1{\langle #1 \rangle} \def\maxim{\mathop{\hbox{\rm maximize}}} \def\maximize#1{\displaystyle\maxim_{#1}} \def\minim{\mathop{\hbox{\rm minimize}}} \def\minimize#1{\displaystyle\minim_{#1}} \def\norm#1{\|#1\|} \def\Null{{\mathbf{null}}} \def\proj{\mathbf{proj}} \def\R{\mathbb R} \def\Rn{\R^n} \def\rank{\mathbf{rank}} \def\range{{\mathbf{range}}} \def\span{{\mathbf{span}}} \def\st{\hbox{\rm subject to}} \def\T{^\intercal} \def\textt#1{\quad\text{#1}\quad} \def\trace{\mathbf{trace}} \]

Goals

  • recognize and formulate the main optimization problem classes
  • understand how to apply standard algorithms for each class
  • recognize if an algorithm succeeded or failed
  • hands-on experience with mathematical software

Emphasis

  • formulating problems
  • algorithms
  • mathematical software

Prerequisites

Required courses

  • CPSC 302 (NUmerical Computation for Algebraic Problems)
  • CPSC 303 (Numerical Computation for Discretization)
  • MATH 307 (Applied Linear Algebra)

Assumed background

  • linear algebra – linear systems, factorizations, eigenvalues
  • multivariate calculus – gradients, Hessians, Taylor series
  • numerical software – eg, Julia, Matlab, Python (numpy, scipy), R

Book

  • Introduction to Nonlinear Optimization: Theory, Algorithms, and Applications with MATLAB, 4th edition, by A. Beck (SIAM, 2014)

Role of optimization

  • fitting a statistical model to data (machine learning)
  • logistics, economics, finance, risk management
  • theory of games and competition
  • theory of computer science and algorithms
  • geometry and analysis

Competing objectives

maximize

  • profit
  • utility
  • accuracy

minimize

  • cost
  • risk
  • error

constraints

  • budget
  • capacity
  • physics
  • fairness
  • safety
  • stability
  • …

Isoperimetric Problem

Queen Dido’s Problem, after the founder of Carthage (814 BC)

  • on then plane, length \(L\) of closed curve and enclosed area \(A\) related by \[L^2\ge 4\pi A\]

  • two points of view: a circle

    • maximizes area for a given length
    • minimizes length for a given area

  • same solution, but two different formulations!

Brachistochrone problem

optimization

  • objective: time for a bead to slide from point \(A\) to point \(B\) under gravity
  • constraints: bead starts at reast at \(A\) and hugs curve

solution

  • cycloid — the curve of least time between two points under gravity
  • posed by Johann Bernoulli (1696) (solved by brother Jacob Bernoulli)

Least squares

  • due to Gauss and Legendre (early 1800s)
  • observations at times \(t_1,\ldots,t_m\): \[ b = (b_1,b_2,\ldots,b_m)\]
  • model has parameters \(x=(x_1,x_2,\ldots,x_n)\), eg, \[m(x,t_i) = x_1 + x_2 t_i + \cdots + x_n t_i^{n-1}\]
  • model may be nonlinear in parameters \(x\)
  • objective is to minimize squared errors \[f(x)=\sum_i[m(x,t_i)-b_i]^2\]

Learning models

model is a simplified abstraction of process

  • model parameters \[x=(x_1,x_2,…,x_n)\in\mathcal{P}\]
  • prediction \[m(x)\in\mathcal{F}\]

least-error principle

optimal parameters \(x^*\) minimizes a distance between the model and the observation

Mathematical Optimization

  • Objective function: \(f:ℝ^n\toℝ\)
  • feasible set (eg, “constraints”): \(\mathcal{C}⊆ℝ^n\)
  • decision variables: \(x=(x_1,x_2,\ldots,x_n)\in\mathcal{C}\)

optimal solution \(x^*\) has smallest (or largest) value of \(f\) among all feasible points, ie, \[ f(x^*) \le f(x) \qquad \forall x\in\mathcal{C} \]

Abstract problem

  • find \(x\in\mathcal{C}\) such that \(f(x)\) is minimal, eg, \[ p^* = \min_{x\in\mathcal{C}}\ f(x) \]
  • optimal solution set \[\mathcal{S}:=\{x\in\mathcal{C}\mid p^*=f(x)\}\]

Varieties of optimization

  • continuous vs discrete
  • linear vs nonlinear
  • convex vs nonconvex
  • smooth vs nonsmooth
  • deterministic vs stochastic
  • global vs local

Steepest descent

  • follow the gradient towards a minimizer
  • measures the objective’s sensitivity to feasible perturbations
  • usually sufficient to devise tractable and implementable algorithms

Linear programming (LP)

Applications

  • resource allocation
  • production planning
  • scheduling
  • network flows
  • optimal transport

History

Quadratic programming (QP)

  • generalizes linear programming
  • includes
    • least-squares over linear and bound constraints (convex case; polynomial-time)
    • integer optimization (nonconvex case; NP-hard)

Applications

What is and why Julia?

println("Hello, world! 1 + 2 = $(1+2)") 
Hello, world! 1 + 2 = 3


@code_llvm(1+2)
;  @ int.jl:87 within `+`
define i64 @"julia_+_862"(i64 signext %0, i64 signext %1) #0 {
top:
  %2 = add i64 %1, %0
  ret i64 %2
}


using Plots
plot(sin, x->sin(2x), 
     0, 2π, 
     legend=false, fill=(0,:lavender),
     size=(600,250))


sum_sqs(y) = mapreduce(x->x^2, +, y) # sum sq 
sum_sqs(1:3)
14

Why Julia?

  • Programming language designed for scientific and technical computing
  • Aims to solve the “two language problem”
  • good package manager and documentation
  • faster than most dynamically-typed languages, eg, Python, Matlab, R
  • easier to use than most statically-typed languages, eg, C, C++, Fortran
  • lots of tutorials available
  • good IDE support, eg, VSCode, Jupyter, Pluto
  • UBC hosts a free JupyterHub server for students (CWL required)

Optimization modeling languages

Julias has two popular optimization modeling packages:

  • JuMP (Julia for Mathematical Programming)
  • Convex.jl (Julia for Disciplined Convex Programming)

\[\min_{x\ge0,y\ge0}\ (1-x)^2 + 100(y-x^2)^2\]

using JuMP, Ipopt
model = Model(Ipopt.Optimizer)
set_silent(model)
@variable(model, x>=0)
@variable(model, y>=0)
@objective(model, Min, (1 - x)^2 + 100 * (y - x^2)^2)
optimize!(model)
solution_summary(model)

******************************************************************************
This program contains Ipopt, a library for large-scale nonlinear optimization.
 Ipopt is released as open source code under the Eclipse Public License (EPL).
         For more information visit https://github.com/coin-or/Ipopt
******************************************************************************

* Solver : Ipopt

* Status
  Result count       : 1
  Termination status : LOCALLY_SOLVED
  Message from the solver:
  "Solve_Succeeded"

* Candidate solution (result #1)
  Primal status      : FEASIBLE_POINT
  Dual status        : FEASIBLE_POINT
  Objective value    : 2.44348e-21
  Dual objective value : 0.00000e+00

* Work counters
  Solve time (sec)   : 1.09451e-02
  Barrier iterations : 16

Got Clicker?

🎱 Who is the Premier of British Columbia?

  1. John Horgan
  2. Andrew Wilkinson
  3. David Eby
  4. Justin Trudeau
  5. David Suzuki

🎱 I took these courses…

  1. CPSC 302 (Numerical Computation for Algebraic Problems)
  2. CPSC 303 (Numerical Computation for Discretization)
  3. MATH 223 (Linear Algebra)
  4. MATH 307 (Applied Linear Algebra)
  5. None of the above

Coursework and evaluation

  • 8 homework assignments (30%)
    • programming and mathematical deriviations
    • typeset submissions, correctness, and writing quality graded
  • midterm exam (30%): ✏️️ and 🗞️, short mathematical problems
  • final exam (40%): multiple choice

Homework

  • work alone 🏃 or in pairs 👯
  • 4 late days allowed (no permission required)
    • but no more than 2 late days for a particular assignment
  • submit your HW solutions to Gradescope
  • no solutions posted online — visit us in office hours
  • typeset solutions using Jupyter or Pluto or LaTeX

Homework 1

  • no late days, no collaborators
  • should feel familiar
  • due next week

Resources

  • see the course home page for schedule
  • visit Canvas for links to
    • Piazza for discussions and announcements
    • Gradescope for homework submission
  • TA and instructor office hours start week 2