Introduction to Mathematical OptimizationIntroduction to Mathematical Optimization Author: Nick Henderson, AJ Friend (Stanford University) Kevin Carlberg (Sandia National Laboratories)

Introduction to Mathematical Optimization

Nick Henderson, AJ Friend (Stanford University)Kevin Carlberg (Sandia National Laboratories)

August 13, 2019

1

Introduction

Introduction 2

Outline

Introduction

Optimization-problem attributes

Introduction 3

Optimization

Optimization find the best choice among a set of options subject to a set of constraintsFormulation in words:

minimize objectiveby varying variablessubject to constraints

Introduction 4

Applications

I Portfolio optimizationI Objective: riskI Variables: amount of capital to allocate to each available assetI Constraints: total amount of capital available

I Transportation problemsI Objective: transportation costI Variables: routes to transport goods between warehouses and outletsI Constraints: outlets receive proper inventory

I Model fitting (statistics and machine learning)I Objective: error in model predictions over a training setI Variables: parameters of the modelI Constraints: model complexity

Introduction 5

Applications

I Control (model predictive control)I Objective: difference between model output and desired state over a time horizonI Variables: control inputs (actuators)I Constraints: control effort (maximum possible actuation force)

I Engineering design (see wing-design example)I Objective: negative performance (maximize performance)I Variables: design parametersI Constraints: manufacturability

Introduction 6

Mathematical optimization: formulation

Standard form:minimize f0(x)subject to fi(x) ≤ 0, i = 1, . . . ,m

hi(x) = 0, i = 1, . . . , p

I x ∈ Rn: optimization/decision variable (to be computed)I f0 : Rn → R: objective/cost functionI fi : Rn → R: inequality constraint functionsI hi : Rn → R: equality constraint functionsI Feasible set: D = {x ∈ Rn | fi(x) ≤ 0, i = 1, . . . ,m, hi(x) = 0, i = 1, . . . , p}I Feasibility problem: Find x ∈ D (determines if the constraints are consistent)

Introduction 7

The big picture

I Bad news: most optimization problems (in full generality) cannot be solvedI Generally NP-hardI Heuristics required, hand-tuning, luck, babysitting

I Good news:I We can do a lot by modeling the problem as a simpler, solvable oneI Excellent computational tools are available:

I Modeling languages to write problems down (CVX, CVXPY, JuMP, AMPL, GAMS)I Solvers to obtain solutions (IPOPT, SNOPT, Gurobi, CPLEX, Sedumi, SDPT3)

I Knowing a few key problem attributes facilitates navigating the large set of possibletools and approaches

Introduction 8

Key challenge

Translate real-world problem into standard formThis requires balancing two competing objectives:1. Representativeness

I Model should closely reflect the actual problemI The solution should be useful

2. SolvabilityI Exercise is useless if a solution cannot be computedI Time-to-solution constraints (e.g., algorithmic trading) limit model complexity

Introduction 9


Optimization-problem attributes 10

Outline

Introduction



Friends and enemies in mathematical optimization

I Key problem attributes:I Convexity: convex v. non-convexI Optimization-variable type: continuous v. discreteI Constraints: unconstrained v. constrainedI Number of optimization variables: low-dimensional v. high-dimensional

I These attributes dictate:I Ability to find the solutionI Problem complexity and computing timeI Appropriate methodsI Relevant software

Always begin by categorizing your problem!


Convex v. non-convexI Convex problems:

I equality constraint functions are affineI objective and inquality constraint functions are convex

g(αx+ βy) ≤ αg(x) + βg(y)

f(x)

x

f(x)

x

D D

f(x)

x

convex non-convex

I Examples:I Linear least squares (later today)I Linear programming (LP): linear objective and constraints (management, finance)I Quadratic programming (QP): quadratic objective, linear constraints.


Convex v. non-convexI Non-convex problems:

I objective function is nonconvex,I inequality constraint functions are non-convex, orI equality constraints are nonlinear

I Main issues:1. Local minimum may not be a global minimium2. Don’t know if we’ve solved the problem (even if we have found the global minimum)

f(x)

x

Figure 1: Local and global solutions for a non-convex objective function.


Convex v. non-convex significance

I ConvexI One unique minimum: local minimizers are global!I Theory: convexity theory is powerfulI Solution process: no algorithm tuning or babysittingI Software: CVXPY, a modeling language for convex optimization

I Non-convexI Possibly many local minima: Local minimum may not be global minimumI Theory: most results ensure convergence to only a local minimum

I This means we have not really solved the problem!I Solution process: often requires significant tuning and babysitting

I For example, use multiple starting points to try to find global minimumI Software: scipy.optimize, a optimization sub-package of SciPy


Continuous v. discrete

I ContinuousI For example, x ∈ Rn

I Often easier to solve because derivative information can be exploitedI Examples

I parameters in a machine-learning modelI asset allocation in portfolio optimizationI position in a coordinate systemI vehicle speed in a model to minimize fuel consumptionI wing thickness in aircraft design


Continuous v. discrete examples

I DiscreteI For example, x ∈ {0, 1, 2, 3, . . .} or x ∈ {0, 1}I Always non-convexI Often NP-hardI Often reformulated as a sequence of continuous problems (e.g., branch and bound)I Sub-types: combinatorial optimization, integer programming

I Examples of discrete variablesI binary selector for facility location, e.g., xij = 1 if and only if resource i is placed in

location j and zero otherwiseI integer representing the number of warehouses to buildI integer representing the number of people allocated to a task


Unconstrained v. constrained (domain)

unconstrained domain(all points considered acceptable)

constrained domain(only green points acceptable)

feasible region


Unconstrained v. constrained (problem)

I Unconstrained problemsminimize f0(x)

I easier to solveI Constrained problems

minimize f0(x)subject to fi(x) ≤ 0, i = 1, . . . ,m

hi(x) = 0, i = 1, . . . , p

I linear equality constraints: can apply null-space/reduced-space methods toreformulate as an unconstrained problem.

I otherwise: can apply interior-point methods, which reformulate as a sequence ofunconstrained problems


Friends and enemies in mathematical optimization (summary)

I Convexity:I convex: local solutions are globalI non-convex: local solutions are not global

I Optimization-variable type:I continuous: gradients facilitate computing the solutionI discrete: cannot compute gradients, NP-hard

I Constraints:I unconstrained: simpler algorithmsI constrained: more complex algorithms; must consider feasibility

I Number of optimization variables:I low-dimensional: can solve even without gradientsI high-dimensional: requires gradients to be solvable in practice

Always begin by categorizing your problem!


Single-objective v. multi-objectiveI What if we care about two competing objectives f1 and f2?

I Example: f1=risk, f2=negative expected returnI Pareto front: set of candidate solutions among which no solution is better than

any other solution in both objectives

f1

f2

Each candidate solution is plotted in terms of both objectives.Pareto-optimal points plotted in red

I Often solved using evolutionary algorithmsI Can also minimize the composite objective function for many different values of a:

minimize a · f1(x) + f2(x)I this captures only points on the convex hull of the Pareto front


This courseTheory, methods, and software for problems exihibiting the characteristics below

I Convexity:I convex : local solutions are globalI non-convex : local solutions are not global

I Optimization-variable type:I continuous : gradients facilitate computing the solutionI discrete: cannot compute gradients, NP-hard

I Constraints:I unconstrained : simpler algorithmsI constrained : more complex algorithms; must consider feasibility

I Number of optimization variables:I low-dimensional : can solve even without gradientsI high-dimensional : requires gradients to be solvable in practice


Introduction to Mathematical OptimizationIntroduction to Mathematical Optimization Author: Nick Henderson, AJ Friend (Stanford University) Kevin Carlberg (Sandia National Laboratories)

Documents