Slide 1
Optimality Conditions for Unconstrained optimizationOne
dimensional optimizationNecessary and sufficient
conditionsMultidimensional optimizationClassification of stationary
pointsNecessary and sufficient conditions for local
optima.Convexity and global optimality
The topic of this lectures are necessary and sufficient
conditions for a point to be an optimum. We will start with a
review of the results known from Calculus for a function of single
variable, in particular dealing with stationary points where the
derivative is zero.
For one dimensional function, there are only three kinds of
stationary points: maxima, minima, and inflection points. For
multidimensional functions there are other kinds, so we will devote
some time to their classification and to the necessary and
sufficient conditions for local optima.
Finally, we will touch on conditions for global optimality, in
particular for the special case of convex functions.1One
dimensional optimizationWe are accustomed to think that if f(x) has
a minimum then f(x)=0 but.
From Calculus we are accustomed to associate a minimum or a
maximum with the vanishing of the first derivative. However, that
applies only to differentiable functions with continuous
derivatives as is illustrated in the figure.21D Optimization
jargonA point with zero derivative is a stationary point. x=5, Can
be a minimum
A maximum
An inflection point
For one dimensional functions stationary points are points where
the derivative vanishes. We consider three functions that have a
stationary point at x=5. One has a minimum ( Y=(x-5)2, the green
curve), one has a maximum (y=10x-x2, the red curve), and one has an
inflection point ( y=0.2(x-5)3, the blue curve).3Optimality
criteria for smooth 1D functions at point x*f(x*)=0 is the
condition for stationarity and a necessary condition for a minimum
or a maximum.f(x*)>0 is sufficient for a minimumf(x*)0 is a
sufficient condition for a minimum, because it guarantees that
close enough to x* (h sufficiently small), f(x)>f*(x).
Similarly, f(x*)=0 is called semi-positive definite, and all its
eigenvalues are non-negative.7Types of stationary pointsPositive
definite: MinimumPositive semi-definite: possibly
minimumIndefinite: Saddle pointNegative semi-definite: possibly
maximumNegative definite: maximum
So we can now offer a complete classification of stationary
points for n-dimensional functions depending on the definiteness of
the Hessian matrix. If the matrix is positive definite we have
sufficient conditions for a minimum, and similarly if the matrix is
negative definite (vTHv=0 for a minimum. In that case the matrix is
called positive simi-definite, all the eigenvalues are
non-negative, and if one ore more is zero, higher derivatives will
determine whether we have a minimum.
Similarly, the necessary condition for a maximum, is that the
matrix is negative semi-definite, which leads to the eigenvalues
being non-positive.
Finally, if the matrix has vTHv>0 for some vectors v, and
vTHv