EL6233 System Optimization Methods Z. P. Jiang ECE D t t R LC214 ECE Department Rm LC214 NYU-Poly. ext. 3646 eeweb.poly.edu/faculty/jiang 1/25/2010 ECE Dept at Poly 1 Spring 2008 Z.P. Jiang. No reproduction permitted without written permission of the author.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
EL6233 System Optimization y p
Methods
Z. P. Jiang
ECE D t t R LC214ECE Department Rm LC214
NYU-Poly. ext. 3646
eeweb.poly.edu/faculty/jiang
1/25/2010 ECE Dept at Poly 1
Spring 2008 Z.P. Jiang. No reproduction permitted without written permission of the author.
Course OutlineCourse Outline
An introductory course to practical OptimizationAn introductory course to practical Optimization
Methods
* linear and nonlinear programming
* l l f i ti* calculus of variations
* dynamic programmingy p g g
* other optimization problems & methods
1/25/2010 ECE Dept at Poly 2
Features of this CourseFeatures of this Course
B l b t th d li ti• Balance between theory and applications
• Mix of academic examples and puzzlesMix of academic examples and puzzles
• Basic homework vs. open-end projectsp p j
• Independent reading vs. team work
• Numerical algorithms
1/25/2010 ECE Dept at Poly 3
Motivational Examples of
O i i iOptimization
! Linear Equations
" Linear Programming
# Least Squares Data Fitting
$ Nonlinear Programming
1/25/2010 ECE Dept at Poly 4
!Linear Equations!Linear Equations
Goal: a first pass to data fittingGoal: a first pass to data fitting
Given three points: (2, 1), (3, 6) and (5, 4), find the
corresponding quadratic functioncorresponding quadratic function
2
1 2 3( ) f t x x t x t! " "
w unknith w 1n ,o 3ix i# #
1/25/2010 ECE Dept at Poly 5
CommentCommentCommentComment
An optimization problem usually involvesAn optimization problem usually involves
Context: 5 pirates robbed 100 diamonds andContext: 5 pirates robbed 100 diamonds and decide to distribute these diamonds by the rules:
• Draw lots to give each pirate a number out of 1, 2, 3, 4, 5.
First No 1 pirate proposes a distribution plan• First, No. 1 pirate proposes a distribution plan for the group of fives to vote. Only when he gets the majority of votes, his proposal will be g j y , p papproved.
Otherwise, he will be thrown into the sea.
1/25/2010 ECE Dept at Poly 17
“Blood Diamond” (cont’d)Blood Diamond (cont d)
• After No. 1 pirate dies, No.2 proposes hisAfter No. 1 pirate dies, No.2 proposes his
distribution plan for the group of four to vote.
Only when he gets the majority of votes, his
proposal will be approved.
Otherwise, he too will be thrown into the sea.
• Continue this way for No. 3 and No. 4
1/25/2010 ECE Dept at Poly 18
“Blood Diamond” (cont’d)Blood Diamond (cont d)
Hypothesis:Hypothesis:
All five pirates are equally smart and can make
their right judgementtheir right judgement.
1/25/2010 ECE Dept at Poly 19
“Blood Diamond” (cont’d)Blood Diamond (cont d)
Problem:Problem:
What should No.1 pirate propose to maximize his
profit?profit?
Solution at the end of this semester &Solution at the end of this semester &
1/25/2010 ECE Dept at Poly 20
PreliminariesPreliminaries
B i N ti f O ti lit• Basic Notions of Optimality
• Convexity• Convexity
• Taylor SeriesTaylor Series
• The General Optimization Algorithmp g
• Rates of Convergence
1/25/2010 ECE Dept at Poly 21
Basic Notions of OptimalityBasic Notions of Optimality
Consider an optimization problemConsider an optimization problem
( ) ( ) performance indexminimize f x P f x!( ) ( )
( ) 0, bj
p
t t t i t
minimize
i
x S
f f
g x i I
-
! - ./0subject to
( ) 0,cons
traints
jg x j J0$ - /1
It is worth noting that
( ) ( ) maximize minimizef x f x''!
1/25/2010 ECE Dept at Poly 22
( ) ( )maximize minimizex S x S
f f- -
FeasibilityFeasibility
• A point satisfying all the constraints is said toA point satisfying all the constraints is said to
be feasible.
• Feasible set S: all feasible points.p
• Active constraints: when
• Inactive constraints: when
( ) 0.jg x !
( ) 0jg x 2Inactive constraints: when
• Boundary: The set of feasible points at which
at least one inequality is active.
( ) 0.jg x 2
at east o e equa ty s act e
• Interior points: All other points than boundary
1/25/2010 ECE Dept at Poly 23
An Example
X31 1 2 3( ) 2 3 6 0
( ) 0
g x x x x! " " ' !
$2 1
3 2
( ) 0
( ) 0
g x x
g x x
! $
! $A x
4 3( ) 0.g x x! $B
Cx
x
X2
Points:
X1
A = (0, 0, 2)
B = (3, 0, 1)
1/25/2010 ECE Dept at Poly 24
C = (1, 1, 1).
MinimizersMinimizers
• Global minimizer ifx3 ( ) ( ), .f x f x x S3 # 4 -• Strict global minimizer if, additionally,
x ( ) ( ), .f x f x x S# 4 -
• Local minimizer if
( ) ( ), , .f x f x x S x x3 35 4 - 6
( ) ( ), , such that | | .f x f x x S x x3 3# 4 - ' 57
• Strict local minimizer if
( ) ( ) s ch that | |f f S3 3 35 4 6 571/25/2010 ECE Dept at Poly 25
( ) ( ), , such that , | | .f x f x x S x x x x5 4 - 6 ' 57
Stationary PointsStationary Points
• For unconstrained problems, a stationary pointFor unconstrained problems, a stationary point
of f(x) is such that
'( ) ( ) 0f
f x x3 38!
• A minimizer (or a maximizer) is a stationary
'( ) ( ) 0f
f x xx
!8!
A minimizer (or a maximizer) is a stationary
point, but not vice versa.
• Saddle point: a stationary point that is not
minimizer nor maximizer.
1/25/2010 ECE Dept at Poly 26
ConvexityConvexity
• Convex set S, ifConvex set S, if
(1 ) for all 0 1.x y S9 " '9 - # 9 #
• Convex function f, if
( (1 ) ) ( ) (1 ) ( )f x y f x f y9 " '9 # 9 " '9
, and 0 1.x y S4 - # 9 #
1/25/2010 ECE Dept at Poly 27
Convex Programming ProblemConvex Programming Problem
For a convex set S and a convex function f over S,For a convex set S and a convex function f over S,
( )minx S
f x-x S-
For example such a set S can be defined via functionsconcave g
: ;For example, such a set S can be defined via functions ,
| ( ) 0
concav
.
e i
n
i
g
S x g x! - $"
Linear Programming,
where both the object
Special case of C
ive and the const
onvex Programming:
raints are linear
1/25/2010 ECE Dept at Poly 28
where both the objective and the constraints are linear.
An Important ResultAn Important Result
A local minimizer of a convex programmingx3A local minimizer of a convex programming
problem is a global minimizer.
Moreover if the cost function is strictly convex
x
Moreover, if the cost function is strictly convex,
then is the unique global minimizer. x3
Proof. Prove by contradiction. See the book.
1/25/2010 ECE Dept at Poly 29
Taylor SeriesTaylor Series
Goal: Approximate a (sufficiently smooth) functionGoal: Approximate a (sufficiently smooth) function
by a polynomial around a point 0.x
2 ( )
:
1( ) ( ) '( ) "( ) ( )
1-variable
nnp
f x p f x pf x p f x f x" " " " " "( )
0 0 0 0 0( ) ( ) ( ) ( ) (
n-variab
)2 !
:les
f x p f x pf x p f x f xn
" ! " " " " "# #
2
0 0 0 0
1( ) ( ) ( ) ( )
2
T Tf x p f x p f x p f x p" ! " < " < "#
1/25/2010 ECE Dept at Poly 30
Notation:
Gradient ( ) :f x<
otat o
Gradient ( ) :
( ) ( ) ( )( )
T
n
f x
f x f x f xf x
<
= >8 8 8< -? @ "
1 2
2
( ) ( ) ( ) ( ) , , ,
n
f f ff x
x x x< ! -? @8 8 8A B
# "
2
2
Jacobian ( ) :
( )
f x
f x
<
= >82 ( ) ( ) .n n
i j
f xf x
x x
C= >8
< ! -? @? @8 8A B"
1/25/2010 ECE Dept at Poly 31
jA B
ExampleExample
2 2
1 1 2 2( ) 3f x x x x x! " '
Compute its gradient ( )f x and<2
Compute its gradient ( )
its Jacobian ( ) (1, 1).
f x and
f x at x
<
< !
1/25/2010 ECE Dept at Poly 32
Necessary ConditionNecessary Condition
A necessary condition for optimization is:A necessary condition for optimization is:
( ) 0f x< !0( ) 0.f x< !
1/25/2010 ECE Dept at Poly 33
Sufficient ConditionsSufficient Conditions
Sufficient conditions for minimization are:Sufficient conditions for minimization are:
0( ) 0f x< !0
2
0
( ) 0,
( ) 0.
f x
f x
<
< 20( )f
1/25/2010 ECE Dept at Poly 34
CommentCommentCommentComment
A matrix is said to be ,
if , 0, it holds that
positive definiten n
n
M
x x
C-
4 - 6
"
" , 0, o ds
0.T
x x
x Mx 2
% &eigenvalueThe of are all such that
s
det 0.
M
I M
D
D ' !% &
1/25/2010 ECE Dept at Poly 35
General Optimization AlgorithmGeneral Optimization Algorithm
Step 0: Specify some initial guess of the solutionStep 0: Specify some initial guess of the solution
0x
Step k (k=0,1,…): If is optimal, STOP.
Otherwise determine an improved estimate of
0x
Otherwise, determine an improved estimate of
the solution: with step
length and search direction1k k k kx x p" ! " 9
k9 kpe gt a d sea c d ect ok9 kp
1/25/2010 ECE Dept at Poly 36
Rates of ConvergenceRates of Convergence
• Does an algorithm converge?Does an algorithm converge?
H f t d it ?• How fast does it converge?
1/25/2010 ECE Dept at Poly 37
: ;Sequence with and , ifrate rate constantkx x r C3E
•
: ;
1 lim , where
k
k kk
k
r
eC e x x
e
"3EF
! 5 F '$
ke
• Linear convergence if r = 1.
1) 0 < C < 1, the sequence converges.
2) C > 1 the sequence diverges2) C > 1, the sequence diverges.
• Superlinear convergence if r = 1 and C = 0.Superlinear convergence if r 1 and C 0.
• Quadratic convergence if r = 2.
1/25/2010 ECE Dept at Poly 38
A Brief Introduction to Lagrange
M l i li M h dMultiplier Method
Lagrange multiplier theory essentially targets
at constrained optimization problems. For example,
% &0
1
max ,
subject to the constraint ( ).
nP f x x
c f x
! -
!
"
1j ( )
To this end introduce the augmented performance inde
f
xTo this end, introduce the augmented performance inde