NLP KKT Practice and Second Order Conditions from Nash and Sofer.

NLP

KKT Practice and Second Order Conditions from Nash and Sofer

Unconstrained

First Order Necessary Condition

Second Order Necessary

Second Order Sufficient

2* min ( *) . . .If x is then FONChold and f x is p s d

* min ( *) 0If x is then f x

2( *) 0 ( *) . .

* min .

If f x and f x is p d

then x is strict local

Easiest Problem

Linear equality constraints

min ( )

. . ,

n

m n m

f x f R

s t Ax b A R b R

KKT Conditions

Note for equality – multipliers are unconstrained

Complementarity not an issue

: ( , ) ( ) '( )

:

( , ) ( ) ' 0x

Lagrangian L x f x Ax b

KKT Ax b

L x u f x A

u unconstrained

Null Space Representation

Let x* be a feasible point, Ax*=b.

Any other feasible point can be written as x=x*+p where Ap=0

The feasible region

{x : x*+p pN(A)}

where N(A) is null space of A

See Section 3.2 of Nash and Sofer for example

mxnA

mAppAN 0|)(

A ofcolumn th theis A where,

somefor |

A of columns by the spanned vectorsofset the)(

i iAq

Aqq

AR

ii

mT

T

Null and Range Spaces

)R(Aq and N(A)p somefor q p x x,

0 Ap because , somefor 0,Appq

)R(Aq and N(A)p

subspaces orthogonal are )R(A and N(A)

T

TT

T

T

Orthogonality

Null Space Review

is the null space matrix of A:

, 0

, 0 ,

Z

p v Zp Av and if

v Av p v Zp

1 2 1 2[11] 0

1, ( )

1

A v v v v

Z p R Zp Null A

Constrained to Unconstrained

You can convert any linear equality constrained optimization problem to an equivalent unconstrained problem

Method 1 substitution

Method 2 using Null space representation and a feasible point.

Example

Solve by substitution

becomes

1 2 2 31 2 22

1 2 3

min

. . 3 4 4

x x x

s t x x x

1 2 2 31 2 22

1 2 3

min

. . 4 3 4

x x x

s t x x x

3

21 2 32 3 2 22

min 4 3 4x x x x

Null Space Method

x*= [4 0 0]’

x=x*+Zv

becomes

1 2 2 31 2 22

1 2 3

min

. . 3 4 4

x x x

s t x x x

3

21 2 31 2 1 22

min 4 3 4v v v v

3 4

1 0

0 1

Z

1 21

12

2

4 3 4 4 3 4

0 1 0

0 0 1

v vv

vv

v

General Method

There exists a Null Space Matrix

The feasible region is:

Equivalent “Reduced” Problem

n rZ R r n m

| *x x Zv

min ( * )v f x Zv

Optimality Conditions

Assume feasible point and convert to null space formulation

2 2 2

( ) ( * )

( ) ' ( * ) ' ( ) 0 *

( ) ' ( * ) ' ( )

k v f x Zv

k v Z f x Zv Z f y where y x Zv

k v Z f x Zv Z Z f y Z

Where is KKT?

KKT implies null space

Null Space implies KKT

Gradient is not in Null(A), thus it must be in Range(A’)

( *) ' 0

'( ( *) ' ) ' ( *) 0 ' ' 0

f x A

Z f x A Z f x because Z A

Lemma 14.1 Necessary Conditions

If x* is a local min of f over {x|Ax=b}, and Z is a null matrix

Or equivalently use KKT Conditions

2

' ( *) 0

' ( *) . . .

Z f x

and Z f x Z is p s d

2

( *) ' 0

*

' ( *) . . .

f x Ahas a solution

Ax b

Z f x Z is p s d

Lemma 14.2 Sufficient Conditions

If x* satisfies (where Z is a basis matrix for Null(A))

then x* is a strict local minimizer

2

*

' ( *) 0

' ( *) . .

Ax b

Z f x

Z f x Z is p d

Lemma 14.2 Sufficient Conditions (KKT form)

If (x*,*) satisfies (where Z is a basis matrix for Null(A))

then x* is a strict local minimizer

2

*

( *) ' 0

' ( *) . .

Ax b

f x A

Z f x Z is p d

Lagrangian Multiplier

* is called the Lagrangian Multiplier

It represents the sensitivity of solution to small perturbations of constraints

*

1

ˆ ˆ( ) ( *) ( *) ' ( *)

ˆ( *) ( *) ' ' *

ˆ

( *) ' * ( *)m

i ii

f x f x x x f x

f x x x A by KKT OC

Now let Ax b

f x f x

Optimality conditions

Consider min (x2+4y2)/2 s.t. x-y=10( ) ' 0

1

4 1

10

* 8, * 8, * 2,

f x A

Ax b

x

y

x y

x y

Optimality conditions

Find KKT point Check SOSC( ) ' 0

1

4 1

10

* 8, * 8, * 2,

f x A

Ax b

x

y

x y

x y

2

2

' [1 1]

1 0( )

0 4

' ( ) . .

So SOSC satisfied

Or we could just observe that

it is a convex program so FONC

are sufficient

Z

f x

Z f x Z is p d

In Class Practice

Find a KKT point

Verify SONC and

SOSC 2 2

1 2

1 2

1min 4

2s.t. 10

x x

x x

2 21 2

1 2

1

2

1

2

1 2

1min 4

2s.t. 10

( ) Ax b

xf(x) A 1 1

4x

1

4 1

10

T

x x

x x

f x A

x

x

x x

Linear Equality Constraints - I

Linear Equality Constraints - II

1 2 1 1 2

2 2

2

1

2

Solve:

x 4 4

4 10

5 10

8

2

8

8* , * 8 KKT point

2

x x x x

x x

x

x

x

x

01

1

40

0111)(

40

01)(

1

1 Z1-1A SOSC

2

2

ZxfZ

xf

T

so SOSC satisfied, and x* is a strict local minimum

Objective is convex, so KKT conditions are sufficient.

Linear Equality Constraints - III

Next Easiest Problem

Linear equality constraints

Constraints form a polyhedron

min ( )

. . ,

n

m n m

f x f R

s t Ax b A R b R

Polyhedron Ax>=b

( *) *

*

Ti i

i

f x A a

Ax b

unconstrained minimum

contour set of function

a1x = b

*)(xf

-a1

Close to Equality Case

a4x = b

a3x = b

a2x = b

a2x = b

-a2

Equality FONC:

Which i are 0? What is the sign of I?

x*

Polyhedron Ax>=b

( *) *

*

Ti i

i

f x A a

Ax b

a1x = b

*)(xf

-a1

Close to Equality Case

a4x = b

a3x = b

a2x = b

a2x = b

-a2

Equality FONC:

Which i are 0? What is the sign of I?

x*

Polyhedron Ax>=b

( *) *

*

( * ) 0

0

Ti i

i

i i i

f x A a

Ax b

A x b

a1x = b

*)(xf

-a1

Inequality Case

a4x = b

a3x = b

a2x = b

a2x = b

-a2

Inequality FONC:

Nonnegative Multipliers imply gradient points to the less thanSide of the constraint.

x*

i

i

0

represents sensitivity

ˆIf 0 then increasing causes the objective to increase.

If b is changed to make feasible region bigger then

the objective will decrease.

i

i i

i

i i

Ax b

if

then A x b

A x b

Lagrangian Multipliers

Lemma 14.3 Necessary Conditions

If x* is a local min of f over {x|Ax≤b}, and Z is a null-space matrix for active constraints then for some vector *

2

( *) ' * 0 or equivalently ' ( *) 0

*

* 0

*'( * ) 0

' ( *) . . .

f x A Z f x

Ax bKKT

Ax b

and Z f x Z is p s d

Lemma 14.5 Sufficient Conditions (KKT form)

If (x*,*) satisfies

2

* Primal feasibility

( *) ' * Dual feasibility

* 0

*'( * ) 0 Complementarity

and SOSC ' ( *) . .

Then x* is a strict local minimizer

Ax b

f x A

Ax b

Z f x Z is p d

Lemma 14.5 Sufficient Conditions (KKT form)

where Z+ is a basis matrix for Null(A

+) and A + corresponds to nondegenerate active constraints)

i.e.+

*j

For the jth row of A

* Active Constraint

0 Nondegenerate

j jA x b

Sufficient Example

Find solution and verify SOSC

2 21 2

1

min 1/ 2( 1) 1/ 2

. . 0 1

x x

s t x

* [1,0]' * [2,0]x

1

2

2

1

2* 1( *)

0

1 0( *)

0 1

Active constraint is 1

A 1 0

xf x

x

f x

x

Linear Inequality Constraints - I

21 22

1

21 22

1

1

1 1min 1

2 2s.t. 0 1

Put in standard form:

1 1min 1

2 2s.t. 1

-x 0

1x* by inspection

0

x x

x

x x

x

Linear Inequality Constraints - II

1

2

2

2* 1( *)

0

1 0( *)

0 1

1 0

1 0

xf x

x

f x

A

T

T1

KKT Conditions

Ax* b

f(x*) A

* 0

*( * ) 0

? since second constraint is inactive

0

2 1f(x*) A 2 0

0 0

1 2KKT Point: x* *

0 0

Ax b

Linear Inequality Constraints - III

2

' 2

Now look at SOSC

0[ 1 0]

1

2*

0

1 0f(x*)

0 1

f(x*) 1 a p.d. matrix

Therefore SOSC are satistified.

X* is a strict local minima

A Z

Z Z

Linear Inequality Constraints - IV

Example

Problem 1 2 2 3

1 2 22

1 2 3

2 3

min

. . 3 4 4

0

x x x

s t x x x

x x

* [8 / 51,28 / 51,28 / 51], * [ 8 / 51, 4 / 51]x

You Try

Solve the problem using above theorems:

2 21 2

1 2

1 2

1

min 1/ 2

. . 2 2

1

0

x x

s t x x

x x

x

If you guess first two constraints active, what happens?

If you guess just first constraint active, what happens?

Why Necessary and Sufficient?

Sufficient conditions are good for? Way to confirm that a candidate point is a minimum (local) But…not every min satisifies any given SC

Necessary tells you: If necessary conditions don’t hold then you know you

don’t have a minimum. Under appropriate assumptions, every point that is a min

satisfies the necessary cond. Good stopping criteria Algorithms look for points that satisfy Necessary

conditions

General Constraints

min ( )

. . ( ) 0i

f x

s t g x i E

min ( )

. . ( ) 0i

f x

s t g x i I

Lagrangian Function

Optimality conditions expressed using

Lagrangian function

and Jacobian matrix

were each row is a gradient of a constraint

1

( , ) ( ) ( ) ( ) ' ( )m

i ii

L x f x g x f x g x

( ) 'g x

Theorem 14.2 Sufficient Conditions Equality (KKT

form)If (x*,*) satisfies

2

( *) 0 Primal feasibility

( *) ( *) ' *

( L(x*, *)=0) Dual feasibility

and SOSC ' ( *) . .

Then x* is a strict local minimizer

xx

x

g x

f x g x

equivalently

Z L x Z is p d

Theorem 14.4 Sufficient Conditions Inequality (KKT)If (x*,*) satisfies

2

( *) 0 Primal feasibility

( *) ( *) ' *

( L(x*, *)=0) Dual feasibility

* 0 (for inequalities only)

* ' ( *) 0 Complementarity

and SOSC ' ( *) . .

Then

xx

x

g x

f x g x

equivalently

g x

Z L x Z is p d

x* is a strict local minimizer

Lemma 14.4 Sufficient Conditions (KKT form)

where Z+ is a basis matrix for Null(A

+) and A + corresponds to Jacobian of nondegenerate active constraints)

i.e.

*j

For the jth row of Jacobian

( *) 0 Active Constraint

0 Nondegenerate

jg x

Sufficient Example

Find solution and verify SOSC

2 21 2

2 21 1

min 1/ 2( 1) 1/ 2

. . 1/ 2 1/ 2 1/ 2

x x

s t x x

* [1,0]' * 2x

10

01*)(

0

2)1()(

0

1 x*Guess

2

1

2

1

2

1 s.t.

2

1)1(

2

1- min

2

2

1

22

21

22

21

xf

x

xxf

xx

xx

Nonlinear Inequality Constraints - I

Nonlinear Inequality Constraints - II

2 21 2

T1 2

1 1

1

1 1 1Has one active constraint:

2 2 2

Jacobian: g(x) x 1 0

( , ) ( ) ' ( )

( , ) ( ) ( )

-2 1

0 0

x

x x

x

L x f x g x

L x f x g x

2 2 2

T

2

( , ) ( ) ' ( )

( , ) ( ) ( )

-1 0 1 0 2

0 1 0 1

1 0 positive definite

0 1

0Z , since g(x) 1 0

1

1 0( , ) 0 1

0 1

xx i ii

Txx

L x f x g x

L x f x g x

Z L x Z

01 positive definite

1

so SOSC satisifed, x* is a strict local minimum

Nonlinear Inequality Constraints - III

Sufficient Example

Find solution and verify SOSC

221 2

min

. . 0

x

s t x x

* [0,0]' * 1x

2

21 2

22 1 2

'

1

min x

s.t. x - x 0

0x* by observation

0

L(x, ) ( ) ' ( )

x (x -x )

L(x, ) ( ) ( )

0 2

1 1

0 0( *, *) 0 * 0 * 1

1 1

K

x

x

f x g x

f x g x

x

L x

0KT Point: x* * 1

0

Nonlinear Inequality Constraints - V

2 2 2

2

1( ) 0 1 Z

0

( , ) ( ) ( )

0 0 2 0 ( 1)

0 0 0 0

2 0

0 0

2 0 1( , ) 1 0 2 positive definite

0 0 0

So SOSC

T

xx i ii

Txx

g x

L x f x g x

Z L x Z

are satisfied, and x* is a strict local minimum.

Nonlinear Inequality Constraints - VI

Theorem 14.1 Necessary Conditions- Equality

If x* is a local min of f over {x|g(x)=0}, Z is a null-space matrix of the Jacobian g(x*)’, and x* is a regular point then

2

there exists *

( *, *) 0 or equivalently ' ( *) 0

( *) 0

' ( *) . . .

x

xx

L x Z f x

g x

and Z L x Z is p s d

Theorem 14.3 Necessary Conditions

If x* is a local min of f over {x|g(x)>=0}, Z is a null-space matrix of the Jacobian g(x*)’, and x* is a regular point then

'

2

there exists *

( *, *) 0 or equivalently ' ( *) 0

( *) 0

* ( *) 0

' ( *) . . .

x

xx

L x Z f x

g x

g x

and Z L x Z is p s d

Regular point

If x* is a regular point with respect to the constraints g(x*) if the gradient of the active constraints are linearly independent.For equality constraints, all constraints are active so

should have linearly independent rows.

( *) 'g x

Necessary Example

Show optimal solution x*=[1,0]’

is regular and find KKT point 1

2 21 2

31 2

max

. . 1

( 1) 0

x

s t x x

x x

1

2 21 2

31 2

min

. . ( ) 1 0

( 1) 0

x

s t x x

x x

KKT point

* [1, 0] '

* [1 / 2, 0] '

x

Constraint Qualifications

Regularity is an example of a constraint qualification CQ.The KKT conditions are based on linearizations of the constraints. CQ guarantees that this linearization is not getting us into trouble. Problem is

KKT point might not exist.There are many other CQ,e.g., for inequalities Slater is there exists g(x)<0.Note CQ not needed for linear constraints.

KKT Summary

X* is local min

KKT Satisfied

X* is global min

CQ

Convex fConvex constraints

SOSC