Chapter 9. Interior Point Methods

Linear Programming 2012 1

Chapter 9. Interior Point Methods

Three major variantsAffine scaling algorithm - easy concept, good performancePotential reduction algorithm - poly timePath following algorithm - poly time, good performance,

theoretically elegant


9.4 The primal path following algorithm min c’x max p’b

Ax = b p’A + s’ = c’ x 0 s 0

Nonnegativity makes the problem difficult, hence use barrier function in the objective and consider unconstrained problem ( in the affine space Ax = b, p’A + s’ = c’ )

Barrier function: B(x) = c’x - j=1n log xj, > 0

B(x) + if xj 0 for some j

Solve min B(x), Ax = b (9.15)

B(x) is strictly convex, hence has unique min point if min ex-ists.


ex) min x, s. t. x 0 B(x) = x - log x,

1 - /x = 0 min at x =

x

B(x)

0 1

- log x


min B(x) = c’x - j=1n log xj

s.t. Ax = b

Let x() is optimal solution given > 0.x(), when varies, is called the central path ( hence the name path following )

It can be shown that lim 0 x() = x* optimal solution to LP. When = , x() is called the analytic center.

For dual problem, the barrier problem ismax p’b + j = 1

m log sj , p’A + s’ = c’ (9.16)

( equivalent to min –p’b - j = 1m log sj , minimizing convex func-

tion)


Figure 9.4: The central path and the analytic center

x(10)

x(1)x(0.1)

x(0.01)

x*

analytic center

c


Results from nonlinear programming(NLP) min f(x)

s.t. gi(x) 0, i = 1, … , m hi(x) = 0. i = 1, … , p

f, gi , hi : Rn R, all twice continuously differentiable( gradient is given as a column vector)

Thm (Karush 1939, Kuhn-Tucker 1951, first order necessary opti-mality condition)If x* is a local minimum for (NLP) and some condition (called con-straint qualification) holds at x*, then there exist u R+

m, v Rp such that

(1) f(x*) + i = 1m ui gi(x*) + i = 1

p vi hi(x*) = 0(2) u 0, gi(x*) 0, i = 1, … , m, i = 1

m ui gi(x*) = 0(3) hi(x*) = 0. i = 1, … , p


Remark:(2) is CS conditions and it implies that ui = 0 for non-active

constraint gi .(1) says f(x*) is a nonnegative linear combination of - gi(x*)

for active constraints and hi(x*) (compare to strong duality theorem in p. 173 and its Figure )

CS conditions for LP are KKT conditionsKKT conditions are necessary conditions for optimality, but it

is also sufficient in some situations. One case is when objec-tive function is convex and constraints are linear, which in-cludes our barrier problem.


Deriving KKT for barrier problem:min B(x) = c’x - j=1

n log xj, s.t. Ax = b ( x > 0 )

f(x) = c - X-1e, hi(x) = ai

( ai is i-th row vector of A, expressed as a column vector and X-1 = diag( 1/x1, … , 1/xn ), e is the vector having 1 in all compo-nents.)Using (Lagrangian) multiplier pi for hi(x), we get c - X-1e = A’p (ignoring the sign of p)Note that hi(x) = ai’x – bi and hi(x) = ai . ( h(x) = Ax – b : Rn Rm) If we define s = X-1e , KKT becomes

A’p + s = c, Ax = b, XSe = e, ( x > 0, s > 0),where S = diag ( s1, … , sn ).


For dual barrier problem,min - p’b - j = 1

m log sj , s.t. p’A + s’ = c’ ( s > 0 )

eS

b)s,p(f 1

i

ii e

A)s,p(h

( Ai is i-th column vector of A and ei is i-th unit vector.)Using (Lagrangian) multiplier – xi for hi(p, s), we get

m

i i

ii e

A)x(

eSb

11

Now i xiei = Xe, hence we have the conditionsA’p + s = c, Ax = b, XSe = e, ( x > 0, s > 0 )

which is the same conditions we obtained from the primal barrier func-tion.


The conditions are given in the text as

Ax() = b, x() 0A’p() + s() = c, s() 0 (9.17)X()S()e = e,

where X() = diag ( x1(), … , xn() ), S() = diag ( s1(), … , sn() ).

Note that when = 0, they are primal, dual feasibility and com-plementary slackness conditions.

Lemma 9.5: If x*, p*, and s* satisfy conditions (9.17), then they are optimal solutions to problems (9.15) and (9.16)


Pf) Let x*, p*, and s* satisfy (9.17), and let x be an arbitrary vec-tor that satisfies x 0 and Ax = b. Then

B(x) = c’x - j=1n log xj

= c’x – (p*)’(Ax – b) - j=1n log xj

= (s*)’x + (p*)’b - j=1n log xj,

n + (p*)’b - j=1n log ( / sj*)

sj*xj - log xj attains min at xj = / sj*.equality holds iff xj = / sj* = xj*

Hence B(x*) B(x) for all feasible x. In particular, x* is the unique optimal solution and x* = x().Similarly for p* and s* for dual barrier problem.


Primal path following algorithm Starting from some 0 and primal and dual feasible x0 > 0, s0 > 0,

p0, find solution of the barrier problem iteratively while 0.

To solve the barrier problem, we use quadratic approximation (2nd order Taylor expansion) of the barrier function and use the minimum of the approximate function as the next iterates.

Taylor expansion is

dX'dd)X'e'c()x(B

ddxx

)x(Bd

x)x(B

)x(B)dx(Bn

j,iji

ji

n

ii

i2

211

1

2

21

1

Also need to satisfy A(x + d) = b Ad = 0


Using KKT, solution to this problem isd() = ( I – X2A’(AX2A’)-1A )( Xe – (1/ )X2c )p() = (AX2A’)-1A ( X2c - Xe )

The duality gap is c’x – p’b = ( p’A + s’ )x – p’(Ax) = s’xHence stop the algorithm if (sk)’xk <

Need a scheme to have initial feasible solution


The primal path following algorithm

1. (Initialization) Start with some primal and dual feasible x0 > 0, s0 > 0, p0, and set k = 0.2. (Optimality test) If (sk)’xk < stop; else go to Step 3.3. Let Xk = diag ( x1

k, … , xnk ), k+1 = k ( 0< <1)

4. (Computation of directions) Solve the linear system k+1 Xk

-2d – A’p = k+1 Xk-1e – c,

Ad = 0,for p and d.

5. (Update of solutions) Letxk+1 = xk + d,pk+1 = p,sk+1 = c – A’p.

6. Let k := k+1 and go to Step 2.


9.5 The primal-dual path following algorithm Find Newton directions both in the primal and dual space.

Instead of finding min of quadratic approximation of barrier func-tion, it finds the solution for KKT system.

Ax() = b, ( x() 0 )A’p() + s() = c, ( s() 0 ) (9.26)X()S()e = e,

System of nonlinear equations because of the last ones. Let F: Rr Rr. Want z* such that F(z*) = 0

We use first order Taylor approximation around zk,F( zk + d) ~ F(zk) + J(zk)d.

Here J(zk) is the r r Jacobian matrix whose (i, j)th element is

k

ji zzz

)z(F


Try to find d that satisfiesF(zk) + J(zk)d = 0

d is called a Newton direction.Here F(z) is given by

eXSecsp'A

bAx)z(F

eeSXcsp'A

bAx

d

dd

XSI'A

A

kkk

kk

k

ks

kp

kx

kk 00

00

This is equivalent to Adx

k = 0 (9.28) A’dp

k + dsk = 0 (9.29)

Skdxk + Xkds

k = ke - XkSke (9.30)


Solution to the previous system isdx

k = Dk ( I – Pk )vk ( k),dp

k = - (ADk2A’)-1ADkvk (k),

dsk = Dk

-1Pkvk (k),where

Dk2 = XkSk

-1,Pk = DkA’ (ADk

2A’)-1ADk , vk (k) = Xk

-1Dk ( ke – XkSke).

Also limit the step length to ensure xk+1 > 0, sk+1 > 0.


The primal-dual path following algorithm1. (Initialization) Start with some feasible x0 > 0, s0 > 0, p0, and set k = 0.2. (Optimality test) If (sk)’xk < stop; else go to Step 3.3. (Computation of Newton directions) Let

k = (sk)’xk / n,Xk = diag ( x1

k, … , xnk ),

Sk = diag ( s1k, … , sn

k ).Solve the linear system (9.28) – (9.30) for dx

k, dpk, and ds

k. 4. (Find step lengths) Let

,)d(

xmin,mini

kx

ki

})d(:i{

kP

ikx

01

,)d(

smin,mini

ks

ki

})d(:i{

kD

iks

01

( 0 < < 1 )


(continued)5. (Solution update) Update the solution vectors according to

xk+1 = xk + Pk dx

k,pk+1 = pk + D

k dpk,

sk+1 = sk + Dk ds

k.6. Let k := k+1 and go to Step 2.


Infeasible primal-dual path following methods

A variation of primal-dual path following.Starts from x0 > 0, s0 > 0, p0, which is not necessarily feasible for either the primal or the dual, i.e. Ax0 b and/or A’p0 + s0 c.Iteration same as the primal-dual path following except feasibility not maintained in each iteration.Excellent performance.


Self-dual method Alternative method to find initial feasible solution w/o using big-M.

Given an initial possibly infeasible point (x0, p0, s0) with x0 > 0 and s0 > 0, consider the problem

minimize ( (x0)’s0 + 1) subject to Ax - b + b = 0

-A’p + c - c - s = 0 (9.33)b’p – c’x + z - = 0

- b’p + c’x - z = - ((x0)’s0 + 1) x 0, 0, s 0, 0

where b = b – Ax0, c = c – A’p0 – s0, z = c’x0 + 1 – b’p0.

This LP is self-dual.Note that ( x, p, s, , , ) = ( x0, p0, s0, 1, 1, 1) is a feasible interior solution to (9.33)


Since both the primal and dual are feasible, they have optimal solutions and the optimal value is 0.

Primal-dual path following method finds an optimal solution ( x*, p*, s*, *, *, *) that satisfies

* = 0, x* + s* > 0, * + * > 0,(s*)’x* = 0, ** = 0

( satisfies strict complementarity )

Can find optimal solution or determine unboundedness depend-ing on the value of *, *. (see Thm 9.8)

Running time : worst case : observed : O( log n log( 0 / ))

))/log(n(O 0

Chapter 9. Interior Point Methods

Documents